Solaris 10 locked room mystery: moving /usr and /var from / to their own zpools.

[reddragdiva]

(If you're not a geek, the text below translates as "axle-maxle manglebratic geeble-geep.")

We have a shiny new T5120! Just getting the thing set up yesterday was a locked-room mystery — you have to set it up on the serial console, and the IT guy spent ages finding a USB serial adapter for his laptop. (No, SPARCs don't take KVMs. But thanks for providing one. Serial please kthx. Network management is way cool too.) But we got it up and running.

We're using the factory installation of Solaris 10 5/09. We're not bothering to wipe and reinstall, though I'm sorely tempted. The default installation is a 12GiB UFS / on c1t0d0s0 and an 8GiB swap on c1t0d0s1. The rest of c1t0d0 (a 146GB disk) is unused, as is c1t1d0. We also put in two more 146GB disks, c1t2d0 and c1t3d0.

We wanted to ZFS it as mirrors. So, on c1t0d0 and c1t1d0 I set up a 16GB rootpool, 30GB usrpool, 16GB varpool and the remaining 66GB as exportpool (which I'll attach c1t2d0 and c1t3d0 to at my leisure). Moved / to the rootpool successfully, mirrored it and made the mirror boot as well. We have mirrored ZFS boot, yay!

Now for the fun part: /usr and /var are on / and I want them on their own partitions. This is proving a locked-room mystery.

I tarred up /var and copied it to /zpool/var . I tried deleting /var and symlinking it to /zpool/var. No dice — /var/run is not deletable. Nor is it unmountable. I tried zfs set mountpoint=/var varpool and it refused to mount it there at boot because /var wasn't empty. /var/run isn't even umountable, even in single-user.

I haven't tried doing the same for /usr as yet — I'm suspecting I'll unlink a command I need to run.

So. How the hell do I move /usr and /var off / to point at their waiting zpools? Other than nuking and reinstalling. I suppose I could boot off a DVD. But there should be a way to solve the locked-room mystery.

(So how much of your job involves hacking into your own damn boxes? Too much of mine.)

Update: Downloading Sol 10u8 (10/09) SPARC DVD right now, to burn or jumpstart from on Monday. Though I'll try hauntedunix's suggestion just to see if it works.

Update 2: It works! (as Apache would say.) Didn't bother wiping and reinstalling, but I've certainly learnt a lot about the dance of disks in ZFS. I now have a 16GB / (including /usr), 16GB /var and the rest /export, all mirrored; and two 8GB raw slices being used as 16GB swap. And the default rootpool swap pool, which I've cut to 1GB. All looks damn fine and I will probably start installing actual stuff to run shortly.

Comments

[gothgeekgirl]

Yes, they do DHCP. I prefer static addresses for servers, though. And yeah, nuke and reinstall. Especially because Sol 10 update 6 (10/08 IIRC) and beyond will boot from ZFS (which I find to be a joy and a wonder) AND because I've never seen a default install filesystem layout that didn't suck moldy donkey dong.

[ideological_cuddle]

You let it do DHCP, look at the dhcpd log to see what IP it picked up, then ssh/https in and set the static IP and other network config. Saves having to stuff around finding a working serial adapter and then dealing with the "settings of the week" club where every damned vendor uses a different speed/parity/etc.

We actually have to pay our Sun reseller in .hk to do a custom install which we then blow away, because they won't sell us the machine without the custom OS install service. Which they bill for separately, of course.

(On the bright side all our resellers do the racking and LOM config so we don't have to care. We then come along and do the cabling.)

[gothgeekgirl]

We live in very different environments. I have a data center full of terminal servers, and I control $ork's DNS servers. I don't think we even run DHCP on our server subnets. (and for those very rare occasions when I need one, I have a USB<->serial adapter that I keep in my laptop bag)

Just in case you were wondering, yes, I'm sufficiently paranoid that I insist on the LOMs being connected to both the terminal servers and the management subnets ;)

[ideological_cuddle]

I'm in an environment where I've got machines in four cities in three countries from Melbourne to Tokyo and points in-between, and not enough staff -- particularly outside Melbourne -- to do anything that's particularly time-intensive.

I have a DHCP server on the deployment subnet for use by Jumpstart. So to do a LOM from scratch -- if we ever did that -- we'd just put the port on the deployment VLAN, spot the IP, configure the LOM over the network, then flip the port to the console subnet. We control the switches, which is a small blessing, so there's no dealing with a separate uncooperative network department for that.

We do have some terminal servers but far too many machines to hook them all up to them. Space is at a premium.

Paranoia is, in this particular place, best directed at fighting the mothership's insistence that telnet and rsh are appropriate means of accessing production systems...

[gothgeekgirl]

*blinks, goes to look at profile, blinks again* Oh hell. Hi there.

I'm still fighting the rsh/telnet battle myself. One day I'm just going to throw a fit and turn them off.

[ideological_cuddle]

(And now I'm trying to figure out if I know you IRL.)

We have them disabled, generally speaking, but whenever there's something the US people have to support we get into an almighty fight about it. Technical management over there stopped being technical sometime around 1990, when it was still barely acceptable to use rlogin and NIS.

(Why yes, they use NIS over there too.)

[ideological_cuddle]

Oh, hang on, I think I figured it out. :)

[gothgeekgirl]

:D

Yes indeed. Some years ago I rescued you from 12 hours of boredom at LAX.

I must admit that I still use NIS because I've got too many other things to handle to do the conversion to LDAP any time soon. But I think I'll manage it before they force me to use Centrify.

[ideological_cuddle]

Indeed. I'm not sure I was quite clear about how much I appreciated that -- I was feeling a little off my game that day.

We don't do NIS but we don't do LDAP either. Individually-managed passwd maps on 200+ hosts! Because it's still less painful than trying to get the Windows/AD people in the US to add the POSIX schema...

[reddragdiva]

Ahahaha you assume we have access to the dhcpd logs. Assuming IT aren't doing it using a Windows box. They like their Windows, they do.

"settings of the week club"? I haven't seen anything but 9600 baud 8N1 in ... years.