Solaris 10 locked room mystery: moving /usr and /var from / to their own zpools.

[reddragdiva]

(If you're not a geek, the text below translates as "axle-maxle manglebratic geeble-geep.")

We have a shiny new T5120! Just getting the thing set up yesterday was a locked-room mystery — you have to set it up on the serial console, and the IT guy spent ages finding a USB serial adapter for his laptop. (No, SPARCs don't take KVMs. But thanks for providing one. Serial please kthx. Network management is way cool too.) But we got it up and running.

We're using the factory installation of Solaris 10 5/09. We're not bothering to wipe and reinstall, though I'm sorely tempted. The default installation is a 12GiB UFS / on c1t0d0s0 and an 8GiB swap on c1t0d0s1. The rest of c1t0d0 (a 146GB disk) is unused, as is c1t1d0. We also put in two more 146GB disks, c1t2d0 and c1t3d0.

We wanted to ZFS it as mirrors. So, on c1t0d0 and c1t1d0 I set up a 16GB rootpool, 30GB usrpool, 16GB varpool and the remaining 66GB as exportpool (which I'll attach c1t2d0 and c1t3d0 to at my leisure). Moved / to the rootpool successfully, mirrored it and made the mirror boot as well. We have mirrored ZFS boot, yay!

Now for the fun part: /usr and /var are on / and I want them on their own partitions. This is proving a locked-room mystery.

I tarred up /var and copied it to /zpool/var . I tried deleting /var and symlinking it to /zpool/var. No dice — /var/run is not deletable. Nor is it unmountable. I tried zfs set mountpoint=/var varpool and it refused to mount it there at boot because /var wasn't empty. /var/run isn't even umountable, even in single-user.

I haven't tried doing the same for /usr as yet — I'm suspecting I'll unlink a command I need to run.

So. How the hell do I move /usr and /var off / to point at their waiting zpools? Other than nuking and reinstalling. I suppose I could boot off a DVD. But there should be a way to solve the locked-room mystery.

(So how much of your job involves hacking into your own damn boxes? Too much of mine.)

Update: Downloading Sol 10u8 (10/09) SPARC DVD right now, to burn or jumpstart from on Monday. Though I'll try hauntedunix's suggestion just to see if it works.

Update 2: It works! (as Apache would say.) Didn't bother wiping and reinstalling, but I've certainly learnt a lot about the dance of disks in ZFS. I now have a 16GB / (including /usr), 16GB /var and the rest /export, all mirrored; and two 8GB raw slices being used as 16GB swap. And the default rootpool swap pool, which I've cut to 1GB. All looks damn fine and I will probably start installing actual stuff to run shortly.

Comments

[ideological_cuddle]

You do it in single-user mode, having booted from media.

Or you *might* be able to do something with LiveUpgrade. It could certainly do this sort of split with UFS, so it may be able to do it with ZFS too.

Or, y'know, you take the 30 minutes it would've taken to just do a fresh install right at the start and save all of this stuffing about. An interactive Solaris install is *dead easy*.

[reddragdiva]

Apparently the answer is to mount the ZFS as legacy and put it /etc/vfstab.

Or, as I'm increasingly coming to think, nuking it and installing from DVD.

[ideological_cuddle]

Unless there's stuff on there you need you're always better off nuking the default install and redoing it.

Also, those machines? They do have network LOM. You have to do the serial dance to configure the network interface on it, but that's about it. They *might* even do DHCP, not sure though.

[reddragdiva]

Yes, I know. I also know that they pull the network manager address out their backside via DHCP, and that the actual Sun-recommended procedure is to nmap your subnet to find the damn thing! Which IT would not have been happy with us doing. Mostly I was annoyed IT didn't have any serial consoles available and presumed everything used a KVM.

[gothgeekgirl]

Yes, they do DHCP. I prefer static addresses for servers, though. And yeah, nuke and reinstall. Especially because Sol 10 update 6 (10/08 IIRC) and beyond will boot from ZFS (which I find to be a joy and a wonder) AND because I've never seen a default install filesystem layout that didn't suck moldy donkey dong.

[ideological_cuddle]

You let it do DHCP, look at the dhcpd log to see what IP it picked up, then ssh/https in and set the static IP and other network config. Saves having to stuff around finding a working serial adapter and then dealing with the "settings of the week" club where every damned vendor uses a different speed/parity/etc.

We actually have to pay our Sun reseller in .hk to do a custom install which we then blow away, because they won't sell us the machine without the custom OS install service. Which they bill for separately, of course.

(On the bright side all our resellers do the racking and LOM config so we don't have to care. We then come along and do the cabling.)

[gothgeekgirl]

Don't bother moving /usr out of /. There's no real disk size limit any more and it could even be downright dangerous. /bin is a link to /usr/bin, nothing's statically linked any more, and /sbin has links into /usr (just 3 commands, but they're there).

[gothgeekgirl]

We live in very different environments. I have a data center full of terminal servers, and I control $ork's DNS servers. I don't think we even run DHCP on our server subnets. (and for those very rare occasions when I need one, I have a USB<->serial adapter that I keep in my laptop bag)

Just in case you were wondering, yes, I'm sufficiently paranoid that I insist on the LOMs being connected to both the terminal servers and the management subnets ;)

[ideological_cuddle]

I'm in an environment where I've got machines in four cities in three countries from Melbourne to Tokyo and points in-between, and not enough staff -- particularly outside Melbourne -- to do anything that's particularly time-intensive.

I have a DHCP server on the deployment subnet for use by Jumpstart. So to do a LOM from scratch -- if we ever did that -- we'd just put the port on the deployment VLAN, spot the IP, configure the LOM over the network, then flip the port to the console subnet. We control the switches, which is a small blessing, so there's no dealing with a separate uncooperative network department for that.

We do have some terminal servers but far too many machines to hook them all up to them. Space is at a premium.

Paranoia is, in this particular place, best directed at fighting the mothership's insistence that telnet and rsh are appropriate means of accessing production systems...

[gothgeekgirl]

*blinks, goes to look at profile, blinks again* Oh hell. Hi there.

I'm still fighting the rsh/telnet battle myself. One day I'm just going to throw a fit and turn them off.

[ideological_cuddle]

(And now I'm trying to figure out if I know you IRL.)

We have them disabled, generally speaking, but whenever there's something the US people have to support we get into an almighty fight about it. Technical management over there stopped being technical sometime around 1990, when it was still barely acceptable to use rlogin and NIS.

(Why yes, they use NIS over there too.)

[ideological_cuddle]

Oh, hang on, I think I figured it out. :)

[gothgeekgirl]

:D

Yes indeed. Some years ago I rescued you from 12 hours of boredom at LAX.

I must admit that I still use NIS because I've got too many other things to handle to do the conversion to LDAP any time soon. But I think I'll manage it before they force me to use Centrify.

[ideological_cuddle]

Indeed. I'm not sure I was quite clear about how much I appreciated that -- I was feeling a little off my game that day.

We don't do NIS but we don't do LDAP either. Individually-managed passwd maps on 200+ hosts! Because it's still less painful than trying to get the Windows/AD people in the US to add the POSIX schema...

[reddragdiva]

Ahahaha you assume we have access to the dhcpd logs. Assuming IT aren't doing it using a Windows box. They like their Windows, they do.

"settings of the week club"? I haven't seen anything but 9600 baud 8N1 in ... years.

[ptomblin-lj.livejournal.com]

Any way to boot from a CD or another disk image, so it isn't using /var while you're trying to fuck with it?

[bob]

id always nuke and reinstall anyway.

[reddragdiva]

Yeah, probably. Involves arranging server room access again. Probably the right answer. How annoying.

[reddragdiva]

I'm increasingly coming around to this way of thinking. The thing is a high-powered compute server and they installed fucking GNOME on it. I think I spent longer removing four gig of crap than it would have taken to download, burn and use the DVD.

[steer.livejournal.com]

Hear you re zfs. I've wasted hours trying to do trivial crap on zfs. It's vexing because in theory zfs is a damn cool idea, it's just the practice which is shit.

Flatmate is zfs zealot despite having lost his entire company's data due to a bug in the BSD implementation of same.

[hauntedunix.livejournal.com]

Why do you need multiple pools? Just have one root pool and multiple zfs filesystems. There's no advantage from using separate zpools in Sol10 if you're only using the same disks.

Anyway - you definitely *can* do this. What happens if you boot from a cd/pxe, then just chroot and manipulate the install that way? Hell - even if that doesn't, zfs set mountpoint=legacy and change the vfstab to the correct values should

[reddragdiva]

It's not ZFS being the problem here - it's the operating system refusing to let me do live surgery on it. ZFS sure as hell beats UFS for what we're doing with it.

[rasilon-x.livejournal.com]

Jumpstart FTW.

[neuro42.livejournal.com]

Single-user mode, I would assume.

[reddragdiva]

I wanted to make sure that if c1t2d0 and c1t3d0 both died, we'd at least have a bootable global zone. (I haven't put zones all over this thing yet.) Perhaps I'm wrong.

"zfs set mountpoint=legacy and change the vfstab to the correct values"

BING BING BING dude, I think we have a winner and if this works I will owe you many a *pint*.