Solaris 10 locked room mystery: moving /usr and /var from / to their own zpools.

(If you're not a geek, the text below translates as "axle-maxle manglebratic geeble-geep.")

We have a shiny new T5120! Just getting the thing set up yesterday was a locked-room mystery — you have to set it up on the serial console, and the IT guy spent ages finding a USB serial adapter for his laptop. (No, SPARCs don't take KVMs. But thanks for providing one. Serial please kthx. Network management is way cool too.) But we got it up and running.

We're using the factory installation of Solaris 10 5/09. We're not bothering to wipe and reinstall, though I'm sorely tempted. The default installation is a 12GiB UFS / on c1t0d0s0 and an 8GiB swap on c1t0d0s1. The rest of c1t0d0 (a 146GB disk) is unused, as is c1t1d0. We also put in two more 146GB disks, c1t2d0 and c1t3d0.

We wanted to ZFS it as mirrors. So, on c1t0d0 and c1t1d0 I set up a 16GB rootpool, 30GB usrpool, 16GB varpool and the remaining 66GB as exportpool (which I'll attach c1t2d0 and c1t3d0 to at my leisure). Moved / to the rootpool successfully, mirrored it and made the mirror boot as well. We have mirrored ZFS boot, yay!

Now for the fun part: /usr and /var are on / and I want them on their own partitions. This is proving a locked-room mystery.

I tarred up /var and copied it to /zpool/var . I tried deleting /var and symlinking it to /zpool/var. No dice — /var/run is not deletable. Nor is it unmountable. I tried zfs set mountpoint=/var varpool and it refused to mount it there at boot because /var wasn't empty. /var/run isn't even umountable, even in single-user.

I haven't tried doing the same for /usr as yet — I'm suspecting I'll unlink a command I need to run.

So. How the hell do I move /usr and /var off / to point at their waiting zpools? Other than nuking and reinstalling. I suppose I could boot off a DVD. But there should be a way to solve the locked-room mystery.

(So how much of your job involves hacking into your own damn boxes? Too much of mine.)

Update: Downloading Sol 10u8 (10/09) SPARC DVD right now, to burn or jumpstart from on Monday. Though I'll try hauntedunix's suggestion just to see if it works.

Update 2: It works! (as Apache would say.) Didn't bother wiping and reinstalling, but I've certainly learnt a lot about the dance of disks in ZFS. I now have a 16GB / (including /usr), 16GB /var and the rest /export, all mirrored; and two 8GB raw slices being used as 16GB swap. And the default rootpool swap pool, which I've cut to 1GB. All looks damn fine and I will probably start installing actual stuff to run shortly.