Solaris 10 locked room mystery: moving /usr and /var from / to their own zpools.

[reddragdiva]

(If you're not a geek, the text below translates as "axle-maxle manglebratic geeble-geep.")

We have a shiny new T5120! Just getting the thing set up yesterday was a locked-room mystery — you have to set it up on the serial console, and the IT guy spent ages finding a USB serial adapter for his laptop. (No, SPARCs don't take KVMs. But thanks for providing one. Serial please kthx. Network management is way cool too.) But we got it up and running.

We're using the factory installation of Solaris 10 5/09. We're not bothering to wipe and reinstall, though I'm sorely tempted. The default installation is a 12GiB UFS / on c1t0d0s0 and an 8GiB swap on c1t0d0s1. The rest of c1t0d0 (a 146GB disk) is unused, as is c1t1d0. We also put in two more 146GB disks, c1t2d0 and c1t3d0.

We wanted to ZFS it as mirrors. So, on c1t0d0 and c1t1d0 I set up a 16GB rootpool, 30GB usrpool, 16GB varpool and the remaining 66GB as exportpool (which I'll attach c1t2d0 and c1t3d0 to at my leisure). Moved / to the rootpool successfully, mirrored it and made the mirror boot as well. We have mirrored ZFS boot, yay!

Now for the fun part: /usr and /var are on / and I want them on their own partitions. This is proving a locked-room mystery.

I tarred up /var and copied it to /zpool/var . I tried deleting /var and symlinking it to /zpool/var. No dice — /var/run is not deletable. Nor is it unmountable. I tried zfs set mountpoint=/var varpool and it refused to mount it there at boot because /var wasn't empty. /var/run isn't even umountable, even in single-user.

I haven't tried doing the same for /usr as yet — I'm suspecting I'll unlink a command I need to run.

So. How the hell do I move /usr and /var off / to point at their waiting zpools? Other than nuking and reinstalling. I suppose I could boot off a DVD. But there should be a way to solve the locked-room mystery.

(So how much of your job involves hacking into your own damn boxes? Too much of mine.)

Update: Downloading Sol 10u8 (10/09) SPARC DVD right now, to burn or jumpstart from on Monday. Though I'll try hauntedunix's suggestion just to see if it works.

Update 2: It works! (as Apache would say.) Didn't bother wiping and reinstalling, but I've certainly learnt a lot about the dance of disks in ZFS. I now have a 16GB / (including /usr), 16GB /var and the rest /export, all mirrored; and two 8GB raw slices being used as 16GB swap. And the default rootpool swap pool, which I've cut to 1GB. All looks damn fine and I will probably start installing actual stuff to run shortly.

Comments

[ideological_cuddle]

You do it in single-user mode, having booted from media.

Or you *might* be able to do something with LiveUpgrade. It could certainly do this sort of split with UFS, so it may be able to do it with ZFS too.

Or, y'know, you take the 30 minutes it would've taken to just do a fresh install right at the start and save all of this stuffing about. An interactive Solaris install is *dead easy*.

[reddragdiva]

Apparently the answer is to mount the ZFS as legacy and put it /etc/vfstab.

Or, as I'm increasingly coming to think, nuking it and installing from DVD.

[gothgeekgirl]

Don't bother moving /usr out of /. There's no real disk size limit any more and it could even be downright dangerous. /bin is a link to /usr/bin, nothing's statically linked any more, and /sbin has links into /usr (just 3 commands, but they're there).

[ptomblin-lj.livejournal.com]

Any way to boot from a CD or another disk image, so it isn't using /var while you're trying to fuck with it?

[reddragdiva]

Yeah, probably. Involves arranging server room access again. Probably the right answer. How annoying.

[bob]

id always nuke and reinstall anyway.

[reddragdiva]

I'm increasingly coming around to this way of thinking. The thing is a high-powered compute server and they installed fucking GNOME on it. I think I spent longer removing four gig of crap than it would have taken to download, burn and use the DVD.

[reddragdiva]

Mind you, 32 threads is probably enough CPU power to run StarOffice ahahaha.

[steer.livejournal.com]

Hear you re zfs. I've wasted hours trying to do trivial crap on zfs. It's vexing because in theory zfs is a damn cool idea, it's just the practice which is shit.

Flatmate is zfs zealot despite having lost his entire company's data due to a bug in the BSD implementation of same.

[reddragdiva]

It's not ZFS being the problem here - it's the operating system refusing to let me do live surgery on it. ZFS sure as hell beats UFS for what we're doing with it.

[blarglefiend.livejournal.com]

ZFS is lovely on Solaris. Everywhere else it's been hacked on top and not properly integrated. Anyone running ZFS in production on FreeBSD deserves to have their data eaten.

[hauntedunix.livejournal.com]

Why do you need multiple pools? Just have one root pool and multiple zfs filesystems. There's no advantage from using separate zpools in Sol10 if you're only using the same disks.

Anyway - you definitely *can* do this. What happens if you boot from a cd/pxe, then just chroot and manipulate the install that way? Hell - even if that doesn't, zfs set mountpoint=legacy and change the vfstab to the correct values should

[reddragdiva]

I wanted to make sure that if c1t2d0 and c1t3d0 both died, we'd at least have a bootable global zone. (I haven't put zones all over this thing yet.) Perhaps I'm wrong.

"zfs set mountpoint=legacy and change the vfstab to the correct values"

BING BING BING dude, I think we have a winner and if this works I will owe you many a *pint*.

[neuro42.livejournal.com]

Single-user mode, I would assume.

[reddragdiva]

Ahahaha nope, tried that!

[bob]

i keeep a usb serial dongle in my rucksuck with my laptop for just suck occasions.
i also have a range of serial adapters

[reddragdiva]

"suck occasions" is about right. I do know that there's no standard whatsoever for USB serial adapters - got one that Just Works under leengux?

[ewx.livejournal.com]

On Linux I’d say “edit fstab and reboot” but it’s 10 years since I even pretended to know how to run a Solaris system so for all I know that’s a laughably naïve suggestion.

[vampwillow]

SolX is - like a few other OSes - summat that wants to be "installed the way you want it fron the get-go" rather than permit changes down the line. (Hence why I have a pack of OS install disks in multiple flavours / releases and various CD readers

[blarglefiend.livejournal.com]

My comment elsewhere suggesting LU was wrong, having looked at the lucreate manpage it doesn't support splitting with a ZFS BE. Oh well.

I'm really not at all sure you need separate *pools* here. That would be an extraordinarily unusual configuration and the installer won't help you do that. My guess is that what you want is separate *filesystems* within rpool, which the installer will help you with.

Also, splitting /usr off? How very 1991. Splitting /var makes sense, you can then apply a quota to it and make sure things like logs don't fill the disk, but /usr *should* be fairly static.

Also-also, 5/09? Grab yourself a copy of 10/09 and install that. And if you expect to build any more of these (or LDOMs on the same box) go read the manpage for 'flarcreate' once you've patched the thing with pca.

[reddragdiva]

I'll probably apply hauntedunix's suggestion just to see if it works, then download the DVD and apply on Monday.

We're gonna go zones with this thing. I'd try LDOMs if I thought I ever wanted to work in a bank, ever ever ;-) One for standby source control (the box this is replacing will remain primary source control), one for CruiseControl. The interesting bit will be if we can tell CruiseControl to actually use the 32 CPUs we bought for it ...

[http://users.livejournal.com/_nicolai_/]

(channelling my inner blunt Scandinavian)

Give up.
/usr must be accessible at boot in singleuser mode, so must /var. They must be accessible by simple UFS means, not ZFS.
Stop trying to overcomplicate things and use some UFS method eg DiskSuite to mirror /usr if you haven't got a raid controller.

[reddragdiva]

"They must be accessible by simple UFS means, not ZFS."

Ew. So Solaris 10 can now boot from ZFS, but not quite properly yet?

You mean... You're NOT a Microsoft Certified Systems Administrator..?

[hairyears.livejournal.com]

I am not a geek, I am a Visual Basic for Applications developer, and the text above translates as "Windows NT server works 'straight out of the box' with just a few mouse clicks'.

Re: You mean... You're NOT a Microsoft Certified Systems Administrator..?

[reddragdiva]

So does Solaris! Until you try to do something that isn't how it comes out of the box. Much like NT, except that in Solaris you can actually do it.