elfdump -a in Solaris

Solaris userland is rather specific... A lot of utilities misses convenient options from GNU/BSD analogs. E.g., we don't have "elfdump -a" here... As always, a bit of scripting solves this problem:

elfdump -c /bin/ls |grep Header |awk ' { print $4; }'  |xargs -n 1 -I '{}' elfdump -N '{}' /bin/ls 

Debian Wheezy experience

We are deploying new Proxmox servers and decided to try Debian 7 (testing). We had some problems with old platform. In particular, Squeeze GRUB is not fresh enough to boot from multipath device and had to be replaced with one from testing. The following issues were discovered with Debian 7:

• Installer gave some strange error during network configuration, however it can be related to missing bnx firmware (awful Debian policy, firmware is in non-free repository)
• QLogic FC adapter also tried to load missing firmware, however the system recognized FC-attached device
• Multipath-tools-boot package is partly broken - it installs /usr/share/initramfs-tools/scripts/init-top/multipath which it seems do nothing, but fails to execute during update-initramfs (so I've just deleted it)
• And last, most annoying issue is absence of perl-suid package in repositories, which prevents Proxmox installation from "deb http://download.proxmox.com/debian squeeze pve"

Last issue was frustrating enough for us to revert to Squeeze with its old but well-known bugs and perculiarities.

Notes on configuring two-nodes proxmox cluster with drbd-backed storage

We had a task to deploy two new visualization servers with possibility of live migration and high availability data. The second means that in case of physical server failure you don't want faulted VMs to be powered up automagically on another node, but just that you can do it by hand in five minutes.

We decided to use proxmox VE 2, because it's free, we have experience of maintaining proxmox 1.9 systems and because it supports live migration without shared storage.

So, we configured two nodes with 4 additional LVM volume groups each: for VZ data for each node (n1vz with one lvm mounted on first node on /var/lib/vz and n2vz with one volume mounted on /var/lib/vz/ on second, n1kvm and n2kvm as VM disk storage on each node, n1kvm is used by VMs running normally on first node, n2kvm - by VMs running on second node). 4 DRBD volumes with primary-primary configuration was created for each of 4 volume groups. Using separate pair of drbd devices for VM's disks makes split brain recovery easier, as explained here. And note, we can't use drbd-mirrored (quazy-shared) disk for VZ storage, because final step of VZ migration includes "rm -rf" after rsyncing container private area.

In such configuration we can do live migration of KVM VMs and VZ. Also we have copy of each VM and VZ for emergencies (falling of one node).

Some difficulties we met were related to LVM and DRBD startup ordering. First one was the following: LVM locked drbd backing storage and drbd couldn't use them. It was solved with correct filter in lvm.conf. The other one was more difficult. Physical volumes n1vz and n2vz available over DRBD couldn't be mounted normally - they should be mounted after initial system startup. Usually firstly starts lvm (and init script makes vgchange -ay, activating volume groups), then drbd, and now we have additional VG, but they are not active.

To solve this problem we are supposed to use hearthbeat. But I am too lazy to study it. So I adopted things more familiar to me - automounter (autofs) to mount /var/lib/vz and udev to make volume groups available on drbd* device appearance. I've added "/- /etc/auto.direct" line to /etc/auto.master and created /etc/auto.direct file, containing:

/var/lib/vz              -fstype=ext4            :/dev/mapper/n1vz-data

Configuration of udev consisted from creation of /etc/udev/rules.d/80-drbd-lvm.rules file, containing:

ACTION=="add|change", SUBSYSTEM=="block",KERNEL=="drbd*", RUN+="/bin/sh -c /sbin/lvm vgscan; /sbin/lvm vgchange -a y'"

I consider this more elegant then just including "vgchange -a y && mount ..." in rc.local.

One thing I hate in FreeBSD...

One thing I hate in FreeBSD - is the speed of maintainence operations.
To install (or upgrade) one port you have to wait half an hour...
To update system with custom kernel/userland parameters, you should wait several hours... Then you call mergemaster, which asks you a dozen questions about config files (and most of changes doesn't impact on the current system, but just in case there is something important, you have to review them).
Comparing to "do-release-upgrade" on ubuntu, the speed of operation is unberable. And binary updates just don't do everithing you need (e.g. dtrace world, necessary ports options...)

Two useful illumos-related links

I've just found two interesting resources:

• planet.illumos.org site - it's always interesting to know what is happening in opensource Solaris. I'm still looking at this project. We'll hardly make something like OpenIndiana our main platform, but for me it's a preferrable alternative to Linux where FreeBSD is not the best sollution (e.g. I'd prefer to use something Solaris-based for hardly loaded file server because of some stability issues with ZFS in FreeBSD and we still should decide what to do with our existing SXCE-based terminal servers). And planet.opensolaris.org became very quiet now..


• src.illumos.org - sometimes you just have to look at some subsystem to understand what has gone wrong. Web access to code is useful thing to have for such cases.
  

And I would like to play with stable OpenIndiana release. In particular it's interesting to see how do it deal with multipath on our EMC Clarion storage.

Weird boot0 behavior

It seems I've found one interesting bug in FreeBSD boot0. The long story is the following. We deploy new systems on IBM blade servers. After some local fixes made by Oleg we made FreeBSD gmultipath work with our EMC Clarion storage. However, we observed strange behavior. Periodically systems refused to boot. We could see only black screen and no usual "F1: FreeBSD" boot0 prompt. This was healed by some tricks - in particular we were able to make two servers boot in all cases by some BIOS and FC adapter settings. However, all these tricks (particular FC adapter settings, disabling Planar SAS and unnecessary boot methods in BIOS) didn't help in last case. It took about a week to make last server boot. At last I've found the certain bug in boot0.

I cassually stepped on this. I've tried to install boot2 loader from current and instead of "bsdlabel -e " executed "bsdlabel -w". And system booted 10 times in a row (to a point before mounting local filesystems). Luckily, "/" filesystem was on the first BSD partition. "bsdlabel -w" rewrote partition table and made one partition on whole slice. Root ("/") could be mounted, other filesystems were lost. So, OS was able to boot to a certain stage. There could be two reasons which influenced boot0 behavior: existence of several partitions and their labels. So I recreated partition table and labeled all BSD partitions as "unused" (instead of "4.4BSD"). This helped. System boots now without any random failures (at least it boots 12 times in a row).
I've created a bug entry for this boot0 failure and started to setup a new FreeBSD server... Why do partition labels influence boot0? I don't know, maybe it's related to OS probing (when boot0 writes "F1: FreeBSD" and not "F1: Disk1"). At least, I'm rather happy: we inclined to install Linux on our blades in case if we wouldn't be able to make FreeBSD booting robust. So I shouldn't learn OpenVZ peculiarities and bugs and can use familiar jail infrastructure (we are going to deploy vnet jails). And ZFS for storing jail images and applications' data. This is a killer feature of FreeBSD which made me struggle for a week :)

POSIX rules, bashism sux, and shell is a great tool...

Just one link. Here I found a lot of useful advices and tricks concerning writing scripts in POSIX shell. As I'm really fond of shell, I'd like to share this link...

Vim-compatible web-based text editor

It is not a joke. Google created vim-compatible Web-based text editor :) I don't feel myself such a geek. It is the best editor in command line, but why should anyone use it in GUI environment, especially in user-oriented OS?

Unix as literature

I've just got an excellent link from my workmate Oleg: The Elements Of Style: UNIX As Literature( the same text in Russian, original is better). It's rather interesting: a man with humanitarian education finds Unix more simple and effective then Windows NT (the article dates to 1999). I think I should mention this article to our students. They complain of Unix-like systems being too uncomfortable for them :)