Adding vdev to raidz pool

There is one fundamental flaw in raidz ZFS pools: you can't add devices to raidz (as opposed to usual raid5). So, could we do anything to expand our pool without full rebuilding? Yes, if we are ready to sacrifice one more disk.
From zpool man page:

A pool can have any number of virtual devices at the top of the config‐
uration (known as "root vdevs"). Data is dynamically distributed across
all top-level devices to balance data among  devices.  As  new  virtual
devices are added, ZFS automatically places data on the newly available
devices.

So, to add some capacity to our pool, we can add one more raidz to it. Let's look at this process. Firstly, let's create test pool from three "disks":

freebsd# foreach i ( 1 2 3 4 5 6)
foreach? dd if=/dev/zero of=/tmp/disk$i bs=100M count=1
foreach? end
freebsd# zpool create testpool raidz /tmp/disk1 /tmp/disk2 /tmp/disk3

Now, let's see what we have:

freebsd# zpool status testpool
  pool: testpool
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        testpool        ONLINE       0     0     0
          raidz1        ONLINE       0     0     0
            /tmp/disk1  ONLINE       0     0     0
            /tmp/disk2  ONLINE       0     0     0
            /tmp/disk3  ONLINE       0     0     0

errors: No known data errors
freebsd# zpool list testpool
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
testpool   286M   156K   286M     0%  ONLINE  -
freebsd# zfs list testpool
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool  95.9K   158M  28.0K  /testpool

Now let's expand this pool:

freebsd# zpool add testpool raidz /tmp/disk4 /tmp/disk5 /tmp/disk6

So, we recieved the following configuration:

freebsd# zpool status testpool
  pool: testpool
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        testpool        ONLINE       0     0     0
          raidz1        ONLINE       0     0     0
            /tmp/disk1  ONLINE       0     0     0
            /tmp/disk2  ONLINE       0     0     0
            /tmp/disk3  ONLINE       0     0     0
          raidz1        ONLINE       0     0     0
            /tmp/disk4  ONLINE       0     0     0
            /tmp/disk5  ONLINE       0     0     0
            /tmp/disk6  ONLINE       0     0     0

errors: No known data errors
freebsd# zpool list testpool
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
testpool   572M   210K   572M     0%  ONLINE  -
freebsd# zfs list testpool
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool   114K   349M  28.0K  /testpool

Let's compare this to raidz from 6 disks:

freebsd# zpool create testpool raidz /tmp/disk1 /tmp/disk2 /tmp/disk3 /tmp/disk4 /tmp/disk5 /tmp/disk6
freebsd# zpool status testpool
  pool: testpool
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        testpool        ONLINE       0     0     0
          raidz1        ONLINE       0     0     0
            /tmp/disk1  ONLINE       0     0     0
            /tmp/disk2  ONLINE       0     0     0
            /tmp/disk3  ONLINE       0     0     0
            /tmp/disk4  ONLINE       0     0     0
            /tmp/disk5  ONLINE       0     0     0
            /tmp/disk6  ONLINE       0     0     0

errors: No known data errors
freebsd# zpool list testpool
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
testpool   572M   147K   572M     0%  ONLINE  -
freebsd# zfs list testpool
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool   112K   443M  34.9K  /testpool

So, we lost more then 21% of space, but expanded our pool without downtime. Firstly, the lost of more then 20% space suprised me, but 20% was expectable (in second case only 5 disks, in first - 4, so 20% is the difference).
Let's see, where did 1 percent go. 443 MB is useful in second case, it means about 11.4 MB (100 /*disk size*/ - 443/*useful space*/ / 5 /*useful disks*/) metadata per disk. In first case we have 349 MB. It is provided by two pools with 2 useful disks in each. So, wasted space per disk is 12.75 MB (100 - 349/2/2). It seems, in this configuration we have a bit more metadata.

PostgreSQL 9.1 create extension and database upgrade

Yesterday I was eager to try some new 9.1 features, in particular to experiment with exclude constraints (I've read they were improved since 9.0). So, I decided to upgrade my test 9.0 installation. It worth mentioning that it was 8.4 installation, previously upgraded to 9.0.
Everything went as expected, even my C functions were successfully recompiled without any warnings.
So, I decided to try extensions. I had hstore module installed in my db, so I issued "create extension" and received the following error:

db=# CREATE EXTENSION hstore WITH SCHEMA public FROM unpackaged;
ERROR:  function hstore_recv(internal) does not exist

I looked at hstore definition in /usr/local/share/postgresql/extension/hstore--1.0.sql, copied definition of this function, created it, but received error saying that some other function doesn't exist.
So I run full script to create unpackaged extension:

# sed -e 's|MODULE_PATHNAME|$libdir/hstore|' < /usr/local/share/postgresql/extension/hstore--1.0.sql | su pgsql -c 'psql -d db ' 

After that it was possible to create extension:

db=# CREATE EXTENSION hstore WITH SCHEMA public FROM unpackaged; 

It seems that the reason for this behavior is that I didn't upgrade hstore module after upgrading database from 8.4 to 9.0...

Extracting domain name from uri with SQL

I don't know if it is effective, but this ugly SQL (in PostgreSQL dialect) extracts domain name part from uri (i.e. translates protocol://something/page into "something"):

SELECT uri, substring (uri from position('//' in uri)+2 for ( char_length(uri) - char_length(substring (uri from position ('/' in substring ( uri from position('//' in uri)+2 )) ) )))  from proxy_data

It seems, it would look better with regexp:

 
select uri, substring ( substring (uri from '//[[:alnum:]\._\-]+') from  3) from proxy_data

However, in later case we should list all possible symbols in domain name in our regexp.

Change user password in script on FreeBSD host

Changing user password in script on FreeBSD host turned out to be very simpe. Use "pw" for this purpose:

echo "user_password" | pw usermod  -V /jails/myjail/etc -h 0 username

KVM is ported to Illumos

Great news. KVM is ported to Illumos by Joyent. Currently only Intel CPUs are supported. I crave for KVM on OpenIndiana with support for AMD CPUs...

SMS notifications in OpenNMS

We just got a lot of problems when our air conditioning system controller in data center went mad and didn't notice the failure of two air conditioners. The 60 Celsius degrees is not the best temperature for servers operations...
After dealing with this situation we decided to setup SMS-notification service at least for temperature in server room and several other parameters. We already had OpenNMS monitoring system configured, so we had to add SMS-notification to our setup.
This task was done in several steps. First of all, we created a script which would send SMS. We used Google Calendar for this purpose. One dedicated user (let's say opennms) was created for our monitoring system and all system administrators imported his calendar with the following notification settings:

• Events reminders - By default remind me via sms 1 minutes before each event


• New Invitations - SMS

Of course, system administrators had to register their phone numbers in Google Calendar.
When we want to send SMS notification, we create new event in opennms's calendar using gcalcli for this purpose.
We used the following script to create a new event (sleep was inserted to prevent mass event creation in case when everything is bad):

#!/bin/sh
H=$(/bin/date "+%H")
M=$(/bin/date "+%M")
sleep 1
event_text="$H:$M $@"
/usr/local/bin/gcalcli --user opennms --pw OurPassword --cals=owner quick "$event_text"  

Then, we described a notification command in /usr/local/opennms/etc/notificationCommands.xml:

    <command binary="true">
        <name>SendSMS</name>
        <execute>/root/bin/send_sms.sh</execute>
        <comment>Send SMS by gcalcli</comment>
        <argument streamed="false">
           <switch>-subject</switch>
        </argument>
        <argument streamed="false">
           <switch>-tm</switch>
        </argument> 
    </command>

Argument elements describe script parameters, here we pass it notice subject and notice body (full list of possible parameters may be found here).

Later we created a destinationPath in /usr/local/opennms/etc/destinationPaths.xml:

    <path name="SMS-Admins" initial-delay="0s">
        <target interval="0s">
            <name xmlns="">admin</name>
            <autoNotify xmlns="">auto</autoNotify>
            <command xmlns="">SendSMS</command>
        </target>
    </path>

You have to set some legal OpenNMS user or group here (in name target's sub-element).

Now you can create some notifications using OpenNMS Web UI. To control outstanding characteristics of certain parameters you should set trigger event to uei.opennms.org/threshold/highThresholdExceeded and set destinationPath for it to "SMS-Admins". You can do it by hand, adding the following entry to /usr/local/opennms/etc/notifications.xml:

    <notification name="High Threshold" status="on" writeable="yes">
        <uei xmlns="">uei.opennms.org/threshold/highThresholdExceeded</uei>
        <description xmlns="">High threshold exceeded</description>
        <!-- some filter -->
        <rule xmlns="">(NODELABEL = 'our label')</rule>
        <destinationPath xmlns="">SMS-Admins</destinationPath>
        <text-message xmlns="">The parameter %parm[ds]% is high on node: %nodelabel%, interface:%interface%. The parameter %parm[ds]% reached a value of %parm[value]% while the threshold is %parm[threshold]%. This threshold for this alert was %parm[threshold]%.</text-message>
        <subject xmlns="">Notice #%noticeid%</subject>
        <numeric-message xmlns="">111-%noticeid%</numeric-message>
    </notification>

The only interesting question I have now is how to do notification only for several thresholds. Just for now I deleted all unimportant (for me) thresholds. It would be cool, however, to specify instead in notification description only thresholds you are interested in...

Drinkers guide to PostgreSQL

Look at thisif you haven't seen it. I liked it. "You tried to use pg_upgrade" costs you 9 glasses of beer...