10 rules for choosing scalable DBMS suitable for your OLTP application

I've just read "Ten Rules for Scalable Performance in Simple Operation Datastores" by Michael Stonebraker and Rick Cattell. We've just discussed the scalability question at work. So, I've found this article quite in time.
The authors state 10 rules for scalable systems performing simple operations over data (read or write a few items).
In short, the rules are:

1. Only shared-nothing system can scale well (in case of adequate table partitioning and replicating read-mostly data). Shared-disk systems suffer from synchronization overheads.
   
2. High level languages can be fast. Compare, e.g. slow relational DBMS and fast navigational - e.g. network or hierarchical - who did win the most of the market share? At first SQL DBMS were considered extremely slow, but just for now optimizer is much smarter then most of the programmers. Also, API matters. However, stored procedures are necessary to achieve adequate performance (eliminating communication and query compilation overhead).
   
3. Your application (or DBMS) should leverage huge memory sizes (64 GB per 16 hosts - a terabyte of main memory is not a dream). However, existing DBMS have a lot of overhead in query processing - locking, logging, maintaining buffer pool, multi-thread locking (latching) overhead).
   
4. HA and automatic recovery is necessary for scalable application. HA includes possibility of automatic data repartitioning, normal operations during DoS, hardware failures, lost network connectivity, application errors, DBMS errors and so on. Sounds like a dream and not fully achivable. But at least we should avoid outages for the most situations and part of recovery actions should be done automatically (e.g. data repartitioning after node failure).
   
5. On-line everything. All operations should be on-line: adding nodes, performing updates, changing data schema and so on.
   
6. Avoid multi-node operations. You should select partitioning key properly to avoid multi-node operations and synchronizations. Such operations just can't be fast.
   
7. Don't build ACID yourself. If you need it (and in most cases you do), use available solutions (DBMSs). CAP theorem is not always a principal thing - ACID is more precious then impossibility of cluster partitioning. Modern LAN technologies allow you build redundant network, and loose of WAN-connected office is not a great problem - using WAN-connected nodes in your cluster transparently is madness because of network delays. If connection breaks, replication may be restarted later.
   
8. The system must not have a lot of knobs. A system should be easy to administer and be able to do auto tuning. I wish my DBMS were so clever :)
   
9. Node performance should be considered. 20 monster nodes is much better then 1000 half-broken outdated junk. And supporting them is usually much cheaper.
   
10. OpenSource makes you free. Without comments. I'd like to eliminate our outdated Oracle installation. We can't update it because it's too expensive, and can't rewrite our system from scratches....
    

I can't help but agreeing with them...

DLink DIR-615

I've just spent about an hour on DLink DIR-615. This piece of ... hardware just didn't want to establish PPTP connection with my provider (BeeLine). Luckily after about two hours of tambourine dance, I'm on-line... Firmware update from 3.02RU to 3.03RU solved my problem. And I can't find how to enable telnet access on this device. It's awful - I have to configure router using my browser...
At least, this device has a very pleasant price - about $50. This is its best feature...

ZFS snapshots or something new on something old

Everyone knows that snapshots are fascinating feature of ZFS. Automatic snapshots in OpenSolaris used to save my mental health. After migration to FreeBSD I thought "It would be easy. There are a lot of decisions for managing ZFS snapshots in FreeBSD". In fact, theare are too many decisions for managing ZFS snapshots in FreeBSD. It's hard to choose one decision. I've looked for example, on sysutils/zfsnap, sysutils/zfs-snapshot-mgmt... But my needs are quite different:

• I'd like the decision for making automatic snapshots to be simple and effective as an axe.
• I'd like to understand how every line of these scripts (for creating and deleting snapshots) work to feel myself comfortable.
• I'm going to use snapshots on my home desktop, and I'd only like to have quite recent copy of my data to recover files after making some silly mistakes.

At last, I decided to create my own solution which was supposed to work as rc script (I boot my computer once or twice per day, and it would be good to have startup snapshot of the system).
The text of script is provided below.

#!/bin/sh

. /etc/rc.subr

name="zfs_snapshot"

start_cmd="create_snapshots"
stop_cmd="drop_snapshots"
zfs_pool=${zfs_snapshot_pool:-"zpool"}
keep_days=${zfs_snapshot_keep_days:-30}

create_snapshots()
{
  echo "Creating zfs snapshots..."

  if [ -z "${zfs_snapshot_filesystems}" ]; then
     snapshot_filesystems=`zfs list -H -o name -t filesystem`;
  else 
     snapshot_filesystems=${zfs_snapshot_filesystems};
  fi

  
 
  DATE=`env LC_ALL=C date -j "+%Y_%m_%d-%H:%M"`
  for fs in $snapshot_filesystems ; do 
   snap="${fs}@AUTO_${DATE}"
   logger -t $name "Creating snapshot $snap"
   zfs snapshot $snap
  done
}

drop_snapshots()
{
  echo "Destroying old automatic zfs snapshots"
 
  current_date=`date "+%s"`
  if [ -z "${zfs_snapshot_filesystems}" ]; then
     snapshot_filesystems=`zfs list -H -o name -t filesystem`;
  else
     snapshot_filesystems=${zfs_snapshot_filesystems};
  fi
 
  for fs in $snapshot_filesystems ; do
     snapshots=`zfs list -H -t snapshot -o name -r $fs |grep "^${fs}@AUTO_"`;
     for snap in $snapshots; do 
       creation_local=`zfs list -H -t snapshot -o creation $snap`
       creation_real=`date -j -f "%a %b %d %k:%M %Y" "${creation_local}" "+%s"`
       
       eval creation_minus=$(( ( ${current_date} - ${creation_real})/3600/24 ))
       if [ $creation_minus -gt $keep_days ]; then
         logger -t $name "Destroying snapshot $snap"
         zfs destroy $snap
       fi
     done
  done

}

load_rc_config $name
run_rc_command $*

There are three parameters for my script - zfs_snapshot_pool, zfs_snapshot_filesystems and zfs_snapshot_keep_days. First or second parameter may be used to specify pool (and all its FS for creating snapshots) or explicitly selecting necessary filesystems. Parameter fs_snapshot_keep_days determines when snapshot is considered outdated.
This simple script is supposed to make my life more comfortable...

Is our favourite government fucken crazy?

It's not a dream of some moron, it's real bill. Hail to Mr. Mikhalkov and Mr. Putin! It seems some ancient morons created quite idiotic law...
1) We have to pay 1 % from the cost of CD/DVD disks, Hard drives and other "data stores" (материальные носители) for some organization which distribute this money among authors.
2) This organization is not even formed.
3) And what if I use my flash stick to backup my home directory or to carry my own presentation or even to setup Ubuntu or FreeBSD on it?
5) And if I've paid this taxes in the store, could I freely distribute copies of some good film (not Mikhalkov's one, but something really interesting)? It seems not. Some contradiction...

If I buy a new flash stick, it will be more expensive by 10-30 roubles (about 30 cents - $1). Not much. But if we summ all these money, we have a good amount to steal.

Hail to our government! The next thing we need is a tax on the amount of trees in the garden. And these taxes we will spread between.... let's suppose... agronomists (they created seeds for you... You might have bought these seeds but who cares).

Chromium was added to FreeBSD ports tree

Chromium was added to FreeBSD ports tree. I'm going to try it... Currently I use Konqueror, but it is not always perfect. And Firefox wants too much memory, works quite slow and (the most annoying for me) doesn't look good with KDE :)

Objects have failed

Interesting view on object-oriented programming languages. Full text is here. I was always wondering how could someone do such a ugly language as C++ from such a nice language as C. The author goes further and says that "programmer in Fortran can write a Fortran program in any language", or something like that. He describes different problems of modern object-oriented languages and he seems to be right. I doubt that FOL-based or LISP-like languages will be successful as general-purpose languages in this decade... However, it would be interesting to see their further develoment.

Unix as literature

I've just got an excellent link from my workmate Oleg: The Elements Of Style: UNIX As Literature( the same text in Russian, original is better). It's rather interesting: a man with humanitarian education finds Unix more simple and effective then Windows NT (the article dates to 1999). I think I should mention this article to our students. They complain of Unix-like systems being too uncomfortable for them :)