One big Oracle ASM adventure

Well, this has happened long ago, but as I'm going to decomission http://dbseminar.r61.net, I think this information should be moved here...

Originally posted on on Tue, 04/07/2009

I had very unpleasant experience with RAC and ASM. I'd like to share it. Yesterday after VMware ESX Server upgrade (and maybe unsuccessfull live motion operations on one of two RAC nodes) I had the following records in asm instance alert log:

SQL> ALTER DISKGROUP ALL MOUNT
Mon Apr  6 15:48:52 2009
NOTE: cache registered group DATA number=1 incarn=0x3ad84292
NOTE: cache registered group FRA number=2 incarn=0x3b084293
NOTE: cache registered group LOGS number=3 incarn=0x3b084294
Mon Apr  6 15:48:52 2009
ERROR: no PST quorum in group 1: required 2, found 0
Mon Apr  6 15:48:52 2009
NOTE: cache dismounting group 1/0x3AD84292 (DATA)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DATA was not mounted
Mon Apr  6 15:48:52 2009
ERROR: no PST quorum in group 2: required 2, found 0
Mon Apr  6 15:48:52 2009
NOTE: cache dismounting group 2/0x3B084293 (FRA)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup FRA was not mounted
Mon Apr  6 15:48:52 2009
ERROR: no PST quorum in group 3: required 2, found 0
Mon Apr  6 15:48:52 2009
NOTE: cache dismounting group 3/0x3B084294 (LOGS)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup LOGS was not mounted

...

Diskgroups disappeared, and asm didn't want to see them. However, I could see /dev/sd* disks, where diskgroups were placed, and permissions were right: disks were owned by oracle user. select * from v$asm_diskgroup ; gave nothing and select path,header_status from v$asm_disk ; said that all disks were in provisioned state. It was awful, but in the end I've found sollution. I've noticed that kfed sees diskgroup and other attribute on the affected disks, but parameter kfdhdb.acdb.ub2spare is not 0 (as I found in Internet, it was its usual state). So I've dumped header info:

$ kfed read /dev/sdf > /tmp/sdf.noop.mod
$ vi /tmp/sdf.noop.mod  #Changed kfdhdb.acdb.ub2spare to 0
$ kfed op=write dev=/dev/sdf text=/tmp/sdf.noop.mod CHKSUM=YES

After this I could do "alter diskgroup fra mount" in asm instance, repeated this procedure for DATA asm diskgroup, after that I could mount diskgroups and recover my database. In conclusion I wish say that we are going to move our fra to OCFS2 fs the next weekend.