Originally posted on on Tue, 04/07/2009
I had very unpleasant experience with RAC and ASM. I'd like to share it. Yesterday after VMware ESX Server upgrade (and maybe unsuccessfull live motion operations on one of two RAC nodes) I had the following records in asm instance alert log:
SQL> ALTER DISKGROUP ALL MOUNT Mon Apr 6 15:48:52 2009 NOTE: cache registered group DATA number=1 incarn=0x3ad84292 NOTE: cache registered group FRA number=2 incarn=0x3b084293 NOTE: cache registered group LOGS number=3 incarn=0x3b084294 Mon Apr 6 15:48:52 2009 ERROR: no PST quorum in group 1: required 2, found 0 Mon Apr 6 15:48:52 2009 NOTE: cache dismounting group 1/0x3AD84292 (DATA) NOTE: dbwr not being msg'd to dismount ERROR: diskgroup DATA was not mounted Mon Apr 6 15:48:52 2009 ERROR: no PST quorum in group 2: required 2, found 0 Mon Apr 6 15:48:52 2009 NOTE: cache dismounting group 2/0x3B084293 (FRA) NOTE: dbwr not being msg'd to dismount ERROR: diskgroup FRA was not mounted Mon Apr 6 15:48:52 2009 ERROR: no PST quorum in group 3: required 2, found 0 Mon Apr 6 15:48:52 2009 NOTE: cache dismounting group 3/0x3B084294 (LOGS) NOTE: dbwr not being msg'd to dismount ERROR: diskgroup LOGS was not mounted ...Diskgroups disappeared, and asm didn't want to see them. However, I could see /dev/sd* disks, where diskgroups were placed, and permissions were right: disks were owned by oracle user. select * from v$asm_diskgroup ; gave nothing and select path,header_status from v$asm_disk ; said that all disks were in provisioned state. It was awful, but in the end I've found sollution. I've noticed that kfed sees diskgroup and other attribute on the affected disks, but parameter kfdhdb.acdb.ub2spare is not 0 (as I found in Internet, it was its usual state). So I've dumped header info:
$ kfed read /dev/sdf > /tmp/sdf.noop.mod $ vi /tmp/sdf.noop.mod #Changed kfdhdb.acdb.ub2spare to 0 $ kfed op=write dev=/dev/sdf text=/tmp/sdf.noop.mod CHKSUM=YESAfter this I could do "alter diskgroup fra mount" in asm instance, repeated this procedure for DATA asm diskgroup, after that I could mount diskgroups and recover my database. In conclusion I wish say that we are going to move our fra to OCFS2 fs the next weekend.
