Voting Disk Lies (CRS-4000)

Add this to the category of annoyingly unhelpful error messages.

I’m working on a mostly-automated process to create a new cluster by cloning another existing cluster. After running OUI (Oracle Universal Installer – called by config.sh to just run config assistants) there is a single ASM diskgroup which contains both the OCR and Voting Disk; however I wanted to switch the voting disks over to some different physical devices.

Upon which I received this errors:

(root)# /oracle/11203/grid/bin/crsctl replace votedisk +CLST3_VOTING
Failed to create voting files on disk group CLST3_VOTING.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.

Lovely… so very informative. And just to be clear, it didn’t complete with errors, it completely failed. Thanks Oracle.

Maybe I used the wrong command to switch the voting disk? Maybe I need to start CRS in the special bootstrap mode? (Remember that the 11g docs apply to 11.2.0.1 only; for 11.2.0.2 and newer you need to use Oracle Support Notes like 1062983.1.) Maybe I should search oracle support for this error message? (Nada.)

Actually the Oracle Support knowledge base was a good idea. Note 1526096.1 didn’t solve my problem but it mentioned this error (CRS-4000) and it gave me a good lead: looking in the ASM alert log.

Aaaaand there’s our answer. :)

Mon Oct 07 10:04:24 2013
NOTE: [crsctl.bin@server1 (TNS V1-V3) 29941] opening OCR file

Mon Oct 07 10:04:24 2013
NOTE: updated gpnp profile ASM diskstring: /dev/mapper/*

Mon Oct 07 10:04:24 2013
NOTE: Creating voting files in diskgroup CLST3_VOTING

Mon Oct 07 10:04:24 2013
NOTE: Voting File refresh pending for group 2/0xb1dd29a2 (CLST3_VOTING)
NOTE: Attempting voting file creation in diskgroup CLST3_VOTING
ERROR: Voting file allocation failed for group CLST3_VOTING
Errors in file /oracle/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_29953.trc:
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher
NOTE: Attempting voting file refresh on diskgroup BMCLST3_VOTING

Would’ve been nice to put that message on the console instead of only dropping it in the alert log – but at least now I’m more likely to remember checking the alert log if the console doesn’t make sense!