Add this to the category of annoyingly unhelpful error messages.
I’m working on a mostly-automated process to create a new cluster by cloning another existing cluster. After running OUI (Oracle Universal Installer – called by config.sh to just run config assistants) there is a single ASM diskgroup which contains both the OCR and Voting Disk; however I wanted to switch the voting disks over to some different physical devices.
Upon which I received this errors:
(root)# /oracle/11203/grid/bin/crsctl replace votedisk +CLST3_VOTING Failed to create voting files on disk group CLST3_VOTING. Change to configuration failed, but was successfully rolled back. CRS-4000: Command Replace failed, or completed with errors.
Lovely... so very informative. And just to be clear, it didn't complete with errors, it completely failed. Thanks Oracle.
Maybe I used the wrong command to switch the voting disk? Maybe I need to start CRS in the special bootstrap mode? (Remember that the 11g docs apply to 184.108.40.206 only; for 220.127.116.11 and newer you need to use Oracle Support Notes like 1062983.1.) Maybe I should search oracle support for this error message? (Nada.)
Actually the Oracle Support knowledge base was a good idea. Note 1526096.1 didn't solve my problem but it mentioned this error (CRS-4000) and it gave me a good lead: looking in the ASM alert log.
Aaaaand there's our answer. :)
Mon Oct 07 10:04:24 2013 NOTE: [crsctl.bin@server1 (TNS V1-V3) 29941] opening OCR file Mon Oct 07 10:04:24 2013 NOTE: updated gpnp profile ASM diskstring: /dev/mapper/* Mon Oct 07 10:04:24 2013 NOTE: Creating voting files in diskgroup CLST3_VOTING Mon Oct 07 10:04:24 2013 NOTE: Voting File refresh pending for group 2/0xb1dd29a2 (CLST3_VOTING) NOTE: Attempting voting file creation in diskgroup CLST3_VOTING ERROR: Voting file allocation failed for group CLST3_VOTING Errors in file /oracle/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_29953.trc: ORA-15221: ASM operation requires compatible.asm of 18.104.22.168.0 or higher NOTE: Attempting voting file refresh on diskgroup BMCLST3_VOTING
Would've been nice to put that message on the console instead of only dropping it in the alert log - but at least now I'm more likely to remember checking the alert log if the console doesn't make sense!