Home > clusterware > CRS-4000: Command Replace failed, or completed with errors.

CRS-4000: Command Replace failed, or completed with errors.

September 18, 2014 Leave a comment Go to comments

Today, I tried to restore OCR and VOTE Diskgroup from backup. I re-created the OCR and VOTE diskgroup with the following command

CREATE DISKGROUP VOTEDG NORMAL REDUNDANCY
DISK ‘/dev/asmdisk/ycfmsvot’
DISK ‘/dev/asmdisk/jcfmsvot’
QUORUM DISK ‘/nfsvoting/nfsvotedk’
ATTRIBUTE ‘compatible.asm’ = ‘11.2.0.0’;

After that, I could restore the OCR from backup. But when I replaced the vote disks on the VOTEDG diskgroup, it showed errors as

root@cfmsvm2:/dev/asmdisk# crsctl replace votedisk +VOTEDG
CRS-4602: Failed 27 to add voting file b594600fe26d4f2dbff4f31c84969822.
CRS-4602: Failed 27 to add voting file 7cb1bfa1d53e4f67bf5e656d38ffb025.
CRS-4602: Failed 27 to add voting file 407128acec804ff5bfdc32047f07c7ba.
Failed to replace voting disk group with +VOTEDG.
CRS-4000: Command Replace failed, or completed with errors.

And the ASM alert showed the messages below

NOTE: [crsctl.bin@cfmsvm2 (TNS V1-V3) 13029] opening OCR file
NOTE: updated gpnp profile ASM diskstring:
2014-09-18 11:35:24.225000 +08:00
NOTE: Creating voting files in diskgroup VOTEDG
NOTE: Voting File refresh pending for group 1/0x84c09605 (VOTEDG)
NOTE: Attempting voting file creation in diskgroup VOTEDG
NOTE: voting file allocation on grp 1 disk VOTEDG_0002
NOTE: voting file allocation on grp 1 disk VOTEDG_0000
2014-09-18 11:35:25.375000 +08:00
NOTE: voting file allocation on grp 1 disk VOTEDG_0001
NOTE: Attempting voting file refresh on diskgroup VOTEDG
NOTE: Voting file relocation is required in diskgroup VOTEDG
NOTE: Attempting voting file relocation on diskgroup VOTEDG
NOTE: voting file deletion on grp 1 disk VOTEDG_0000
NOTE: voting file deletion on grp 1 disk VOTEDG_0001
NOTE: voting file deletion on grp 1 disk VOTEDG_0002
NOTE: Failed voting file relocation on diskgroup VOTEDG

Finally, I found the QUORUM DISK ‘/nfsvoting/nfsvotedk’ size was not equal to size of  disk ‘/dev/asmdisk/ycfmsvot’.

root@cfmsvm2:/nfsvoting# ls -al
total 409604
drwxr-xr-x   2 grid     oinstall      64 Sep  3 14:34 .
drwxr-xr-x  25 root     root          29 Sep 17 16:13 ..
-rwxrwxr-x   1 grid     oinstall 209715200 Sep 18 11:38 nfsvotedk <– the size was about 200M

 

Fix Procedure

1. Drop the quorum disk

SQL>  alter system set asm_diskstring=’/dev/asmdisk/*’,’/nfsvoting/*’;

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot
VOTEDG_0002                    VOTEDG_0002                    /nfsvoting/nfsvotedk

SQL> ALTER DISKGROUP VOTEDG DROP QUORUM DISK ‘VOTEDG_0002’;

Diskgroup altered.

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot

2. Re-create the quorum disk

dd if=/dev/zero of=nfsvotedk bs=1024k count=10240

3. Re-add the quorum disk

SQL> ALTER DISKGROUP VOTEDG ADD  QUORUM DISK ‘/nfsvoting/nfsvotedk’;

Diskgroup altered.

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
/dev/asmdisk/jasmdk1
/dev/asmdisk/yasmdk1
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot
VOTEDG_0002                    VOTEDG_0002                    /nfsvoting/nfsvotedk

4. Replace VOTEDISK  successfully

bash-4.1$ crsctl replace votedisk +VOTEDG
Successful addition of voting disk 49612e9331d64ff5bf59c81c665e201c.
Successful addition of voting disk 3f45109f98254f62bff4947648761a16.
Successful addition of voting disk 4dff0f244aeb4feabf2df797dbdab091.
Successfully replaced voting disk group with +VOTEDG.
CRS-4266: Voting file(s) successfully replaced

 

 

 

 

Advertisements
Categories: clusterware
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: