ORA-01157: cannot identify/lock data file 44 – see DBWR trace file

August 31, 2015 Leave a comment

Problems

  1. Application server showed java.sql.SQLException: ORA-01157: cannot identify/lock data file 44 – see DBWR trace file
  2. Node 2 of RAC was restarted 2 days ago and the diskgroups were umounted.
  3. But the corresponding tablespaces on the database were online
  4. And select file_name to show following errors

SQL> select file_name from dba_data_files;
ERROR:
ORA-01157: cannot identify/lock data file 44 – see DBWR trace file
ORA-01110: data file 44: ‘+DATA4/ispro/datafile/fttxwkf_tbs.1021.885226031’

Fixed

  1. Login ASM
  2. Mount the diskgroup
  3. The state of diskgroup were changed to mounted

SQL> select name, state from  v$asm_diskgroup;

NAME                           STATE
—————————— ———–
DATA1                          CONNECTED
DATA2                          MOUNTED
DATA3                          MOUNTED
DATA4                          MOUNTED

4. But select file_name, still show following errors

SQL> select file_name from dba_data_files;
ERROR:
ORA-01157: cannot identify/lock data file 44 – see DBWR trace file
ORA-01110: data file 44: ‘+DATA4/ispro/datafile/fttxwkf_tbs.1021.885226031’

5. Try online tablespace again by the following command, the problem got fixed since the state of DATA4 changed from MOUNTED to CONNECTED

SQL>  alter tablespace FTTXWKF_TBS online;

Tablespace altered.

SQL> select name, state from  v$asm_diskgroup;

NAME                           STATE
—————————— ———–
DATA1                          CONNECTED
DATA2                          MOUNTED
DATA3                          MOUNTED
DATA4                          CONNECTED

Categories: Uncategorized

sendmail show “domain.com: Name server timeout”

April 9, 2015 Leave a comment

After I setup sendmail relay on the Red Hat Enterprise Linux Server 5.5 based on some online docuements, it always tells the following steps

1. Change /etc/mail/sendmail.mc entry “dnl define(`SMART_HOST’, `smtp.your.provider’)dnl” to “define(`SMART_HOST’, `[mailserver1][mailserver2]’)dnl”

2. Remake the /etc/mail/sendmail.cf by the command make

3. Restart the sendmail services

# service sendmail restart

But when I did sendmail testing, it showed error below. I searched on the internet, there are no any workable clue.

domain.com: Name server timeout

… Transient parse error — message queued for future delivery

… queued

Finally, I found that the procedure for setup sendmail relay above, it assums that DNS has been setup. Therefore, if there is no DNS, extra setup procedures are required.

1. Uncomment the entry below in /etc/mail/submit.mc

define(`confDIRECT_SUBMISSION_MODIFIERS’,`C’)dnl

2. Create /etc/mail/service.switch with the content below

####### start of file #########
hosts files
aliases files
####### end of file ###########

3. Restart sendmail

Categories: Uncategorized

Error accessing package DBMS_APPLICATION_INFO

February 4, 2015 Leave a comment

After migrating oracle database from 7.3.4.0 to Oracle 12c1, some old oracle 7.3.4 clients showed the errors below

Error accessing package DBMS_APPLICATION_INFO
ERROR:
ORA-06550: line 1, column 40:
PLS-00553: character set name is not recognized
ORA-06550: line 0, column 0:
PL/SQL: Compilation unit analysis terminated

When asking oracle support, of course, they said that it was not support and upgraded to latest clients.Luckly, I found that it was caused by the database character set. After I recreated the database with the default characeter set  WE8MSWIN1252. The problem was fixed.

Categories: ORA-XXX

Dynamic reconfiguration of the virtual device on domain failed with error code (-122).

November 20, 2014 Leave a comment

Problem Symptons

root@control:~# ldm remove-vdisk guestdk guest
Dynamic reconfiguration of the virtual device on domain guest
failed with error code (-122).
The OS on domain guest did not report a reason for the failure.
Check the logs on that OS instance for any further information.
Failed to remove vdisk instance

root@control:~# ldm remove-vdisk -f guestdk guest
Dynamic reconfiguration of the virtual device on domain guest
failed with error code (-122).
The OS on domain guest did not report a reason for the failure.
Check the logs on that OS instance for any further information.
Failed to remove vdisk instance
The -f option has no effect because the device cannot be removed safely

Cause: The disk is still in use in domain even though the ZFS filesystem is unmounted.

Fixed: Have to export the zpool of the ZFS filesystem in the guest domain

Procedures:
Login Guest:
root@guest:~# zfs unmount /data
root@guest:~# zpool export data
Login Control Domain
root@control:~#  ldm remove-vdisk -f guestdk guest

Categories: Uncategorized

[ctssd(1883)]CRS-2409:The clock on host XXXXX is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

November 13, 2014 Leave a comment

Today, just finished to install Oracle 12c1 RAC on Solars 11 SPRACH. The GI alertxxxx.log showed the message below

[ctssd(1883)]CRS-2409:The clock on host <hostname> is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

From the oracle support, it explained that it caused by the ntp slew_always not setup to true.

root@XXXX:~# svcprop ntp |grep slew

config/slew_always boolean false
root@XXXX:~# svccfg -s svc:/network/ntp:default setprop config/slew_always=true
root@XXXX:~# svcadm refresh ntp
root@XXX:~# svcprop ntp | grep slew
config/slew_always boolean true

Then, restart the GI for refresh ctssd.

Categories: Uncategorized

ORA-15031: disk specification matches no disks

September 18, 2014 Leave a comment

Added a disk into a diskgroup, it showed errors as below

SQL> ALTER DISKGROUP VOTEDG ADD  QUORUM DISK ‘/nfsvoting/nfsvotedk’;
ALTER DISKGROUP VOTEDG ADD  QUORUM DISK ‘/nfsvoting/nfsvotedk’
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15031: disk specification ‘/nfsvoting/nfsvotedk’ matches no disks
ORA-15014: path ‘/nfsvoting/nfsvotedk’ is not in the discovery set

 

The problem was caused by the asm_diskstring was not set. I set the asm_diskstring. The problem get fixed.
SQL>  alter system set asm_diskstring=’/dev/asmdisk/*’,’/nfsvoting/*’;

System altered.

SQL> show parameter asm_dis

NAME                                 TYPE        VALUE
———————————— ———– ——————————
asm_diskgroups                       string      VOTEDG
asm_diskstring                       string      /dev/asmdisk/*, /nfsvoting/*
SQL> ALTER DISKGROUP VOTEDG ADD  QUORUM DISK ‘/nfsvoting/nfsvotedk’;

Diskgroup altered.

Categories: ASM

CRS-4000: Command Replace failed, or completed with errors.

September 18, 2014 Leave a comment

Today, I tried to restore OCR and VOTE Diskgroup from backup. I re-created the OCR and VOTE diskgroup with the following command

CREATE DISKGROUP VOTEDG NORMAL REDUNDANCY
DISK ‘/dev/asmdisk/ycfmsvot’
DISK ‘/dev/asmdisk/jcfmsvot’
QUORUM DISK ‘/nfsvoting/nfsvotedk’
ATTRIBUTE ‘compatible.asm’ = ‘11.2.0.0’;

After that, I could restore the OCR from backup. But when I replaced the vote disks on the VOTEDG diskgroup, it showed errors as

root@cfmsvm2:/dev/asmdisk# crsctl replace votedisk +VOTEDG
CRS-4602: Failed 27 to add voting file b594600fe26d4f2dbff4f31c84969822.
CRS-4602: Failed 27 to add voting file 7cb1bfa1d53e4f67bf5e656d38ffb025.
CRS-4602: Failed 27 to add voting file 407128acec804ff5bfdc32047f07c7ba.
Failed to replace voting disk group with +VOTEDG.
CRS-4000: Command Replace failed, or completed with errors.

And the ASM alert showed the messages below

NOTE: [crsctl.bin@cfmsvm2 (TNS V1-V3) 13029] opening OCR file
NOTE: updated gpnp profile ASM diskstring:
2014-09-18 11:35:24.225000 +08:00
NOTE: Creating voting files in diskgroup VOTEDG
NOTE: Voting File refresh pending for group 1/0x84c09605 (VOTEDG)
NOTE: Attempting voting file creation in diskgroup VOTEDG
NOTE: voting file allocation on grp 1 disk VOTEDG_0002
NOTE: voting file allocation on grp 1 disk VOTEDG_0000
2014-09-18 11:35:25.375000 +08:00
NOTE: voting file allocation on grp 1 disk VOTEDG_0001
NOTE: Attempting voting file refresh on diskgroup VOTEDG
NOTE: Voting file relocation is required in diskgroup VOTEDG
NOTE: Attempting voting file relocation on diskgroup VOTEDG
NOTE: voting file deletion on grp 1 disk VOTEDG_0000
NOTE: voting file deletion on grp 1 disk VOTEDG_0001
NOTE: voting file deletion on grp 1 disk VOTEDG_0002
NOTE: Failed voting file relocation on diskgroup VOTEDG

Finally, I found the QUORUM DISK ‘/nfsvoting/nfsvotedk’ size was not equal to size of  disk ‘/dev/asmdisk/ycfmsvot’.

root@cfmsvm2:/nfsvoting# ls -al
total 409604
drwxr-xr-x   2 grid     oinstall      64 Sep  3 14:34 .
drwxr-xr-x  25 root     root          29 Sep 17 16:13 ..
-rwxrwxr-x   1 grid     oinstall 209715200 Sep 18 11:38 nfsvotedk <– the size was about 200M

 

Fix Procedure

1. Drop the quorum disk

SQL>  alter system set asm_diskstring=’/dev/asmdisk/*’,’/nfsvoting/*’;

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot
VOTEDG_0002                    VOTEDG_0002                    /nfsvoting/nfsvotedk

SQL> ALTER DISKGROUP VOTEDG DROP QUORUM DISK ‘VOTEDG_0002’;

Diskgroup altered.

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot

2. Re-create the quorum disk

dd if=/dev/zero of=nfsvotedk bs=1024k count=10240

3. Re-add the quorum disk

SQL> ALTER DISKGROUP VOTEDG ADD  QUORUM DISK ‘/nfsvoting/nfsvotedk’;

Diskgroup altered.

SQL> select NAME, FAILGROUP, path from v$asm_disk;

NAME                           FAILGROUP                      PATH
—————————— —————————— —————————————-
/dev/asmdisk/jasmdk1
/dev/asmdisk/yasmdk1
VOTEDG_0001                    VOTEDG_0001                    /dev/asmdisk/jcfmsvot
VOTEDG_0000                    VOTEDG_0000                    /dev/asmdisk/ycfmsvot
VOTEDG_0002                    VOTEDG_0002                    /nfsvoting/nfsvotedk

4. Replace VOTEDISK  successfully

bash-4.1$ crsctl replace votedisk +VOTEDG
Successful addition of voting disk 49612e9331d64ff5bf59c81c665e201c.
Successful addition of voting disk 3f45109f98254f62bff4947648761a16.
Successful addition of voting disk 4dff0f244aeb4feabf2df797dbdab091.
Successfully replaced voting disk group with +VOTEDG.
CRS-4266: Voting file(s) successfully replaced

 

 

 

 

Categories: clusterware
Follow

Get every new post delivered to your Inbox.