Archive

Archive for December, 2010

Add static route in Solaris 10

December 31, 2010 Leave a comment

# route -p add 192.168.0.0 192.168.130.254 1

This will create /etc/inet/static_routes and the content as below

# File generated by route(1M) – do not edit.
192.168.0.0 192.168.130.254 1

 

Categories: Solaris Admin

/svn/repos/conf/authz:165:Option must end with ‘:’ or ‘=’

December 15, 2010 Leave a comment

Yesterday, I created some subdirectories for new projects. After a while, some developers reported that they couldn’t logon the SVN Server. Then I checked the SVN Server:/svn/repos/conf/authz file and inspected the line 165 as below. I found that there were read/write permission missing on the line. By adding read/write permission. The problem was fixed.

[/ir-v2]
@gp_ir =

Categories: SVN

OPatch failed with error code 73

December 6, 2010 3 comments

Yesterday, I installed CRS PSU 9294495 ( 11.1.0.7.4). I successfully installed on the first node. I got failed on the second node since the opatch got the message: OPatch failed with error code 73.

I turned on OPATCH_DEBUG ( =true), we got details error below.

Registering the caller : OPatch
OracleHomeInventory::createInventoryObj() gets OUIInstallAreaControl object
OracleHomeInventory::createInventoryObj() gets OUIInstallInventory object
OracleHomeInventory::createInventoryObj() gets OUIOracleHomeInfo object
   OracleHomeInfo::lock() fails, and there is no retry supported.
OracleHomeInventory::createInventoryObj() gets a null OUIOracleHomeInfo object
OracleHomeInventory::createInventoryObj() tries to print a list of Oracle Homes on this system
OracleHomeInventory::createInventoryObj() Your Oracle Home path: "/opt/oracle/product/11g/crs"
List of Homes on this system:

  Home name= oracle_11ghome, Location= "/opt/oracle/product/11g"
OracleHomeInventory::createInventoryObj() construction done
LsInventory::loadAndPrintInventory()
Retrieving inventory from Oracle Home...
OracleHomeInventory::load()
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
Cleaning up the directory : "/opt/oracle/product/11g/crs/.patch_storage/patch_unzip"...

OPatch failed with error code 73

1. It showed that opatch failure may caused by inventory. Then I compared the inventory on both nodes, I found the inventory.xml is not same.

At node 1:
ls -l /opt/oracle/product/oraInventory/ContentsXML
total 6
-rw-rw----   1 oracle   dba          260 Dec  4 22:51 comps.xml
-rw-rw----   1 oracle   dba          683 May  5  2009 inventory.xml
-rw-rw----   1 oracle   dba          270 Dec  4 22:51 libs.xml

At node 2: 
$  ls -l /opt/oracle/product/oraInventory/ContentsXML
total 8
-rw-r--r--   1 oracle   dba          260 Dec  5 00:50 comps.xml
-rw-r--r--   1 oracle   dba          495 Dec  4 23:48 inventory.xml
-rw-r--r--   1 oracle   dba          270 Dec  5 00:50 libs.xml

2. Compare the contents of them, I found that there was missing CRS Home List in node 2 inventory. 

At node 1:

$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2006, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
 <SAVED_WITH>11.1.0.6.0</SAVED_WITH>
 <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraCrs11g_home" LOC="/opt/oracle/product/11g/crs" TYPE="O" IDX="1" CRS="true">
 <NODE_LIST>
 <NODE NAME="qapdb01"/>
 <NODE NAME="qapdb03"/>
 </NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/opt/oracle/product/11g" TYPE="O" IDX="2">
 <NODE_LIST>
 <NODE NAME="qapdb01"/>
 <NODE NAME="qapdb03"/>
 </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

At node 2:
$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2008, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
 <SAVED_WITH>11.1.0.7.0</SAVED_WITH>
 <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="oracle_11ghome" LOC="/opt/oracle/product/11g" TYPE="O" IDX="1">
 <NODE_LIST>
 <NODE NAME="qapdb01"/>
 <NODE NAME="qapdb03"/>
 </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

Fixed

I copy the inventory.xml at node 1 to node 2. It worked.
 











Categories: Patching, TroubleShooting

Is patch supported rolling

December 2, 2010 Leave a comment

Here there are three patches I need to apply RAC cluster database. I want to know if they support rolling

drwxr-xr-x   4 oracle   dba          512 Mar  9  2009 7578879
drwxr-xr-x   4 oracle   dba          512 Mar  9  2009 7627678
drwxr-xr-x   4 oracle   dba          512 Apr  2  2009 7628357

$  opatch query  -is_rolling_patch 7627678
Invoking OPatch 11.2.0.1.0

Oracle Interim Patch Installer version 11.2.0.1.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/oracle/product/11g
Central Inventory : /opt/oracle/oraInventory
from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.2.0.1.0
OUI version       : 11.1.0.7.0
OUI location      : /opt/oracle/product/11g/oui
Log file location : /opt/oracle/product/11g/cfgtoollogs/opatch/opatch2010-12-02_17-03-10PM.log

Patch history file: /opt/oracle/product/11g/cfgtoollogs/opatch/opatch_history.txt

——————————————————————————–
Patch is a rolling patch: true

OPatch succeeded.
$  opatch query  -is_rolling_patch 7578879
Invoking OPatch 11.2.0.1.0

Oracle Interim Patch Installer version 11.2.0.1.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/oracle/product/11g
Central Inventory : /opt/oracle/oraInventory
from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.2.0.1.0
OUI version       : 11.1.0.7.0
OUI location      : /opt/oracle/product/11g/oui
Log file location : /opt/oracle/product/11g/cfgtoollogs/opatch/opatch2010-12-02_17-03-23PM.log

Patch history file: /opt/oracle/product/11g/cfgtoollogs/opatch/opatch_history.txt

——————————————————————————–
Patch is a rolling patch: false

OPatch succeeded.

Categories: Patching Tags:

Fail to start nodeapps if there is no default gateway

December 2, 2010 Leave a comment

Since there was no default gateway on the host, we failed to start node apps as following

$ srvctl stop nodeapps -n sgvmdb01
$ srvctl start  nodeapps -n sgvmdb01
sgvmdb01:ora.sgvmdb01.vip:checkIf: Default gateway is not defined (host=sgvmdb01)
sgvmdb01:ora.sgvmdb01.vip:Interface vnet1 checked failed (host=sgvmdb01)
sgvmdb01:ora.sgvmdb01.vip:Failed to start VIP 172.31.16.190 (host=sgvmdb01)
sgvmdb01:ora.sgvmdb01.vip:checkIf: Default gateway is not defined (host=sgvmdb01)
sgvmdb01:ora.sgvmdb01.vip:Interface vnet1 checked failed (host=sgvmdb01)
sgvmdb01:ora.sgvmdb01.vip:Failed to start VIP 172.31.16.190 (host=sgvmdb01)
CRS-1006: No more members to consider
CRS-0215: Could not start resource ‘ora.sgvmdb01.vip’.

Workaround

We can set FAIL_WHEN_DEFAULTGW_NOT_FOUND=0 in $CRS_HOME/bin/racgvip

# set it to 0 for checkIf() to return success if default gateway is not found,
# otherwise set it to 1
FAIL_WHEN_DEFAULTGW_NOT_FOUND=0

 

 

 


 

 

 

Categories: clusterware

Others cluster nodes reboot when master node reboot

December 2, 2010 1 comment

Today in early morning, we did set diagwait to 13 seconds to fix ntp causing problem. We set the diagwait at one of clusternodes by shutting down and restarting instances and clusterware. It went well. Finally, we restarted the master to make effect to master node by restart, then others nodes immediately reboot.  Then we check the clusterware log $CRS_HOME/log/wvpdb08/cssd and finding the following errors

[    CSSD]2010-12-02 05:18:52.356 [19] >ERROR:   ASSERT clssgm.c 1604

[    CSSD]2010-12-02 05:18:52.356 [19] >ERROR:   clssgmGetGrock: Group ID of 624625 exceeds max value for global groups
[    CSSD]2010-12-02 05:18:52.356 [19] >TRACE:   clssgmDiscOmonReady: omon was posted for member 2

Finally, we checked with oracle support. It reported it was bug and was fixed as the suggestion

Issue is solved in PSU#2 (11.1.0.7.2) on top of 11.1.0.7.
Please review the following notes to get latest PSU:

Note 810663.1 11.1.0.X CRS Bundle Patch Information
Note 854428.1 Patch Set Updates for Oracle Products

Categories: clusterware