Only One Node Starts in a 2 RAC Cluster.

Recently we had a server move from one DC to other.Posting the issue faced on the one of the system.

Configuration as below,

1).AIX Machine
2).11.1.0.7 Database
3).2 Node Cluster.

We had a clear shutdown before the move.And upon starting the Cluster & Database, only one Node comes up.Which ever starts first will have full running CRS/CSS Stack ,the other node fails to start CSS and throws following error in the crsd.log.

20xx-xx-xx xx:xx:xx.xxx: [ COMMCRS][903]clsc_connect: (600000000033e030) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_njzdb12_crs))
20xx-xx-xx xx:xx:xx.xxx: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
20xx-xx-xx xx:xx:xx.xxx: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good.

Googling for the above error mostly points to the issue with /tmp/.oracle or /var/tmp/.oracle directory.The files under which has been deleted when the CRS was running.It was true that one of my colleague has deleted those log files.The easy fix told was to reboot both the machine.We tried but no luck.

Upon checking the cssd logs i could see the below error.

ocssd.log:[ CSSD]2010-03-05 17:48:21.908 [84704144] >TRACE: clssnmReadDskHeartbeat: node 3, vm-lnx-rds1173, has a disk HB, but no network HB, DHB has rcfg 0, wrtcnt, 2, LATS 1185024, lastSeqNo 2, timestamp 1267791501/1961474

Which just flashed to my brain to check the cluster interconnect.Yes.The Cluster Interconnect is not pinging between the servers.Informed to Sysadmins and got it fixed.Then it came out quite well.

Simple UTL_FILE Script to create & write in a file

Below PL/SQL block creates a file named 'sample.txt' with 'welcome' word in it under the Oracle Directory Named UTL_DIR [Can be created using CREATE DIRECTORY]

We can use utl_file_dir instead of CREATE DIRECTORY in that case we need to specify the whole path of the UNIX directory instead of the just the Directory name[i.e.., UTL_DIR]

Make sure you have necessary permissions.

Using utl_file_dir is not a recommended one.

Declare
file1 utl_file.file_type;
Begin
file1:= utl_file.fopen('UTL_DIR','sample.txt','w');
utl_file.put_line(file1,'Welcome' );
utl_file.fclose(file1);
end;
/

Parameters to Format the Output from DBA_SOURCE.

Set the Below Variables:

set verify off
set feedback off
set lines 132
set pages 0
set heading off
set space 0
set recsep off
column text format a79
column line noprint

select DECODE(line,1,'create or replace ','')||text, line
from dba_source where owner = upper('&&1')
and type = upper('&&2')
and name = upper('&&3')


REFER MOS NOTE:SCRIPT:REGENERATING PACKAGE and PROCEDURE CODE from DBA_SOURCE 
[ID 1012473.7]