Unix Useful Commands - Part 2

1). Find whether OS is 64/32 Bit Kernel in UNIX

OS	Command
AIX	getconf -a | grep –i KERN OR bootinfo -K OR file /usr/lib/boot/unix
HP-UX	getconf -a | grep -i KERN
Solaris	isainfo –kv
Linux	uname –a

2).Find Free Physical Memory in UNIX

OS	Command
AIX	bootinfo -r
HP-UX	grep -i Physical /var/adm/syslog/syslog.log
Solaris	prtconf
Linux	free -m

3).Find CPU details in UNIX

OS	Command
AIX	lsdev -Cc processor
HP-UX	ioscan –fnC processor
Solaris	psrinfo -v
Linux	cat /proc/cpuinfo

4).Find SWAP Size details in UNIX

OS	Command
AIX	lsps –a
HP-UX	swapinfo -a
Solaris	swap –l
Linux	free -m

Clusterware Processes in 11g RAC R2 Environment

i).Cluster Ready Services (CRS)

$ ps -ef | grep crs | grep -v grep

root 25863 1 1 Oct27 ? 11:37:32 /opt/oracle/grid/product/11.2.0/bin/crsd.bin reboot

crsd.bin => The above process is responsible for start, stop, monitor and failover of resource. It maintains OCR and also restarts the resources when the failure occurs.

This is applicable for RAC systems. For Oracle Restart and ASM ohasd is used.

ii).Cluster Synchronization Service (CSS)

$ ps -ef | grep -v grep | grep css

root 19541 1 0 Oct27 ? 00:05:55 /opt/oracle/grid/product/11.2.0/bin/cssdmonitor

root 19558 1 0 Oct27 ? 00:05:45 /opt/oracle/grid/product/11.2.0/bin/cssdagent

oragrid 19576 1 6 Oct27 ? 2-19:13:56 /opt/oracle/grid/product/11.2.0/bin/ocssd.bin

cssdmonitor => Monitors node hangs(via oprocd functionality) and monitors OCCSD process hangs (via oclsomon functionality) and monitors vendor clusterware(via vmon functionality).This is the multi threaded process that runs with elavated priority.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdmonitor

cssdagent => Spawned by OHASD process.Previously(10g) oprocd, responsible for I/O fencing.Killing this process would cause node reboot.Stops,start checks the status of occsd.bin daemon

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent

occsd.bin => Manages cluster node membership runs as oragrid user.Failure of this process results in node restart.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent --> ocssd --> ocssd.bin

iii) Event Management (EVM)

$ ps -ef | grep evm | grep -v grep

oragrid 24623 1 0 Oct27 ? 00:30:25 /opt/oracle/grid/product/11.2.0/bin/evmd.bin

oragrid 25934 24623 0 Oct27 ? 00:00:00 /opt/oracle/grid/product/11.2.0/bin/evmlogger.bin -o /opt/oracle/grid/product/11.2.0/evm/log/evmlogger.info -l /opt/oracle/grid/product/11.2.0/evm/log/evmlogger.log

evmd.bin => Distributes and communicates some cluster events to all of the cluster members so that they are aware of the cluster changes.

evmlogger.bin => Started by EVMD.bin reads the configuration files and determines what events to subscribe to from EVMD and it runs user defined actions for those events.

iv).Oracle Root Agent

$ ps -ef | grep -v grep | grep orarootagent

root 19395 1 0 Oct17 ? 12:06:57 /opt/oracle/grid/product/11.2.0/bin/orarootagent.bin

root 25853 1 1 Oct17 ? 16:30:45 /opt/oracle/grid/product/11.2.0/bin/orarootagent.bin

orarootagent.bin => A specialized oraagent process that helps crsd manages resources owned by root, such as the network, and the Grid virtual IP address.

The above 2 process are actually threads which looks like processes. This is a Linux specific

v).Cluster Time Synchronization Service (CTSS)

$ ps -ef | grep ctss | grep -v grep

root 24600 1 0 Oct27 ? 00:38:10 /opt/oracle/grid/product/11.2.0/bin/octssd.bin reboot

octssd.bin => Provides Time Management in a cluster for Oracle Clusterware

vi).Oracle Agent

$ ps -ef | grep -v grep | grep oraagent

oragrid 5337 1 0 Nov14 ? 00:35:47 /opt/oracle/grid/product/11.2.0/bin/oraagent.bin

oracle 8886 1 1 10:25 ? 00:00:05 /opt/oracle/grid/product/11.2.0/bin/oraagent.bin

oragrid 19481 1 0 Oct27 ? 01:45:19 /opt/oracle/grid/product/11.2.0/bin/oraagent.bin

oraagent.bin => Extends clusterware to support Oracle-specific requirements and complex resources. This process runs server callout scripts when FAN events occur. This process was known as RACG in Oracle Clusterware 11g Release 1 (11.1).

ORACLE HIGH AVAILABILITY SERVICES STACK

i) Cluster Logger Service

$ ps -ef | grep -v grep | grep ologgerd

root 24856 1 0 Oct27 ? 01:43:48 /opt/oracle/grid/product/11.2.0/bin/ologgerd -m mg5hfmr02a -r -d /opt/oracle/grid/product/11.2.0/crf/db/mg5hfmr01a

ologgerd => Receives information from all the nodes in the cluster and persists in a CHM repository-based database. This service runs on only two nodes in a cluster

ii).System Monitor Service (osysmond)

$ ps -ef | grep -v grep | grep osysmond

root 19528 1 0 Oct27 ? 09:42:16 /opt/oracle/grid/product/11.2.0/bin/osysmond

osysmond => The monitoring and operating system metric collection service that sends the data to the cluster logger service. This service runs on every node in a cluster

iii). Grid Plug and Play (GPNPD):

$ ps -ef | grep gpn

oragrid 19502 1 0 Oct27 ? 00:21:13 /opt/oracle/grid/product/11.2.0/bin/gpnpd.bin

gpnpd.bin => Provides access to the Grid Plug and Play profile, and coordinates updates to the profile among the nodes of the cluster to ensure that all of the nodes have the most recent profile.

iv).Grid Interprocess Communication (GIPC):

$ ps -ef | grep -v grep | grep gipc

oragrid 19516 1 0 Oct27 ? 01:51:41 /opt/oracle/grid/product/11.2.0/bin/gipcd.bin

gipcd.bin => A support daemon that enables Redundant Interconnect Usage.

v). Multicast Domain Name Service (mDNS):

$ ps -ef | grep -v grep | grep dns

oragrid 19493 1 0 Oct27 ? 00:01:18 /opt/oracle/grid/product/11.2.0/bin/mdnsd.bin

mdnsd.bin => Used by Grid Plug and Play to locate profiles in the cluster, as well as by GNS to perform name resolution. The mDNS process is a background process on Linux and UNIX and on Windows.

vi).Oracle Grid Naming Service (GNS)

$ ps -ef | grep -v grep | grep gns

gnsd.bin => Handles requests sent by external DNS servers, performing name resolution for names defined by the cluster.

Troubleshooting :Cursor :Pin S Wait Event

Recently we had performance issue with a batch job on a 3 Node RAC running on Linux.The job got hung from the database side, all we could see is session waiting on the Cursor pin S wait event. is not moving anywhere

When I tried to find out the BLOCKING_SESSION & BLOCKING_INSTANCE in the v$session. The column was blank.!!

Here is how the issue got fixed.

select sid,serial#,p1,p2,p3 from v$session where sid=178;
SID SERIAL# P1 P2 P3
---------- ---------- ---------- ---------- -------------------- ---------- ---------- ---------- ----------
178 12856 3552538230 6.4425E+11 2.5761E+10

Get the P3 Value from the above query and then substitute in the below query.You will the get Location

select decode(trunc(2.5376E+10/4294967296),0,trunc(2.5376E+10/65536),trunc(2.5376E+10/4294967296)) LOCATION_ID from dual;

LOCATION_ID
----------------------
5

The below query may not be useful in debugging. But could help to find more about the wait event.

SELECT MUTEX_TYPE, LOCATION FROM x$mutex_sleep WHERE mutex_type like 'Cursor Pin%'
and location_id=&&LOCATION_ID;
Enter value for location_id: 5
old 4: and location_id=&&LOCATION_ID
new 4: and location_id=5
MUTEX_TYPE LOCATION
--------------------------------------------
Cursor Pin kkslce [KKSCHLPIN2]

select MUTEX_TYPE,LOCATION,REQUESTING_SESSION,BLOCKING_SESSION from V$MUTEX_SLEEP_HISTORY where MUTEX_IDENTIFIER=3552538230;

MUTEX_TYPE LOCATION REQUESTING_SESSION BLOCKING_SESSION
------------------------------------------------------------------------------------------------ ------------------ ----------------
Cursor Pin kkslce [KKSCHLPIN2] 178 150

Here P3 from the v$session is value for MUTEX_IDENTIFIER.

Lets see what session 150 is doing.

select SID,SERIAL#,STATUS,ACTION,STATE,EVENT from v$session where sid=150;

SID SERIAL# STATUS ACTION STATE EVENT
---------------- ----------------------------------------------------------------------------------------------------------
150 2905 KILLED JDBC Thin Client WAITED SHORT TIME library cache lock

Here we see that session is actually killed but still holding the lock.

select p.pid,p.SPID,s.SID from v$process p,v$session s where s.paddr = p.addr and s.SID = 150

SPID is the Operating system identifier (OS-PID)
SID is the Oracle session identifier
PID is the Oracle process identifier

Find the Oracle process and kill it at OS level.Once done.the Batch resumed smoothly.

How To Find PSU Version ?

Patch Set Updates are referenced by their 5-place version number. You may use the below commands depending on hte ORACLE_HOME.

Database PSU:

$opatch lsinventory -bugs_fixed | grep -i 'DATABASE PSU' 

CRS (Cluster Ready Services) PSU:

$opatch lsinventory -bugs_fixed | grep -i 'TRACKING BUG' | grep -i 'PSU'

GI (Grid Infrastructure) PSU:

$opatch lsinventory -bugs_fixed | grep -i 'GI PSU' 

Enterprise Manager Agent PSU:

$ opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER AGENT' | grep -i 'PSU'

Enterprise Manager OMS PSU:

$opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER OMS' | grep -i 'PSU'

WebLogic Server PSU:

Enter the following commands, where WL_HOME is the path of the WebLogic home:

$ . $WL_HOME/server/bin/SetWLSEnv.sh  

$ java weblogic.version 

R12 Tech Stack Tit Bits

1) Why do we have 2 homes in R12[10.1.2 and 10.1.3] ?


10.1.2 is also called C Home. Where Forms and Reports are run but it doesn’t have latest OC4J. Latest OC4J is obtained from 10.1.3 Home (a.k.a Java Home). Forms and Reports are not included in each Application Server releases (say 10.1.2, 10.1.3)  because they have become a matured product and they don’t need as many release cycles as others. So with both 10.1.2 and 10.1.3 we are using latest Java and C Code Stacks.

• By default Forms 10.1.2 is the Forms Servlet. Although, socket mode is still supported, and may be useful in high-latency, bandwidth-constrained WAN environments


• The 8.0.6 and iAS Oracle Homes(8.1.7) of 11i have been replaced with AS 10.1.2 and  AS 10.1.3 Homes respectively.


• 10.1.3 does not include Forms and Reports. These are delivered through the 10.1.2 Oracle Home.


• The AS 10.1.3 http server and OC4J container are used to run the Forms servlet, although the Forms runtime process is forked into an AS 10.1.2  environment. This avoids having to run both 10.1.2 and 10.1.3 instances of the http server. AS Development are supporting the use of  10.1.2 forms java code with the 10.1.3 Application servers.


• The Oracle E-Business Suite modules (packaged in the file formsapp.ear) are deployed into the OC4J-Forms instance running out of the Oracle AS 10.1.3 ORACLE_HOME, while the frmweb executable is invoked out of the Oracle AS 10.1.2 ORACLE_HOME.


• All major services are started out of the Oracle AS 10.1.3 ORACLE_HOME.


• Web services include HTTP + OC4J + OPMN Components.


• Self-Service applications are purely designed in HTML and JAVA Scripts.


• AOL/J supplies the Oracle Application Framework with underlying security and applications Java services.


• Communication continues between the Forms applet (Running in the user's Web browser) and the Forms runtime process, via the Listener servlet  until the Forms session ends.


• The Forms Listener servlet acts as a broker between the Java client and the Forms runtime process. It takes connection requests from Java client processes and initiates a Forms runtime process on their behalf.


• The Forms runtime process(frmweb) manages application logic and processing. It maintains a connection to the database on behalf of the Java client. It uses the same forms, menus, and library files that were used for running in client/server mode. The Forms runtime process plays two roles: then it communicates with the client browser, it acts as a server by managing requests from client browsers and it sends metadata to the client to describe the user interface; when it is communicating with the database server, it acts as a client by querying the database server for requested data. 

Is R12/11i Application Tier 32 or 64 Bit ?

11i/R12 E-Business Suite Code on the Application Tier are tightly integrated with Oracle Application Server(10g).Apps code are built and linked directly against the AS (and DB Client) libraries which are 32-bit Compatible Applications.So E-Biz Application Tier is 32 bit Compiled even on the 64 Bit OS .

Only Advances Planning Module has an exception in this case, Where Oracle delivers both 32 and 64 Bit this is due to the special memory requirement by the product. 

Strange Misleading Error[XML -2018/ AC-10006] while R12 Clone

During the recent Multi Node to Single Node R12 Clone, Encountered an strange error. When doing the database portion of the clone. Below command 'adclonectx.pl' creates the Context file

$ perl adclonectx.pl contextfile=$ORACLE_HOME/appsutil/SOURCE_CONTEXT_FILE.xml template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp pairsfile=$ORACLE_HOME/appsutil/clone/pairsfile.txt initialnode

When running the same command, It dumped the below error,

file:/tmp/tmpCtxClone.xml: XML-20108: (Fatal Error) Start of root element expected.

AC-10006: Exception - org.xml.sax.SAXParseException: file:/tmp/tmpCtxClone.xml: XML-20108: (Fatal Error) Start of root element expected. thrown while creating OAVars object for file: /tmp/tmpCtxClone.xml

The new database context file has been created :
/opt/oracle/product/11.1.0_SID/appsutil/SID_HostName.xml

At first site, I suspected that the issue is with format of the source xml file. Hence compared with the working XML file. Result is clean. Below portion of the error struck me

Thrown while creating OAVars object for file: /tmp//dummy.xml

Cause :

The /tmp is 100% full.

Fix:

Either remove the old files in /tmp directory OR export TEMP=/new/location where there is plenty of free space.