Today, We faced above issue where srvctl start database was failing on one node. It was failing with “CRS-5802: Unable to start the agent process” . There was no clue in database alert log what was causing the problem.
Following issue was reported on Sun Solaris OS / 19.11 Binary Version
oracle@oracle_test:~$ srvctl start database -d oracledb PRCR-1079 : Failed to start resource ora.oracledb.db CRS-2674: Start of 'ora.oracledb.db' on 'oracle_test2' failed CRS-2678: 'ora.oracledb.db' on 'oracle_test2' has experienced an unrecoverable failure CRS-0267: Human intervention required to resume its availability. CRS-5802: Unable to start the agent process oracle@oracle_test:~$
If you check database’s status in cluster – It shows starting… but it never started for me.
oracle@oracle_test2:~$
/GRIDHOME/oracle/app/product/grid/19.3.0/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE oracle_test STABLE
ONLINE ONLINE oracle_test2 STABLE
ora.helper
OFFLINE OFFLINE oracle_test STABLE
OFFLINE OFFLINE oracle_test2 IDLE,STABLE
ora.net1.network
ONLINE ONLINE oracle_test STABLE
ONLINE ONLINE oracle_test2 STABLE
ora.ons
ONLINE ONLINE oracle_test STABLE
ONLINE ONLINE oracle_test2 STABLE
ora.proxy_advm
OFFLINE OFFLINE oracle_test STABLE
OFFLINE OFFLINE oracle_test2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ARCH.dg(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oracle_test2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oracle_test STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oracle_test STABLE
ora.MGMTDB.dg(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.MGMTLSNR
1 ONLINE ONLINE oracle_test 169.254.25.89 10.254
.64.168,STABLE
ora.OCR_VOTE.dg(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.REDO.dg(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 Started,STABLE
2 ONLINE ONLINE oracle_test Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oracle_test2 STABLE
2 ONLINE ONLINE oracle_test STABLE
ora.cvu
1 ONLINE ONLINE oracle_test STABLE
ora.oracle_test.vip
1 ONLINE ONLINE oracle_test STABLE
ora.oracle_test2.vip
1 ONLINE ONLINE oracle_test2 STABLE
ora.mgmtdb
1 ONLINE ONLINE oracle_test Open,STABLE
ora.qosmserver
1 ONLINE ONLINE oracle_test STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE oracle_test2 STABLE
ora.scan2.vip
1 ONLINE ONLINE oracle_test STABLE
ora.scan3.vip
1 ONLINE ONLINE oracle_test STABLE
ora.oracledb.db
1 ONLINE ONLINE oracle_test Open,Readonly,HOME=/
DBHOME/oracle/app/pr
oduct/19.3.0/dbhome_
1,STABLE
2 ONLINE OFFLINE oracle_test2 STARTING
--------------------------------------------------------------------------------
oracle@oracle_test2:~$
-bash-5.1#
After searching on MOS, came arcos – ‘srvctl start database’ Fails With ‘CRS-5802: Unable to start the agent process’ as the Agent Log is Owned by Wrong User (Doc ID 1957360.1) which suggest to check for ownership of following location –
$(orabase)/crsdata/<hostname>/output/ crsd_oraagent_<database user>OUT.trc $(orabase)/crsdata/<hostname>/output/crsd_oraagent_<database user>.pid
And looking at the node 2’s ownership was different then ownership of node 1.
-bash-5.1# ls -ltr
total 1295
-rw-r--r-- 1 grid oinstall 6 Apr 5 12:58 crsd_oraagent_oracle.pid
-rw-r--r-- 1 grid oinstall 1872 Apr 9 19:49 crsd_oraagent_oracleOUT.trc
-rw-r--r-- 1 grid oinstall 2637 Apr 12 14:09 crsd_scriptagent_gridOUT.trc
-rw-r--r-- 1 grid oinstall 2147 Apr 12 14:09 crsd_jagent_gridOUT.trc
-rw-r--r-- 1 grid oinstall 6 Apr 12 14:09 crsd_scriptagent_grid.pid
-rw-r--r-- 1 grid oinstall 6 Apr 12 14:09 crsd_jagent_grid.pid
-rw-r--r-- 1 grid oinstall 1695 Apr 12 14:13 ologgerdOUT.trc
-rw-r--r-- 1 grid oinstall 6 Apr 12 14:13 ologgerd.pid
-rw-r--r-- 1 grid oinstall 335 Apr 12 17:29 crswrapexece.log
-rw-r--r-- 1 grid oinstall 3758 Apr 12 17:29 ohasdOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ohasd.pid
-rw-r--r-- 1 grid oinstall 4629 Apr 12 17:29 ohasd_orarootagent_rootOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ohasd_orarootagent_root.pid
-rw-r--r-- 1 grid oinstall 7779 Apr 12 17:29 ohasd_oraagent_gridOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ohasd_oraagent_grid.pid
-rw-r--r-- 1 grid oinstall 184295 Apr 12 17:29 mdnsdOUT.trc
-rw-r--r-- 1 grid oinstall 3741 Apr 12 17:29 evmdOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 mdnsd.pid
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 evmd.pid
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 gpnpd.pid
-rw-r--r-- 1 grid oinstall 4476 Apr 12 17:29 gpnpdOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 evmlogger.pid
-rw-r--r-- 1 grid oinstall 5310 Apr 12 17:29 evmloggerOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 gipcd.pid
-rw-r--r-- 1 grid oinstall 5005 Apr 12 17:29 gipcdOUT.trc
-rw-r--r-- 1 grid oinstall 14034 Apr 12 17:29 ohasd_cssdmonitor_rootOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ohasd_cssdmonitor_root.pid
-rw-r--r-- 1 grid oinstall 13974 Apr 12 17:29 ohasd_cssdagent_rootOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 osysmond.pid
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ohasd_cssdagent_root.pid
-rw-r--r-- 1 grid oinstall 10858 Apr 12 17:29 osysmondOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:29 ocssd.pid
-rw-r--r-- 1 grid oinstall 233292 Apr 12 17:29 ocssdOUT.trc
-rw-r--r-- 1 grid oinstall 3357 Apr 12 17:30 octssdOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:30 octssd.pid
-rw-r--r-- 1 grid oinstall 2909 Apr 12 17:30 crsdOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:30 crsd.pid
-rw-r--r-- 1 grid oinstall 3533 Apr 12 17:30 crsd_orarootagent_rootOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:30 crsd_orarootagent_root.pid
-rw-r--r-- 1 grid oinstall 5997 Apr 12 17:33 crsd_oraagent_gridOUT.trc
-rw-r--r-- 1 grid oinstall 5 Apr 12 17:33 crsd_oraagent_grid.pid
-bash-5.1# ps -ef | grep pmon
grid 6650 1 0 17:34:19 ? 0:02 asm_pmon_+ASM2
root 41393 40748 0 09:34:57 pts/5 0:00 grep pmon
As result, we changed the ownership of following two files on problematic node.
-bash-5.1# ls -ltr crsd_oraagent_oracleOUT.trc crsd_oraagent_oracle.pid -rw-r--r-- 1 grid oinstall 6 Apr 5 12:58 crsd_oraagent_oracle.pid -rw-r--r-- 1 grid oinstall 1872 Apr 9 19:49 crsd_oraagent_oracleOUT.trc -bash-5.1# chown oracle:oinstall crsd_oraagent_oracleOUT.trc crsd_oraagent_oracle.pid
-bash-5.1# -bash-5.1# ls -ltr crsd_oraagent* -rw-r--r-- 1 oracle oinstall 6 Apr 5 12:58 crsd_oraagent_oracle.pid -rw-r--r-- 1 oracle oinstall 1872 Apr 9 19:49 crsd_oraagent_oracleOUT.trc -rw-r--r-- 1 grid oinstall 5997 Apr 12 17:33 crsd_oraagent_gridOUT.trc -rw-r--r-- 1 grid oinstall 5 Apr 12 17:33 crsd_oraagent_grid.pid -bash-5.1#
Post changes, database instance came up on node 2 without any issues.
srvctl start instance -d oracledb -i oracledb1