Hey Readers, Today I am going to share all the checks we do before patching Oracle RAC / ASM Standalone oracle 19c database server. Some of the steps may not be required if you are doing manual patching.
Table of Contents
GI and RDBMS owner – Password shouldn’t be lock or expired
As we know that analyze command usually check connectivity between nodes for both RDBMS and Grid owner with password less SSH. So it becomes mandatory to have your OS user ( GI or RDBMS owners) to be in active state. If not please take help from system administrator to unlock.
Grid Infrastructure owner – OS user through which you have installed Grid Infrastructure.
RDBMS owner – OS user through which you have installed RDBMS binaries.
Password-less SSH for GI Owner and RDBMS Owner
This point is valid for RAC, please make sure you are able to do ssh to both the nodes from both the nodes before initiating the patching. This should work for Grid and oracle users both.
On node 1 $ssh node_1_hostname date $ssh node_2_hostname date On Node 2 $ssh node_1_hostname date $ssh node_2_hostname date
How to Configure Password-less SSH
Backup the existing files on both the nodes
$cd ~ $cd .ssh $mkdir backup $mv id_* backup $mv auth* backup $mv know* backup
Now, Run ssh-keygen -t rsa on both the nodes
$ssh-keygen -t rsa
Create a new file named – authorized_keys on both the nodes as follows –
$cd ~/.ssh $cat id_rsa.pub >> authorized_keys
Now copy the content of authorized_keys file from one node to another and vice versa.
Node 1 cat authorized_keys ---> Copy the content Node 2 cat authorized_keys Vi authorized_keys paste the content of authorized_keys from node 1 Node 1 vi authorized_keys ---> paste the content of authorized_keys from node 2
Now try to do ssh to itself and other node
On node 1 $ssh node_1_hostname date $ssh node_2_hostname date On Node 2 $ssh node_1_hostname date $ssh node_2_hostname date
Repeat the steps for other user if you have different users for GI and RDBMS owner.
Check checksum of the patch.zip file before unzipping it
This was weird usually a zip file doesn’t allow to unzip if partial file is transferred. But I had similar issue when I tried to unzip the file. Please note file size was same, but cksum of the file on the node which got patched perfectly and other node on which we were trying to patch were different.
OPATCHAUTO-72083 OPATCHAUTO-72146
Analyze command failed with following error –
OPatchauto session is initiated at Sun Apr 10 17:22:05 2022 OPATCHAUTO-72083: Performing bootstrap operations failed. OPATCHAUTO-72083: The bootstrap execution failed because OPATCHAUTO-72146: Failed to load patch OPATCHAUTO-72146: Failed while collecting patch information for patch /oradump/PATCHES/PATCH_JAN2022/33509923. OPATCHAUTO-72146: Check the log for more information.. OPATCHAUTO-72083: Fix the reported problem and re-run opatchauto. OPatchauto session completed at Sun Apr 10 17:22:13 2022 Time taken to complete the session 0 minute, 9 seconds opatchauto bootstrapping failed with error code 255.
To understand what caused it we checked the cksum of the files from source and target and we see change in the checksum of the file.
Source –
grid@test_home:../PATCHES/PATCH_JAN2022$ cksum p33509923_190000_SOLARIS64.zip 2957318235 2098695269 p33509923_190000_SOLARIS64.zip grid@test_home:../PATCHES/PATCH_JAN2022$
Target –
oracle@test_home2:/oradump/GRID_BACKUP/PATCHES$ cksum p33509923_190000_SOLARIS64.zip 2864029616 2098695269 p33509923_190000_SOLARIS64.zip oracle@test_home2:/oradump/GRID_BACKUP/PATCHES$
We re-uploaded the patch file on the server to resolve the issue.
OPatch utility should be same across all the nodes
Check the version of the OPatch utility , Make sure you upgrade opatch utility on both the nodes and in both homes ( GRID HOME and RDBMS HOME).
You can check out how to upgrade OPatch utility –
If you have different version then analyze of patch will fail with following error –
OPATCHAUTO-72088: OPatch version check failed. OPATCHAUTO-72088: OPatch software version in homes selected for patching are different.
OPATCHAUTO-72088: OPatch software version in homes selected for patching are different.
OPATCHAUTO-72088: Please install same OPatch software in all homes.
2022-04-09 23:56:10,474 INFO [1] com.oracle.glcm.patch.auto.db.product.validation.validators.OOPPatchTargetValidator - OOP patch target validation skipped
2022-04-09 23:56:10,474 INFO [1] com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport - Space available after session: 79647 MB
2022-04-09 23:56:10,504 SEVERE [1] com.oracle.glcm.patch.auto.OPatchAuto - OPatchAuto failed.
com.oracle.glcm.patch.auto.OPatchAutoException: OPATCHAUTO-72088: OPatch version check failed.
OPATCHAUTO-72088: OPatch software version in homes selected for patching are different.
OPATCHAUTO-72088: Please install same OPatch software in all homes.
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.loadTopology(DBBaseProductSupport.java:233)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBProductSupport.loadTopology(DBProductSupport.java:69)
at com.oracle.glcm.patch.auto.OPatchAuto.loadTopology(OPatchAuto.java:1732)
at com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:730)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:397)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:344)
at com.oracle.glcm.patch.auto.OPatchAuto.main(OPatchAuto.java:212)
2022-04-09 23:56:10,505 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter - Reporting console output : Message{id='null', message='OPATCHAUTO-72088: OPatch version check failed.
OPATCHAUTO-72088: OPatch software version in homes selected for patching are different.
OPATCHAUTO-72088: Please install same OPatch software in all homes.'}
2022-04-09 23:56:10,505 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter - Reporting console output : Message{id='null', message='OPatchAuto failed.'}
Choose the correct user to unzip PSU/RU zip file.
I usually unzip PSU/RU zip file using GRID HOME Owner, If you have different user for GRID HOME and RDBMS Home.
One of my colleague was trying to patch and get got following error –
Preparing to bring down database service on home /DBHOME/oracle/app/product/19.3.0 Successfully prepared home /DBHOME/oracle/app/product/19.3.0 to bring down database service Bringing down database service on home /DBHOME/oracle/app/product/19.3.0 Database service successfully brought down on home /DBHOME/oracle/app/product/19.3.0 Performing prepatch operations on CRS - bringing down CRS service on home /GRIDHOME/oracle/app/product/grid/19.3.0 Prepatch operation log file location: /GRIDHOME/grid/app/orabase/crsdata/ijibmpf-c4b5-cl2/crsconfig/hapatch_2022-04-11_03-11-57PM.log CRS service brought down successfully on home /GRIDHOME/oracle/app/product/grid/19.3.0 Start applying binary patch on home /DBHOME/oracle/app/product/19.3.0 Successfully executed command: /usr/sbin/slibclean Binary patch applied successfully on home /DBHOME/oracle/app/product/19.3.0 Start applying binary patch on home /GRIDHOME/oracle/app/product/grid/19.3.0 Successfully executed command: /usr/sbin/slibclean Failed while applying binary patches on home /GRIDHOME/oracle/app/product/grid/19.3.0 Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures: Patch Target : ijibmpf-c4b5-cl2->/GRIDHOME/oracle/app/product/grid/19.3.0 Type[siha] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /GRIDHOME/oracle/app/product/grid/19.3.0, host: ijibmpf-c4b5-cl2. Command failed: /GRIDHOME/oracle/app/product/grid/19.3.0/OPatch/opatchauto apply /ora11204/PATCHES/33509923 -oh /GRIDHOME/oracle/app/product/grid/19.3.0 -target_type has -binary -invPtrLoc /GRIDHOME/oracle/app/product/grid/19.3.0/oraInst.loc -jre /GRIDHOME/oracle/app/product/grid/19.3.0/OPatch/jre -persistresult /GRIDHOME/oracle/app/product/grid/19.3.0/opatchautocfg/db/sessioninfo/sessionresult_ijibmpf-c4b5-cl2_siha_13.ser -analyzedresult /GRIDHOME/oracle/app/product/grid/19.3.0/opatchautocfg/db/sessioninfo/sessionresult_analyze_ijibmpf-c4b5-cl2_siha_13.ser Command failure output: ==Following patches FAILED in apply: Patch: /ora11204/PATCHES/33509923/33529556 Log: /GRIDHOME/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-11_15-25-21PM_1.log Reason: Failed during Analysis: CheckPatchApplicable Failed, [ Prerequisite Status: FAILED, Prerequisite output: The details are: Patch 33529556: Copy Action: Source File "/ora11204/PATCHES/33509923/33529556/files/bin/ocrcheck.bin" does not exists or is not readable 'oracle.has.crs, 19.0.0.0.0': Cannot copy file from 'ocrcheck.bin' to '/GRIDHOME/oracle/app/product/grid/19.3.0/bin/ocrcheck.bin' Copy Action: Source File "/ora11204/PATCHES/33509923/33529556/files/bin/ocrconfig.bin" does not exists or is ... After fixing the cause of failure Run opatchauto resume ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed. OPatchauto session completed at Mon Apr 11 15:36:34 2022 Time taken to complete the session 28 minutes, 32 seconds opatchauto failed with error code 42
When we check if file exists in Patch. And it was available but when we check the file permission then we understood what caused the problem
# ls -ltr /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ocrcheck.bin -rwxr-xr-x 1 grid oinstall 777209 May 15 2021 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ocrcheck.bin # ls -ltr "/ora11204/PATCHES/33509923/33529556/files/bin/ocrconfig.bin" -rwx------ 1 oracle oinstall 923983 Jan 12 04:25 /ora11204/PATCHES/33509923/33529556/files/bin/ocrconfig.bin #
My colleague unzipped the file using oracle user instead of grid.
# cd /ora11204/PATCHES/33509923/33529556/files/bin # ls -ltr total 6021024 -rwxr-xr-x 1 oracle oinstall 199707 Jan 12 04:24 ocrdump.bin -rwxr-xr-x 1 oracle oinstall 224343642 Jan 12 04:24 appagent.bin -rwx------ 1 oracle oinstall 779409 Jan 12 04:25 ocrcheck.bin -rwxr-xr-x 1 oracle oinstall 170606705 Jan 12 04:25 evmwatch.bin -rwxr-xr-x 1 oracle oinstall 5725 Jan 12 04:25 rhpctl -rwxr-xr-x 1 oracle oinstall 758026 Jan 12 04:25 racgmain -rwxr-x--- 1 oracle oinstall 12793 Jan 12 04:25 osysmond.bin -rwxr-xr-x 1 oracle oinstall 255096 Jan 12 04:25 oifcfg.bin -rwxr-xr-x 1 oracle oinstall 123191 Jan 12 04:25 odig -rwxr-xr-x 1 oracle oinstall 406311 Jan 12 04:25 octssd.bin -rwxr-xr-x 1 oracle oinstall 169306860 Jan 12 04:25 evminfo.bin -rwxr-xr-x 1 oracle oinstall 92103 Jan 12 04:25 cssvfupgd.bin -rwxr-xr-x 1 oracle oinstall 13438 Jan 12 04:25 crstmpl.scr -rwxr-xr-x 1 oracle oinstall 41352 Jan 12 04:25 clsecho.bin -rwxr-xr-x 1 oracle oinstall 9395 Jan 12 04:25 gnsd -rwxr-xr-x 1 oracle oinstall 172208036 Jan 12 04:25 evmd.bin -rwxr-xr-x 1 oracle oinstall 2346841 Jan 12 04:25 ocssd.bin -rwxr-xr-x 1 oracle oinstall 2642 Jan 12 04:25 ghappctl.pl -rwxr-xr-x 1 oracle oinstall 22149 Jan 12 04:25 clsdiaglsnr.bin -rwxr-xr-x 1 oracle oinstall 109692 Jan 12 04:25 oifcfg_ifls -rwxr-xr-x 1 oracle oinstall 55953 Jan 12 04:25 odnsd.bin -rwx------ 1 oracle oinstall 923983 Jan 12 04:25 ocrconfig.bin -rwxr-xr-x 1 oracle oinstall 169880023 Jan 12 04:25 emcrsp.bin -rwxr-xr-x 1 oracle oinstall 11600 Jan 12 04:25 oclskd.bin -rwxr-xr-x 1 oracle oinstall 170592640 Jan 12 04:25 evmshow.bin -rwxr-xr-x 1 oracle oinstall 9546 Jan 12 04:25 crscdpd.bin -rwxr-xr-x 1 oracle oinstall 1072932 Jan 12 04:26 ocssdrim.bin -rwxr-xr-x 1 oracle oinstall 169342360 Jan 12 04:26 evmsort.bin -rwxr-xr-x 1 oracle oinstall 16865 Jan 12 04:26 crsrename.pl -rwxr-xr-x 1 oracle oinstall 224450371 Jan 12 04:26 orarootagent.bin -rwxr-x--- 1 oracle oinstall 12763 Jan 12 04:26 ologgerd -rwxr-xr-x 1 oracle oinstall 31308 Jan 12 04:26 lsnodes.bin -rwxr-xr-x 1 oracle oinstall 21035 Jan 12 04:26 gsd -rwxr-xr-x 1 oracle oinstall 15051246 Jan 12 04:26 crsctl.bin -rwxr-xr-x 1 oracle oinstall 119190 Jan 12 04:26 oradnssd -rwxr-xr-x 1 oracle oinstall 169231013 Jan 12 04:26 evmmklib.bin -rwxr-xr-x 1 oracle oinstall 224251495 Jan 12 04:26 scriptagent.bin -rwxr-xr-x 1 oracle oinstall 1646519 Jan 12 04:26 oranetmonitor.bin -rwxr-xr-x 1 oracle oinstall 349410 Jan 12 04:26 gpnpd.bin -rwxr-xr-x 1 oracle oinstall 2859437 Jan 12 04:26 crsd.bin -rwxr-xr-x 1 oracle oinstall 25130 Jan 12 04:26 cemutls.bin -rwxr-xr-x 1 oracle oinstall 224460246 Jan 12 04:26 oraagent.bin -rwxr-xr-x 1 oracle oinstall 224434503 Jan 12 04:27 cssdmonitor -rwxr-xr-x 1 oracle oinstall 16437 Jan 12 04:27 cemutlo.bin -rwxr-xr-x 1 oracle oinstall 2882602 Jan 12 04:27 ohasd.bin -rwxr-xr-x 1 oracle oinstall 7326 Jan 12 04:27 jwcctl.pl -rwxr-xr-x 1 oracle oinstall 169348204 Jan 12 04:27 evmmkbin.bin -rwxr-xr-x 1 oracle oinstall 8284 Jan 12 04:27 cdpcfg.pl -rwxr-xr-x 1 oracle oinstall 8562 Jan 12 04:27 cdpc.bin -rwxr-xr-x 1 oracle oinstall 170662912 Jan 12 04:27 evmlogger.bin -rwxr-xr-x 1 oracle oinstall 6456 Jan 12 04:27 zdmcli -rwxr-xr-x 1 oracle oinstall 4749 Jan 12 04:27 srvconfig -rwxr-xr-x 1 oracle oinstall 10972 Jan 12 04:27 sclsspawn -rwxr-xr-x 1 oracle oinstall 37577 Jan 12 04:27 rhpplsnr_lib.pm -rwxr-xr-x 1 oracle oinstall 2412 Jan 12 04:27 rhpplsnr.pl -rwxr-xr-x 1 oracle oinstall 17548 Jan 12 04:27 racgvip -rwxr-xr-x 1 oracle oinstall 119190 Jan 12 04:27 oradnssd.bin -rwxr-xr-x 1 oracle oinstall 6703 Jan 12 04:27 oracsswd.bin
As a solution we moved the existing unzip folder of RU and re-unzipped the file using grid user. And with completion of unzip we resumed the opatchauto
# mv /ora11204/PATCHES/33509923 /ora11204/PATCHES/33509923_backup # su - grid $ cd /ora11204/PATCHES/ $ unzip p33509923_190000_SOLARIS64.zip $ exit # opatchauto resume
How to Patch both homes without using OH parameter.
Why opatchauto apply command doesn’t patch Oracle home but only patch GRID HOME ?
You would say I have central inventory entry, I updated oraInst.loc but still system doesn’t patch/ analyze command doesn’t show prechecks for ORACLE HOME. This is due to the fact that Cluster doesn’t have any db entry on it. So to over come this I usually add a fake db entry. as follows –
oracle@oracle_test:/oracle/GRID_PATCH$srvctl add database -d oracledbworld -o /DBHOME/oracle/app/product/19.3.0 oracle@oracle_test:/oracle/GRID_PATCH$ srvctl config database oracledbworld oracle@oracle_test:/oracle/GRID_PATCH$ srvctl config database -d oracledbworld Database unique name: oracledbworld Database name: Oracle home: /DBHOME/oracle/app/product/19.3.0 Oracle user: oracle Spfile: Password file: Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Disk Groups: Services: OSDBA group: oinstall OSOPER group: oinstall Database instance: oracledbworld oracle@oracle_test:/oracle/GRID_PATCH$ srvctl add instance -d <dbname> -i <instance_name> -n <hostname> oracle@oracle_test:/oracle/GRID_PATCH$ srvctl add instance -d <dbname> -i <instance_name> -n <hostname>
Following is the analyze output without any database added in cluster –
OPatchauto session is initiated at Sun Apr 10 19:10:18 2022 System initialization log file is /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchautodb/systemconfig2022-04-10_07-10-24PM.log. Session log file is /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/opatchauto2022-04-10_07-10-32PM.log The id for this session is LXWT Executing OPatch prereq operations to verify patch applicability on home /u01/oracle/app/product/grid/19.3.0 Patch applicability verified successfully on home /u01/oracle/app/product/grid/19.3.0 Executing patch validation checks on home /u01/oracle/app/product/grid/19.3.0 Patch validation checks successfully completed on home /u01/oracle/app/product/grid/19.3.0 OPatchAuto successful. --------------------------------Summary-------------------------------- Analysis for applying patches has completed successfully: Host:oracle_test SIHA Home:/u01/oracle/app/product/grid/19.3.0 Version:19.0.0.0.0 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /oracle/GRID_PATCH/33509923/33529556 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-10-58PM_1.log Patch: /oracle/GRID_PATCH/33509923/33534448 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-10-58PM_1.log Patch: /oracle/GRID_PATCH/33509923/33575402 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-10-58PM_1.log Patch: /oracle/GRID_PATCH/33509923/33239955 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-10-58PM_1.log Patch: /oracle/GRID_PATCH/33509923/33515361 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-10-58PM_1.log OPatchauto session completed at Sun Apr 10 19:12:25 2022 Time taken to complete the session 2 minutes, 7 seconds oracle@oracle_test:/oracle/GRID_PATCH$
Following is the analyze output with a database added in cluster –
OPatchauto session is initiated at Sun Apr 10 19:19:24 2022 System initialization log file is /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchautodb/systemconfig2022-04-10_07-19-29PM.log. Session log file is /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/opatchauto2022-04-10_07-19-37PM.log The id for this session is K3KG Executing OPatch prereq operations to verify patch applicability on home /DBHOME/oracle/app/product/19.3.0 Patch applicability verified successfully on home /DBHOME/oracle/app/product/19.3.0 Executing patch validation checks on home /DBHOME/oracle/app/product/19.3.0 Patch validation checks successfully completed on home /DBHOME/oracle/app/product/19.3.0 Verifying SQL patch applicability on home /DBHOME/oracle/app/product/19.3.0 SQL patch applicability verified successfully on home /DBHOME/oracle/app/product/19.3.0 Executing OPatch prereq operations to verify patch applicability on home /u01/oracle/app/product/grid/19.3.0 Patch applicability verified successfully on home /u01/oracle/app/product/grid/19.3.0 Executing patch validation checks on home /u01/oracle/app/product/grid/19.3.0 Patch validation checks successfully completed on home /u01/oracle/app/product/grid/19.3.0 OPatchAuto successful. --------------------------------Summary-------------------------------- Analysis for applying patches has completed successfully: Host:oracle_test SIDB Home:/DBHOME/oracle/app/product/19.3.0 Version:19.0.0.0.0 ==Following patches were SKIPPED: Patch: /oracle/GRID_PATCH/33509923/33534448 Reason: This patch is not applicable to this specified target type - "oracle_database" Patch: /oracle/GRID_PATCH/33509923/33575402 Reason: This patch is not applicable to this specified target type - "oracle_database" Patch: /oracle/GRID_PATCH/33509923/33239955 Reason: This patch is not applicable to this specified target type - "oracle_database" ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /oracle/GRID_PATCH/33509923/33529556 Log: /DBHOME/oracle/app/product/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-20-04PM_1.log Patch: /oracle/GRID_PATCH/33509923/33515361 Log: /DBHOME/oracle/app/product/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-20-04PM_1.log Host:oracle_test SIHA Home:/u01/oracle/app/product/grid/19.3.0 Version:19.0.0.0.0 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /oracle/GRID_PATCH/33509923/33529556 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-21-30PM_1.log Patch: /oracle/GRID_PATCH/33509923/33534448 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-21-30PM_1.log Patch: /oracle/GRID_PATCH/33509923/33575402 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-21-30PM_1.log Patch: /oracle/GRID_PATCH/33509923/33239955 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-21-30PM_1.log Patch: /oracle/GRID_PATCH/33509923/33515361 Log: /u01/oracle/app/product/grid/19.3.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-04-10_19-21-30PM_1.log OPatchauto session completed at Sun Apr 10 19:22:58 2022 Time taken to complete the session 3 minutes, 35 seconds oracle@oracle_test:/oracle/GRID_PATCH$
OraInst.loc and Central Inventory Plays an Important role
Central inventory tells system what all Oracle Products are installed on the system and OraInst.loc tells system where is central inventory. Refer following document to understand What is Central Inventory in Oracle 19c and how to recreate central Inventory – https://www.oracledbworld.com/oracle-error/how-to-recreate-central-inventory-in-oracle/
Cannot detect any valid Grid Infrastructure setup
When I was trying to apply an patch on Old server I got following error – Cannot detect any valid Grid Infrastructure setup, I thought central inventory doesn’t have any entry for the database. But that wasn’t the case. It was oraInst.loc which was pointing to OEM central inventory ( For some reason OEM team keep a different central inventory, I never asked them why ) .
Exception in thread "main" oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Cannot detect any valid Grid Infrastructure setup.
at oracle.dbsysmodel.driver.sdk.productdriver.ClusterInformationLoader.detectCrsHome(ClusterInformationLoader.java:331)
at oracle.dbsysmodel.driver.sdk.productdriver.ClusterInformationLoader.loadInventory(ClusterInformationLoader.java:81)
at oracle.dbsysmodel.driver.sdk.productdriver.ClusterInformationLoader.getInstance(ClusterInformationLoader.java:70)
at oracle.dbsysmodel.driver.sdk.productdriver.remote.HomeListCollector.executeAction(HomeListCollector.java:31)
at oracle.dbsysmodel.driver.sdk.productdriver.remote.RemoteOperationHelper.main(RemoteOperationHelper.java:26)
Command output:
at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.checkForException(OUIDriver.java:872)
at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.handleExceptions(OUIDriver.java:858)
at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteHostData(OUIDriver.java:817)
at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteNodeData(OUIDriver.java:716)
at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchNodeNamesfor122(OUIDriver.java:687)
at com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver.identifyNodes(AbstractCrsProductDriver.java:212)
at com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver.fetchNodeNames(AbstractCrsProductDriver.java:185)
at com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver.isShared(AbstractCrsProductDriver.java:610)
at com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver.makeHome(AbstractCrsProductDriver.java:397)
at com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver.findOrMakeHome(AbstractCrsProductDriver.java:454)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.makeDBInstances(CrsProductDriver.java:1438)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.makeRACDatabase(CrsProductDriver.java:1044)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.makeDBs(CrsProductDriver.java:923)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.buildCRSSystemInstance(CrsProductDriver.java:514)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.buildSystemInstance(CrsProductDriver.java:562)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.createSystemInstance(TopologyCreator.java:261)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.process(TopologyCreator.java:177)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.main(TopologyCreator.java:121)
2022-04-09 22:17:02,920 SEVERE [1] com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator - Failure reason::null
Make sure central inventory location has your both – Grid and Oracle homes are attached and oraInst.loc reflecting the same.
oracle@oracle_test:/oracle/GRID_PATCH$ opatch lsinv | grep -i cent Central Inventory /DBHOME/oracle/oraInventory oracle@oracle_test:/oracle/GRID_PATCH$ cat /var/opt/oracle/oraInst.loc inventory_loc=/DBHOME/oracle/oraInventory inst_group=oinstall oracle@oracle_test:/oracle/GRID_PATCH$
If any of your home is not attached with central inventory then follow following link to add it in your central inventory. If your central inventory is corrupted then you can follow => how to recreate central inventory in oracle 19c
oracle@oracle_test:../oracle/app/product/19.3.0$ cd $ORACLE_HOME/oui/bin oracle@test_home3:...cle/app/product/19.3.0/oui/bin$ ./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/oracle/app/product/19.3.0" ORACLE_HOME_NAME=Ora19cHome Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 168188 MB Passed The inventory pointer is located at /var/opt/oracle/oraInst.loc You can find the log of this install session at: /u01/oracle/oraInventory/logs/AttachHome2022-04-10_07-17-21PM.log 'AttachHome' was successful. oracle@oracle_test:...cle/app/product/19.3.0/oui/bin$
oui-patch.xml (Permission denied)
It better to add following precautionary step if you are doing patching on Oracle 19c RAC setup.
$ opatch lsinventory | grep -i centr or $ cat /var/opt/oracle/oraInst.loc or cat /etc/oraInst.loc (based on your OS flavor) $ ls -ltr <central_file_location_from_above_command>/ContentsXML/oui-patch.xml Make sure your system has this file on both the nodes and give proper permission. I actually gave 777 on the file first time, but 660 also works.
One of the problematic thing identified in Oracle 19c fresh installed RAC setup where node 2 usually doesn’t have oui-patch.xml due to the fact that GUI installation was done from Node 1. This bug is fixed in Oracle 21.
Solution provided in oracle doc – opatchauto apply Results java.io.FileNotFoundException: <oraInventory>/ContentsXML/oui-patch.xml (Permission denied) Error in Non-OUI Nodes (Doc ID 2582139.1) didn’t help me. Doc suggest to copy the file from first node to second node with proper permissions in central inventory and execute – opatchauto resume.
Unfortunately it never worked for me, it has already changed by Oracle Home binaries and corrupted my local inventory.
You have left with only two options –
1.Restore the home from backup and start the patching once again on DB home
2. There are non-supported blog available which suggest you to copy the content of local inventory folder, copy the oui-patch.xml and try to rollback applied patches and re-try. It worked for me, but for my new system I chose to do clean up , restore the backup, start patching again.
[Apr 9, 2022 10:15:00 PM] [INFO] [OPSR-TIME] Finished modifying the system for patch 33515361
[Apr 9, 2022 10:15:00 PM] [INFO] ApplySession adding interim patch '33515361' to inventory
[Apr 9, 2022 10:15:00 PM] [INFO] [OPSR-TIME] Saving patch 33515361 to inventory
[Apr 9, 2022 10:15:00 PM] [INFO] Is batchMode enable? false
[Apr 9, 2022 10:15:00 PM] [INFO] Before homeOp.processPatch()
[Apr 9, 2022 10:15:00 PM] [INFO] After homeOp.processPatch()
[Apr 9, 2022 10:15:00 PM] [INFO] Saving inventory after applying each patch
[Apr 9, 2022 10:15:02 PM] [WARNING] OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
[Apr 9, 2022 10:15:02 PM] [INFO] Restoring "/DBHOME/oracle/app/product/19.3.0/dbhome_1" to the state prior to running NApply...
[Apr 9, 2022 10:15:02 PM] [INFO] Restoring files: copy recurse from /DBHOME/oracle/app/product/19.3.0/dbhome_1/.patch_storage/NApply/2022-04-09_22-11-20PM/backup to /DBHOME/oracle/app/product/19.3.0/dbhome_1
[Apr 9, 2022 10:15:25 PM] [INFO] Removed patch "33515361" with UPI + "24574896" from OUI inventory memory..
[Apr 9, 2022 10:15:25 PM] [INFO] Saving inventory after applying each patch
[Apr 9, 2022 10:15:25 PM] [INFO] Stack Description: java.lang.RuntimeException: OUI session not initialized
at oracle.sysman.oui.patch.impl.HomeOperationsImpl.saveInventory(HomeOperationsImpl.java:372)
at oracle.glcm.opatch.common.api.install.HomeOperationsShell.saveInventory(HomeOperationsShell.java:216)
at oracle.opatch.ipm.IPMRWServices.saveInstallInventory(IPMRWServices.java:914)
at oracle.opatch.OPatchSession.restorePatchesInventory(OPatchSession.java:1434)
at oracle.opatch.MergedPatchObject.restoreOH(MergedPatchObject.java:1276)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:1466)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:373)
at oracle.opatch.opatchutil.NApply.process(NApply.java:353)
at oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.opatch.UtilSession.process(UtilSession.java:355)
at oracle.opatch.OPatchSession.process(OPatchSession.java:2660)
at oracle.opatch.OPatch.process(OPatch.java:870)
at oracle.opatch.OPatch.main(OPatch.java:927)
[Apr 9, 2022 10:15:25 PM] [SEVERE] OUI-67115:OPatch failed to restore OH '/DBHOME/oracle/app/product/19.3.0/dbhome_1'. Consult OPatch document to restore the home manually before proceeding.
[Apr 9, 2022 10:15:25 PM] [WARNING] OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/DBHOME/oracle/app/product/19.3.0/dbhome_1/.patch_storage/NApply/2022-04-09_22-11-20PM"
[Apr 9, 2022 10:15:26 PM] [SEVERE] OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
[Apr 9, 2022 10:15:26 PM] [INFO] --------------------------------------------------------------------------------
[Apr 9, 2022 10:15:26 PM] [INFO] The following warnings have occurred during OPatch execution:
[Apr 9, 2022 10:15:26 PM] [INFO] 1) OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
[Apr 9, 2022 10:15:26 PM] [INFO] 2) OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/DBHOME/oracle/app/product/19.3.0/dbhome_1/.patch_storage/NApply/2022-04-09_22-11-20PM"
[Apr 9, 2022 10:15:26 PM] [INFO] --------------------------------------------------------------------------------
[Apr 9, 2022 10:15:26 PM] [INFO] Finishing UtilSession at Sat Apr 09 22:15:26 IST 2022
[Apr 9, 2022 10:15:26 PM] [INFO] Log file location: /DBHOME/oracle/app/product/19.3.0/dbhome_1/cfgtoollogs/opatch/opatch2022-04-09_22-11-20PM_1.log
[Apr 9, 2022 10:15:26 PM] [INFO] Stack Description: java.lang.RuntimeException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
at oracle.sysman.oui.patch.impl.HomeOperationsImpl.saveInventory(HomeOperationsImpl.java:381)
at oracle.glcm.opatch.common.api.install.HomeOperationsShell.saveInventory(HomeOperationsShell.java:216)
at oracle.opatch.ipm.IPMRWServices.saveInstallInventory(IPMRWServices.java:914)
at oracle.opatch.ipm.IPMRWServices.savePatch(IPMRWServices.java:487)
at oracle.opatch.ApplySession.apply(ApplySession.java:959)
at oracle.opatch.ApplySession.processLocal(ApplySession.java:4111)
at oracle.opatch.ApplySession.process(ApplySession.java:5093)
at oracle.opatch.ApplySession.process(ApplySession.java:4955)
at oracle.opatch.OPatchACL.processApply(OPatchACL.java:310)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:1430)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:373)
at oracle.opatch.opatchutil.NApply.process(NApply.java:353)
at oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.opatch.UtilSession.process(UtilSession.java:355)
at oracle.opatch.OPatchSession.process(OPatchSession.java:2660)
at oracle.opatch.OPatch.process(OPatch.java:870)
at oracle.opatch.OPatch.main(OPatch.java:927)
Caused by: java.lang.Error: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
... 21 more
Caused by: java.lang.RuntimeException: ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
... 21 more
Caused by: oracle.glcm.opatch.common.api.install.HomeOperationsException: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
... 21 more
Caused by: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
at oracle.sysman.oii.oiii.OiiiInstallInventory.writeXML(OiiiInstallInventory.java:1253)
at oracle.sysman.oii.oiii.OiiiInstallInventory.saveInventory(OiiiInstallInventory.java:1039)
at oracle.sysman.oui.patch.impl.HomeOperationsImpl.saveInventory(HomeOperationsImpl.java:378)
... 20 more
Caused by: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
at oracle.sysman.oui.patch.xmlHandler.PatchXML.save(PatchXML.java:202)
at oracle.sysman.oui.patch.xmlHandler.PatchXML.save(PatchXML.java:93)
at oracle.sysman.oui.patch.impl.PatchInventoryImpl.saveOneOffs(PatchInventoryImpl.java:97)
at oracle.sysman.oii.oiii.OiiiInstallXMLWriter.writeOneOffs(OiiiInstallXMLWriter.java:695)
at oracle.sysman.oii.oiii.OiiiInstallInventory.writeXML(OiiiInstallInventory.java:1249)
... 22 more
Caused by: java.io.FileNotFoundException: /GRIDHOME/oracle/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
at oracle.sysman.oui.patch.xmlHandler.PatchXML.save(PatchXML.java:172)
... 26 more
I tried to cover all the issue, problems me or my colleagues faced so far while patching. Hope this will help you to avoid problems while patching 🙂