Today, We will learn upgrade in rolling fashion, which means, you will have one instance of the database available to the end user. I will cover each and every slide and steps I followed in detail.
In case you are looking for how to upgrade oracle grid infrastructure (ORACLE RESTART) from 12c to 19c
Before we start, In this example,
Platform – Sun Solaris
Current Version of Oracle Grid Infrastructure – 12.1.0.2
To be Upgraded to release – 19.3.0.0.0
Table of Contents
Pre-requisite for Oracle Grid Infrastructure from 12c to 19c
- You have 81GB space available in the mount point. So I would recommend you to take at least 100GB mount point to successfully complete the Grid Infrastructure from 12c to 19c
- You have downloaded and copied the Oracle Grid Infrastructure 19c binaries on new location under new home. Link from where I downloaded the binaries as follows –
https://www.oracle.com/database/technologies/oracle19c-solarisx86-downloads.html - You have all the disk group’s ASM COMPATIBLE set to 12 at least.
What is DRY Run and when we can do Dry Run
From Oracle 19c, Oracle Grid Infrastructure Installer allow us to use Dry Run option where it checks for System readiness like – system configuration, network details, storage details and patches. And it will display it errors in the logfile in the end.
Note – Dry Run doesn’t perform actual upgrade.
To execute dry run for upgrade there is switch called “-dryRunForUpgrade” which you have to use while invoking gridsetup.sh. And best part is – It doesn’t impact live setup which is running from previous version. DRY Run is only available in RAC setup.
./gridSetup.sh -dryRunForUpgradeAs always, It’s recommended to runcluvfy to identify the if all the prechecks are passed.
./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /GRID_HOME/grid/app/12.1.0.2/gridhome -dest_crshome /GRIDHOME/oracle/app/product/grid/19.3.0 -dest_version 19.0.0.0.0 -fixup -verbose
....TRIMMED OUTPUT...
Verifying loopback network interface address ...PASSED
Verifying Privileged group consistency for upgrade ...PASSED
Verifying CRS user Consistency for upgrade ...PASSED
Verifying Clusterware Version Consistency ...
Verifying cluster upgrade state ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying Check incorrectly sized ASM Disks ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...PASSED
Verifying User Equivalence ...PASSED
Verifying IP hostmodel ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying Multiuser services check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED
Pre-check for cluster services setup was successful.Step by Step How to execute Oracle Grid Infrastructure dry-run upgrade mode
Lets begin
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>./gridSetup.sh -dryRunForUpgrade
Launching Oracle Grid Infrastructure Setup Wizard...
The response file for this session can be found at:
/GRIDHOME/oracle/app/product/grid/19.3.0/install/response/grid_2020-08-07_07-02-14PM.rsp
You can find the log of this install session at:
/DBHOME/oracle/oraInventory/logs/GridSetupActions2020-08-07_07-02-14PM/gridSetupActions2020-08-07_07-02-14PM.log
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>Chose Upgrade Oracle Grid Infrastructure
Here we will check if it has identified our Oracle Gird Infrastructure Home and Node Name properly. And press Next
Oracle Grid Infrastructure 19c Installer is checking ssh connectivity across the nodes –
Press Next
Enter Oracle Base and Press Next
We don’t get root credentials here, but in case you get it then you can put root password so that GUI will not ask you to run root script instead it will run itself.
Wait for it to complete –
Check for all issues and try to fix it.
Here I am choosing Ignore ALL and Press Next
Finally it will show you the summary and last option to verify everything and chance to change anything if you have missed it. Press Next
Let it complete.
Now you have a new dialogue box which ask you to run the script as “root” user.
Make sure you run the rootupgrade.sh only on that node where you executed the gridsetup.sh
-bash-5.0# sh /GRIDHOME/oracle/app/product/grid/19.3.0/rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /GRIDHOME/oracle/app/product/grid/19.3.0
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Performing Dry run of the Grid Infrastructure upgrade.
Using configuration parameter file: /GRIDHOME/oracle/app/product/grid/19.3.0/crs/install/crsconfig_params
The log of current session can be found at:
/GRIDHOME1/oracle/orabase/crsdata/SUN-SOLARIS-00/crsconfig/rootcrs_SUN-SOLARIS-00_2020-08-07_09-49-27AM.log
2020/08/07 09:50:51 CLSRSC-464: Starting retrieval of the cluster configuration data
2020/08/07 09:51:21 CLSRSC-729: Checking whether CRS entities are ready for upgrade, cluster upgrade will not be attempted now. This operation may take a few minutes.
2020/08/07 10:04:17 CLSRSC-693: CRS entities validation completed successfully.Once rootupgrade.sh is completed. Come back to GUI and select OK
Step by Step How to execute Oracle Grid Infrastructure from 12c to 19c
Prechecks ->
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>crsctl query crs softwareversion -all
Oracle Clusterware version on node [SUN-SOLARIS-00] is [12.1.0.2.0]
Oracle Clusterware version on node [SUN-SOLARIS-01] is [12.1.0.2.0]
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>grid@SUN-SOLARIS-01/GRIDHOME/oracle/app/product/grid/19.3.0>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.DATA.dg
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.OCR_VOTE.dg
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.REDO1.dg
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.REDO2.dg
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.asm
ONLINE ONLINE SUN-SOLARIS-00 Started,STABLE
ONLINE ONLINE SUN-SOLARIS-01 Started,STABLE
ora.net1.network
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.ons
ONLINE ONLINE SUN-SOLARIS-00 STABLE
ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE SUN-SOLARIS-00 169.254.245.101,STAB
LE
ora.test.db
1 ONLINE ONLINE SUN-SOLARIS-00 Open,Readonly,STABLE
2 ONLINE ONLINE SUN-SOLARIS-01 Open,Readonly,STABLE
ora.cvu
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.SUN-SOLARIS-00.vip
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.SUN-SOLARIS-01.vip
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.mgmtdb
1 ONLINE ONLINE SUN-SOLARIS-00 Open,STABLE
ora.oc4j
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.scan1.vip
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.scan2.vip
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.scan3.vip
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
--------------------------------------------------------------------------------
grid@SUN-SOLARIS-01/GRIDHOME/oracle/app/product/grid/19.3.0>We have to unset all the environment variable like – ORACLE_HOME,ORACLE_BASE and ORACLE_SID
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>echo $ORACLE_HOME
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>echo $ORACLE_SID
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>echo $ORACLE_BASE
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>./gridSetup.sh
Launching Oracle Grid Infrastructure Setup Wizard...
The response file for this session can be found at:
/GRIDHOME/oracle/app/product/grid/19.3.0/install/response/grid_2020-08-07_09-17-48PM.rsp
You can find the log of this install session at:
/DBHOME/oracle/oraInventory/logs/UpdateNodeList2020-08-07_09-17-48PM.log
You can find the log of this install session at:
/DBHOME/oracle/oraInventory/logs/UpdateNodeList2020-08-07_09-17-48PM.log
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0>
We ignore all the warning.
Once the rootupgrade.sh pop up appears. You can go ahead and stop the database on the node where you are planning to run it. First we are running on first node. Hence we stopped the instances on first node.
srvctl stop home -o /DBHOME/oracle/app/11.2.0.4/dbhome -n SUN-SOLARIS-00 -s /DBHOME/oracle/app/11.2.0.4/dbhome/DB_SHUT_IMMEDIATE.txt -t immediateRootupgrade.sh on Node 1
-bash-5.0# sh /GRIDHOME/oracle/app/product/grid/19.3.0/rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /GRIDHOME/oracle/app/product/grid/19.3.0
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /GRIDHOME/oracle/app/product/grid/19.3.0/crs/install/crsconfig_params
The log of current session can be found at:
/GRIDHOME1/oracle/orabase/crsdata/SUN-SOLARIS-00/crsconfig/rootcrs_SUN-SOLARIS-00_2020-08-07_10-43-47AM.log
2020/08/07 10:44:22 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2020/08/07 10:44:23 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/08/07 10:44:24 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2020/08/07 10:44:29 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2020/08/07 10:44:29 CLSRSC-464: Starting retrieval of the cluster configuration data
2020/08/07 10:44:41 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
2020/08/07 10:55:01 CLSRSC-693: CRS entities validation completed successfully.
2020/08/07 10:55:26 CLSRSC-515: Starting OCR manual backup.
2020/08/07 10:55:38 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2020/08/07 10:55:53 CLSRSC-516: OCR manual backup successful.
2020/08/07 11:03:28 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2020/08/07 11:03:28 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.
2020/08/07 11:03:29 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2020/08/07 11:03:51 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2020/08/07 11:03:52 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
2020/08/07 11:03:55 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
2020/08/07 11:04:06 CLSRSC-363: User ignored prerequisites during installation
2020/08/07 11:04:43 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
2020/08/07 11:04:44 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.
2020/08/07 11:06:49 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/08/07 11:06:50 CLSRSC-482: Running command: '/GRID_HOME/grid/app/12.1.0.2/gridhome/bin/crsctl start rollingupgrade 19.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2020/08/07 11:06:56 CLSRSC-482: Running command: '/GRIDHOME/oracle/app/product/grid/19.3.0/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /GRID_HOME/grid/app/12.1.0.2/gridhome -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '
ASM configuration upgraded in local node successfully.
2020/08/07 11:07:10 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2020/08/07 11:07:32 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2020/08/07 11:08:09 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2020/08/07 11:08:27 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
2020/08/07 11:08:34 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
2020/08/07 11:09:07 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
2020/08/07 11:09:46 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
2020/08/07 11:10:08 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
2020/08/07 11:10:29 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
2020/08/07 11:10:30 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
2020/08/07 11:12:36 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
2020/08/07 11:16:27 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
2020/08/07 11:16:48 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
clscfg: EXISTING configuration version 5 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2020/08/07 11:27:18 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/08/07 11:27:51 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
2020/08/07 11:28:03 CLSRSC-474: Initiating upgrade of resource types
2020/08/07 11:33:54 CLSRSC-475: Upgrade of resource types successfully initiated.
2020/08/07 11:34:47 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
2020/08/07 11:35:29 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
-bash-5.0#Check for errors and make sure rootupgrade.sh is successful before moving ahead. Once Successful, do post check on node 1 –
-bash-5.0# ps -ef | grep d.bin
grid 28835 1 0 22:24:34 ? 3:13 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ocssd.bin
root 29904 1 0 22:25:06 ? 0:12 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/octssd.bin reboot
grid 28571 1 0 22:24:26 ? 0:14 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/gipcd.bin
root 28725 1 0 22:24:31 ? 0:37 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/osysmond.bin
grid 28202 1 0 22:24:21 ? 0:21 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/evmd.bin
grid 28197 1 0 22:24:21 ? 0:04 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/mdnsd.bin
root 27475 1 0 22:24:03 ? 2:47 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
grid 28319 1 0 22:24:23 ? 0:06 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/gpnpd.bin
root 25559 27469 0 22:35:59 pts/26 0:00 grep d.bin
root 3150 1 0 22:26:34 ? 2:38 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/crsd.bin reboot
-bash-5.0#grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0/bin>./crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE SUN-SOLARIS-00 Started,STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.crf
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.crsd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.cssd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.cssdmonitor
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.ctssd
1 ONLINE ONLINE SUN-SOLARIS-00 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.evmd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.gipcd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.gpnpd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.mdnsd
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
ora.storage
1 ONLINE ONLINE SUN-SOLARIS-00 STABLE
--------------------------------------------------------------------------------
grid@SUN-SOLARIS-00/GRIDHOME/oracle/app/product/grid/19.3.0/bin>grid@SUN-SOLARIS-01/GRIDHOME/grid>crsctl query crs softwareversion -all
Oracle Clusterware version on node [SUN-SOLARIS-00] is [19.0.0.0.0]
Oracle Clusterware version on node [SUN-SOLARIS-01] is [12.1.0.2.0]
grid@SUN-SOLARIS-01/GRIDHOME/grid>Once everything looks clean, start the database on node 1.
srvctl start home -o /DBHOME/oracle/app/11.2.0.4/dbhome -n SUN-SOLARIS-00 -s /DBHOME/oracle/app/11.2.0.4/dbhome/DB_SHUT_IMMEDIATE.txt
Now Stop the database instance on node 2. If it’s live machine, get it checked with application team if they are able to connect to the database.
srvctl stop home -o /DBHOME/oracle/app/11.2.0.4/dbhome -n SUN-SOLARIS-01 -s /DBHOME/oracle/app/11.2.0.4/dbhome/DB_SHUT_IMMEDIATE2.txt -t immediateRootupgrade.sh on Node 2
-bash-5.0# hostname
SUN-SOLARIS-01
-bash-5.0# date
Friday, August 7, 2020 at 11:01:09 PM IST
-bash-5.0# sh /GRIDHOME/oracle/app/product/grid/19.3.0/rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /GRIDHOME/oracle/app/product/grid/19.3.0
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /GRIDHOME/oracle/app/product/grid/19.3.0/crs/install/crsconfig_params
The log of current session can be found at:
/GRIDHOME1/oracle/orabase/crsdata/SUN-SOLARIS-01/crsconfig/rootcrs_SUN-SOLARIS-01_2020-08-07_12-02-11AM.log
2020/08/07 12:02:41 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2020/08/07 12:02:41 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/08/07 12:02:42 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2020/08/07 12:02:45 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2020/08/07 12:02:46 CLSRSC-464: Starting retrieval of the cluster configuration data
2020/08/07 12:07:04 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2020/08/07 12:19:43 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2020/08/07 12:19:44 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
2020/08/07 12:19:44 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
2020/08/07 12:19:47 CLSRSC-363: User ignored prerequisites during installation
2020/08/07 12:19:52 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
2020/08/07 12:19:53 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.
ASM configuration upgraded in local node successfully.
2020/08/07 12:20:21 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2020/08/07 12:20:23 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2020/08/07 12:20:31 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
2020/08/07 12:20:37 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
2020/08/07 12:20:51 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
2020/08/07 12:20:52 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
2020/08/07 12:20:58 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
2020/08/07 12:21:03 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
2020/08/07 12:21:04 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
2020/08/07 12:22:09 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
2020/08/07 12:23:59 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
2020/08/07 12:24:04 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
2020/08/07 12:25:52 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 19 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2020/08/07 12:26:37 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
Start upgrade invoked..
Upgrading CRS managed objects
Upgrading 92 CRS resources
Completed upgrading CRS resources
Upgrading 69 CRS types
Completed upgrading CRS types
Upgrading 15 CRS server pools
Completed upgrading CRS server pools
Upgrading 1 CRS server categories
Completed upgrading CRS server categories
Upgrading 2 old servers
Completed upgrading servers
CRS upgrade has completed.
2020/08/07 12:27:37 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2020/08/07 12:27:37 CLSRSC-482: Running command: '/GRIDHOME/oracle/app/product/grid/19.3.0/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Oracle ASM was successfully upgraded.
Started to upgrade CRS.
CRS was successfully upgraded.
Started to upgrade Oracle ACFS.
Oracle ACFS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 19.0.0.0.0.
2020/08/07 12:29:15 CLSRSC-479: Successfully set Oracle Clusterware active version
2020/08/07 12:31:25 CLSRSC-476: Finishing upgrade of resource types
2020/08/07 12:35:09 CLSRSC-477: Successfully completed upgrade of resource types
2020/08/07 12:45:50 CLSRSC-716: RHP could not be configured in local mode. Continuing with the Grid Infrastructure configuration to complete the install or upgrade.
2020/08/07 12:46:40 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
Successfully updated XAG resources.
2020/08/07 12:48:54 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded-bash-5.0# ps -ef | grep d.bin
grid 26630 1 0 23:24:20 ? 0:06 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/mdnsd.bin
root 27079 1 0 23:24:28 ? 0:40 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/osysmond.bin
root 974 1 0 23:28:04 ? 1:53 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/crsd.bin reboot
root 26348 1 0 23:24:10 ? 2:05 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
grid 26634 1 0 23:24:20 ? 1:48 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/evmd.bin
grid 26718 1 0 23:24:22 ? 0:09 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/gpnpd.bin
grid 26970 1 0 23:24:24 ? 0:22 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/gipcd.bin
grid 27131 1 0 23:24:30 ? 0:49 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/ocssd.bin
root 27424 1 0 23:25:01 ? 1:37 /GRIDHOME/oracle/app/product/grid/19.3.0/bin/octssd.bin reboot
root 1539 5020 0 23:51:13 pts/3 0:00 grep d.bin
-bash-5.0#-bash-5.0# ./crsctl query crs softwareversion -all
Oracle Clusterware version on node [SUN-SOLARIS-00] is [19.0.0.0.0]
Oracle Clusterware version on node [SUN-SOLARIS-01] is [19.0.0.0.0]-bash-5.0# ./crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2676783881].-bash-5.0# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.crf
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.crsd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.cssd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.ctssd
1 ONLINE ONLINE SUN-SOLARIS-01 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.evmd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.gipcd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.gpnpd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.mdnsd
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
ora.storage
1 ONLINE ONLINE SUN-SOLARIS-01 STABLE
--------------------------------------------------------------------------------
-bash-5.0#
========================Start database on node 2 as well now –
srvctl start home -o /DBHOME/oracle/app/11.2.0.4/dbhome -n SUN-SOLARIS-01 -s /DBHOME/oracle/app/11.2.0.4/dbhome/DB_SHUT_IMMEDIATE2.txt Go back to GUI, and Press OK
Here you might be thinking that step number has been reset to 1 how ? You can find it in this post -> https://www.oracledbworld.com/upgrade/how-to-complete-grid-infrastructure-upgrade-when-gui-is-lost/
Once all the database services are started on both the server – Kindly run the post check and release the system
ps -ef | grep pmon
ps -ef | grep d.bin
ps -ef | grep tns
srvctl status database -d <dbname>
check alert log of both the nodes (DB and ASM)