Yesterday, one of my colleague reported an issue where we were doing HAS installation. It was an AIX 7.1 OS on which were trying to install 19.14 Grid Infra and Oracle Binary. All prechecks were completed. And everything was looking good. But when we executed root.sh it failed with – CLSRSC-196: ACFS driver install actions failed.
# sh /GRIDHOME/oracle/app/product/grid/19.3.0/root.sh Performing root user operation. The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /GRIDHOME/oracle/app/product/grid/19.3.0 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /GRIDHOME/oracle/app/product/grid/19.3.0/crs/install/crsconfig_params The log of current session can be found at: /GRIDHOME/oracle/orabase/crsdata/oracledbworld/crsconfig/roothas_2022-02-24_09-57-18PM.log 0513-095 The request for subsystem refresh was completed successfully. LOCAL ADD MODE Creating OCR keys for user 'grid', privgrp 'oinstall'.. Operation successful. LOCAL ONLY MODE Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'system'.. Operation successful. CRS-4664: Node ijpritestp780-03-l3 successfully pinned. 2022/02/24 22:00:47 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab' 2022/02/24 22:02:40 CLSRSC-196: ACFS driver install actions failed
We checked the roothas.log file to identify what happened and what caused CLSRSC-196: ACFS driver install actions failed issue And when checked got something new – >
Authorizations:
Methods:
exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors:
0509-150 Dependent module libhasgen12.so could not be loaded.
0509-022 Cannot load module libhasgen12.so.
0509-026 System error: A file or directory in the path name does not exist.
> Setting install_files_loc to default for main > ACFS-9180: Sub-command is 'install' > ACFS-9505: Using acfsutil executable from location: '/GRIDHOME/oracle/app/product/grid/19.3.0/usm/install/cmds/bin/acfsutil' > ACFS-9176: Entering 'install' > ACFS-9300: ADVM/ACFS distribution files found. > ACFS-9176: Entering 'uld usm drvs' > ACFS-9176: Entering 'run driver method' > ACFS-9182: Variable 'driver' has value 'oracleoks.ext' > ACFS-9178: Return code = 0 > ACFS-9177: Return from 'run driver method' > ACFS-9178: Return code = USM_SUCCESS > ACFS-9177: Return from 'uld usm drvs' > ACFS-9176: Entering 'fail mode' > ACFS-9178: Return code = 0 > ACFS-9177: Return from 'fail mode' > ACFS-9176: Entering 'uninstall' > ACFS-9176: Entering 'uld usm drvs' > ACFS-9176: Entering 'run driver method' > ACFS-9182: Variable 'driver' has value 'oracleoks.ext' > ACFS-9178: Return code = 0 > ACFS-9177: Return from 'run driver method' > ACFS-9178: Return code = USM_SUCCESS > ACFS-9177: Return from 'uld usm drvs' > ACFS-9314: Removing previous ADVM/ACFS installation. > Entering rmaixsetup > This may take several minutes. Please wait ... > Executing command: setkst > Successfully updated the Kernel Authorization Table. > Successfully updated the Kernel Role Table. > Successfully updated the Kernel Command Table. > Successfully updated the Kernel Device Table. > Successfully updated the Kernel Object Domain Table. > Successfully updated the Kernel Domains Table. > Successfully updated the Kernel RBAC log level. > Return code = 0 > oracle_devmgmt attributes via lsrole: > 3004-733 Role "oracle_devmgmt" does not exist. > The authorizations' atributes via lsauth > 1420-001 Authorization "oracle" does not exist. > 1420-001 Authorization "oracle.dev" does not exist. > 1420-001 Authorization "oracle.dev.acfsctl" does not exist. > 1420-001 Authorization "oracle.dev.advmctl" does not exist. > 1420-001 Authorization "oracle.dev.advmvol" does not exist. > 1420-001 Authorization "oracle.dev.acfsctl.def" does not exist. > 1420-001 Authorization "oracle.dev.acfsctl.cfg" does not exist. > 1420-001 Authorization "oracle.dev.acfsctl.udef" does not exist. > 1420-001 Authorization "oracle.dev.acfsctl.ucfg" does not exist. > 1420-001 Authorization "oracle.dev.advmctl.def" does not exist. > 1420-001 Authorization "oracle.dev.advmctl.cfg" does not exist. > 1420-001 Authorization "oracle.dev.advmctl.udef" does not exist. > 1420-001 Authorization "oracle.dev.advmctl.ucfg" does not exist. > 1420-001 Authorization "oracle.dev.advmvol.cfg" does not exist. > 1420-001 Authorization "oracle.dev.advmvol.ucfg" does not exist. > The security attributes for the device methods via lssecattr: > 1420-012 "/usr/lib/methods/defacfsctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/udefacfsctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/cfgacfsctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/ucfgacfsctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/defadvmctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/udefadvmctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/cfgadvmctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/ucfgadvmctl" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/cfgadvmvol" does not exist in the privileged command database. > 1420-012 "/usr/lib/methods/ucfgadvmvol" does not exist in the privileged command database. > Executing command: chuser roles and default_roles > Return code = 0 > The user's attributes roles and default_roles via lsuser: > default_roles= > roles= > Kernel security table information from lskst: > Role oracle_devmgmt: > Authorizations: > Methods: > exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors: > 0509-150 Dependent module libhasgen12.so could not be loaded. > 0509-022 Cannot load module libhasgen12.so. > 0509-026 System error: A file or directory in the path name does not exist. > ACFS-9361: Removing device 'acfsctl' failed with error code '255'. > ACFS-9347: Unable to set permission bits (0644) on: '/etc/oracledrivers.conf'. > ACFS-9294: updating file /etc/oracledrivers.conf > ACFS-9178: Return code = USM_FAIL > ACFS-9177: Return from 'uninstall' > ACFS-9305: ADVM/ACFS installation cannot proceed: > ACFS-9306: Failed to uninstall previous installation. > ACFS-9178: Return code = USM_FAIL > ACFS-9177: Return from 'install' > ACFS-9176: Entering 'acroot ex' > ACFS-9178: Return code = 1 > ACFS-9177: Return from 'acroot ex' >End Command output 2022-02-24 22:02:40: /GRIDHOME/oracle/app/product/grid/19.3.0/bin/acfsroot install -t2 ... failed 2022-02-24 22:02:40: USM driver install status is 0 2022-02-24 22:02:40: Executing cmd: /GRIDHOME/oracle/app/product/grid/19.3.0/bin/clsecho -p has -f clsrsc -m 196 2022-02-24 22:02:40: Executing cmd: /GRIDHOME/oracle/app/product/grid/19.3.0/bin/clsecho -p has -f clsrsc -m 196 2022-02-24 22:02:41: Command output: > CLSRSC-196: ACFS driver install actions failed >End Command output 2022-02-24 22:02:41: CLSRSC-196: ACFS driver install actions failed 2022-02-24 22:02:41: Invoking "/GRIDHOME/oracle/app/product/grid/19.3.0/bin/cluutil -ckpt -oraclebase /GRIDHOME/oracle/orabase -writeckpt -name ROOTHAS_ACFSINST -state FAIL" 2022-02-24 22:02:41: trace file=/GRIDHOME/oracle/orabase/crsdata/oracledbworld/crsconfig/cluutil8.log 2022-02-24 22:02:41: Running as user grid: /GRIDHOME/oracle/app/product/grid/19.3.0/bin/cluutil -ckpt -oraclebase /GRIDHOME/oracle/orabase -writeckpt -name ROOTHAS_ACFSINST -state FAIL 2022-02-24 22:02:42: Removing file /tmp/MyuNAen7SK 2022-02-24 22:02:42: Successfully removed file: /tmp/MyuNAen7SK 2022-02-24 22:02:42: pipe exit code: 0 2022-02-24 22:02:42: /bin/su successfully executed 2022-02-24 22:02:42: 2022-02-24 22:02:42: Succeeded in writing the checkpoint:'ROOTHAS_ACFSINST' with status:FAIL
Next thing we checked was alert.log in crs/trace. ACFS-9306: Failed to uninstall previous installation.
2022-02-24 22:00:07.679 [CLSECHO(4129628)]CLSRSC-0567: Beginning Oracle Grid Infrastructure configuration. 2022-02-24 22:00:28.494 [OCRCONFIG(5113006)]CRS-2101: The OLR was formatted using version 4. 2022-02-24 22:00:38.049 [CLSCFG(3735910)]CRS-1001: The OCR was formatted using version 4. 2022-02-24 22:00:54.132 [CLSECHO(2950874)]CRS-6021: Could not set system resource limits for Clusterware: "ulimit -u 65536". 2022-02-24 22:00:55.136 [OHASD(4325952)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 4325952 2022-02-24 22:00:55.306 [OHASD(4325952)]CRS-0714: Oracle Clusterware Release 19.0.0.0.0. 2022-02-24 22:00:55.479 [OHASD(4325952)]CRS-2112: The OLR service started on node oracledbworld. 2022-02-24 22:00:55.640 [OHASD(4325952)]CRS-1301: Oracle High Availability Service started on node oracledbworld. 2022-02-24 22:01:59.771 [ORAAGENT(4785594)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 4785594 2022-02-24 22:02:01.065 [EVMD(4849842)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 4849842 2022-02-24 22:02:24.691 [CLSECHO(4129782)]ACFS-9300: ADVM/ACFS distribution files found. 2022-02-24 22:02:28.814 [CLSECHO(3146968)]ACFS-9314: Removing previous ADVM/ACFS installation. 2022-02-24 22:02:30.851 [CLSECHO(3342922)]ACFS-9361: Removing device 'acfsctl' failed with error code '255'. 2022-02-24 22:02:38.381 [CLSECHO(4129634)]ACFS-9294: updating file /etc/oracledrivers.conf 2022-02-24 22:02:39.054 [CLSECHO(4916364)]ACFS-9305: ADVM/ACFS installation cannot proceed: 2022-02-24 22:02:39.402 [CLSECHO(3736266)]ACFS-9306: Failed to uninstall previous installation.
Initially, I thought it is missing library file issue. And hence thought of raising an SR for this.
$ls -ltr $ORACLE_HOME/lib/libhasgen12.so ls: 0653-341 The file /GRIDHOME/oracle/app/product/grid/19.3.0/lib/libhasgen12.so does not exist.
But with few searches on Oracle Support, Found an oracle document which was pointing to same issue. I was told that server which is released to us was a fresh server, but looking at the output it wasn’t. If you look at the output of strings of udefacfsctl.bin you will see the USM_VERSION to be 12.1.0.2 whereas I am installing 19.14
# strings /usr/lib/methods/udefacfsctl.bin @(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59 @(#)61 1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21 ▒▒▒▒▒ 12.1.0.2.0 USM_VERSION-12.1.0.2.0.0 >>>>>>>>>>>>>>>>>>>>>> =======> Here
# /usr/lib/methods/ucfgacfsctl -l ofsctl exec(): 0509-036 Cannot load program /usr/lib/methods/ucfgacfsctl.bin because of the following errors: 0509-150 Dependent module libhasgen12.so could not be loaded. 0509-022 Cannot load module libhasgen12.so. 0509-026 System error: A file or directory in the path name does not exist. #
# cd /GRIDHOME/oracle/app/product/grid/19.3.0/ # acfsroot version_check ksh: acfsroot: not found. # cd bin # ./acfsroot version_check ACFS-9316: Valid ADVM/ACFS distribution media detected at: '/GRIDHOME/oracle/app/product/grid/19.3.0/usm/install/AIX/V7x/powerpc/bin' # ./acfsdriverstate installed ACFS-9203: true # ./acfsdriverstate loaded ACFS-9204: false # ./acfsdriverstate supported ACFS-9200: Supported # ./acfsdriverstate version ACFS-9129: ADVM/ACFS not installed #
Change the ORACLE_HOME to new home in following listed files.
# vi /usr/lib/methods/ucfgacfsctl # vi /usr/lib/methods/ucfgacfsctl # vi /usr/lib/methods/ucfgadvmctl # vi /usr/lib/methods/udefacfsctl # vi /usr/lib/methods/udefadvmctl
Then as per the Doc ID 1929899.1 , we executed following commands –
# /usr/lib/methods/ucfgacfsctl -l ofsctl # /usr/lib/methods/ucfgadvmctl -l advmctl # /usr/lib/methods/udefacfsctl -l ofsctl # /usr/lib/methods/udefadvmctl -l advmctl
And once it was completed successfully, we removed all the files as per doc ( Doc ID 1929899.1 )
# /usr/sbin/rmauth -h grid 1420-001 Authorization "grid" does not exist. # rmrole oracle_devmgmt 3004-733 Role "oracle_devmgmt" does not exist. # setkst Successfully updated the Kernel Authorization Table. Successfully updated the Kernel Role Table. Successfully updated the Kernel Command Table. Successfully updated the Kernel Device Table. Successfully updated the Kernel Object Domain Table. Successfully updated the Kernel Domains Table. Successfully updated the Kernel RBAC log level. # rm /usr/lib/drivers/oracle* rm: Remove /usr/lib/drivers/oracleacfs.ext? yes rm: Remove /usr/lib/drivers/oracleadvm.ext? yes rm: Remove /usr/lib/drivers/oracleoks.ext? yes # alias rm=rm # rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs* # rm -rf /sbin/helpers/acfs # rm /usr/sbin/acfsutil* /usr/sbin/advmutil* # rm /sbin/acfsutil* /sbin/advmutil* #
Now it was time to re-execute the root.sh to see if it is working for us or not.
# sh /GRIDHOME/oracle/app/product/grid/19.3.0/root.sh Performing root user operation. The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /GRIDHOME/oracle/app/product/grid/19.3.0 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ... The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /GRIDHOME/oracle/app/product/grid/19.3.0/crs/install/crsconfig_params The log of current session can be found at: /GRIDHOME/oracle/orabase/crsdata/oracledbworld/crsconfig/roothas_2022-02-25_00-09-14AM.log 2022/02/25 00:10:41 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab' oracledbworld 2022/02/25 00:12:23 /GRIDHOME/oracle/orabase/crsdata/oracledbworld/olr/backup_20220225_001223.olr 2965637783 2022/02/25 00:12:25 CLSRSC-327: Successfully configured Oracle Restart for a standalone server #
And installation was completed. And we did post check for has installation. And found to be successful too.
$ ./cluvfy stage -post hacfg -verbose Verifying Oracle Restart Integrity ...PASSED Verifying OLR Integrity ...PASSED Post-check for Oracle Restart configuration was successful. CVU operation performed: stage -post hacfg Date: Feb 25, 2022 1:22:29 AM CVU home: /GRIDHOME/oracle/app/product/grid/19.3.0/ User: grid $
This way we completed the installation. It took 2 hours but we completed the installation that was important for us.
Reference –
| AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED ( Doc ID 1929899.1 ) |