Scan Listener – Not All Endpoints Registered

Hello Readers, today we faced a new issue where one of our scan listener after reboot were giving us error. Resource was in Intermediate state. Rest all the scan listener were online. Scan Listener – Not All Endpoints Registered And there was no services were registered to Scan listener 1 .

We have changed the server IP and hostname from the output which will be shared here on the website.

ISSUE – Scan Listener – Not All Endpoints Registered

Here we check the cluster status. And here was the output, apart from that all listeners where online and services were registered.

$crsctl stat res -t
.
.
.
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
       1        ONLINE  INTERMEDIATE uat780-l3              Not All Endpoints Registered,STABLE
ora.LISTENER_SCAN2.lsnr
       1        ONLINE  ONLINE       uat780-l3              STABLE
ora.LISTENER_SCAN3.lsnr
       1        ONLINE  ONLINE       uat780-02-lp1          STABLE
.
.
.

We checked the listener log of scan listener 1 and here it was giving us following error –

25-JUL-2020 18:49:10 * remote_registration_address * 0
Error listening on:
(ADDRESS=(PROTOCOL=TCP)(RATE_LIMIT=YES)(HOST=192.50.6.163)(PORT=1988))
25-JUL-2020 18:49:11 * service_register * LsnrAgt * 12542
TNS-12542: TNS:address already in use
  TNS-12560: TNS:protocol adapter error
   TNS-00512: Address already in use
    IBM/AIX RISC System/6000 Error: 67: Address already in use

Scan details before shutting down the cluster services and releasing it to server team.

Scan 1 --> 192.50.6.163
Scan 2 --> 192.50.6.164
Scan 3 --> 192.50.6.165

Cause of Issue –>

Then we checked the stats of resource for ora.scan1.vip and found something

$crsctl stat res ora.scan1.vip -p
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
ACL=owner:root:rwx,pgrp:system:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=120
CLEAN_TIMEOUT=60
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle SCAN VIP resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_USR_ORA_STATIC_VIP=192.50.6.163
GEN_USR_ORA_VIP=
HOSTING_MEMBERS=
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NETNUM=1
NETWORK_COUNT=
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
PLACEMENT=restricted
RELOCATE_BY_DEPENDENCY=1
RESTART_ATTEMPTS=0
SCAN_NAME=uat-scan
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=ora.hub.category
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.net1.network) weak(global:ora.gns)
dispersion:active(type:ora.scan_vip.type) pullup(global:ora.net1.network)
START_TIMEOUT=120
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)
STOP_TIMEOUT=0
TYPE_VERSION=1.1
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=
USR_ORA_VIP=192.50.6.164

There was some issue with above output as well. Check these IPs which are showing different IPs.
GEN_USR_ORA_STATIC_VIP=192.50.6.163
USR_ORA_VIP=192.50.6.164 —> scan_listener2 IP address

I checked for output for rest two scan listener and we came to conclusion that these parameters should have same values.

What we tried –> Scan Listener – Not All Endpoints Registered

We tried to stop and start the scan listener 1 but it didn’t work.
We tried to relocate it on node 2 it didn’t work either.
And we tried to set the values to null on GEN_USR_ORA_STATIC_VIP and USR_ORA_VIP but that also didn’t work for us.

1. Stop scan listener 1

$ srvctl stop scan -scannumber 1 -f             (we are running 12.1.0.2 GI) 

2.  As root user modify SCAN vip1 GEN_USR_ORA_STATIC_VIP parameter to be null value:

# GRID_HOME/bin/crsctl modify resource ora.scan1.vip -attr "GEN_USR_ORA_STATIC_VIP=" -unsupported
# GRID_HOME/bin/crsctl modify resource ora.scan1.vip -attr "USR_ORA_VIP=" -unsupported
 
3. Start scan listener 1 

$ srvctl start scan -scannumber 1               (we are running 12.1.0.2 GI) 

Solution –>

Then we planned to set those values explicitly. And it worked for us.

1. Stop scan listener 1

$ srvctl stop scan -scannumber 1 -f             (we are running 12.1.0.2 GI) 

2.  As root user modify SCAN vip1 GEN_USR_ORA_STATIC_VIP parameter to be null value:

# GRID_HOME/bin/crsctl modify resource ora.scan1.vip -attr "GEN_USR_ORA_STATIC_VIP=192.50.6.163" -unsupported
# GRID_HOME/bin/crsctl modify resource ora.scan1.vip -attr "USR_ORA_VIP=192.50.6.163" -unsupported
 
3. Start scan listener 1 

$ srvctl start scan -scannumber 1               (we are running 12.1.0.2 GI) 

Post these changes , we can see all the services were registered on scan 1. Phew 😛 After 3 hours of searching and testing we could Identify the issue and solve it for us. Now we were having a different issue here. This server’s oracle central inventory was corrupted.

Reference -> Not All Endpoints Registered for SCAN Listener (Doc ID 2391077.1)

Thanks for reading. Happy Learning. 🙂