Skip to content

Add Missing VIP on RAC Node 2

October 30, 2013

Add Missing VIP on RAC Node 2

While Installing Grid, it failed for Cluster verification Utility :

Manually ran Cluvfy to check for the issues:

 

./runcluvfy.sh stage -post crsinst -n mgracsolsrv64bit1, mgracsolsrv64bit2 -verbose | tee /export/home/grid/cvulogs/node1.post-crs-check.txt

Checking existence of VIP node application

  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mgracsolsrv64bit2  yes                       unknown                   failed
  mgracsolsrv64bit1  yes                       online                    passed
Result: Check failed

From /etc/hosts

#VIRTUAL

192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com

192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com

Node 2 failed to create the VIP , Let’s Check Existing VIPS:

 

mgracsolsrv64bit2:/export/home/grid: crsctl stat res -t |egrep  ‘net|vip’

ora.net1.network

ora.mgracsolsrv64bit1.vip

ora.scan1.vip

ora.scan2.vip

ora.scan3.vip

 

mgracsolsrv64bit2:/export/home/grid: srvctl config nodeapps -a

VIP exists.:mgracsolsrv64bit1

VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0

 

OR

mgracsolsrv64bit1:/export/home/grid: srvctl config vip -n mgracsolsrv64bit1

VIP exists.:mgracsolsrv64bit1

VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0

 

Node 1 : Network interfaces:

 

mgracsolsrv64bit1:/export/home/grid: ifconfig-a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.56.20 netmask ffffff00 broadcast 192.168.56.255

e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2

        inet 192.168.56.30 netmask ffffff00 broadcast 192.168.56.255

e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

        inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255

e1000g2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4

        inet 192.168.56.99 netmask ffffff00 broadcast 192.168.56.255

 

From above srvctl config command we see VIP is running on node 1 as network interface e1000g0:1.

 

:1 => stands for Virtual IP

 

Even crsctl (cluster) resources showed only One VIP and Nodeapp status show VIP does not exist for node 2 as below.

 

mgracsolsrv64bit2:/export/home/grid: crsctl stat res –t

….

ora.mgracsolsrv64bit1.vip

      1        ONLINE  ONLINE       mgracsolsrv64bit1

….

 

mgracsolsrv64bit2:/export/home/grid: srvctl status nodeapps

VIP mgracsolsrv64bit1-vip is enabled

VIP mgracsolsrv64bit1-vip is running on node: mgracsolsrv64bit1

Network is enabled

Network is running on node: mgracsolsrv64bit1

Network is running on node: mgracsolsrv64bit2

GSD is disabled

GSD is not running on node: mgracsolsrv64bit1

GSD is not running on node: mgracsolsrv64bit2

ONS is enabled

ONS daemon is running on node: mgracsolsrv64bit1

ONS daemon is running on node: mgracsolsrv64bit2

eONS is enabled

eONS daemon is running on node: mgracsolsrv64bit1

eONS daemon is running on node: mgracsolsrv64bit2

 

mgracsolsrv64bit2:/u01/app/grid: srvctl status nodeapps -n mgracsolsrv64bit2

-n <node_name> option has been deprecated.

 

Network is enabled

Network is running on node: mgracsolsrv64bit2

GSD is disabled

GSD is not running on node: mgracsolsrv64bit2

ONS is enabled

ONS daemon is running on node: mgracsolsrv64bit2

eONS is enabled

eONS daemon is running on node: mgracsolsrv64bit2

PRKO-2165 : VIP does not exist on node(s) : mgracsolsrv64bit2

 

 

Let’s us ADD VIP Manually

 

1)    Make sure VIP is not pingable

mgracsolsrv64bit2:/u01/app/oraInventory/logs: ping-s mgracsolsrv64bit2-vip

PING mgracsolsrv64bit2-vip: 56 data bytes

^C

—-mgracsolsrv64bit2-vip PING Statistics—-

59 packets transmitted, 0 packets received, 100% packet loss

 

As above it did not give any response so I have to CTRL+C.

 

2)    Check Current network interface, in my case node 2.

 

mgracsolsrv64bit2:/u01/app/oraInventory/logs: ifconfig-a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.56.21 netmask ffffff00 broadcast 192.168.56.255

e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

        inet 192.168.5.2 netmask ffff0000 broadcast 192.168.255.255

 

3)    As per /etc/hosts find your VIP name

 

From /etc/hosts

#VIRTUAL

192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com

192.168.56.31 mgracsolsrv64bit2-vipmgracsolsrv64bit2-vip.mgdom.com

 

4)    Add the VIP now as user root from Node2

 

Usage: srvctl add nodeapps { { -n <node_name> -A <name|ip>/<netmask>/[if1[|if2…]] } | { -S <subnet>/<netmask>/[if1[|if2…]] } } [-p <portnum>] [-m <multicast-ip-address>] [-e <eons-listen-port>] [-l <ons-local-port>]  [-r <ons-remote-port>] [-t <host>[:<port>][,<host>[:<port>]…]] [-v]

    -n <node_name>           Node name

    -A <vip_name|ip>/<netmask>/[if1[|if2…]]  VIP address spec for node applications

    -S <subnet>/<netmask>/[if1[|if2…]]  NET address spec for network

    -m <multicast-ip-address>   The multicast IP address for eONS

    -p <multicast-portnum>    The port number for eONS

    -e <eons-listen-port>     Local listen port for eONS daemon (Default port number is 2016)

    -l <ons-local-port>      ONS listening port for local client connections

    -r <ons-remote-port>     ONS listening port for connections from remote hosts

    -t <host>[:<port>][,<host>[:<port>]…]  List of remote host/port pairs for ONS daemons outside this cluster

    -h                       Print usage

    -v                       Verbose output

 

srvctladd nodeapps -n mgracsolsrv64bit2 -A mgracsolsrv64bit2-vip/255.255.255.0/e1000g0

PRCN-2018 : Current user grid is not a privileged user

 

su – root

Password:

Oracle Corporation      SunOS 5.10      Generic Patch   January 2005

 

mgracsolsrv64bit2:[root]$ /u01/app/11.2.0.1/grid/bin/srvctladd nodeapps -n mgracsolsrv64bit2 -A mgracsolsrv64bit2-vip/255.255.255.0/e1000g0

mgracsolsrv64bit2:[root]$

 

OR

 

http://docs.oracle.com/cd/E11882_01/rac.112/e41960/srvctladmin.htm#RACAD7736

 

srvctl add vip -n node_name -A {name|ip}/netmask[/if1[if2|…]]

     [-k network_number] [-v]

 

srvctl add vip -n mgracsolsrv64bit2 -A mgracsolsrv64bit2-vip/255.255.255.0/e1000g0 -k 1

 

 

 

 

5)    Check the Status of VIP

 

mgracsolsrv64bit2:/export/home/grid: crsctlstat res ora.mgracsolsrv64bit2.vip

NAME=ora.mgracsolsrv64bit2.vip

TYPE=ora.cluster_vip_net1.type

TARGET=OFFLINE

STATE=OFFLINE

mgracsolsrv64bit2:/export/home/grid: srvctl config nodeapps -a

VIP exists.:mgracsolsrv64bit1

VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0

VIP exists.:mgracsolsrv64bit2

VIP exists.: /mgracsolsrv64bit2-vip/192.168.56.31/255.255.255.0/e1000g0

 

mgracsolsrv64bit2:/export/home/grid: ifconfig-a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.56.21 netmask ffffff00 broadcast 192.168.56.255

e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

        inet 192.168.5.2 netmask ffff0000 broadcast 192.168.255.255

 

6)    From above we see nodeapps show VIP now, but ifconfig command still does not showed the VIP, it happened because we have not started the VIP/nodeapps.

 

mgracsolsrv64bit2:/export/home/grid: srvctl start nodeapps -n mgracsolsrv64bit2

PRKO-2421 : Network resource is already started on node(s): mgracsolsrv64bit2

PRKO-2422 : ONS is already started on node(s): mgracsolsrv64bit2

PRKO-2423 : eONS is already started on node(s): mgracsolsrv64bit2

 

OR

 

mgracsolsrv64bit2:/export/home/grid: srvctl start vip-n mgracsolsrv64bit2

PRKO-2420 : VIP is already started on node(s): mgracsolsrv64bit2

 

7)    Now check the resource status

 

mgracsolsrv64bit2:/export/home/grid: crsctlstat res ora.mgracsolsrv64bit2.vip

NAME=ora.mgracsolsrv64bit2.vip

TYPE=ora.cluster_vip_net1.type

TARGET=ONLINE

STATE=ONLINE on mgracsolsrv64bit2

 

mgracsolsrv64bit2:/export/home/grid: ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.56.21 netmask ffffff00 broadcast 192.168.56.255

e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2

        inet 192.168.56.31 netmask ffffff00 broadcast 192.168.56.255

e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

        inet 192.168.5.2 netmask ffff0000 broadcast 192.168.255.255

 

 

From above we see the VIP is running and its running on network interface e1000g0:1 similar to node 1

 

:1=> stands for Virtual IP

Now Ping the VIP which should be pingable Now :

mgracsolsrv64bit1:/export/home/grid: ping -s 192.168.56.30 1 3
PING 192.168.56.30: 1 data bytes
9 bytes from mgracsolsrv64bit1-vip (192.168.56.30): icmp_seq=0.
9 bytes from mgracsolsrv64bit1-vip (192.168.56.30): icmp_seq=1.
9 bytes from mgracsolsrv64bit1-vip (192.168.56.30): icmp_seq=2.

—-192.168.56.30 PING Statistics—-
3 packets transmitted, 3 packets received, 0% packet loss

mgracsolsrv64bit1:/export/home/grid: ping -s 192.168.56.31 1 3
PING 192.168.56.31: 1 data bytes
9 bytes from mgracsolsrv64bit2-vip (192.168.56.31): icmp_seq=0.
9 bytes from mgracsolsrv64bit2-vip (192.168.56.31): icmp_seq=1.
9 bytes from mgracsolsrv64bit2-vip (192.168.56.31): icmp_seq=2.

—-192.168.56.31 PING Statistics—-
3 packets transmitted, 3 packets received, 0% packet loss

8)    Re-run clufvy to Test existence of VIP node application

 

./runcluvfy.sh stage -post crsinst -n mgracsolsrv64bit1,mgracsolsrv64bit2 -verbose | tee /export/home/grid/cvulogs/node1.post-crs-check.txt

 

Checking node application existence...
Checking existence of VIP node application
  Node Name         Required                  Status                    Comment
  ------------      ------------------------  ------------------------  ----------
  mgracsolsrv64bit2  yes                       online                    passed
  mgracsolsrv64bit1  yes                       online                    passed
Result: Check passed.

 

 

9)    Resource Details :

 

crsctl stat res ora.net1.network -p

 

mgracsolsrv64bit1:/export/home/grid: crsctlstat res ora.net1.network -p

NAME=ora.net1.network

TYPE=ora.network.type

ACL=owner:root:rwx,pgrp:root:r-x,other::r–,group:oinstall:r-x,user:grid:r-x

ACTION_FAILURE_TEMPLATE=

ACTION_SCRIPT=

AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%

ALIAS_NAME=

AUTO_START=restore

CHECK_INTERVAL=1

DEFAULT_TEMPLATE=

DEGREE=1

DESCRIPTION=Oracle Network resource

ENABLED=1

LOAD=1

LOGGING_LEVEL=1

NLS_LANG=

NOT_RESTARTING_TEMPLATE=

OFFLINE_CHECK_INTERVAL=60

PROFILE_CHANGE_TEMPLATE=

RESTART_ATTEMPTS=5

SCRIPT_TIMEOUT=60

START_DEPENDENCIES=

START_TIMEOUT=0

STATE_CHANGE_TEMPLATE=

STOP_DEPENDENCIES=

STOP_TIMEOUT=0

UPTIME_THRESHOLD=1d

USR_ORA_AUTO=

USR_ORA_ENV=

USR_ORA_IF=e1000g0

USR_ORA_NETMASK=255.255.255.0

USR_ORA_SUBNET=192.168.56.0

VERSION=11.2.0.1.0

 

 

mgracsolsrv64bit1:/export/home/grid: crsctlstat res ora.mgracsolsrv64bit1.vip -p

NAME=ora.mgracsolsrv64bit1.vip

TYPE=ora.cluster_vip_net1.type

ACL=owner:root:rwx,pgrp:root:r-x,other::r–,group:oinstall:r-x,user:grid:r-x

ACTION_FAILURE_TEMPLATE=

ACTION_SCRIPT=

ACTIVE_PLACEMENT=1

AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%

AUTO_START=restore

CARDINALITY=1

CHECK_INTERVAL=1

DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=vip)

DEGREE=1

DESCRIPTION=Oracle VIP resource

ENABLED=1

FAILOVER_DELAY=0

FAILURE_INTERVAL=0

FAILURE_THRESHOLD=0

HOSTING_MEMBERS=mgracsolsrv64bit1

LOAD=1

LOGGING_LEVEL=1

NLS_LANG=

NOT_RESTARTING_TEMPLATE=

OFFLINE_CHECK_INTERVAL=0

PLACEMENT=favored

PROFILE_CHANGE_TEMPLATE=

RESTART_ATTEMPTS=0

SCRIPT_TIMEOUT=60

SERVER_POOLS=*

START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)

START_TIMEOUT=0

STATE_CHANGE_TEMPLATE=

STOP_DEPENDENCIES=hard(ora.net1.network)

STOP_TIMEOUT=0

UPTIME_THRESHOLD=1h

USR_ORA_ENV=

USR_ORA_VIP=mgracsolsrv64bit1-vip

VERSION=11.2.0.1.0

=======================================================================

Click Here for the PDF

Advertisements

From → RAC

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: