Skip to content

Identify your RAC Network details.

November 3, 2013

As per defined IP Addresses in /etc/hosts, Let’s find the same information using various commands available from GRID and OS.

Let’s verify the same if Grid (cluster) picked up the correct IP’s.

NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.56.99   mgsrv-dns         mgsrv-dns.mgdom.com       loghost

#PUBLIC
192.168.56.20   mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21   mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com

#PRIVATE
192.168.05.01  mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02  mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com

#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com

###########################################################################################################
1) To Find Cluster Name
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster

###########################################################################################################
2) Find the PUBLIC and VIP Node name
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i
mgracsolsrv64bit1       1       mgracsolsrv64bit1-vip
mgracsolsrv64bit2       2       mgracsolsrv64bit2-vip

$ /u01/app/11.2.0.1/grid/bin/srvctl config nodeapps -a
VIP exists.:mgracsolsrv64bit1
VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0
VIP exists.:mgracsolsrv64bit2
VIP exists.: /mgracsolsrv64bit2-vip/192.168.56.31/255.255.255.0/e1000g0

$ /u01/app/11.2.0.1/grid/bin/crsctl stat res ora.mgracsolsrv64bit1.vip -p |egrep ‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’
NAME=ora.mgracsolsrv64bit1.vip
TYPE=ora.cluster_vip_net1.type
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
USR_ORA_VIP=mgracsolsrv64bit1-vip
VERSION=11.2.0.1.0

###########################################################################################################
3) To find Private IP Details
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i -l -p
mgracsolsrv64bit1       1       192.168.5.1     mgracsolsrv64bit1-vip    => (Middle one is Private IP – 192.168.5.1)

The Oracle Interface Configuration Tool (oifcfg) is used define and administer network interfaces such as the public and private interfaces.
———————————————————————————————————————————————

Oracle only store network interface name and subnet ID in OCR, not the netmask. Oifcfg command can be used for such change, oifcfg commands only require to run on 1 of the cluster node, not all. When netmask is changed, the associated subnet ID is also changed.

Interfaces used  are : e1000g0  , e1000g1 , oifcfg would not show exact IP’s,  instead it shows SubnetID  in OCR.

Subnet Info in Oracle Clusterware – OCR – To find out what’s in OCR:

Both public and private network information are stored in OCR.

$ /u01/app/11.2.0.1/grid/bin/oifcfg getif
e1000g0  192.168.56.0  global  public
e1000g1  192.168.0.0  global  cluster_interconnect       —-> Private Interface

Note:
The first column is the network adapter name
The second column is the subnet ID
The third column is always “global” and should not be changed
The last column indicates whether it’s public or cluster_interconnect(private) in Oracle Clusterware

To include the subnet mask, append the -n option to the -p option:
Shows the available interfaces that you can configure with setif.
The –p parameter displays the type of interface which can be PRIVATE, PUBLIC or UNKNOWN.
The iflist command queries the operating system to find which network interfaces are present on this node.
————————————————————————————————————–

$ ifconfig -a 
...
...
 e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3                 ==> Private IP
 inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255
...

$ /u01/app/11.2.0.1/grid/bin/oifcfg iflist -p -n
e1000g0  192.168.56.0  PUBLIC   255.255.255.0
e1000g1  192.168.0.0   PUBLIC 255.255.0.0  —–> Private interface on different Subnet 255.255.0.0
e1000g2  192.168.56.0  PUBLIC   255.255.255.0

Note :

oifcfg getif => Does not works, if  Oracle Clusterware (CRS) is not running.

$ ps -ef |grep d.bin |wc -l
0

$ oifcfg getif
PRIF-10: failed to initialize the cluster registry

oifcfg iflist => Works, even if  Oracle Clusterware (CRS) is not running.

$ oifcfg iflist -p -n
e1000g0  192.168.56.0  PUBLIC  255.255.255.0
e1000g1  192.168.0.0  PUBLIC  255.255.0.0

$ cat /etc/hosts |grep 192.168.5.1
192.168.5.1  mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com

Note:
The first column is the network adapter name
The second column is the subnet ID
The third column indicates whether it’s private, public or unknown according to RFC standard, it has NOTHING to do whether it’s used as private or public network in Oracle Clusterware

The last column is the netmask

$ netstat -i

Name      Mtu  Net/Dest              Address                Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0       8232 loopback              localhost              51795  0     51795  0     0      0
e1000g0 1500 mgracsolsrv64bit1       mgracsolsrv64bit1      6288   0     4673   0     0      0
e1000g1 1500 mgracsolsrv64bit1-priv  mgracsolsrv64bit1-priv 121009 0     83308  0     0      0
e1000g2 1500 mgsrv-dns               mgsrv-dns              4493   0     201    0     0      0

$ netstat -in

Name      Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
 lo0       8232 127.0.0.0     127.0.0.1      52598  0     52598  0     0      0
 e1000g0 1500 192.168.56.0  192.168.56.20    6356   0     4734   0     0      0
 e1000g1 1500 192.168.0.0   192.168.5.1      123476 0     85456  0     0      0
 e1000g2 1500 192.168.56.0  192.168.56.99    4562   0     202    0     0      0

mgracsolsrv64bit1:/export/home/grid: ifconfig -a

 lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
 inet 127.0.0.1 netmask ff000000
 e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2                ==> Public IP
 inet 192.168.56.20 netmask ffffff00 broadcast 192.168.56.255
 e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2    ==> SCAN IP 1
 inet 192.168.56.81 netmask ffffff00 broadcast 192.168.56.255
 e1000g0:3: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2    ==> SCAN IP 2
 inet 192.168.56.82 netmask ffffff00 broadcast 192.168.56.255
 e1000g0:5: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2    ==> VIP
 inet 192.168.56.30 netmask ffffff00 broadcast 192.168.56.255
 e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3                 ==> Private IP
 inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255
 e1000g2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4                 ==> DNS Server IP
 inet 192.168.56.99 netmask ffffff00 broadcast 192.168.56.255

$ netstat -r

Routing Table: IPv4

Destination           Gateway                  Flags  Ref     Use     Interface
 -------------------- --------------------     ----- ----- ---------- ---------
 default              mgsrv-dns                UG        1          0
 192.168.0.0          mgracsolsrv64bit1-priv   U         1         97 e1000g1
 192.168.56.0         mgracsolsrv64bit1        U         1         90 e1000g0
 192.168.56.0         mgsrv-dns                U         1          0 e1000g2
 192.168.56.0         mgracsolsrv64bit1        U         1          0 e1000g0:1    -> SCANIP 1 - 192.168.56.81
 192.168.56.0         mgracsolsrv64bit1        U         1          0 e1000g0:3    -> SCANIP 2 - 192.168.56.82
 192.168.56.0         mgracsolsrv64bit1        U         1          0 e1000g0:5    -> VIP      - 192.168.56.30
 224.0.0.0            mgracsolsrv64bit1        U         1          0 e1000g0
 localhost            localhost                UH       59      21132 lo0

$ netstat -rnv

IRE Table: IPv4

Destination             Mask           Gateway          Device        Mxfrg Rtt   Ref Flg  Out  In/Fwd
 -------------------- --------------- -------------------- ------     ----- ----- --- --- ----- ------
 default              0.0.0.0         192.168.56.99                   1500*    0   1     UG    0      0
 192.168.0.0          255.255.0.0     192.168.5.1          e1000g1    1500*    0   1     U    100     0
 192.168.56.0         255.255.255.0   192.168.56.20        e1000g0    1500*    0   1     U    93      0
 192.168.56.0         255.255.255.0   192.168.56.99        e1000g2    1500*    0   1     U     0      0
 192.168.56.0         255.255.255.0   192.168.56.20        e1000g0:1  1500*    0   1     U     0      0
 192.168.56.0         255.255.255.0   192.168.56.20        e1000g0:3  1500*    0   1     U     0      0
 192.168.56.0         255.255.255.0   192.168.56.20        e1000g0:5  1500*    0   1     U     0      0
 224.0.0.0            240.0.0.0       192.168.56.20        e1000g0    1500*    0   1     U     0      0
 127.0.0.1            255.255.255.255 127.0.0.1            lo0        8232*    0  59     UH   21725   0

###########################################################################################################
4) To Find SCAN IP
###########################################################################################################

$ nslookup mgrac-scan
Server:         192.168.56.99
Address:        192.168.56.99#53

Name:   mgrac-scan.mgdom.com
Address: 192.168.56.82
Name:   mgrac-scan.mgdom.com
Address: 192.168.56.83
Name:   mgrac-scan.mgdom.com
Address: 192.168.56.81

$ dig mgrac-scan.mgdom.com +noall +answer

; <<>> DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.81
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.82
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.83

$ /u01/app/11.2.0.1/grid/bin/srvctl config scan

SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /192.168.56.83/192.168.56.83
SCAN VIP name: scan2, IP: /192.168.56.81/192.168.56.81
SCAN VIP name: scan3, IP: /192.168.56.82/192.168.56.82

When /etc/hosts is configured with SCAN names:

$ cat /etc/hosts |grep -i scan
# SCAN
192.168.56.81 mgracscan1
192.168.56.82 mgracscan2
192.168.56.83 mgracscan3

$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82

$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

##################################################################################################################
Before You Start Grid Installation Please follow below to configure the required IP Addresses.
##################################################################################################################

Even if you are using a DNS,Oracle recommends that you add lines to the /etc/hosts file on each node,specifying the public IP, VIP and private addresses.
Configure the /etc/hosts file so that it is similar to the following example:

NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.56.99   mgsrv-dns         mgsrv-dns.mgdom.com       loghost

#PUBLIC
192.168.56.20   mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21   mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com

#PRIVATE
192.168.05.01  mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02  mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com

#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com

=================================================================================
NODE-2
=========
mgracsolsrv64bit2:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost

#PUBLIC
192.168.56.20   mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21   mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com

#PRIVATE
192.168.05.01  mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02  mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com

#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com

=================================================================================
Public – Requires one interface or network Adapter/card
=================================================================================
Determine the public host name for each node in the cluster.
1) For the public host name, use the primary host name of each node.
2) In other words, use the name displayed by the hostname command for example: mgracsolsrv64bit1
3) Static IP address
4) Configured before installation for each node, and resolvable to that node before installation.
5) It should be On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses.
6) A public IP is registered on DNS
7) Assign One IP address with an associated host name (or network name) registered in the DNS for the public interface.
8) Record the host name and IP address in the system hosts file also, /etc/hosts.
9) Each node needs a public IP address This can be an interface bonded using IPMP.

=================================================================================
Private – Requires one interface or network Adapter/card
=================================================================================
Determine the private hostname for each node in the cluster.
1) This private hostname does not need to be resolvable through DNS and should be entered in the /etc/hosts file.
2) The private IP should NOT be accessable to servers not participating in the local cluster. (Only RAC Nodes should ping each other using Priv IP)
3) The private network should be on standalone dedicated switch(es).
4) The private network should NOT be part of a larger overall network topology.
5) The private network should be deployed on Gigabit Ethernet or better
6) It is recommended that redundant NICs are configured For Solaris either Sun Trunking (OS based) or Sun IPMP (OS based)
More information: <<Note: 283107.1>>
7) IPMP in general. When IPMP is used for the interconnect: <<Note: 368464.1>>

NOTE: If IPMP is used for public and/or cluster interconnect, critical merge patch 9729439 should be applied to both Grid Infrastructure and RDBMS Oracle homes.

8) Static IP address
9) Configured before installation, but on a separate, private network, with its own subnet, that is not resolvable except by other cluster member nodes.
10) A private IP only known to the servers in the RAC configuration, to be used by the interconnect.
11) Configure the private interconnect network interface cards to have a private node name and a private IP address.
12) This can be an interface bonded using IPMP.
13) Choose a private IP address that is in the address range 10.*.*.* or 192.168.*.*
Private     10.0.0.1     Hosts file
Private     10.0.0.2     Hosts file
14) The private IP address should be on a separate subnet than the public IP address.
15) Oracle strongly recommends using a physically separate, private network (You can use a vlan on a shared switch if you must).
16) You should ensure that the private IP addresses are reachable only by the cluster member nodes.

17) Use Non Routable network addresses for private interconnect:
Class A: 10.0.0.0 to 10.255.255.255
Class B: 172.16.0.0 to 172.31.255.255
Class C: 192.168.0.0 to 192.168.255.255

http://www.subnet-calculator.com/

http://www.subnetmask.info/

http://jodies.de/ipcalc

http://www.subnetonline.com/pages/subnet-calculators/ip-subnet-calculator.php

Or Google search – “subnet calculator”

=======================================================================================
VIP – NOT required separate Network adapter/card , it uses existing public interface.
=======================================================================================
1) The virtual IP address and the network name must not be currently in use, (But should not be accessible by a ping command or NOT Pingable)
2) The virtual IP address must be on the same subnet as your public IP address.
3) The virtual host name for each node should be registered with your DNS.
4) Static IP address
5) Configured before installation for each node, The IP address and host name are currently unused (it can be registered in a DNS, but should not be accessible by a ping command).
6) It should On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses.
7) A virtual IP is registered on DNS, but NOT defined in the servers. It will be defined later during Oracle Clusterware Install
8) Assign One virtual IP (VIP) address with an associated host name registered in a DNS.
9) Record the host name and VIP address in the system hosts file, /etc/hosts.

===========================================================================================
SCAN – VIP  – Define in corporate DNS (Domain Name Service), this again uses existing public Interface
===========================================================================================
1) Determine your cluster name. The cluster name should satisfy the following conditions:
2) The cluster name is globally unique throughout your host domain.
3) The cluster name is at least 1 character long and less than 15 characters long.
4) The cluster name must consist of the same character set used for host names: single-byte alphanumeric
characters (a to z, A to Z, and 0 to 9) and hyphens (-).

5) Define the SCAN in your corporate DNS (Domain Name Service), You must ask your network administrator to create a single name,
that resolves to three IP addresses using a round robin algorithm.
6) The IP addresses must be on the same subnet as your public network in the cluster.
7) SCAN VIPs must NOT be in the /etc/hosts file, it must be resolved by DNS.

$ /u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster

$ nslookup mgrac-scan
Server:         192.168.56.99
Address:        192.168.56.99#53

Name:   mgrac-scan.mgdom.com
Address: 192.168.56.81
Name:   mgrac-scan.mgdom.com
Address: 192.168.56.82
Name:   mgrac-scan.mgdom.com
Address: 192.168.56.83

$ dig mgrac-scan.mgdom.com +noall +answer

; <<>> DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.83
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.81
mgrac-scan.mgdom.com.   600     IN      A       192.168.56.82

$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82

$ /u01/app/11.2.0.1/grid/bin/srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

Verify SCAN Ip’s Registered with resource
===========================================
$ /u01/app/11.2.0.1/grid/bin/crsctl stat res ora.scan1.vip -p |egrep ‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’

NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
SCAN_NAME=mgrac-scan
START_DEPENDENCIES=hard(ora.net1.network) dispersion:active(type:ora.scan_vip.type) pullup(ora.net1.network)
USR_ORA_VIP=192.168.56.83
VERSION=11.2.0.1.0

Refer few notes:
=================

http://technet.microsoft.com/en-us/library/cc958825.aspx

http://technet.microsoft.com/en-us/library/cc958832.aspx
How to Modify Private Network Information in Oracle Clusterware (Doc ID 283684.1)
How to Modify Public Network Information including VIP in Oracle Clusterware (Doc ID 276434.1)
How to Modify SCAN Setting or SCAN Listener Port after Installation (Doc ID 972500.1)

Advertisements

From → Oracle, RAC, Unix

5 Comments
  1. Gurudutt permalink

    Good information and all in one place Malesh. Keep blogging. Thanks.

    Like

  2. Pradipto permalink

    Really useful blog, thank for all these sharing

    Like

Trackbacks & Pingbacks

  1. RAC Network details – oracleras

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: