Identify your RAC Network details.
As per defined IP Addresses in /etc/hosts, Let’s find the same information using various commands available from GRID and OS.
Let’s verify the same if Grid (cluster) picked up the correct IP’s.
NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
192.168.56.99 mgsrv-dns mgsrv-dns.mgdom.com loghost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
###########################################################################################################
1) To Find Cluster Name
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster
###########################################################################################################
2) Find the PUBLIC and VIP Node name
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i
mgracsolsrv64bit1 1 mgracsolsrv64bit1-vip
mgracsolsrv64bit2 2 mgracsolsrv64bit2-vip
$ /u01/app/11.2.0.1/grid/bin/srvctl config nodeapps -a
VIP exists.:mgracsolsrv64bit1
VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0
VIP exists.:mgracsolsrv64bit2
VIP exists.: /mgracsolsrv64bit2-vip/192.168.56.31/255.255.255.0/e1000g0
$ /u01/app/11.2.0.1/grid/bin/crsctl stat res ora.mgracsolsrv64bit1.vip -p |egrep ‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’
NAME=ora.mgracsolsrv64bit1.vip
TYPE=ora.cluster_vip_net1.type
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
USR_ORA_VIP=mgracsolsrv64bit1-vip
VERSION=11.2.0.1.0
###########################################################################################################
3) To find Private IP Details
###########################################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i -l -p
mgracsolsrv64bit1 1 192.168.5.1 mgracsolsrv64bit1-vip => (Middle one is Private IP – 192.168.5.1)
The Oracle Interface Configuration Tool (oifcfg) is used define and administer network interfaces such as the public and private interfaces.
———————————————————————————————————————————————
Oracle only store network interface name and subnet ID in OCR, not the netmask. Oifcfg command can be used for such change, oifcfg commands only require to run on 1 of the cluster node, not all. When netmask is changed, the associated subnet ID is also changed.
Interfaces used are : e1000g0 , e1000g1 , oifcfg would not show exact IP’s, instead it shows SubnetID in OCR.
Subnet Info in Oracle Clusterware – OCR – To find out what’s in OCR:
Both public and private network information are stored in OCR.
$ /u01/app/11.2.0.1/grid/bin/oifcfg getif
e1000g0 192.168.56.0 global public
e1000g1 192.168.0.0 global cluster_interconnect —-> Private Interface
Note:
The first column is the network adapter name
The second column is the subnet ID
The third column is always “global” and should not be changed
The last column indicates whether it’s public or cluster_interconnect(private) in Oracle Clusterware
To include the subnet mask, append the -n option to the -p option:
Shows the available interfaces that you can configure with setif.
The –p parameter displays the type of interface which can be PRIVATE, PUBLIC or UNKNOWN.
The iflist command queries the operating system to find which network interfaces are present on this node.
————————————————————————————————————–
$ ifconfig -a ... ... e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 ==> Private IP inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255 ...
$ /u01/app/11.2.0.1/grid/bin/oifcfg iflist -p -n
e1000g0 192.168.56.0 PUBLIC 255.255.255.0
e1000g1 192.168.0.0 PUBLIC 255.255.0.0 —–> Private interface on different Subnet 255.255.0.0
e1000g2 192.168.56.0 PUBLIC 255.255.255.0
Note :
oifcfg getif => Does not works, if Oracle Clusterware (CRS) is not running.
$ ps -ef |grep d.bin |wc -l
0
$ oifcfg getif
PRIF-10: failed to initialize the cluster registry
oifcfg iflist => Works, even if Oracle Clusterware (CRS) is not running.
$ oifcfg iflist -p -n
e1000g0 192.168.56.0 PUBLIC 255.255.255.0
e1000g1 192.168.0.0 PUBLIC 255.255.0.0
$ cat /etc/hosts |grep 192.168.5.1
192.168.5.1 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
Note:
The first column is the network adapter name
The second column is the subnet ID
The third column indicates whether it’s private, public or unknown according to RFC standard, it has NOTHING to do whether it’s used as private or public network in Oracle Clusterware
The last column is the netmask
$ netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue lo0 8232 loopback localhost 51795 0 51795 0 0 0 e1000g0 1500 mgracsolsrv64bit1 mgracsolsrv64bit1 6288 0 4673 0 0 0 e1000g1 1500 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv 121009 0 83308 0 0 0 e1000g2 1500 mgsrv-dns mgsrv-dns 4493 0 201 0 0 0
$ netstat -in
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue lo0 8232 127.0.0.0 127.0.0.1 52598 0 52598 0 0 0 e1000g0 1500 192.168.56.0 192.168.56.20 6356 0 4734 0 0 0 e1000g1 1500 192.168.0.0 192.168.5.1 123476 0 85456 0 0 0 e1000g2 1500 192.168.56.0 192.168.56.99 4562 0 202 0 0 0
mgracsolsrv64bit1:/export/home/grid: ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 ==> Public IP inet 192.168.56.20 netmask ffffff00 broadcast 192.168.56.255 e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2 ==> SCAN IP 1 inet 192.168.56.81 netmask ffffff00 broadcast 192.168.56.255 e1000g0:3: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2 ==> SCAN IP 2 inet 192.168.56.82 netmask ffffff00 broadcast 192.168.56.255 e1000g0:5: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2 ==> VIP inet 192.168.56.30 netmask ffffff00 broadcast 192.168.56.255 e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 ==> Private IP inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255 e1000g2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 ==> DNS Server IP inet 192.168.56.99 netmask ffffff00 broadcast 192.168.56.255
$ netstat -r
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default mgsrv-dns UG 1 0 192.168.0.0 mgracsolsrv64bit1-priv U 1 97 e1000g1 192.168.56.0 mgracsolsrv64bit1 U 1 90 e1000g0 192.168.56.0 mgsrv-dns U 1 0 e1000g2 192.168.56.0 mgracsolsrv64bit1 U 1 0 e1000g0:1 -> SCANIP 1 - 192.168.56.81 192.168.56.0 mgracsolsrv64bit1 U 1 0 e1000g0:3 -> SCANIP 2 - 192.168.56.82 192.168.56.0 mgracsolsrv64bit1 U 1 0 e1000g0:5 -> VIP - 192.168.56.30 224.0.0.0 mgracsolsrv64bit1 U 1 0 e1000g0 localhost localhost UH 59 21132 lo0
$ netstat -rnv
IRE Table: IPv4
Destination Mask Gateway Device Mxfrg Rtt Ref Flg Out In/Fwd -------------------- --------------- -------------------- ------ ----- ----- --- --- ----- ------ default 0.0.0.0 192.168.56.99 1500* 0 1 UG 0 0 192.168.0.0 255.255.0.0 192.168.5.1 e1000g1 1500* 0 1 U 100 0 192.168.56.0 255.255.255.0 192.168.56.20 e1000g0 1500* 0 1 U 93 0 192.168.56.0 255.255.255.0 192.168.56.99 e1000g2 1500* 0 1 U 0 0 192.168.56.0 255.255.255.0 192.168.56.20 e1000g0:1 1500* 0 1 U 0 0 192.168.56.0 255.255.255.0 192.168.56.20 e1000g0:3 1500* 0 1 U 0 0 192.168.56.0 255.255.255.0 192.168.56.20 e1000g0:5 1500* 0 1 U 0 0 224.0.0.0 240.0.0.0 192.168.56.20 e1000g0 1500* 0 1 U 0 0 127.0.0.1 255.255.255.255 127.0.0.1 lo0 8232* 0 59 UH 21725 0
###########################################################################################################
4) To Find SCAN IP
###########################################################################################################
$ nslookup mgrac-scan
Server: 192.168.56.99
Address: 192.168.56.99#53
Name: mgrac-scan.mgdom.com
Address: 192.168.56.82
Name: mgrac-scan.mgdom.com
Address: 192.168.56.83
Name: mgrac-scan.mgdom.com
Address: 192.168.56.81
$ dig mgrac-scan.mgdom.com +noall +answer
; <<>> DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com. 600 IN A 192.168.56.81
mgrac-scan.mgdom.com. 600 IN A 192.168.56.82
mgrac-scan.mgdom.com. 600 IN A 192.168.56.83
$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /192.168.56.83/192.168.56.83
SCAN VIP name: scan2, IP: /192.168.56.81/192.168.56.81
SCAN VIP name: scan3, IP: /192.168.56.82/192.168.56.82
When /etc/hosts is configured with SCAN names:
$ cat /etc/hosts |grep -i scan
# SCAN
192.168.56.81 mgracscan1
192.168.56.82 mgracscan2
192.168.56.83 mgracscan3
$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82
$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
##################################################################################################################
Before You Start Grid Installation Please follow below to configure the required IP Addresses.
##################################################################################################################
Even if you are using a DNS,Oracle recommends that you add lines to the /etc/hosts file on each node,specifying the public IP, VIP and private addresses.
Configure the /etc/hosts file so that it is similar to the following example:
NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
192.168.56.99 mgsrv-dns mgsrv-dns.mgdom.com loghost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
=================================================================================
NODE-2
=========
mgracsolsrv64bit2:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
=================================================================================
Public – Requires one interface or network Adapter/card
=================================================================================
Determine the public host name for each node in the cluster.
1) For the public host name, use the primary host name of each node.
2) In other words, use the name displayed by the hostname command for example: mgracsolsrv64bit1
3) Static IP address
4) Configured before installation for each node, and resolvable to that node before installation.
5) It should be On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses.
6) A public IP is registered on DNS
7) Assign One IP address with an associated host name (or network name) registered in the DNS for the public interface.
8) Record the host name and IP address in the system hosts file also, /etc/hosts.
9) Each node needs a public IP address This can be an interface bonded using IPMP.
=================================================================================
Private – Requires one interface or network Adapter/card
=================================================================================
Determine the private hostname for each node in the cluster.
1) This private hostname does not need to be resolvable through DNS and should be entered in the /etc/hosts file.
2) The private IP should NOT be accessable to servers not participating in the local cluster. (Only RAC Nodes should ping each other using Priv IP)
3) The private network should be on standalone dedicated switch(es).
4) The private network should NOT be part of a larger overall network topology.
5) The private network should be deployed on Gigabit Ethernet or better
6) It is recommended that redundant NICs are configured For Solaris either Sun Trunking (OS based) or Sun IPMP (OS based)
More information: <<Note: 283107.1>>
7) IPMP in general. When IPMP is used for the interconnect: <<Note: 368464.1>>
NOTE: If IPMP is used for public and/or cluster interconnect, critical merge patch 9729439 should be applied to both Grid Infrastructure and RDBMS Oracle homes.
8) Static IP address
9) Configured before installation, but on a separate, private network, with its own subnet, that is not resolvable except by other cluster member nodes.
10) A private IP only known to the servers in the RAC configuration, to be used by the interconnect.
11) Configure the private interconnect network interface cards to have a private node name and a private IP address.
12) This can be an interface bonded using IPMP.
13) Choose a private IP address that is in the address range 10.*.*.* or 192.168.*.*
Private 10.0.0.1 Hosts file
Private 10.0.0.2 Hosts file
14) The private IP address should be on a separate subnet than the public IP address.
15) Oracle strongly recommends using a physically separate, private network (You can use a vlan on a shared switch if you must).
16) You should ensure that the private IP addresses are reachable only by the cluster member nodes.
17) Use Non Routable network addresses for private interconnect:
Class A: 10.0.0.0 to 10.255.255.255
Class B: 172.16.0.0 to 172.31.255.255
Class C: 192.168.0.0 to 192.168.255.255
http://www.subnet-calculator.com/
http://www.subnetonline.com/pages/subnet-calculators/ip-subnet-calculator.php
Or Google search – “subnet calculator”
=======================================================================================
VIP – NOT required separate Network adapter/card , it uses existing public interface.
=======================================================================================
1) The virtual IP address and the network name must not be currently in use, (But should not be accessible by a ping command or NOT Pingable)
2) The virtual IP address must be on the same subnet as your public IP address.
3) The virtual host name for each node should be registered with your DNS.
4) Static IP address
5) Configured before installation for each node, The IP address and host name are currently unused (it can be registered in a DNS, but should not be accessible by a ping command).
6) It should On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses.
7) A virtual IP is registered on DNS, but NOT defined in the servers. It will be defined later during Oracle Clusterware Install
8) Assign One virtual IP (VIP) address with an associated host name registered in a DNS.
9) Record the host name and VIP address in the system hosts file, /etc/hosts.
===========================================================================================
SCAN – VIP – Define in corporate DNS (Domain Name Service), this again uses existing public Interface
===========================================================================================
1) Determine your cluster name. The cluster name should satisfy the following conditions:
2) The cluster name is globally unique throughout your host domain.
3) The cluster name is at least 1 character long and less than 15 characters long.
4) The cluster name must consist of the same character set used for host names: single-byte alphanumeric
characters (a to z, A to Z, and 0 to 9) and hyphens (-).
5) Define the SCAN in your corporate DNS (Domain Name Service), You must ask your network administrator to create a single name,
that resolves to three IP addresses using a round robin algorithm.
6) The IP addresses must be on the same subnet as your public network in the cluster.
7) SCAN VIPs must NOT be in the /etc/hosts file, it must be resolved by DNS.
$ /u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster
$ nslookup mgrac-scan
Server: 192.168.56.99
Address: 192.168.56.99#53
Name: mgrac-scan.mgdom.com
Address: 192.168.56.81
Name: mgrac-scan.mgdom.com
Address: 192.168.56.82
Name: mgrac-scan.mgdom.com
Address: 192.168.56.83
$ dig mgrac-scan.mgdom.com +noall +answer
; <<>> DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com. 600 IN A 192.168.56.83
mgrac-scan.mgdom.com. 600 IN A 192.168.56.81
mgrac-scan.mgdom.com. 600 IN A 192.168.56.82
$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82
$ /u01/app/11.2.0.1/grid/bin/srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
Verify SCAN Ip’s Registered with resource
===========================================
$ /u01/app/11.2.0.1/grid/bin/crsctl stat res ora.scan1.vip -p |egrep ‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
SCAN_NAME=mgrac-scan
START_DEPENDENCIES=hard(ora.net1.network) dispersion:active(type:ora.scan_vip.type) pullup(ora.net1.network)
USR_ORA_VIP=192.168.56.83
VERSION=11.2.0.1.0
Refer few notes:
=================
http://technet.microsoft.com/en-us/library/cc958825.aspx
http://technet.microsoft.com/en-us/library/cc958832.aspx
How to Modify Private Network Information in Oracle Clusterware (Doc ID 283684.1)
How to Modify Public Network Information including VIP in Oracle Clusterware (Doc ID 276434.1)
How to Modify SCAN Setting or SCAN Listener Port after Installation (Doc ID 972500.1)
Maleshg 
Good information and all in one place Malesh. Keep blogging. Thanks.
LikeLike
Thanks Gurudutt 🙂 !!
LikeLike
Really useful blog, thank for all these sharing
LikeLike
Thanks Pradipto !!
LikeLike