This is tenth part of my RAC lab series. I will show you step by step how to prepare for and how to install Oracle 12c Grid Infrastructure

RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database

 
 
Oracle 12c official documentation
 

Please refer to the official 12.1 documentation and CLUVFY documentation:

 
 
 
Prepare installation files for grid and database software
 

I need shared directory between host and VMs to to keep oracle installation files there. From RAC nodes I can then grab those files and unzip them into /u01/app/grid/grid-software and /u01/app/oracle/oracle-software. VboxAddition software will help to establish thos shared dirs.

  • Prepare directory for shared dir
maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mkdir -p /opt/vboxshared/Oracle12.1_install
 
  • Move files downloaded from Oracle to that directory
maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mv /home/maciek/Downloads/V46*.zip /opt/vboxshared/Oracle12.1_install
 
  • Set this folder as shared with the vms
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n1 --name SharedWithHost --hostpath /opt/vboxshared --automount
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n2 --name SharedWithHost --hostpath /opt/vboxshared --automount

# check if directory has been mounted in the VM
maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep Shared
SharedWithHost      135981224 43987764  91993460  33% /media/sf_SharedWithHost
 

You can access the directory as root, or grant vboxsf group to the user that should get to the directory. I will use root sudo to manipulate within it

 
  • Unpack software first on the first node oel72rac1n1
# grid infrastructure software
root@oel7rac1n1.dba24.pl ~ $ cd /u01/app/grid/grid-software/
root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46096-01_1of2.zip
root@oel7rac1n1.dba24.pl ~ $unzip /m edia/sf_SharedWithHost/Oracle12.1_install/V46096-01_2of2.zip
root@oel7rac1n1.dba24.pl ~ $chown grid:oinstall -R .
root@oel7rac1n1 grid-software ~ ls
grid

# oracle database software
root@oel7rac1n1.dba24.pl ~ $cd /u01/app/oracle/oracle-software/
root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46095-01_1of2.zip
root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46095-01_2of2.zip
root@oel7rac1n1.dba24.pl ~ $chown oracle:oinstall -R .
root@oel7rac1n1 oracle-software ~ ls
database
 
 
Configure SSH equivalency
 
  • Generate ssh keys on both nodes for oracle and grid user
# grid user

[grid@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
80:86:19:46:95:f8:b0:0b:2e:7d:9f:b2:dd:75:93:52 grid@oel7rac1n1
The key's randomart image is:
+--[ DSA 1024]----+
| .+o..           |
| .o+..           |
|  o+o .          |
|. ...  .         |
|.o .    S   E    |
|..o .      . .   |
|.  . . .  o +    |
|    ..o. . o .   |
|    .o. .        |
+-----------------+

[grid@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa): 
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
27:3b:bf:81:cb:c8:c0:af:af:19:50:aa:2c:8d:7b:99 grid@oel7rac1n2
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|    .            |
|   o             |
|  o     S .      |
|.+ o     =       |
|+..o+   + .      |
|..E  * o + .     |
|..  ++= o o.     |
+-----------------+

# oracle user
[oracle@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
34:08:9b:d3:1c:42:6e:ee:c6:b3:2b:27:0e:76:31:90 oracle@oel7rac1n1
The key's randomart image is:
+--[ DSA 1024]----+
|   .+ .          |
|  .. B o         |
| E  * + o        |
|  .o . . .       |
|   o.   S        |
|   oo            |
|.. .=            |
|..+..o           |
| ..+o.           |
+-----------------+

[oracle@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa): 
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
be:c2:28:ab:c9:c3:98:fd:60:85:60:92:4c:83:1e:09 oracle@oel7rac1n2
The key's randomart image is:
+--[ DSA 1024]----+
|Eo.              |
|++.              |
|=o.              |
|oo .             |
|  . .   S        |
|   .   .         |
|ooo  o  .        |
|+=o.. o  .       |
|ooo+.  ..        |
+-----------------+

 
  • Distribute public key among nodes
# grid user
[grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_key
[grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

# grid - Now exchange public keys - you need to authorize with password this time
[grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authorized_keys'
[grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authorized_keys'

# grid - check authorized_keys on both nodes
[grid@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKSXX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7AAAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/DQMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9fnMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2QTw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXVGsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1
ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIXJ9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9LVxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAAFQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOPed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+QAAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLazdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQnfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2

[grid@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIXJ9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9LVxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAAFQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOPed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+QAAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLazdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQnfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2
ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKSXX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7AAAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/DQMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9fnMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2QTw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXVGsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1


# oracle user
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

# oracle - Now exchange public keys - you need to authorize with password this time
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authorized_keys'
[oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authorized_keys'

# oracle check authorized_keys on both nodes
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hzKZAnZpCIZ3LKInPUX5chRACzEFij3IOQ0jI8zCN6FtYq0+1uvCe5+zrnBHaMiHKALhxAAAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblqG+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTWaiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4vN4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n1
ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcgKHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHvO+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbdNJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/OfpHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/qPIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbKuBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubvKvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2

[oracle@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcgKHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHvO+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbdNJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/OfpHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/qPIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbKuBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubvKvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2
ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hzKZAnZpCIZ3LKInPUX5chRACzEFij3IOQ0jI8zCN6FtYq0+1uvCe5+zrnBHaMiHKALhxAAAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblqG+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTWaiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4vN4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n1
 

Remember that you should have set passwords for oracle and grid on all the nodes

 
  • Test bidirectional ssh connectivity without passwords both from oracle and grid user.
    Establish connections to short and long server names (fqdn) in the public network (192.168.1.x)
 
 
 
Cluster verification before GI installation – CLUVFY
 

I strongly recommend using cluvfy tool to check if the cluster is ready before running the installation process. I does similar checks as OUI, but you won’t be stuck in the middle of the process if something is wrong. Cluster verification utility provides a lot of options to check your environment before you run clusterware, database installations or upgrades. The tool is evolving so expect more to come in later versions

 
  • System requirements verification: cluvfy comp sys

Use the cluvfy comp sys component verification command to check that the minimum system requirements are met for the specified product on all the specified nodes.

[grid@oel7rac1n1 bin]$ ./cluvfy comp sys -n oel7rac1n1,oel7rac1n2 -p crs -r 12.1

Verifying system requirement
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oel7rac1n2:/sbin,oel7rac1n2:/tmp"
Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oel7rac1n1:/sbin,oel7rac1n1:/tmp"
Check for multiple users with UID value 2002 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"
Check for multiple users with UID value 0 passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed
Time zone consistency check passed
Verification of system requirement was successful. 
 

Verification successful.

 
  • Verify storage using: cluvfy comp ssa

Use the cluvfy comp ssa component verification command to discover and check the sharing of the specified storage locations. CVU checks sharing for nodes in the node list.

# Check disks for OCR

[grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-asm-crs-03 -t ocr_vdisk
Verifying shared storage accessibility 

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-crs-01          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-crs-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op3-rac1-asm-crs-03          oel7rac1n1 oel7rac1n2   


Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful. 


## Check disks for DATA

[grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t data

Verifying shared storage accessibility 

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-01         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-02         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-03         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-04         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-01          oel7rac1n1 oel7rac1n2   
  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-03          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-04          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-01         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-02         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-03         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-04         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-01          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-03          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-04          oel7rac1n1 oel7rac1n2   


Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful.
 

Shared storage verified successfully.

 
  • Network verification using: cluvfy stage -post hwos

Use the cluvfy stage -post hwos stage verification command to perform network and storage verifications on the specified nodes in the cluster before installing Oracle software. This command also checks for supported storage types and checks each one for sharing.

As we have already checked the disks we are just interested in the network checks, ignore storage information – storage checks will faile

grid@oel7rac1n1 bin]$ ./cluvfy stage -post hwos -n oel7rac1n1,oel7rac1n2 

Performing post-checks for hardware and operating system setup 

Checking node reachability...
Node reachability check passed from node "oel7rac1n1"


Checking user equivalence...
User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.10.0"

Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.20.0"

Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.21.0"

Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.30.0"

Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.31.0"


Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
oel7rac1n1 enp0s3:192.168.1.21
oel7rac1n2 enp0s3:192.168.1.23

Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s8:192.168.10.21
oel7rac1n2 enp0s8:192.168.10.23

Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s9:192.168.20.21
oel7rac1n2 enp0s9:192.168.20.23
Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s10:192.168.21.21
oel7rac1n2 enp0s10:192.168.21.23

Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s16:192.168.30.21
oel7rac1n2 enp0s16:192.168.30.23

Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s17:192.168.31.21
oel7rac1n2 enp0s17:192.168.31.23
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed for subnet "192.168.20.0".
Subnet mask consistency check passed for subnet "192.168.21.0".
Subnet mask consistency check passed for subnet "192.168.30.0".
Subnet mask consistency check passed for subnet "192.168.31.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Check for multiple users with UID value 0 passed 
Time zone consistency check passed

Checking shared storage accessibility...

No shared storage found


Shared storage check failed on nodes "oel7rac1n1,oel7rac1n2"

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Post-check for hardware and operating system setup was unsuccessful on all the nodes. 

 

Verification successful (ingore storage failed checks)

 
  • Verify storage using: cluvfy comp ssa

Use the cluvfy comp ssa component verification command to discover and check the sharing of the specified storage locations. CVU checks sharing for nodes in the node list.

# Check disks for OCR

[grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-asm-crs-03 -t ocr_vdisk
Verifying shared storage accessibility 

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-crs-01          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-crs-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op3-rac1-asm-crs-03          oel7rac1n1 oel7rac1n2   


Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful. 


## Check disks for DATA

[grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t data

Verifying shared storage accessibility 

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-01         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-02         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-03         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-data-04         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-01          oel7rac1n1 oel7rac1n2   
  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-03          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op1-rac1-asm-fra-04          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-01         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-02         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-03         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-data-04         oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-01          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-02          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-03          oel7rac1n1 oel7rac1n2   

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/asm/op2-rac1-asm-fra-04          oel7rac1n1 oel7rac1n2   


Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful.
 

Shared storage verified successfully.

 
  • Nodes connectivity verification: cluvfy comp nodecon

Use the cluvfy comp nodecon component verification command to check the connectivity among the nodes specified in the node list. If you provide an interface list, then CVU checks the connectivity using only the specified interfaces.

[grid@oel7rac1n1 bin]$ ./cluvfy comp nodecon -n oel7rac1n1,oel7rac1n2

Verifying node connectivity

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.10.0"

Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.20.0"

Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.21.0"

Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.30.0"

Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.31.0"


Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
oel7rac1n1 enp0s3:192.168.1.21
oel7rac1n2 enp0s3:192.168.1.23

Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s8:192.168.10.21
oel7rac1n2 enp0s8:192.168.10.23

Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s9:192.168.20.21
oel7rac1n2 enp0s9:192.168.20.23

Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s10:192.168.21.21
oel7rac1n2 enp0s10:192.168.21.23

Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s16:192.168.30.21
oel7rac1n2 enp0s16:192.168.30.23

Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s17:192.168.31.21
oel7rac1n2 enp0s17:192.168.31.23
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed for subnet "192.168.20.0".
Subnet mask consistency check passed for subnet "192.168.21.0".
Subnet mask consistency check passed for subnet "192.168.30.0".
Subnet mask consistency check passed for subnet "192.168.31.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Verification of node connectivity was successful.

 

Node connectivity verification successful

 
  • check if servers are ready for GI installation – final check: cluvfy stage [-pre | -post] crsinst

Use the cluvfy stage -pre crsinst command with either the -file, -n, -flex, or -upgrade parameters to check the specified nodes before installing or upgrading Oracle Clusterware.
Use the cluvfy stage -post crsinst command to check the specified nodes after installing Oracle Clusterware.

From what you will see, the last check incorporates all the previous checks, but it’s good to run them all just to make sure we have and understand all the components

[grid@oel7rac1n1 ~]$ cd /u01/app/grid/cluvfy/bin/
[grid@oel7rac1n1 bin]$ ./cluvfy stage -pre crsinst -n oel7rac1n1,oel7rac1n2

Performing pre-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "oel7rac1n1"


Checking user equivalence...
User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.10.0"

Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.20.0"

Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.21.0"

Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.30.0"

Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.31.0"


Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
oel7rac1n1 enp0s3:192.168.1.21
oel7rac1n2 enp0s3:192.168.1.23

Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s8:192.168.10.21
oel7rac1n2 enp0s8:192.168.10.23

Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s9:192.168.20.21
oel7rac1n2 enp0s9:192.168.20.23

Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s10:192.168.21.21
oel7rac1n2 enp0s10:192.168.21.23

Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s16:192.168.30.21
oel7rac1n2 enp0s16:192.168.30.23
Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are:
oel7rac1n1 enp0s17:192.168.31.21
oel7rac1n2 enp0s17:192.168.31.23
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed for subnet "192.168.20.0".
Subnet mask consistency check passed for subnet "192.168.21.0".
Subnet mask consistency check passed for subnet "192.168.30.0".
Subnet mask consistency check passed for subnet "192.168.31.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oel7rac1n2:/sbin,oel7rac1n2:/tmp"
Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oel7rac1n1:/sbin,oel7rac1n1:/tmp"
Check for multiple users with UID value 2002 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Kernel parameter check passed for "panic_on_oops"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
Port availability check passed for ports "6200,6100"

Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)"
Port availability check passed for ports "42424"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...
No NTP Daemons or Services were found to be running

Clock synchronization check using Network Time Protocol(NTP) passed

Core file name pattern consistency check passed.

User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
The DNS response time for an unreachable node is within acceptable limit on all nodes
checking DNS response from all servers in "/etc/resolv.conf"

Check for integrity of file "/etc/resolv.conf" passed

Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was successful. 

 

OK!! we are ready to install the cluster software

 
 
 
OUI Grid Installation
 
  • Run OUI installer
[grid@oel7rac1n1 ~]$ cd /u01/app/grid/grid-software/grid/
[grid@oel7rac1n1 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 21611 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 16895 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-12-02_05-18-35PM. Please wait ...

 
  • Choose to install and configure GI for a cluster

1-step-1-screenshot-from-2016-12-02-17-20-21

  • Choose configure standard cluster

2-step-2-screenshot-from-2016-12-02-17-20-41

  • Choose advance installation

3-step-3-screenshot-from-2016-12-02-17-21-02

  • Choose preferred languages to install

4-step-4-screenshot-from-2016-12-02-17-21-08

  • Set cluster name, SCAN name and port. Disable GNS

5-step-5-screenshot-from-2016-12-02-17-22-54

  • Provide list of nodes to be used as cluster nodes.

6-step-6a-screenshot-from-2016-12-02-17-30-44

  • Verify ssh user equivalency

7-step-6b-ssh-connectivity-screenshot-from-2016-12-02-17-30-58

  • Declare cluster networks

8-step-7a-screenshot-from-2016-12-02-17-32-37

  • Declare cluster networks

9-step-7b-screenshot-from-2016-12-02-17-32-41

  • Use Flex ASM for storage

10-step-8-screenshot-from-2016-12-02-17-33-20

  • Set disk string in order to get our prepared disks to the list

11-step-9a-change-discoverypathscreenshot-from-2016-12-02-17-34-20

  • Create crs diskgroup using /dev/asm/*asm*crs* disks

Ooops! As you can see OUI can’t take care of the quorum group, so it complains that the crs-03 disk presented from openfiler3 is too small and with different size than crs-01 and crs-02.
The same problem also exists in 11g R2.

 

Some interesting notes:
http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
http://www.redstk.com/voting-disk-setup-in-a-12c-grid-infrastructure-extended-rac-environment/

 

12-step-9b-screenshot-from-2016-12-02-17-44-10

  • We need to create temporary disk on openfiler3 with size of 10GB as crs-02 and crs-03. We need to present it to cluster nodes and prepare on those VMs.

    Create new volume (click to enlarge)

13-openfiler3-additional-lun-temp-screenshot-from-2016-12-02-18-31-11

  • Check volumes now (click to enlarge)

14-openfiler3-view-of-additional-3rd-temp-10gb-lun-for-crs-screenshot-from-2016-12-02-18-31-24

  • Map new lun to the existing target on openfiler3 (click to enlarge)

15-openfiler3-lun-mapped-to-iscsi-target-screenshot-from-2016-12-02-18-31-37

  • Check the new lun on openfiler3 system level
[root@openfiler3 ~]# cat /proc/net/iet/volume
tid:1 name:iqn.2006-01.com.openfiler:op3.rac1.asm
        lun:0 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-crs-03
        lun:1 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-tmp
 
  • Discover disk on the cluster nodes
[root@oel7rac1n1 ~]# rescan-scsi-bus.sh 
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
      Vendor: VBOX     Model: CD-ROM           Rev: 1.0
      Type:   CD-ROM                           ANSI SCSI revision: 05
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: ATA      Model: VBOX HARDDISK    Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 1 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 2 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 02
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 3 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 03
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 4 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 04
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 5 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 05
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 6 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 06
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
     Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 7 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 07
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 8 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 08
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 9 0 0 0 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 1 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 2 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 02
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 3 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 03
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 4 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 04
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 5 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 05
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 6 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 06
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 7 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 07
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 8 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 08
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
Scanning host 11 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 11 0 0 0 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 11 0 0 1 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
1 new or changed device(s) found.
        [11:0:0:1]
0 remapped or resized device(s) found.
0 device(s) removed.

[root@oel7rac1n2 ~]# rescan-scsi-bus.sh 
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
      Vendor: VBOX     Model: CD-ROM           Rev: 1.0
      Type:   CD-ROM                           ANSI SCSI revision: 05
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: ATA      Model: VBOX HARDDISK    Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 1 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 2 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 02
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 3 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 03
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 4 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 04
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 5 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 05
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 6 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 06
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 7 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 07
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 7 0 0 8 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 08
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 9 0 0 0 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 1 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 2 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 02
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 3 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 03
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 4 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 04
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 5 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 05
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 6 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 06
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 7 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 07
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 9 0 0 8 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 08
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
Scanning host 11 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 11 0 0 0 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 00
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
 Scanning for device 11 0 0 1 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
      Vendor: OPNFILER Model: VIRTUAL-DISK     Rev: 0
      Type:   Direct-Access                    ANSI SCSI revision: 04
1 new or changed device(s) found.
        [11:0:0:1]
0 remapped or resized device(s) found.
0 device(s) removed.

 

New device found [11:0:0:1] on both nodes, this is our temporary LUN.

 
  • Find device name assigned to the new disk
[root@oel7rac1n1 ~]# lsscsi   
[1:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0
[3:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda
[7:0:0:0]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdb
[7:0:0:1]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdc
[7:0:0:2]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdd
[7:0:0:3]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sde
[7:0:0:4]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdf
[7:0:0:5]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdg
[7:0:0:6]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdh
[7:0:0:7]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdi
[7:0:0:8]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdj
[9:0:0:0]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdk
[9:0:0:1]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdl
[9:0:0:2]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdm
[9:0:0:3]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdn
[9:0:0:4]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdo
[9:0:0:5]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdp
[9:0:0:6]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdq
[9:0:0:7]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdr
[9:0:0:8]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sds
[11:0:0:0]   disk    OPNFILER VIRTUAL-DISK     0     /dev/sdt
[11:0:0:1]   disk    OPNFILER VIRTUAL-DISK     0     /dev/sdu

[root@oel7rac1n2 ~]# lsscsi 
[1:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0
[3:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda
[7:0:0:0]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdb
[7:0:0:1]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdc
[7:0:0:2]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdd
[7:0:0:3]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sde
[7:0:0:4]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdf
[7:0:0:5]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdg
[7:0:0:6]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdh
[7:0:0:7]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdi
[7:0:0:8]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdj
[9:0:0:0]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdk
[9:0:0:1]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdl
[9:0:0:2]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdm
[9:0:0:3]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdo
[9:0:0:4]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdp
[9:0:0:5]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdq
[9:0:0:6]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdr
[9:0:0:7]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sds
[9:0:0:8]    disk    OPNFILER VIRTUAL-DISK     0     /dev/sdt
[11:0:0:0]   disk    OPNFILER VIRTUAL-DISK     0     /dev/sdn
[11:0:0:1]   disk    OPNFILER VIRTUAL-DISK     0     /dev/sdu
 

/dev/sdu – new disk device name on both nodes

 
 
  • Create partition on /dev/sdu on the first node
[root@oel7rac1n1 ~]# parted /dev/sdu mklabel gpt mkpart primary "1 -1"
Information: You may need to update /etc/fstab.

[root@oel7rac1n1 ~]# fdisk -l /dev/sdu
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sdu: 10.5 GB, 10502537216 bytes, 20512768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048     20510719    9,8G  Microsoft basic primary
 
  • Discover new partition on the second node
 
[root@oel7rac1n2 ~]# kpartx -a /dev/sdu
 
 
  • Find scsi ID
[root@oel7rac1n1 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu
14f504e46494c45524b6878344b792d4869724a2d4c414875

[root@oel7rac1n2 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu
14f504e46494c45524b6878344b792d4869724a2d4c414875

 
  • Add the following line to /etc/udev/rules.d/90-oracle-asmdevices.rules on both nodes.
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45524b6878344b792d4869724a2d4c414875", SYMLINK+="asm/op3-rac1-asm-tmp", OWNER="grid", GROUP="asmadmin", MODE="0660"
 
  • Reload udev and check asm disk (owner,group,privs)
[root@oel7rac1n1 ~]#udevadm control --reload-rules
[root@oel7rac1n1 ~]#udevadm trigger

[root@oel7rac1n2 ~]#udevadm control --reload-rules
[root@oel7rac1n2 ~]#udevadm trigger

[root@oel7rac1n1 ~]# ls -l /dev/asm/*|grep tmp
lrwxrwxrwx 1 root root 7 12-02 18:47 /dev/asm/op3-rac1-asm-tmp ->; ../sdu1
[root@oel7rac1n1 ~]# ls -l /dev/sdu1
brw-rw---- 1 grid asmadmin 65, 65 12-02 18:47 /dev/sdu1

[root@oel7rac1n2 ~]# ls -l /dev/asm/*|grep tmp
lrwxrwxrwx 1 root root 7 12-02 18:46 /dev/asm/op3-rac1-asm-tmp -> ../sdu1
[root@oel7rac1n2 ~]# ls -l /dev/sdu1 
brw-rw---- 1 grid asmadmin 65, 65 12-02 18:46 /dev/sdu1

 
  • Create CRS diskgroup using temporary disk from openfiler3

16-step-9-again-now-adding-newly-added-temp-lun-screenshot-from-2016-12-02-18-56-07

  • Set passwords for ASM

17-step-10-passwords-screenshot-from-2016-12-02-18-56-39

  • Don’t configure IPMI

18-step-11-screenshot-from-2016-12-02-18-56-52

  • Do not register with EM Cloud Control unless you have it already

19-step-12-screenshot-from-2016-12-02-18-56-58

  • Set asm groups – leave default

20-step-13-screenshot-from-2016-12-02-18-57-05

  • Set ORACLE_BASE and ORACLE_HOME directories

21-step-14-screenshot-from-2016-12-02-18-57-38

  • Set oraInventory directory

22-step-15-screenshot-from-2016-12-02-18-57-53

  • We wan to run root scripts manually

23-step-16-screenshot-from-2016-12-02-18-58-57

 
  • Prerequisits being checked

24-step-17a-screenshot-from-2016-12-02-18-59-04

  • Prerequisits being checked

25-step-17b-screenshot-from-2016-12-02-18-59-25

  • It complains about /dev/shm size

26-step-17c-screenshot-from-2016-12-02-19-01-50

  • As you can see it looks like OUI bug as size is ok

27-step-17d-screenshot-from-2016-12-02-19-01-40

  • Confirm you want to ignore this warning

28-step-17e-screenshot-from-2016-12-02-19-02-06

  • Installation plan summary screen

29-step-18-screenshot-from-2016-12-02-19-02-25

  • Installation process

30-step-19a-screenshot-from-2016-12-02-19-03-56

  • Run scripts as root

31-step-19b-rootscripts-screenshot-from-2016-12-02-19-25-28

 
# root scripts
[root@oel7rac1n1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.


[root@oel7rac1n2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@oel7rac1n1 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2016/12/02 19:27:19 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2016/12/02 19:27:50 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2016/12/02 19:27:52 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2016/12/02 19:28:32 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded

ASM created and started successfully.

Disk Group CRS created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 1c193c21750c4fe3bfa28c6b5df261b3.
Successful addition of voting disk 54cd207d0d744fd8bf2714d7776f5ad2.
Successful addition of voting disk 355efd32461e4f3fbf50305bfac1657e.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: oel7rac1n1
CRS-6016: Resource auto-start has completed for server oel7rac1n1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 19:34:26 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'oel7rac1n1'
CRS-2676: Start of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cvu' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ons' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.net1.network' on 'oel7rac1n1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7rac1n1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CRS.dg' is already running on 'oel7rac1n1'
CRS-6017: Processing resource auto-start for servers: oel7rac1n1
CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.oc4j' on 'oel7rac1n1'
CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cvu' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.scan2.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.scan3.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ons' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'oel7rac1n1' succeeded
CRS-6016: Resource auto-start has completed for server oel7rac1n1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 19:38:37 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

# node2 root.sh
[root@oel7rac1n2 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2016/12/02 20:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2016/12/02 20:18:12 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2016/12/02 20:18:13 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2016/12/02 20:19:43 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n2'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n2'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n2'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n2' succeeded
CRS-6017: Processing resource auto-start for servers: oel7rac1n2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n2'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n2'
CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2'
CRS-2676: Start of 'ora.ons' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7rac1n2' succeeded
CRS-6016: Resource auto-start has completed for server oel7rac1n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 20:23:53 CLSRSC-343: Successfully started Oracle Clusterware stack

2016/12/02 20:24:22 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

  • Installation in progress

32-step-19c-screenshot-from-2016-12-02-20-28-34

  • Installation finished successfully.

33-step-20-screenshot-from-2016-12-02-21-22-45

 
 
Postinstallation checks
 
 
  • Verify Grid Infrastructure after the installation: cluvfy stage -post crsinst
 
[grid@oel7rac1n1 bin]$ ./cluvfy stage -post crsinst -n oel7rac1n1,oel7rac1n2 -method root
Enter "ROOT" password:

Performing post-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "oel7rac1n1"


Checking user equivalence...
User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n2,oel7rac1n1
TCP connectivity check passed for subnet "192.168.1.0"


Check: Node connectivity using interfaces on subnet "192.168.31.0"
Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.31.0"


Check: Node connectivity using interfaces on subnet "192.168.20.0"
Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n2,oel7rac1n1
TCP connectivity check passed for subnet "192.168.20.0"


Check: Node connectivity using interfaces on subnet "192.168.21.0"
Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n2,oel7rac1n1
TCP connectivity check passed for subnet "192.168.21.0"


Check: Node connectivity using interfaces on subnet "192.168.30.0"
Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n2,oel7rac1n1
TCP connectivity check passed for subnet "192.168.30.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "192.168.20.0".
Subnet mask consistency check passed for subnet "192.168.21.0".
Subnet mask consistency check passed for subnet "192.168.30.0".
Subnet mask consistency check passed for subnet "192.168.31.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.20.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.20.0" for multicast communication with multicast group "224.0.0.251" passed.

Checking subnet "192.168.21.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.21.0" for multicast communication with multicast group "224.0.0.251" passed.

Checking subnet "192.168.30.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.30.0" for multicast communication with multicast group "224.0.0.251" passed.

Checking subnet "192.168.31.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.31.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking whether the ASM filter driver is active and consistent on all nodes
ASM filter driver library is not installed on any of the cluster nodes.
ASM filter driver configuration was found consistent across all the cluster nodes.
Time zone consistency check passed

Checking Cluster manager integrity...


Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


UDev attributes check for OCR locations started...
UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
UDev attributes check passed for Voting Disk locations

Default user file creation mask check passed

Checking cluster integrity...


Cluster integrity check passed
hecking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


Checking daemon liveness...
Liveness check passed for "CRS daemon"

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+CRS/rac1/OCRFILE/registry.255.929561455" is available on all the nodes

Uniqueness check for OCR device passed

Checking the version of OCR...
OCR of correct Version "4" exists

Checking data integrity of OCR...
Data integrity check for OCR passed

OCR integrity check passed

Checking CRS integrity...

Clusterware version consistency passed.

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of ONS node application (optional)
ONS node application check passed


Checking Single Client Access Name (SCAN)...

Checking TCP connectivity to SCAN listeners...
TCP connectivity to SCAN listeners exists on all cluster nodes

Checking name resolution setup for "rac1-scan.dba24.pl"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking SCAN IP addresses...
Check of SCAN IP addresses passed

Verification of SCAN VIP and listener setup passed

Checking OLR integrity...
Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed

WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Task ASM Integrity check started...

Checking if connectivity exists across cluster nodes on the ASM network

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.30.0"
Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.30.0"


Check: Node connectivity using interfaces on subnet "192.168.31.0"
Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2
TCP connectivity check passed for subnet "192.168.31.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.30.0".
Subnet mask consistency check passed for subnet "192.168.31.0".
Subnet mask consistency check passed.

Node connectivity check passed

Network connectivity check across cluster nodes on the ASM network passed
Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all specified nodes
Disk Group Check passed. At least one Disk Group configured

Task ASM Integrity check passed...

User "grid" is not part of "root" group. Check passed
Oracle Clusterware is installed on all nodes.
CTSS resource check passed
Query of CTSS for time offset passed

CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Post-check for cluster services setup was successful.
 

All checks passed, seems that GI has been installed properly!!

 
 
Modify CRS, add DATA, FRA diskgroups
 
  • Run asmca as grid

34-asmca-modify-crs-a-screenshot-from-2016-12-02-21-32-51

  • Try to add op3-rac1-asm-crs-03 disk as Quorum failgroup

35-asmca-modify-crs-b-screenshot-from-2016-12-02-21-32-51

  • It complains again. Wow! Bug or a feature?

36-asmca-modify-crs-c-screenshot-from-2016-12-02-21-32-51

 
  • Replace temporary disk with crs-03 using sqlplus
# first let's see votes
[grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
 

Add op3-rac1-asm-crs-03 to the CRS diskgroup as quorum disk

 
[grid@oel7rac1n1 bin]$ ./sqlplus / as sysasm

SQL> alter diskgroup crs add QUORUM FAILGROUP fg3 DISK '/dev/asm/op3-rac1-asm-crs-03';

Diskgroup altered.
 

And what happened? nothing, vote stil on tmp

 
[grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
 

Remove temporary disk

 
SQL> alter diskgroup crs drop disk CRS_0002;
Diskgroup altered.
 

Result?

 
[grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   0588f4e97cbb4fb8bfb28fd5d18a46d0 (/dev/asm/op3-rac1-asm-crs-03) [CRS]
Located 3 voting disk(s).
 

This is what we want!

 
  • Create DATA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup op2 to disks that come from openfiler2. Set AU_SIZE to 4GB

37-data_wlasciwy_screenshot-from-2016-12-05-14-37-39

  • Create FRA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup op2 to disks that come from openfiler2. Set AU_SIZE to 1GB

38-fra_createscreenshot-from-2016-12-05-14-39-48

  • Diskgroups successfully created.

39-diskgroups-created-screenshot-from-2016-12-05-14-40-53

 

Grid Infrastructure installed successfully!

 
 
 
Alternate approach to ASM diskgroups manipulation using ascmca -silent, chdg, mkdg
 

I used asmca or sqlplus to create and modify the diskgroup, but there are some other ways. Please find examples below.

  • ASMCA – silent

As you can read in the doc https://docs.oracle.com/database/121/OSTMG/GUID-877EB0F8-E9CA-4C97-965C-AACBB256B12D.htm#OSTMG94309

asmca silent allows only this operations:
-convertToFlexASM
-configureASM
-upgradeASM
-configureParameter
-deleteASM
-createDiskGroup
-addDisk
-createVolume
-createACFS

Let’s add the crs-03 quorum disk to the crs as we did with sqlplus

[grid@oel7rac1n1 bin]$ ./asmca -silent -addDisk -diskGroupName crs -diskList '/dev/asm/op3-rac1-asm-crs-03' -failuregroup fg3 -quorum

Disks added successfully to disk group crs

[grid@oel7rac1n1 bin]$ ./asmcmd -p
ASMCMD [+] > lsdsk -k
Total_MB  Free_MB  OS_MB  Name      Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
   10014     6965  10014  CRS_0001  CRS_0001   REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-crs-01
   10014     6965  10014  CRS_0000  CRS_0000   REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-crs-02
     510      507    510  CRS_0003  FG3        QUORUM          System                         UNKNOWN  /dev/asm/op3-rac1-asm-crs-03
   10014     6966  10014  CRS_0002  CRS_0002   REGULAR         System                         UNKNOWN  /dev/asm/op3-rac1-asm-tmp
ASMCMD [+] > exit

# check how votes look like
[grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   a25677e1db034f6cbf5b2f88daf9d4c7 (/dev/asm/op3-rac1-asm-tmp) [CRS]

As we cannot use asmca silent mode to drop disk from diskgroup let’s use chdg command in ASMCMD. First prepare xml file

# so lets use chdg to drop /dev/asm/op3-rac1-asm-tmp
# create file /home/grid/change_crs.xml
<chdg name="crs" power="3">
  <drop>
    <dsk name="CRS_0002"></dsk>
  </drop>
</chdg>

Run chdg

ASMCMD [+] > chdg /home/grid/change_crs.xml
Diskgroup altered.

ASMCMD [+] > lsdsk -k
Total_MB  Free_MB  OS_MB  Name      Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
   10014     5530  10014  CRS_0001  CRS_0001   REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-crs-01
   10014     5530  10014  CRS_0000  CRS_0000   REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-crs-02
     510      475    510  CRS_0003  FG3        QUORUM          System                         UNKNOWN  /dev/asm/op3-rac1-asm-crs-03

[grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
 2. ONLINE   54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
 3. ONLINE   3d51b07f8a6e4f2abf8f7b4a506f67b6 (/dev/asm/op3-rac1-asm-crs-03) [CRS]
Located 3 voting disk(s).

That’s it!!

 
  • Use ASMCA silent mode to create DATA and FRA
[grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName DATA -diskList /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04 -failuregroup op1 -diskList /dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatible.asm 12.1 -compatible.rdbms 11.2

Disk Group DATA created successfully.

[grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName FRA -diskList /dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04 -failuregroup op1 -diskList /dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatible.asm 12.1 -compatible.rdbms 11.2

Disk Group FRA created successfully.

Check new dgs

ASMCMD [+] > lsdsk -k -G DATA
Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    2014     1987   2014  DATA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-01
    2014     1990   2014  DATA_0001  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-02
    2014     1986   2014  DATA_0002  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-03
    2014     1988   2014  DATA_0003  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-04
    2014     1991   2014  DATA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-01
    2014     1984   2014  DATA_0005  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-02
    2014     1988   2014  DATA_0006  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-03
    2014     1988   2014  DATA_0007  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-04
ASMCMD [+] > lsdsk -k -G FRA
Total_MB  Free_MB  OS_MB  Name      Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    1022      997   1022  FRA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-01
    1022      994   1022  FRA_0001  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-02
    1022      995   1022  FRA_0002  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-03
    1022      997   1022  FRA_0003  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-04
    1022      996   1022  FRA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-01
    1022      998   1022  FRA_0005  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-02
    1022      994   1022  FRA_0006  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-03
    1022      995   1022  FRA_0007  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-04
ASMCMD [+] > lsattr -l -G DATA
Name                     Value
access_control.enabled   FALSE
access_control.umask     066
au_size                  1048576
cell.smart_scan_capable  FALSE
compatible.asm           12.1.0.0.0
compatible.rdbms         11.2.0.0.0
content.check            FALSE
content.type             data
disk_repair_time         3.6h
failgroup_repair_time    24.0h
idp.boundary             auto
idp.type                 dynamic
phys_meta_replicated     true
sector_size              512
thin_provisioned         FALSE

ASMCMD [+] > lsattr -l -G FRA
Name                     Value
access_control.enabled   FALSE
access_control.umask     066
au_size                  1048576
cell.smart_scan_capable  FALSE
compatible.asm           12.1.0.0.0
compatible.rdbms         11.2.0.0.0
content.check            FALSE
content.type             data
disk_repair_time         3.6h
failgroup_repair_time    24.0h
idp.boundary             auto
idp.type                 dynamic
phys_meta_replicated     true
sector_size              512
thin_provisioned         FALSE

 
  • Use sqlplus to create DATA and FRA
SQL> create diskgroup DATA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op1-rac1-asm-data-01','/dev/asm/op1-rac1-asm-data-02','/dev/asm/op1-rac1-asm-data-03','/dev/asm/op1-rac1-asm-data-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-data-01','/dev/asm/op2-rac1-asm-data-02','/dev/asm/op2-rac1-asm-data-03','/dev/asm/op2-rac1-asm-data-04' ATTRIBUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M';

Diskgroup created.

SQL> create diskgroup FRA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op1-rac1-asm-fra-01','/dev/asm/op1-rac1-asm-fra-02','/dev/asm/op1-rac1-asm-fra-03','/dev/asm/op1-rac1-asm-fra-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-fra-01','/dev/asm/op2-rac1-asm-fra-02','/dev/asm/op2-rac1-asm-fra-03','/dev/asm/op2-rac1-asm-fra-04' ATTRIBUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M';

Diskgroup created.

Check results

SMCMD [+] > lsdsk -k -G DATA
Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    2014     1984   2014  DATA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-01
    2014     1989   2014  DATA_0001  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-02
    2014     1989   2014  DATA_0002  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-03
    2014     1989   2014  DATA_0003  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-04
    2014     1989   2014  DATA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-01
    2014     1988   2014  DATA_0005  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-02
    2014     1986   2014  DATA_0006  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-03
    2014     1988   2014  DATA_0007  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-04
ASMCMD [+] > lsdsk -k -G FRA
Total_MB  Free_MB  OS_MB  Name      Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    1022      997   1022  FRA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-01
    1022      995   1022  FRA_0001  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-02
    1022      998   1022  FRA_0002  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-03
    1022      993   1022  FRA_0003  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-fra-04
    1022      996   1022  FRA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-01
    1022      998   1022  FRA_0005  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-02
    1022      995   1022  FRA_0006  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-03
    1022      994   1022  FRA_0007  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-fra-04
 
  • Another way to do it is to build the diskgroups with mkdg command

Create file /home/grid/mkdg_data.xml

# /home/grid/mkdg_data.xml

<dg name="data" redundancy="normal">
  <fg name="op1">
    <dsk string="/dev/asm/op1-rac1-asm-data-01"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-data-02"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-data-03"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-data-04"></dsk>
  </fg>
  <fg name="op2">
    <dsk string="/dev/asm/op2-rac1-asm-data-01"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-data-02"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-data-03"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-data-04"></dsk>
  </fg>
  <a name="compatible.asm" value="12.1"></a>
  <a name="compatible.rdbms" value="11.2"></a>
  <a name="au_size" value="4M"></a>
</dg>

Create file /home/grid/mkdg_fra.xml

#/home/grid/mkdg_fra.xml
<dg name="fra" redundancy="normal">
  <fg name="op1">
    <dsk string="/dev/asm/op1-rac1-asm-fra-01"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-fra-02"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-fra-03"></dsk>
    <dsk string="/dev/asm/op1-rac1-asm-fra-04"></dsk>
  </fg>
  <fg name="op2">
    <dsk string="/dev/asm/op2-rac1-asm-fra-01"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-fra-02"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-fra-03"></dsk>
    <dsk string="/dev/asm/op2-rac1-asm-fra-04"></dsk>
  </fg>
  <a name="compatible.asm" value="12.1"></a>
  <a name="compatible.rdbms" value="11.2"></a>
  <a name="au_size" value="1M"></a>
</dg>

Create diskgroups using mkdg

ASMCMD [+] > mkdg /home/grid/mkdg_data.xml
ASMCMD [+] > mkdg /home/grid/mkdg_fra.xml

Check results

ASMCMD [+] > lsdsk -k -G DATA
Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    2012     1980   2014  DATA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-01
    2012     1968   2014  DATA_0001  DATA_0001  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-02
    2012     1984   2014  DATA_0002  DATA_0002  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-03
    2012     1976   2014  DATA_0003  DATA_0003  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-04
    2012     1964   2014  DATA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-01
    2012     1980   2014  DATA_0005  DATA_0005  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-02
    2012     1964   2014  DATA_0006  DATA_0006  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-03
    2012     1968   2014  DATA_0007  DATA_0007  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-04

ASMCMD [+] > lsdsk -k -G DATA
Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
    2012     1980   2014  DATA_0000  OP1        REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-01
    2012     1968   2014  DATA_0001  DATA_0001  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-02
    2012     1984   2014  DATA_0002  DATA_0002  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-03
    2012     1976   2014  DATA_0003  DATA_0003  REGULAR         System                         UNKNOWN  /dev/asm/op1-rac1-asm-data-04
    2012     1964   2014  DATA_0004  OP2        REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-01
    2012     1980   2014  DATA_0005  DATA_0005  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-02
    2012     1964   2014  DATA_0006  DATA_0006  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-03
    2012     1968   2014  DATA_0007  DATA_0007  REGULAR         System                         UNKNOWN  /dev/asm/op2-rac1-asm-data-04

Despite proper configuration in the xml files disks did not get their proper failgroup assignment. I will try out this approach later again – after patching with latest PSU.

 
 
 
 
What do you think?? Please post your comments :)
Source: My experience and WorlWideWeb

About the author

 
maciej tokar
Maciej Tokar

An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU

 
View Maciej Tokar's profile on LinkedIn         logoDB4
LinkedIn Auto Publish Powered By : XYZScripts.com