This is tenth part of my RAC lab series. I will show you step by step how to prepare for and how to install Oracle 12c Grid Infrastructure
RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database
Oracle 12c official documentation
Please refer to the official 12.1 documentation and CLUVFY documentation:
-
Grid Infrastructure Installation Guide
https://docs.oracle.com/database/121/CWLIN/toc.htm -
Clusterware Administration and Deployment Guide – Cluster Verification Utility Reference
https://docs.oracle.com/database/121/CWADD/cvu.htm#CWADD1100
Prepare installation files for grid and database software
I need shared directory between host and VMs to to keep oracle installation files there. From RAC nodes I can then grab those files and unzip them into /u01/app/grid/grid-software and /u01/app/oracle/oracle-software. VboxAddition software will help to establish thos shared dirs.
- Prepare directory for shared dir
maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mkdir -p /opt/vboxshared/Oracle12.1_install
- Move files downloaded from Oracle to that directory
maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mv /home/maciek/Downloads/V46*.zip /opt/vboxshared/Oracle12.1_install
- Set this folder as shared with the vms
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n1 --name SharedWithHost --hostpath /opt/vboxshared --automount vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n2 --name SharedWithHost --hostpath /opt/vboxshared --automount # check if directory has been mounted in the VM maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep Shared SharedWithHost 135981224 43987764 91993460 33% /media/sf_SharedWithHost
You can access the directory as root, or grant vboxsf group to the user that should get to the directory. I will use root sudo to manipulate within it
- Unpack software first on the first node oel72rac1n1
# grid infrastructure software root@oel7rac1n1.dba24.pl ~ $ cd /u01/app/grid/grid-software/ root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46096-01_1of2.zip root@oel7rac1n1.dba24.pl ~ $unzip /m edia/sf_SharedWithHost/Oracle12.1_install/V46096-01_2of2.zip root@oel7rac1n1.dba24.pl ~ $chown grid:oinstall -R . root@oel7rac1n1 grid-software ~ ls grid # oracle database software root@oel7rac1n1.dba24.pl ~ $cd /u01/app/oracle/oracle-software/ root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46095-01_1of2.zip root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46095-01_2of2.zip root@oel7rac1n1.dba24.pl ~ $chown oracle:oinstall -R . root@oel7rac1n1 oracle-software ~ ls database
Configure SSH equivalency
- Generate ssh keys on both nodes for oracle and grid user
# grid user [grid@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/grid/.ssh/id_dsa): Created directory '/home/grid/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_dsa. Your public key has been saved in /home/grid/.ssh/id_dsa.pub. The key fingerprint is: 80:86:19:46:95:f8:b0:0b:2e:7d:9f:b2:dd:75:93:52 grid@oel7rac1n1 The key's randomart image is: +--[ DSA 1024]----+ | .+o.. | | .o+.. | | o+o . | |. ... . | |.o . S E | |..o . . . | |. . . . o + | | ..o. . o . | | .o. . | +-----------------+ [grid@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/grid/.ssh/id_dsa): Created directory '/home/grid/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_dsa. Your public key has been saved in /home/grid/.ssh/id_dsa.pub. The key fingerprint is: 27:3b:bf:81:cb:c8:c0:af:af:19:50:aa:2c:8d:7b:99 grid@oel7rac1n2 The key's randomart image is: +--[ DSA 1024]----+ | | | | | . | | o | | o S . | |.+ o = | |+..o+ + . | |..E * o + . | |.. ++= o o. | +-----------------+ # oracle user [oracle@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): Created directory '/home/oracle/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: 34:08:9b:d3:1c:42:6e:ee:c6:b3:2b:27:0e:76:31:90 oracle@oel7rac1n1 The key's randomart image is: +--[ DSA 1024]----+ | .+ . | | .. B o | | E * + o | | .o . . . | | o. S | | oo | |.. .= | |..+..o | | ..+o. | +-----------------+ [oracle@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): Created directory '/home/oracle/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: be:c2:28:ab:c9:c3:98:fd:60:85:60:92:4c:83:1e:09 oracle@oel7rac1n2 The key's randomart image is: +--[ DSA 1024]----+ |Eo. | |++. | |=o. | |oo . | | . . S | | . . | |ooo o . | |+=o.. o . | |ooo+. .. | +-----------------+
- Distribute public key among nodes
# grid user [grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_key [grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys # grid - Now exchange public keys - you need to authorize with password this time [grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authorized_keys' [grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authorized_keys' # grid - check authorized_keys on both nodes [grid@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKSXX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7AAAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/DQMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9fnMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2QTw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXVGsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1 ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIXJ9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9LVxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAAFQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOPed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+QAAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLazdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQnfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2 [grid@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIXJ9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9LVxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAAFQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOPed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+QAAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLazdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQnfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2 ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKSXX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7AAAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/DQMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9fnMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2QTw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXVGsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1 # oracle user [oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys [oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys # oracle - Now exchange public keys - you need to authorize with password this time [oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authorized_keys' [oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authorized_keys' # oracle check authorized_keys on both nodes [oracle@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hzKZAnZpCIZ3LKInPUX5chRACzEFij3IOQ0jI8zCN6FtYq0+1uvCe5+zrnBHaMiHKALhxAAAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblqG+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTWaiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4vN4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n1 ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcgKHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHvO+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbdNJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/OfpHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/qPIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbKuBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubvKvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2 [oracle@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcgKHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHvO+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbdNJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/OfpHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/qPIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbKuBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubvKvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2 ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hzKZAnZpCIZ3LKInPUX5chRACzEFij3IOQ0jI8zCN6FtYq0+1uvCe5+zrnBHaMiHKALhxAAAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblqG+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTWaiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4vN4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n1
Remember that you should have set passwords for oracle and grid on all the nodes
-
Test bidirectional ssh connectivity without passwords both from oracle and grid user.
Establish connections to short and long server names (fqdn) in the public network (192.168.1.x)
Cluster verification before GI installation – CLUVFY
I strongly recommend using cluvfy tool to check if the cluster is ready before running the installation process. I does similar checks as OUI, but you won’t be stuck in the middle of the process if something is wrong. Cluster verification utility provides a lot of options to check your environment before you run clusterware, database installations or upgrades. The tool is evolving so expect more to come in later versions
- System requirements verification: cluvfy comp sys
Use the cluvfy comp sys component verification command to check that the minimum system requirements are met for the specified product on all the specified nodes.
[grid@oel7rac1n1 bin]$ ./cluvfy comp sys -n oel7rac1n1,oel7rac1n2 -p crs -r 12.1 Verifying system requirement Total memory check passed Available memory check passed Swap space check passed Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oel7rac1n2:/sbin,oel7rac1n2:/tmp" Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oel7rac1n1:/sbin,oel7rac1n1:/tmp" Check for multiple users with UID value 2002 passed User existence check passed for "grid" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "grid" in group "oinstall" [as Primary] passed Membership check for user "grid" in group "dba" passed Run level check passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" Kernel parameter check passed for "panic_on_oops" Package existence check passed for "binutils" Package existence check passed for "compat-libcap1" Package existence check passed for "compat-libstdc++-33(x86_64)" Package existence check passed for "libgcc(x86_64)" Package existence check passed for "libstdc++(x86_64)" Package existence check passed for "libstdc++-devel(x86_64)" Package existence check passed for "sysstat" Package existence check passed for "gcc" Package existence check passed for "gcc-c++" Package existence check passed for "ksh" Package existence check passed for "make" Package existence check passed for "glibc(x86_64)" Package existence check passed for "glibc-devel(x86_64)" Package existence check passed for "libaio(x86_64)" Package existence check passed for "libaio-devel(x86_64)" Package existence check passed for "nfs-utils" Check for multiple users with UID value 0 passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Time zone consistency check passed Verification of system requirement was successful.
Verification successful.
- Verify storage using: cluvfy comp ssa
Use the cluvfy comp ssa component verification command to discover and check the sharing of the specified storage locations. CVU checks sharing for nodes in the node list.
# Check disks for OCR [grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-asm-crs-03 -t ocr_vdisk Verifying shared storage accessibility Checking shared storage accessibility... Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-crs-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-crs-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op3-rac1-asm-crs-03 oel7rac1n1 oel7rac1n2 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2" Verification of shared storage accessibility was successful. ## Check disks for DATA [grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t data Verifying shared storage accessibility Checking shared storage accessibility... Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2" Verification of shared storage accessibility was successful.
Shared storage verified successfully.
- Network verification using: cluvfy stage -post hwos
Use the cluvfy stage -post hwos stage verification command to perform network and storage verifications on the specified nodes in the cluster before installing Oracle software. This command also checks for supported storage types and checks each one for sharing.
As we have already checked the disks we are just interested in the network checks, ignore storage information – storage checks will faile
grid@oel7rac1n1 bin]$ ./cluvfy stage -post hwos -n oel7rac1n1,oel7rac1n2 Performing post-checks for hardware and operating system setup Checking node reachability... Node reachability check passed from node "oel7rac1n1" Checking user equivalence... User equivalence check passed for user "grid" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.1.0" Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.10.0" Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.20.0" Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.21.0" Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.30.0" Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.31.0" Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: oel7rac1n1 enp0s3:192.168.1.21 oel7rac1n2 enp0s3:192.168.1.23 Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s8:192.168.10.21 oel7rac1n2 enp0s8:192.168.10.23 Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s9:192.168.20.21 oel7rac1n2 enp0s9:192.168.20.23 Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s10:192.168.21.21 oel7rac1n2 enp0s10:192.168.21.23 Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s16:192.168.30.21 oel7rac1n2 enp0s16:192.168.30.23 Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s17:192.168.31.21 oel7rac1n2 enp0s17:192.168.31.23 Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed for subnet "192.168.10.0". Subnet mask consistency check passed for subnet "192.168.20.0". Subnet mask consistency check passed for subnet "192.168.21.0". Subnet mask consistency check passed for subnet "192.168.30.0". Subnet mask consistency check passed for subnet "192.168.31.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Check for multiple users with UID value 0 passed Time zone consistency check passed Checking shared storage accessibility... No shared storage found Shared storage check failed on nodes "oel7rac1n1,oel7rac1n2" Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Post-check for hardware and operating system setup was unsuccessful on all the nodes.
Verification successful (ingore storage failed checks)
- Verify storage using: cluvfy comp ssa
Use the cluvfy comp ssa component verification command to discover and check the sharing of the specified storage locations. CVU checks sharing for nodes in the node list.
# Check disks for OCR [grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-asm-crs-03 -t ocr_vdisk Verifying shared storage accessibility Checking shared storage accessibility... Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-crs-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-crs-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op3-rac1-asm-crs-03 oel7rac1n1 oel7rac1n2 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2" Verification of shared storage accessibility was successful. ## Check disks for DATA [grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asmdev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t data Verifying shared storage accessibility Checking shared storage accessibility... Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-data-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op1-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-data-04 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-01 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-02 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-03 oel7rac1n1 oel7rac1n2 Disk Sharing Nodes (2 in count) ------------------------------------ ------------------------ /dev/asm/op2-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2" Verification of shared storage accessibility was successful.
Shared storage verified successfully.
- Nodes connectivity verification: cluvfy comp nodecon
Use the cluvfy comp nodecon component verification command to check the connectivity among the nodes specified in the node list. If you provide an interface list, then CVU checks the connectivity using only the specified interfaces.
[grid@oel7rac1n1 bin]$ ./cluvfy comp nodecon -n oel7rac1n1,oel7rac1n2 Verifying node connectivity Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.1.0" Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.10.0" Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.20.0" Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.21.0" Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.30.0" Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.31.0" Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: oel7rac1n1 enp0s3:192.168.1.21 oel7rac1n2 enp0s3:192.168.1.23 Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s8:192.168.10.21 oel7rac1n2 enp0s8:192.168.10.23 Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s9:192.168.20.21 oel7rac1n2 enp0s9:192.168.20.23 Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s10:192.168.21.21 oel7rac1n2 enp0s10:192.168.21.23 Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s16:192.168.30.21 oel7rac1n2 enp0s16:192.168.30.23 Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s17:192.168.31.21 oel7rac1n2 enp0s17:192.168.31.23 Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed for subnet "192.168.10.0". Subnet mask consistency check passed for subnet "192.168.20.0". Subnet mask consistency check passed for subnet "192.168.21.0". Subnet mask consistency check passed for subnet "192.168.30.0". Subnet mask consistency check passed for subnet "192.168.31.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Verification of node connectivity was successful.
Node connectivity verification successful
- check if servers are ready for GI installation – final check: cluvfy stage [-pre | -post] crsinst
Use the cluvfy stage -pre crsinst command with either the -file, -n, -flex, or -upgrade parameters to check the specified nodes before installing or upgrading Oracle Clusterware.
Use the cluvfy stage -post crsinst command to check the specified nodes after installing Oracle Clusterware.
From what you will see, the last check incorporates all the previous checks, but it’s good to run them all just to make sure we have and understand all the components
[grid@oel7rac1n1 ~]$ cd /u01/app/grid/cluvfy/bin/ [grid@oel7rac1n1 bin]$ ./cluvfy stage -pre crsinst -n oel7rac1n1,oel7rac1n2 Performing pre-checks for cluster services setup Checking node reachability... Node reachability check passed from node "oel7rac1n1" Checking user equivalence... User equivalence check passed for user "grid" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.1.0" Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.10.0" Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.20.0" Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.21.0" Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.30.0" Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.31.0" Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: oel7rac1n1 enp0s3:192.168.1.21 oel7rac1n2 enp0s3:192.168.1.23 Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s8:192.168.10.21 oel7rac1n2 enp0s8:192.168.10.23 Interfaces found on subnet "192.168.20.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s9:192.168.20.21 oel7rac1n2 enp0s9:192.168.20.23 Interfaces found on subnet "192.168.21.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s10:192.168.21.21 oel7rac1n2 enp0s10:192.168.21.23 Interfaces found on subnet "192.168.30.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s16:192.168.30.21 oel7rac1n2 enp0s16:192.168.30.23 Interfaces found on subnet "192.168.31.0" that are likely candidates for a private interconnect are: oel7rac1n1 enp0s17:192.168.31.21 oel7rac1n2 enp0s17:192.168.31.23 Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed for subnet "192.168.10.0". Subnet mask consistency check passed for subnet "192.168.20.0". Subnet mask consistency check passed for subnet "192.168.21.0". Subnet mask consistency check passed for subnet "192.168.30.0". Subnet mask consistency check passed for subnet "192.168.31.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Total memory check passed Available memory check passed Swap space check passed Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oel7rac1n2:/sbin,oel7rac1n2:/tmp" Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oel7rac1n1:/sbin,oel7rac1n1:/tmp" Check for multiple users with UID value 2002 passed User existence check passed for "grid" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "grid" in group "oinstall" [as Primary] passed Membership check for user "grid" in group "dba" passed Run level check passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" Kernel parameter check passed for "panic_on_oops" Kernel parameter check passed for "panic_on_oops" Package existence check passed for "binutils" Package existence check passed for "compat-libcap1" Package existence check passed for "compat-libstdc++-33(x86_64)" Package existence check passed for "libgcc(x86_64)" Package existence check passed for "libstdc++(x86_64)" Package existence check passed for "libstdc++-devel(x86_64)" Package existence check passed for "sysstat" Package existence check passed for "gcc" Package existence check passed for "gcc-c++" Package existence check passed for "ksh" Package existence check passed for "make" Package existence check passed for "glibc(x86_64)" Package existence check passed for "glibc-devel(x86_64)" Package existence check passed for "libaio(x86_64)" Package existence check passed for "libaio-devel(x86_64)" Package existence check passed for "nfs-utils" Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)" Port availability check passed for ports "6200,6100" Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)" Port availability check passed for ports "42424" Check for multiple users with UID value 0 passed Current group ID check passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Starting Clock synchronization checks using Network Time Protocol(NTP)... No NTP Daemons or Services were found to be running Clock synchronization check using Network Time Protocol(NTP) passed Core file name pattern consistency check passed. User "grid" is not part of "root" group. Check passed Default user file creation mask check passed Checking integrity of file "/etc/resolv.conf" across nodes "domain" and "search" entries do not coexist in any "/etc/resolv.conf" file The DNS response time for an unreachable node is within acceptable limit on all nodes checking DNS response from all servers in "/etc/resolv.conf" Check for integrity of file "/etc/resolv.conf" passed Time zone consistency check passed Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Checking daemon "avahi-daemon" is not configured and running Daemon not configured check passed for process "avahi-daemon" Daemon not running check passed for process "avahi-daemon" Starting check for /dev/shm mounted as temporary file system ... Check for /dev/shm mounted as temporary file system passed Starting check for /boot mount ... Check for /boot mount passed Starting check for zeroconf check ... Check for zeroconf check passed Pre-check for cluster services setup was successful.
OK!! we are ready to install the cluster software
OUI Grid Installation
- Run OUI installer
[grid@oel7rac1n1 ~]$ cd /u01/app/grid/grid-software/grid/ [grid@oel7rac1n1 grid]$ ./runInstaller Starting Oracle Universal Installer... Checking Temp space: must be greater than 415 MB. Actual 21611 MB Passed Checking swap space: must be greater than 150 MB. Actual 16895 MB Passed Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-12-02_05-18-35PM. Please wait ...
- Choose to install and configure GI for a cluster
- Choose configure standard cluster
- Choose advance installation
- Choose preferred languages to install
- Set cluster name, SCAN name and port. Disable GNS
- Provide list of nodes to be used as cluster nodes.
- Verify ssh user equivalency
- Declare cluster networks
- Declare cluster networks
- Use Flex ASM for storage
- Set disk string in order to get our prepared disks to the list
- Create crs diskgroup using /dev/asm/*asm*crs* disks
Ooops! As you can see OUI can’t take care of the quorum group, so it complains that the crs-03 disk presented from openfiler3 is too small and with different size than crs-01 and crs-02.
The same problem also exists in 11g R2.
Some interesting notes:
http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
http://www.redstk.com/voting-disk-setup-in-a-12c-grid-infrastructure-extended-rac-environment/
-
We need to create temporary disk on openfiler3 with size of 10GB as crs-02 and crs-03. We need to present it to cluster nodes and prepare on those VMs.
Create new volume (click to enlarge)
- Check volumes now (click to enlarge)
- Map new lun to the existing target on openfiler3 (click to enlarge)
- Check the new lun on openfiler3 system level
[root@openfiler3 ~]# cat /proc/net/iet/volume
tid:1 name:iqn.2006-01.com.openfiler:op3.rac1.asm
lun:0 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-crs-03
lun:1 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-tmp
- Discover disk on the cluster nodes
[root@oel7rac1n1 ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: VBOX Model: CD-ROM Rev: 1.0
Type: CD-ROM ANSI SCSI revision: 05
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 1 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 2 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 02
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 3 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 03
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 4 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 04
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 5 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 05
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 6 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 06
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 7 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 07
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 8 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 08
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 9 0 0 0 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 1 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 2 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 02
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 3 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 03
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 4 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 04
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 5 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 05
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 6 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 06
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 7 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 07
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 8 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 08
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning host 11 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 11 0 0 0 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 11 0 0 1 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
1 new or changed device(s) found.
[11:0:0:1]
0 remapped or resized device(s) found.
0 device(s) removed.
[root@oel7rac1n2 ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: VBOX Model: CD-ROM Rev: 1.0
Type: CD-ROM ANSI SCSI revision: 05
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 1 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 2 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 02
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 3 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 03
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 4 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 04
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 5 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 05
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 6 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 06
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 7 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 07
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 7 0 0 8 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 08
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 9 0 0 0 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 1 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 2 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 02
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 3 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 03
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 4 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 04
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 5 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 05
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 6 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 06
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 7 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 07
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 9 0 0 8 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 08
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning host 11 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 11 0 0 0 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 11 0 0 1 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
1 new or changed device(s) found.
[11:0:0:1]
0 remapped or resized device(s) found.
0 device(s) removed.
New device found [11:0:0:1] on both nodes, this is our temporary LUN.
- Find device name assigned to the new disk
[root@oel7rac1n1 ~]# lsscsi [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0 [3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda [7:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb [7:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc [7:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd [7:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sde [7:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf [7:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg [7:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh [7:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi [7:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj [9:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk [9:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl [9:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm [9:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdn [9:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdo [9:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdp [9:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdq [9:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdr [9:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sds [11:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdt [11:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdu [root@oel7rac1n2 ~]# lsscsi [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0 [3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda [7:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb [7:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc [7:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd [7:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sde [7:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf [7:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg [7:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh [7:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi [7:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj [9:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk [9:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl [9:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm [9:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdo [9:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdp [9:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdq [9:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdr [9:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sds [9:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdt [11:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdn [11:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdu
/dev/sdu – new disk device name on both nodes
- Create partition on /dev/sdu on the first node
[root@oel7rac1n1 ~]# parted /dev/sdu mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~]# fdisk -l /dev/sdu WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion. Disk /dev/sdu: 10.5 GB, 10502537216 bytes, 20512768 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt # Start End Size Type Name 1 2048 20510719 9,8G Microsoft basic primary
- Discover new partition on the second node
[root@oel7rac1n2 ~]# kpartx -a /dev/sdu
- Find scsi ID
[root@oel7rac1n1 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu 14f504e46494c45524b6878344b792d4869724a2d4c414875 [root@oel7rac1n2 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu 14f504e46494c45524b6878344b792d4869724a2d4c414875
- Add the following line to /etc/udev/rules.d/90-oracle-asmdevices.rules on both nodes.
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45524b6878344b792d4869724a2d4c414875", SYMLINK+="asm/op3-rac1-asm-tmp", OWNER="grid", GROUP="asmadmin", MODE="0660"
- Reload udev and check asm disk (owner,group,privs)
[root@oel7rac1n1 ~]#udevadm control --reload-rules [root@oel7rac1n1 ~]#udevadm trigger [root@oel7rac1n2 ~]#udevadm control --reload-rules [root@oel7rac1n2 ~]#udevadm trigger [root@oel7rac1n1 ~]# ls -l /dev/asm/*|grep tmp lrwxrwxrwx 1 root root 7 12-02 18:47 /dev/asm/op3-rac1-asm-tmp ->; ../sdu1 [root@oel7rac1n1 ~]# ls -l /dev/sdu1 brw-rw---- 1 grid asmadmin 65, 65 12-02 18:47 /dev/sdu1 [root@oel7rac1n2 ~]# ls -l /dev/asm/*|grep tmp lrwxrwxrwx 1 root root 7 12-02 18:46 /dev/asm/op3-rac1-asm-tmp -> ../sdu1 [root@oel7rac1n2 ~]# ls -l /dev/sdu1 brw-rw---- 1 grid asmadmin 65, 65 12-02 18:46 /dev/sdu1
- Create CRS diskgroup using temporary disk from openfiler3
- Set passwords for ASM
- Don’t configure IPMI
- Do not register with EM Cloud Control unless you have it already
- Set asm groups – leave default
- Set ORACLE_BASE and ORACLE_HOME directories
- Set oraInventory directory
- We wan to run root scripts manually
- Prerequisits being checked
- Prerequisits being checked
- It complains about /dev/shm size
- As you can see it looks like OUI bug as size is ok
- Confirm you want to ignore this warning
- Installation plan summary screen
- Installation process
- Run scripts as root
# root scripts
[root@oel7rac1n1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oel7rac1n2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oel7rac1n1 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2016/12/02 19:27:19 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/12/02 19:27:50 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/12/02 19:27:52 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2016/12/02 19:28:32 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
ASM created and started successfully.
Disk Group CRS created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 1c193c21750c4fe3bfa28c6b5df261b3.
Successful addition of voting disk 54cd207d0d744fd8bf2714d7776f5ad2.
Successful addition of voting disk 355efd32461e4f3fbf50305bfac1657e.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: oel7rac1n1
CRS-6016: Resource auto-start has completed for server oel7rac1n1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 19:34:26 CLSRSC-343: Successfully started Oracle Clusterware stack
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'oel7rac1n1'
CRS-2676: Start of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cvu' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ons' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.net1.network' on 'oel7rac1n1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7rac1n1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7rac1n1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CRS.dg' is already running on 'oel7rac1n1'
CRS-6017: Processing resource auto-start for servers: oel7rac1n1
CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.oc4j' on 'oel7rac1n1'
CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'oel7rac1n1'
CRS-2676: Start of 'ora.cvu' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.scan2.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.scan3.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1'
CRS-2676: Start of 'ora.ons' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'oel7rac1n1' succeeded
CRS-6016: Resource auto-start has completed for server oel7rac1n1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 19:38:37 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
# node2 root.sh
[root@oel7rac1n2 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2016/12/02 20:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/12/02 20:18:12 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/12/02 20:18:13 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2016/12/02 20:19:43 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n2'
CRS-2676: Start of 'ora.asm' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n2'
CRS-2676: Start of 'ora.storage' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n2'
CRS-2676: Start of 'ora.crf' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n2'
CRS-2676: Start of 'ora.crsd' on 'oel7rac1n2' succeeded
CRS-6017: Processing resource auto-start for servers: oel7rac1n2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n2'
CRS-2672: Attempting to start 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7rac1n1'
CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n2'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n2'
CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2'
CRS-2676: Start of 'ora.ons' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7rac1n2' succeeded
CRS-6016: Resource auto-start has completed for server oel7rac1n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 20:23:53 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/12/02 20:24:22 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
- Installation in progress
- Installation finished successfully.
Postinstallation checks
- Verify Grid Infrastructure after the installation: cluvfy stage -post crsinst
[grid@oel7rac1n1 bin]$ ./cluvfy stage -post crsinst -n oel7rac1n1,oel7rac1n2 -method root Enter "ROOT" password: Performing post-checks for cluster services setup Checking node reachability... Node reachability check passed from node "oel7rac1n1" Checking user equivalence... User equivalence check passed for user "grid" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Check: Node connectivity using interfaces on subnet "192.168.1.0" Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n2,oel7rac1n1 TCP connectivity check passed for subnet "192.168.1.0" Check: Node connectivity using interfaces on subnet "192.168.31.0" Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.31.0" Check: Node connectivity using interfaces on subnet "192.168.20.0" Node connectivity passed for subnet "192.168.20.0" with node(s) oel7rac1n2,oel7rac1n1 TCP connectivity check passed for subnet "192.168.20.0" Check: Node connectivity using interfaces on subnet "192.168.21.0" Node connectivity passed for subnet "192.168.21.0" with node(s) oel7rac1n2,oel7rac1n1 TCP connectivity check passed for subnet "192.168.21.0" Check: Node connectivity using interfaces on subnet "192.168.30.0" Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n2,oel7rac1n1 TCP connectivity check passed for subnet "192.168.30.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed for subnet "192.168.20.0". Subnet mask consistency check passed for subnet "192.168.21.0". Subnet mask consistency check passed for subnet "192.168.30.0". Subnet mask consistency check passed for subnet "192.168.31.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.20.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.20.0" for multicast communication with multicast group "224.0.0.251" passed. Checking subnet "192.168.21.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.21.0" for multicast communication with multicast group "224.0.0.251" passed. Checking subnet "192.168.30.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.30.0" for multicast communication with multicast group "224.0.0.251" passed. Checking subnet "192.168.31.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.31.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Checking whether the ASM filter driver is active and consistent on all nodes ASM filter driver library is not installed on any of the cluster nodes. ASM filter driver configuration was found consistent across all the cluster nodes. Time zone consistency check passed Checking Cluster manager integrity... Checking CSS daemon... Oracle Cluster Synchronization Services appear to be online. Cluster manager integrity check passed UDev attributes check for OCR locations started... UDev attributes check passed for OCR locations UDev attributes check for Voting Disk locations started... UDev attributes check passed for Voting Disk locations Default user file creation mask check passed Checking cluster integrity... Cluster integrity check passed hecking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations Checking daemon liveness... Liveness check passed for "CRS daemon" Checking OCR config file "/etc/oracle/ocr.loc"... OCR config file "/etc/oracle/ocr.loc" check successful Disk group for ocr location "+CRS/rac1/OCRFILE/registry.255.929561455" is available on all the nodes Uniqueness check for OCR device passed Checking the version of OCR... OCR of correct Version "4" exists Checking data integrity of OCR... Data integrity check for OCR passed OCR integrity check passed Checking CRS integrity... Clusterware version consistency passed. CRS integrity check passed Checking node application existence... Checking existence of VIP node application (required) VIP node application check passed Checking existence of NETWORK node application (required) NETWORK node application check passed Checking existence of ONS node application (optional) ONS node application check passed Checking Single Client Access Name (SCAN)... Checking TCP connectivity to SCAN listeners... TCP connectivity to SCAN listeners exists on all cluster nodes Checking name resolution setup for "rac1-scan.dba24.pl"... Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Checking SCAN IP addresses... Check of SCAN IP addresses passed Verification of SCAN VIP and listener setup passed Checking OLR integrity... Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed WARNING: This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR. OLR integrity check passed Checking Oracle Cluster Voting Disk configuration... Oracle Cluster Voting Disk configuration check passed Task ASM Integrity check started... Checking if connectivity exists across cluster nodes on the ASM network Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Check: Node connectivity using interfaces on subnet "192.168.30.0" Node connectivity passed for subnet "192.168.30.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.30.0" Check: Node connectivity using interfaces on subnet "192.168.31.0" Node connectivity passed for subnet "192.168.31.0" with node(s) oel7rac1n1,oel7rac1n2 TCP connectivity check passed for subnet "192.168.31.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.30.0". Subnet mask consistency check passed for subnet "192.168.31.0". Subnet mask consistency check passed. Node connectivity check passed Network connectivity check across cluster nodes on the ASM network passed Starting check to see if ASM is running on all cluster nodes... ASM Running check passed. ASM is running on all specified nodes Disk Group Check passed. At least one Disk Group configured Task ASM Integrity check passed... User "grid" is not part of "root" group. Check passed Oracle Clusterware is installed on all nodes. CTSS resource check passed Query of CTSS for time offset passed CTSS is in Active state. Proceeding with check of clock time offsets on all nodes... Check of clock time offsets passed Oracle Cluster Time Synchronization Services check passed Checking VIP configuration. Checking VIP Subnet configuration. Check for VIP Subnet configuration passed. Checking VIP reachability Check for VIP reachability passed. Post-check for cluster services setup was successful.
All checks passed, seems that GI has been installed properly!!
Modify CRS, add DATA, FRA diskgroups
- Run asmca as grid
- Try to add op3-rac1-asm-crs-03 disk as Quorum failgroup
- It complains again. Wow! Bug or a feature?
- Replace temporary disk with crs-03 using sqlplus
# first let's see votes [grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS] 2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS] 3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Add op3-rac1-asm-crs-03 to the CRS diskgroup as quorum disk
[grid@oel7rac1n1 bin]$ ./sqlplus / as sysasm SQL> alter diskgroup crs add QUORUM FAILGROUP fg3 DISK '/dev/asm/op3-rac1-asm-crs-03'; Diskgroup altered.
And what happened? nothing, vote stil on tmp
[grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS] 2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS] 3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Remove temporary disk
SQL> alter diskgroup crs drop disk CRS_0002; Diskgroup altered.
Result?
[grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS] 2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS] 3. ONLINE 0588f4e97cbb4fb8bfb28fd5d18a46d0 (/dev/asm/op3-rac1-asm-crs-03) [CRS] Located 3 voting disk(s).
This is what we want!
- Create DATA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup op2 to disks that come from openfiler2. Set AU_SIZE to 4GB
- Create FRA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup op2 to disks that come from openfiler2. Set AU_SIZE to 1GB
- Diskgroups successfully created.
Grid Infrastructure installed successfully!
Alternate approach to ASM diskgroups manipulation using ascmca -silent, chdg, mkdg
I used asmca or sqlplus to create and modify the diskgroup, but there are some other ways. Please find examples below.
- ASMCA – silent
As you can read in the doc https://docs.oracle.com/database/121/OSTMG/GUID-877EB0F8-E9CA-4C97-965C-AACBB256B12D.htm#OSTMG94309
asmca silent allows only this operations:
-convertToFlexASM
-configureASM
-upgradeASM
-configureParameter
-deleteASM
-createDiskGroup
-addDisk
-createVolume
-createACFS
Let’s add the crs-03 quorum disk to the crs as we did with sqlplus
[grid@oel7rac1n1 bin]$ ./asmca -silent -addDisk -diskGroupName crs -diskList '/dev/asm/op3-rac1-asm-crs-03' -failuregroup fg3 -quorum
Disks added successfully to disk group crs
[grid@oel7rac1n1 bin]$ ./asmcmd -p
ASMCMD [+] > lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
10014 6965 10014 CRS_0001 CRS_0001 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-crs-01
10014 6965 10014 CRS_0000 CRS_0000 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-crs-02
510 507 510 CRS_0003 FG3 QUORUM System UNKNOWN /dev/asm/op3-rac1-asm-crs-03
10014 6966 10014 CRS_0002 CRS_0002 REGULAR System UNKNOWN /dev/asm/op3-rac1-asm-tmp
ASMCMD [+] > exit
# check how votes look like
[grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
3. ONLINE a25677e1db034f6cbf5b2f88daf9d4c7 (/dev/asm/op3-rac1-asm-tmp) [CRS]
As we cannot use asmca silent mode to drop disk from diskgroup let’s use chdg command in ASMCMD. First prepare xml file
# so lets use chdg to drop /dev/asm/op3-rac1-asm-tmp
# create file /home/grid/change_crs.xml
<chdg name="crs" power="3">
<drop>
<dsk name="CRS_0002"></dsk>
</drop>
</chdg>
Run chdg
ASMCMD [+] > chdg /home/grid/change_crs.xml
Diskgroup altered.
ASMCMD [+] > lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
10014 5530 10014 CRS_0001 CRS_0001 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-crs-01
10014 5530 10014 CRS_0000 CRS_0000 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-crs-02
510 475 510 CRS_0003 FG3 QUORUM System UNKNOWN /dev/asm/op3-rac1-asm-crs-03
[grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CRS]
2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CRS]
3. ONLINE 3d51b07f8a6e4f2abf8f7b4a506f67b6 (/dev/asm/op3-rac1-asm-crs-03) [CRS]
Located 3 voting disk(s).
That’s it!!
- Use ASMCA silent mode to create DATA and FRA
[grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName DATA -diskList /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-asm-data-03,/dev/asm/op1-rac1-asm-data-04 -failuregroup op1 -diskList /dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/asm/op2-rac1-asm-data-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatible.asm 12.1 -compatible.rdbms 11.2 Disk Group DATA created successfully. [grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName FRA -diskList /dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04 -failuregroup op1 -diskList /dev/asm/op2-rac1-asm-fra-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatible.asm 12.1 -compatible.rdbms 11.2 Disk Group FRA created successfully.
Check new dgs
ASMCMD [+] > lsdsk -k -G DATA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
2014 1987 2014 DATA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-01
2014 1990 2014 DATA_0001 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-02
2014 1986 2014 DATA_0002 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-03
2014 1988 2014 DATA_0003 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-04
2014 1991 2014 DATA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-01
2014 1984 2014 DATA_0005 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-02
2014 1988 2014 DATA_0006 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-03
2014 1988 2014 DATA_0007 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-04
ASMCMD [+] > lsdsk -k -G FRA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
1022 997 1022 FRA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-01
1022 994 1022 FRA_0001 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-02
1022 995 1022 FRA_0002 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-03
1022 997 1022 FRA_0003 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-04
1022 996 1022 FRA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-01
1022 998 1022 FRA_0005 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-02
1022 994 1022 FRA_0006 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-03
1022 995 1022 FRA_0007 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-04
ASMCMD [+] > lsattr -l -G DATA
Name Value
access_control.enabled FALSE
access_control.umask 066
au_size 1048576
cell.smart_scan_capable FALSE
compatible.asm 12.1.0.0.0
compatible.rdbms 11.2.0.0.0
content.check FALSE
content.type data
disk_repair_time 3.6h
failgroup_repair_time 24.0h
idp.boundary auto
idp.type dynamic
phys_meta_replicated true
sector_size 512
thin_provisioned FALSE
ASMCMD [+] > lsattr -l -G FRA
Name Value
access_control.enabled FALSE
access_control.umask 066
au_size 1048576
cell.smart_scan_capable FALSE
compatible.asm 12.1.0.0.0
compatible.rdbms 11.2.0.0.0
content.check FALSE
content.type data
disk_repair_time 3.6h
failgroup_repair_time 24.0h
idp.boundary auto
idp.type dynamic
phys_meta_replicated true
sector_size 512
thin_provisioned FALSE
- Use sqlplus to create DATA and FRA
SQL> create diskgroup DATA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op1-rac1-asm-data-01','/dev/asm/op1-rac1-asm-data-02','/dev/asm/op1-rac1-asm-data-03','/dev/asm/op1-rac1-asm-data-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-data-01','/dev/asm/op2-rac1-asm-data-02','/dev/asm/op2-rac1-asm-data-03','/dev/asm/op2-rac1-asm-data-04' ATTRIBUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M'; Diskgroup created. SQL> create diskgroup FRA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op1-rac1-asm-fra-01','/dev/asm/op1-rac1-asm-fra-02','/dev/asm/op1-rac1-asm-fra-03','/dev/asm/op1-rac1-asm-fra-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-fra-01','/dev/asm/op2-rac1-asm-fra-02','/dev/asm/op2-rac1-asm-fra-03','/dev/asm/op2-rac1-asm-fra-04' ATTRIBUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M'; Diskgroup created.
Check results
SMCMD [+] > lsdsk -k -G DATA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
2014 1984 2014 DATA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-01
2014 1989 2014 DATA_0001 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-02
2014 1989 2014 DATA_0002 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-03
2014 1989 2014 DATA_0003 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-04
2014 1989 2014 DATA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-01
2014 1988 2014 DATA_0005 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-02
2014 1986 2014 DATA_0006 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-03
2014 1988 2014 DATA_0007 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-04
ASMCMD [+] > lsdsk -k -G FRA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
1022 997 1022 FRA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-01
1022 995 1022 FRA_0001 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-02
1022 998 1022 FRA_0002 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-03
1022 993 1022 FRA_0003 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-fra-04
1022 996 1022 FRA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-01
1022 998 1022 FRA_0005 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-02
1022 995 1022 FRA_0006 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-03
1022 994 1022 FRA_0007 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-fra-04
- Another way to do it is to build the diskgroups with mkdg command
Create file /home/grid/mkdg_data.xml
# /home/grid/mkdg_data.xml
<dg name="data" redundancy="normal">
<fg name="op1">
<dsk string="/dev/asm/op1-rac1-asm-data-01"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-data-02"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-data-03"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-data-04"></dsk>
</fg>
<fg name="op2">
<dsk string="/dev/asm/op2-rac1-asm-data-01"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-data-02"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-data-03"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-data-04"></dsk>
</fg>
<a name="compatible.asm" value="12.1"></a>
<a name="compatible.rdbms" value="11.2"></a>
<a name="au_size" value="4M"></a>
</dg>
Create file /home/grid/mkdg_fra.xml
#/home/grid/mkdg_fra.xml
<dg name="fra" redundancy="normal">
<fg name="op1">
<dsk string="/dev/asm/op1-rac1-asm-fra-01"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-fra-02"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-fra-03"></dsk>
<dsk string="/dev/asm/op1-rac1-asm-fra-04"></dsk>
</fg>
<fg name="op2">
<dsk string="/dev/asm/op2-rac1-asm-fra-01"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-fra-02"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-fra-03"></dsk>
<dsk string="/dev/asm/op2-rac1-asm-fra-04"></dsk>
</fg>
<a name="compatible.asm" value="12.1"></a>
<a name="compatible.rdbms" value="11.2"></a>
<a name="au_size" value="1M"></a>
</dg>
Create diskgroups using mkdg
ASMCMD [+] > mkdg /home/grid/mkdg_data.xml ASMCMD [+] > mkdg /home/grid/mkdg_fra.xml
Check results
ASMCMD [+] > lsdsk -k -G DATA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
2012 1980 2014 DATA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-01
2012 1968 2014 DATA_0001 DATA_0001 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-02
2012 1984 2014 DATA_0002 DATA_0002 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-03
2012 1976 2014 DATA_0003 DATA_0003 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-04
2012 1964 2014 DATA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-01
2012 1980 2014 DATA_0005 DATA_0005 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-02
2012 1964 2014 DATA_0006 DATA_0006 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-03
2012 1968 2014 DATA_0007 DATA_0007 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-04
ASMCMD [+] > lsdsk -k -G DATA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
2012 1980 2014 DATA_0000 OP1 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-01
2012 1968 2014 DATA_0001 DATA_0001 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-02
2012 1984 2014 DATA_0002 DATA_0002 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-03
2012 1976 2014 DATA_0003 DATA_0003 REGULAR System UNKNOWN /dev/asm/op1-rac1-asm-data-04
2012 1964 2014 DATA_0004 OP2 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-01
2012 1980 2014 DATA_0005 DATA_0005 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-02
2012 1964 2014 DATA_0006 DATA_0006 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-03
2012 1968 2014 DATA_0007 DATA_0007 REGULAR System UNKNOWN /dev/asm/op2-rac1-asm-data-04
Despite proper configuration in the xml files disks did not get their proper failgroup assignment. I will try out this approach later again – after patching with latest PSU.
What do you think?? Please post your comments 
Source: My experience and WorlWideWeb
About the author
Maciej Tokar
An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU
