This is sixth part of my RAC lab series. I will show you step by step how to prepare your OEL 7.x OS for GI and RAC installation. VM with OEL 7.2 in part 3 of the series will be used to show this.

RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database

 
 
Oracle 12c official documentation
 

Please refer to the official 12.1 documentation to verify current OS requirements:

 
 
Oracle Enterprise Linux 7.2 Network Basic Configuration
 
  • If oel72rac1n1 VM is not running please start it using vboxmanage in the cli.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage startvm oel72rac1n1
Waiting for VM "oel72rac1n1" to power on...
VM "oel72rac1n1" has been successfully started.
 
  • Before I do anything with the OS, I need VM’s network to be functional.
    Let’s check with vboxmanage showvminfo what kind of interfaces we got for this VM, and what they MAC addresses are.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel72rac1n1"|grep NIC

NIC 1:           MAC: 0800276A189E, Attachment: Bridged Interface 'wlx98ded00b5b05', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: den
y, Bandwidth group: none
NIC 2:           MAC: 080027DB1740, Attachment: Internal Network 'storage-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: den
y, Bandwidth group: none
NIC 3:           MAC: 080027D59C97, Attachment: Internal Network 'priv1-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny,
 Bandwidth group: none
NIC 4:           MAC: 080027AE9147, Attachment: Internal Network 'priv2-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny,
 Bandwidth group: none
NIC 5:           MAC: 08002748771B, Attachment: Internal Network 'apriv1-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny
, Bandwidth group: none
NIC 6:           MAC: 0800271A1EE2, Attachment: Internal Network 'apriv2-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny
, Bandwidth group: none
 
  • Let’s clean it up manually a little to get clear output:
NIC 1:  MAC: 0800276A189E Bridged Interface 'wlx98ded00b5b05'
NIC 2:  MAC: 080027DB1740 Internal Network 'storage-internal'
NIC 3:  MAC: 080027D59C97 Internal Network 'priv1-internal'
NIC 4:  MAC: 080027AE9147 Internal Network 'priv2-internal'
NIC 5:  MAC: 08002748771B Internal Network 'apriv1-internal'
NIC 6:  MAC: 0800271A1EE2 Internal Network 'apriv2-internal'
 
 
  • Now compare macs to find out NIC vs dev name in the using the “ip addr” command output text (on the VM)
2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:6a:18:9e brd ff:ff:ff:ff:ff:ff
3: enp0s8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:db:17:40 brd ff:ff:ff:ff:ff:ff
4: enp0s9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d5:9c:97 brd ff:ff:ff:ff:ff:ff
5: enp0s10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:ae:91:47 brd ff:ff:ff:ff:ff:ff
6: enp0s16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:48:77:1b brd ff:ff:ff:ff:ff:ff
7: enp0s17: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:1a:1e:e2 brd ff:ff:ff:ff:ff:ff
 
  • What gives us after some analysis a list of interface <-> network relationship
NIC 1:  MAC: 0800276A189E Bridged Interface 'wlx98ded00b5b05' enp0s3 dhcp
NIC 2:  MAC: 080027DB1740 Internal Network 'storage-internal' enp0s8 192.168.10.x
NIC 3:  MAC: 080027D59C97 Internal Network 'priv1-internal' enp0s9 192.168.20.x
NIC 4:  MAC: 080027AE9147 Internal Network 'priv2-internal' enp0s10 192.168.21.x
NIC 5:  MAC: 08002748771B Internal Network 'apriv1-internal' enp0s16 192.168.30.x
NIC 6:  MAC: 0800271A1EE2 Internal Network 'apriv2-internal' enpos17 192.168.31.x

So now as I know that I need to get IP from dhcp for bridged interface I take MAC address of the bridged interface (0800276A189E) and register a static IP for it in my DHCP server (192.168.1.21).

 

After having done that, I can proceed with configuration of the network interfaces

 
  • First we need to remove incorrectly configured connections – done by the installer.
    Look how it looks now:
[maciek@oel72rac1n1 ~]# sudo nmcli con show
NAME     UUID                                  TYPE            DEVICE
enp0s10  528e61d5-c88d-4ebc-bd73-bbece03229f0  802-3-ethernet  --
enp0s16  41e299d3-b303-4aef-a7ab-6356166a7af8  802-3-ethernet  --
enp0s3   2093693b-13c3-4fbc-99f2-5b4eee6895b5  802-3-ethernet  --
enp0s8   4460b313-5b21-4dce-b816-ea780ffc120e  802-3-ethernet  --
enp0s9   3e0d7ed2-4109-4682-a79a-7b370b91dd75  802-3-ethernet  --
enp0s17  9b951f52-e1bb-4863-a6d5-64b2c7670d60  802-3-ethernet  --
 
  • Remove those networks – not sure the reason they were configured like this, but I want to get rid of them and configure networks from scratch
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s3
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s8
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s9
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s10
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s16
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s17
 
  • After having it cleared, let’s add the networks back with proper config
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name bridged ifname enp0s3 type ethernet
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name storage-internal ifname enp0s8 type ethernet ip4 "192.168.10.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name priv1-internal ifname enp0s9 type ethernet ip4 "192.168.20.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name priv2-internal ifname enp0s10 type ethernet ip4 "192.168.21.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name apriv1-internal ifname enp0s16 type ethernet ip4 "192.168.30.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name apriv2-internal ifname enp0s17 type ethernet ip4 "192.168.31.21/24"

[maciek@oel72rac1n1 ~]# sudo nmcli con show
NAME              UUID                                  TYPE            DEVICE
priv1-internal    0acddf7c-be94-49ec-8d05-2fc6bf10d25a  802-3-ethernet  enp0s9
storage-internal  4d92bf55-a505-405d-b4a2-fd36cfbef270  802-3-ethernet  enp0s8
bridged           5851f1af-fb84-4bc1-bc3f-394ab7524714  802-3-ethernet  enp0s3
apriv1-internal   87d2f222-d768-4fd6-be6d-96a10d198922  802-3-ethernet  enp0s16
apriv2-internal   c23f2a48-d4e7-4c4d-8c67-170ef5ae9118  802-3-ethernet  enp0s17
priv2-internal    518cd49c-51fb-482f-9bf1-19af3c462a6d  802-3-ethernet  enp0s10

[maciek@oel72rac1n1 ~]# sudo ip addr|grep -v inet6|grep -v lft
1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:6a:18:9e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.21/24 brd 192.168.1.255 scope global enp0s3
3: enp0s8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:db:17:40 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.21/24 brd 192.168.10.255 scope global enp0s8
4: enp0s9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d5:9c:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.21/24 brd 192.168.20.255 scope global enp0s9
5: enp0s10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:ae:91:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.21.21/24 brd 192.168.21.255 scope global enp0s10
6: enp0s16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:48:77:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.21/24 brd 192.168.30.255 scope global enp0s16
7: enp0s17: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:1a:1e:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.31.21/24 brd 192.168.31.255 scope global enp0s17

As you can see all IPs are set, the bridged interface enp0s3 has got it’s IP from the DHCP server

 
  • It is good to perform some basic test now. First let’s try to ping some url on the internet and then openfiler1 interface on the storage network.
# check internet first
[maciek@oel72rac1n1 ~]# sudo ping onet.pl
PING onet.pl (213.180.141.140) 56(84) bytes of data.
64 bytes from sg1.any.onet.pl (213.180.141.140): icmp_seq=1 ttl=57 time=260 ms
# OK!!!

# Now check if openfiler1 is visible
[maciek@oel72rac1n1 ~]# sudo ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data.
64 bytes from 192.168.10.10: icmp_seq=1 ttl=64 time=1.28 ms
 

Great!!!, the rest of the networks to be checked after the second rac node is up and running

 
 
Installation of required missing packages and OEL 7.2 system update
 
 

We will get back to the network configuration later, but now let’s add some software to the system. Remember, we have installed just the minimal software pack for the system, we lack some packages required according to the oracle documentation and some others that can be useful.

  • Let’s check available software groups including hidden groups
[maciek@oel72rac1n1 ~]$ sudo yum grouplist  hidden
Loaded plugins: ulninfo
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
ol7_UEKR3                                                                                    | 1.2 kB  00:00:00
ol7_latest                                                                                   | 1.4 kB  00:00:00
(1/5): ol7_UEKR3/x86_64/updateinfo                                                           |  69 kB  00:00:00
(2/5): ol7_latest/x86_64/group                                                               | 681 kB  00:00:03
(3/5): ol7_latest/x86_64/updateinfo                                                          | 1.2 MB  00:00:06
(4/5): ol7_UEKR3/x86_64/primary                                                              |  22 MB  00:00:42
(5/5): ol7_latest/x86_64/primary                                                             |  21 MB  00:00:40
ol7_UEKR3                                                                                                   525/525
ol7_latest                                                                                              17829/17829
Available Environment Groups:
   Minimal Install
   Infrastructure Server
   File and Print Server
   Basic Web Server
   Virtualization Host
   Server with GUI
Available Groups:
   Additional Development
   Anaconda Tools
   Backup Client
   Backup Server
   Base
   Common NetworkManager submodules
   Compatibility Libraries
   Conflicts (Server)
   Console Internet Tools
   Core
   DNS Name Server
   Debugging Tools
   Desktop Debugging and Performance Tools
   Development Tools
   Dial-up Networking Support
   Directory Client
   Directory Server
   E-mail Server
   Emacs
   FTP Server
   File and Storage Server
   Fonts
   GNOME
   Graphical Administration Tools
   Graphics Creation Tools
   Guest Agents
   Guest Desktop Agents
   Hardware Monitoring Utilities
   High Availability
   Identity Management Server
   Infiniband Support
   Input Methods
   Internet Browser
   Java Platform
   KDE
   Large Systems Performance
   Legacy UNIX Compatibility
   Legacy X Window System Compatibility
   Load Balancer
   Mainframe Access
   MariaDB Database Client
   MariaDB Database Server
   Multimedia
   MySQL Database Client
   MySQL Database Server
   Network File System Client
   Network Infrastructure Server
   Networking Tools
   PHP Support
   Performance Tools
   Perl Support
   Perl for Web
   Platform Development
   PostgreSQL Database Client
   PostgreSQL Database Server
   Print Server
   Printing Client
   Python
   Remote Desktop Clients
   Remote Management for Linux
   Resilient Storage
   Ruby Support
   Scientific Support
   Security Tools
   Smart Card Support
   System Administration Tools
   System Management
   Technical Writing
   Virtualization Client
   Virtualization Hypervisor
   Virtualization Platform
   Virtualization Tools
   Web Server
   Web Servlet Engine
   X Window System
Done
 
  • What we generally need to install is the base group – a lot of packages are installed by this group.
[maciek@oel72rac1n1 ~]$ sudo yum groupinstall base
Loaded plugins: ulninfo
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Resolving Dependencies
--&gt; Running transaction check
---&gt; Package abrt-addon-ccpp.x86_64 0:2.1.11-45.0.1.el7 will be installed
--&gt; Processing Dependency: abrt = 2.1.11-45.0.1.el7 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: abrt-libs = 2.1.11-45.0.1.el7 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: gdb &gt;= 7.6.1-63 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: libreport-python for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: elfutils for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: libsatyr.so.3()(64bit) for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64
--&gt; Processing Dependency: libabrt.so.0()(64bit) for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64

......... lines removed for better clarity .........................

--&gt; Processing Dependency: iw for package: crda-3.13_2016.02.08-1.el7.x86_64
---&gt; Package cryptsetup.x86_64 0:1.7.2-1.el7 will be installed
--&gt; Processing Dependency: cryptsetup-libs(x86-64) = 1.7.2-1.el7 for package: cryptsetup-1.7.2-1.el7.x86_64
---&gt; Package cyrus-sasl-plain.x86_64 0:2.1.26-20.el7_2 will be installed
--&gt; Processing Dependency: cyrus-sasl-lib(x86-64) = 2.1.26-20.el7_2 for package: cyrus-sasl-plain-2.1.26-20.el7_2.x86_64
---&gt; Package dmraid.x86_64 0:1.0.0.rc16-28.el7 will be installed
--&gt; Processing Dependency: dmraid-events for package: dmraid-1.0.0.rc16-28.el7.x86_64
---&gt; Package dosfstools.x86_64 0:3.0.20-9.el7 will be installed
---&gt; Package ed.x86_64 0:1.9-4.el7 will be installed
---&gt; Package fprintd-pam.x86_64 0:0.5.0-4.0.el7_0 will be installed
--&gt; Processing Dependency: fprintd = 0.5.0-4.0.el7_0 for package: fprintd-pam-0.5.0-4.0.el7_0.x86_64
---&gt; Package hunspell.x86_64 0:1.3.2-15.el7 will be installed
--&gt; Processing Dependency: hunspell-en-US for package: hunspell-1.3.2-15.el7.x86_64
--&gt; Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================
 Package                             Arch           Version                                Repository          Size
====================================================================================================================
Installing for group install "Base":
 abrt-addon-ccpp                     x86_64         2.1.11-45.0.1.el7                      ol7_latest         190 k
 abrt-addon-python                   x86_64         2.1.11-45.0.1.el7                      ol7_latest         100 k
 abrt-cli                            x86_64         2.1.11-45.0.1.el7                      ol7_latest          85 k
 abrt-console-notification           x86_64         2.1.11-45.0.1.el7                      ol7_latest          86 k
 at                                  x86_64         3.1.13-22.el7                          ol7_latest          50 k
 attr                                x86_64         2.4.46-12.el7                          ol7_latest          65 k
 bash-completion                     noarch         1:2.1-6.el7                            ol7_latest          85 k

......... lines removed for better clarity .........................
 yum                                 noarch         3.4.3-150.0.1.el7                      ol7_latest         1.2 M

Transaction Summary
====================================================================================================================
Install  67 Packages (+101 Dependent packages)
Upgrade              (  13 Dependent packages)

Total download size: 61 M
Is this ok [y/d/N]:

Complete!
 
  • Install some additional required packages
[maciek@oel72rac1n1 ~]$ sudo yum install ksh rsh nfs-utils java telnet
[maciek@oel72rac1n1 ~]$ sudo yum install xorg-x11-xauth xterm xinetd libXxf86misc libdmx xorg-x11-utils
[maciek@oel72rac1n1 ~]$ sudo yum install device-mapper-multipath sg3_utils* iscsi-initiator-utils
[maciek@oel72rac1n1 ~]$ sudo yum install compat-libstdc++-33.i686 compat-libstdc++-33.x86_64
[maciek@oel72rac1n1 ~]$ sudo yum install libstdc++-devel.i686 libstdc++-devel.x86_64
[maciek@oel72rac1n1 ~]$ sudo yum install gcc gcc-c++.x86_64 compat-libcap1.x86_64 glibc-devel.i686
[maciek@oel72rac1n1 ~]$ sudo yum install libaio.i686 libaio-devel.i686 libaio-devel.x86_64
[maciek@oel72rac1n1 ~]$ sudo yum install libXext.i686 libXtst.i686 libXp.i686 libXp.x86_64
[maciek@oel72rac1n1 ~]$ sudo yum install elfutils-libelf-devel elfutils-libelf-devel.i686 unixODBC-devel unixODBC-devel.i686 unixODBC
[maciek@oel72rac1n1 ~]$ sudo yum install compat-db47 compat-gcc-44 compat-gcc-44-c++ compat-db47.i686
[maciek@oel72rac1n1 ~]$ sudo yum install oracle-rdbms-server-12cR1-preinstall.x86_64
 
  • Install packages required for VboxAdditions
[maciek@oel72rac1n1 ~]$ sudo yum install kernel-uek-devel-*uek.x86_64
 
  • Update whole OS, the update process will update packages and linux kernel.
    Make sure you have ol7_UEKR3 public repository enabled in /etc/yum.repos.d/
[maciek@oel72rac1n1 ~]$ sudo yum update
 
  • Check your kernel version before you restart the server after the update has completed
[maciek@oel72rac1n1 ~]$ uname -a
Linux oel72rac1n1.dba24.pl 3.8.13-98.7.1.el7uek.x86_64 #2 SMP Wed Nov 25 13:51:41 PST 2015 x86_64 x86_64 x86_64 GNU/Linux
 
  • After restart you check it again
[maciek@oel72rac1n1 ~]$ uname -a

Linux oel72rac1n1.dba24.pl 3.8.13-118.14.1.el7uek.x86_64 #2 SMP Mon Oct 31 17:32:03 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@oel72rac1n1 log]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)
 

Wow, during the update process, OS version has been upgraded from 7.2 – 7.3 so the server and VM name is no longer valid. I need to change it maybe to oel7rac1n1 to avoid doubts after any future updates that upgrade my OS to higher version:)

 
 
Hostname and VM name change
 
  • Ok so first we change hostname
maciek@oel7rac1n1 ~ $ sudo hostnamectl set-hostname oel7rac1n1
 
  • Then change domain by adding one line to the /etc/hosts for 192.168.1.21 IP. Additionally let’s get rid of ipv6 ::1 entry – IPv6 is going to be disabled in the OS
maciek@oel7rac1n1 ~ $ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# after the tweak
maciek@oel7rac1n1 ~ $ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.1.21 oel7rac1n1.dba24.pl oel7rac1n1
 
  • Relog again as maciek and verify
maciek@oel7rac1n1.dba24.pl ~ $ hostname -f
oel7rac1n1.dba24.pl
maciek@oel7rac1n1.dba24.pl ~ $ hostname
oel7rac1n1
 

We got what we expected.

 
  • Change VM name
maciek@oel7rac1n1.dba24.pl ~ $ sudo shutdown -h now
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage modifyvm "oel72rac1n1" --name "oel7rac1n1"
 
  • Check how the files related to the VM looks like now
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"|grep oel7
Name:            oel7rac1n1
Config file:     /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.vbox
Log folder:      /vbox-repo1/metadata/RacLab1/oel7rac1n1/Logs
SATA Controller (1, 0): /vbox-repo1/disks/oel72rac1n1_localOSdisk1.vdi (UUID: be004bfa-0069-452c-b9b5-2448a988cd3b)
Capture file:       /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.webm
 

Well, vm configuration files’ names has changed, but local disk still bears the old server name

 
  • Change name of the disk (medium)
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage modifymedium /vbox-repo1/disks/oel72rac1n1_localOSdisk1.vdi --move /vbox-repo1/disks/oel7rac1n1_localOSdisk1.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Move medium with UUID be004bfa-0069-452c-b9b5-2448a988cd3b finished
 
  • Let’s see how it looks now. As you can see Virtualbox copes with changing of the file name automatically. New media file name has been registerd in the VM’s configuration
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"|grep oel7
Name:            oel7rac1n1
Config file:     /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.vbox
Log folder:      /vbox-repo1/metadata/RacLab1/oel7rac1n1/Logs
SATA Controller (1, 0): /vbox-repo1/disks/oel7rac1n1_localOSdisk1.vdi (UUID: be004bfa-0069-452c-b9b5-2448a988cd3b)
Capture file:       /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.webm
 
  • Start up VM again after the modification.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage startvm oel72rac1n1
Waiting for VM "oel72rac1n1" to power on...
VM "oel72rac1n1" has been successfully started.
 
 
Install additional software
 

I would like to have also some additional pkgs installed like rlwrap, Vbox Guest Additions

 
  • RLWRAP Installation From EPEL

In order to configure EPEL yum repository download and install EPEL repository package

maciek@oel7rac1n1.dba24.pl ~ $ sudo wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
maciek@oel7rac1n1.dba24.pl ~ $ sudo rpm -Uvh epel-release-7-8.noarch.rpm
 

With the repository in place you can install rlwrap using the following command.

 
maciek@oel7rac1n1.dba24.pl ~ $ sudo yum install rlwrap
 

Let’s see the installation process in action

 
maciek@oel7rac1n1.dba24.pl ~ $ wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
--2016-11-15 16:03:58--  http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
Resolving www.mirrorservice.org (www.mirrorservice.org)... 212.219.56.184, 2001:630:341:12::184
Connecting to www.mirrorservice.org (www.mirrorservice.org)|212.219.56.184|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14612 (14K) [application/x-redhat-package-manager]
Saving to: ‘epel-release-7-8.noarch.rpm’

100%[===================================================================================&gt;] 14.612      --.-K/s   in 0,06s

2016-11-15 16:03:59 (240 KB/s) - ‘epel-release-7-8.noarch.rpm’ saved [14612/14612]

maciek@oel7rac1n1.dba24.pl ~ $ sudo rpm -Uvh epel-release-7-8.noarch.rpm
warning: epel-release-7-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:epel-release-7-8                 ################################# [100%]

maciek@oel7rac1n1.dba24.pl ~ $ sudo yum install rlwrap
Loaded plugins: langpacks, ulninfo
epel/x86_64/metalink                                                                                  |  21 kB  00:00:00
epel                                                                                                  | 4.3 kB  00:00:00
ol7_UEKR4                                                                                             | 1.2 kB  00:00:00
ol7_latest                                                                                            | 1.4 kB  00:00:00
epel/x86_64/updateinfo         FAILED
http://ftp.ps.pl/pub/Linux/fedora-epel/7/x86_64/repodata/7eeedbbcd4abfb06739248bf68ee306e4ec49718c86557b1bacc28f456dfd8ba-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
(1/3): epel/x86_64/group_gz                                                                           | 170 kB  00:00:01
(2/3): epel/x86_64/updateinfo                                                                         | 673 kB  00:00:05
(3/3): epel/x86_64/primary_db                                                                         | 4.3 MB  00:00:16
Resolving Dependencies
--&gt; Running transaction check
---&gt; Package rlwrap.x86_64 0:0.42-1.el7 will be installed
--&gt; Processing Dependency: perl(Data::Dumper) for package: rlwrap-0.42-1.el7.x86_64
--&gt; Running transaction check
---&gt; Package perl-Data-Dumper.x86_64 0:2.145-3.el7 will be installed
--&gt; Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================
 Package                           Arch                    Version                         Repository                   Size
=============================================================================================================================
Installing:
 rlwrap                            x86_64                  0.42-1.el7                      epel                         96 k
Installing for dependencies:
 perl-Data-Dumper                  x86_64                  2.145-3.el7                     ol7_latest                   47 k

Transaction Summary
=============================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 143 k
Installed size: 306 k
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7Server/epel/packages/rlwrap-0.42-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for rlwrap-0.42-1.el7.x86_64.rpm is not installed
(1/2): rlwrap-0.42-1.el7.x86_64.rpm                                                                   |  96 kB  00:00:00
(2/2): perl-Data-Dumper-2.145-3.el7.x86_64.rpm                                                        |  47 kB  00:00:00
-----------------------------------------------------------------------------------------------------------------------------
Total                                                                                        260 kB/s | 143 kB  00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-8.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : perl-Data-Dumper-2.145-3.el7.x86_64                                                                       1/2
  Installing : rlwrap-0.42-1.el7.x86_64                                                                                  2/2
  Verifying  : perl-Data-Dumper-2.145-3.el7.x86_64                                                                       1/2
  Verifying  : rlwrap-0.42-1.el7.x86_64                                                                                  2/2

Installed:
  rlwrap.x86_64 0:0.42-1.el7

Dependency Installed:
  perl-Data-Dumper.x86_64 0:2.145-3.el7

Complete!
 

Rlwrap has been installed successfully!

 
  • Install VBox Guest Additions

Accordingly to the Virtualbox documentation chapter 4.1

“As mentioned in Section 1.2, “Some terminology”, the Guest Additions are designed to be installed inside a virtual machine after the guest operating system has been installed. They consist of device drivers and system applications that optimize the guest operating system for better performance and usability. Please see Section 3.1, “Supported guest operating systems” for details on what guest operating systems are fully supported with Guest Additions by VirtualBox.

The VirtualBox Guest Additions for all supported guest operating systems are provided as a single CD-ROM image file which is called VBoxGuestAdditions.iso. This image file is located in the installation directory of VirtualBox. To install the Guest Additions for a particular VM, you mount this ISO file in your VM as a virtual CD-ROM and install from there.”

The Guest Additions offer the following features:

  • Mouse pointer integration
  • Shared folders
  • Better video support
  • Seamless windows
  • Time synchronization
  • Shared clipboard
  • Automated logons (credentials passing)

For more infor please read documentation:
https://www.virtualbox.org/manual/ch04.html

# Attach VBox Guest Addition ISO to the rac node
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage storageattach "oel7rac1n1" --storagectl "IDE Controller"  --port 1 --device 0 --type dvddrive --medium /usr/share/virtualbox/VBoxGuestAdditions.iso

# Mount cdrom on host
maciek@oel7rac1n1.dba24.pl ~ $ sudo mount /dev/sr0 /media/
mount: /dev/sr0 is write-protected, mounting read-only

maciek@oel7rac1n1.dba24.pl /media $ cd /media
maciek@oel7rac1n1.dba24.pl /media $ sudo sh VBoxLinuxAdditions.run --nox11
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.8 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Building Guest Additions kernel modules.
vboxadd.sh: Starting the VirtualBox Guest Additions.

# shutdown VM operating system and remove iso file from drive
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage storageattach "oel7rac1n1" --storagectl "IDE Controller"  --port 1 --device 0 --type dvddrive --medium emptydrive

# startup the VM again and check VB Guest Addition status
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"
......
Facility "VirtualBox Base Driver": active/running (last update: 2016/11/17 14:37:46 UTC)
Facility "VirtualBox System Service": active/running (last update: 2016/11/17 14:37:50 UTC)
Facility "Seamless Mode": not active (last update: 2016/11/17 14:37:46 UTC)
Facility "Graphics Mode": not active (last update: 2016/11/17 14:37:46 UTC)

# Graphic and seamless modes not supported since no X11 server installed on the host

# On the OS directly check if service is running
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status vboxadd-service.service
● vboxadd-service.service
   Loaded: loaded (/opt/VBoxGuestAdditions-5.1.8/init/vboxadd-service; enabled; vendor preset: disabled)
   Active: active (running) since czw 2016-11-17 15:43:45 CET; 47min ago
   CGroup: /system.slice/vboxadd-service.service
           └─774 /usr/sbin/VBoxService --pidfile /var/run/vboxadd-service.sh

lis 17 16:43:44 oel7rac1n1 systemd[1]: Starting vboxadd-service.service...
lis 17 16:43:44 oel7rac1n1 vboxadd-service[764]: vboxadd-service.sh: Starting VirtualBox Guest Addition service.
lis 17 16:43:44 oel7rac1n1 vboxadd-service.sh[775]: VirtualBox Guest Addition service started.
lis 17 15:43:45 oel7rac1n1 systemd[1]: Started vboxadd-service.service.

# How to check running VMs? It is really simple (taken where only openfiler1 was running)
vboxuser1@macieksrv.dba24.pl ~ $ VBoxManage list runningvms
"openfiler1" {002ac87b-755b-46e5-897d-35ebbd5f70f2}
 
 
DNS configuration
 

Dns names resolving configuration is very important for RAC to work properly.
You need to take the most care of the SCAN addresses – they have to be available from a dns server. As far as I know other addresses can be resolved locally. Grid Installer checks if dns is configured properly so this is a mandatory step.

 
  • Because I know all the IPs in advance I will add them to the /etc/hosts file and then use dnsmasq tool to serve as local dns on RAC each node
# Add to /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

# Public interfaces
192.168.1.21 oel7rac1n1.dba24.pl oel7rac1n1
192.168.1.23 oel7rac1n2.dba24.pl oel7rac1n2

# VIP addresses
192.168.1.22 oel7rac1n1-vip.dba24.pl oel7rac1n1-vip
192.168.1.24 oel7rac1n2-vip.dba24.pl oel7rac1n2-vip

# interconnect network for haip subnet1
192.168.20.21 oel7rac1n1-priv1.dba24.pl oel7rac1n1-priv1
192.168.20.23 oel7rac1n2-priv1.dba24.pl oel7rac1n2-priv1

# interconnect network for haip subnet2
192.168.21.21 oel7rac1n1-priv2.dba24.pl oel7rac1n1-priv2
192.168.21.23 oel7rac1n2-priv2.dba24.pl oel7rac1n2-priv2

# asm network subnet1
192.168.30.21 oel7rac1n1-apriv1.dba24.pl oel7rac1n1-apriv1
192.168.30.23 oel7rac1n2-apriv1.dba24.pl oel7rac1n2-apriv1

# asm network subnet2
192.168.31.21 oel7rac1n1-apriv2.dba24.pl oel7rac1n1-apriv2
192.168.31.23 oel7rac1n2-apriv2.dba24.pl oel7rac1n2-apriv2

# storage network 
192.168.10.21 oel7rac1n1-spriv.dba24.pl oel7rac1n1-spriv
192.168.10.23 oel7rac1n2-spriv.dba24.pl oel7rac1n2-spriv

192.168.1.101 rac1-scan.dba24.pl rac1-scan
192.168.1.102 rac1-scan.dba24.pl rac1-scan
192.168.1.103 rac1-scan.dba24.pl rac1-scan

# Openfilers

192.168.1.10 openfiler1
192.168.1.11 openfiler2
192.168.1.12 openfiler3

192.168.10.10 openfiler1-spriv
192.168.10.11 openfiler2-spriv
192.168.10.12 openfiler3-spriv
 
  • As you can see in /etc/resolv.conf system queries gateway 192.168.1.1 for IP addresses by default
maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/resolv.conf

# Generated by NetworkManager
search speedport.ip
nameserver 192.168.1.1
 
  • For local addresses I want to use dnsmasq that will run on 127.0.0.1. I need to block dns that bridged network gets from dhcp and set DNS ip to localhost only. This local dnsmasq ip will be the only one in the /etc/resolve.conf
maciek@oel7rac1n1.dba24.pl ~ $ sudo nmcli con mod bridged ipv4.ignore-auto-dns yes
maciek@oel7rac1n1.dba24.pl ~ $ sudo nmcli con mod bridged ipv4.dns 127.0.0.1
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart network
 
  • As you can see below now localhost is gonna be asked for any addresses so all the queries will go through dnsmasq daemon
maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1
 
  • Now we need to enable dnsmasq. You can read about this daemon here: http://www.dnsmasq.org/
    DNSmasq serves as dns cache running on a local host’s 53rd port, you need to run it on every node in the cluster. DNSmasq serves adresses from /etc/hosts and forwards queries that it cannot resolve locally to the configured DNS servers (reads /etc/resolv.conf to get the list)

First in order to avoid dns queries for addresses in dba24.pl domain, I will add it as local domain that should be resolved from /etc/hosts only. I will change a line in /etc/dnsmasq.conf. Find that part below

# Add local-only domains here, queries in these domains are answered
# from /etc/hosts or DHCP only.
#local=/localnet/

and change into

# Add local-only domains here, queries in these domains are answered
# from /etc/hosts or DHCP only.
local=/dba24.pl/
 

Second I will add file /etc/dnsmasq.d/externalDNS.conf with content

maciek@oel7rac1n4.dba24.pl ~ $ cat /etc/dnsmasq.d/externalDNS.conf
server=192.168.1.1
 

Next change /etc/dnsmasq.conf by adding the line for resolv-file option pointing to the externalDNS file.

# Change this line if you want dns to get its upstream servers from
# somewhere other that /etc/resolv.conf
#resolv-file=
resolv-file=/etc/dnsmasq.d/externalDNS.conf
 
  • Now enable and start dnsmasq
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl enable dnsmasq
Created symlink from /etc/systemd/system/multi-user.target.wants/dnsmasq.service to /usr/lib/systemd/system/dnsmasq.service.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl start dnsmasq
 
  • Check dnsmasq status
maciek@oel7rac1n2.dba24.pl ~ $ sudo systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since pią 2016-12-02 13:59:46 CET; 3s ago
 Main PID: 11243 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           └─11243 /usr/sbin/dnsmasq -k

gru 02 13:59:46 oel7rac1n2 systemd[1]: Started DNS caching server..
gru 02 13:59:46 oel7rac1n2 systemd[1]: Starting DNS caching server....
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: started, version 2.66 cachesize 150
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: using nameserver 192.168.1.1#53
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: using local addresses only for domain dba24.pl
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: no servers found in /etc/dnsmasq.d/externalDNS.conf, will retry
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: read /etc/hosts - 39 addresses


 
  • Test if dns is working properly
# Test DNS
# local addresses
maciek@oel7rac1n1.dba24.pl ~ $ nslookup oel7rac1n1
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   oel7rac1n1
Address: 192.168.1.21

maciek@oel7rac1n1.dba24.pl ~ $ nslookup oel7rac1n2.dba24.pl
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   oel7rac1n2.dba24.pl
Address: 192.168.1.23

# internet address
maciek@oel7rac1n1.dba24.pl ~ $ nslookup facebook.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   facebook.com
Address: 31.13.92.36

# And last but not least verify the holy scan name
maciek@oel7rac1n1.dba24.pl ~ $ nslookup rac1-scan
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac1-scan
Address: 192.168.1.103
Name:   rac1-scan
Address: 192.168.1.101
Name:   rac1-scan
Address: 192.168.1.102

maciek@oel7rac1n1.dba24.pl ~ $ nslookup rac1-scan.dba24.pl
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac1-scan.dba24.pl
Address: 192.168.1.103
Name:   rac1-scan.dba24.pl
Address: 192.168.1.101
Name:   rac1-scan.dba24.pl
Address: 192.168.1.102

Beautifully, works like a charm!!!!

 
 
Configure services
 
 
  • SSH

Add two lines at the end of /etc/ssh/sshd_config and restart sshd server



AddressFamily inet
UseDNS no

First line property is required in a case ipv6 is to be turned off (I will do it soon)
Second line speeds up login operations to the system via ssh – no reverse dns query to find out the hostname for IP of a client

Now we can restart sshd

maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart sshd
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since czw 2016-11-17 16:06:16 CET; 5s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 2801 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2802 (sshd)
   CGroup: /system.slice/sshd.service
           └─2802 /usr/sbin/sshd

lis 17 16:06:16 oel7rac1n1 systemd[1]: Starting OpenSSH server daemon...
lis 17 16:06:16 oel7rac1n1 sshd[2802]: Server listening on 0.0.0.0 port 22.
lis 17 16:06:16 oel7rac1n1 systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
lis 17 16:06:16 oel7rac1n1 systemd[1]: Started OpenSSH server daemon.
 
  • TUNED

I want to disable tuned, don’t need to change performance settings on the fly

maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl stop tuned
[sudo] password for maciek:
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl disable tuned
Removed symlink /etc/systemd/system/multi-user.target.wants/tuned.service.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status tuned
● tuned.service - Dynamic System Tuning Daemon
   Loaded: loaded (/usr/lib/systemd/system/tuned.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

lis 17 15:43:56 oel7rac1n1 systemd[1]: Starting Dynamic System Tuning Daemon...
lis 17 15:43:57 oel7rac1n1 systemd[1]: Started Dynamic System Tuning Daemon.
lis 17 16:18:58 oel7rac1n1 systemd[1]: Stopping Dynamic System Tuning Daemon...
lis 17 16:18:59 oel7rac1n1 systemd[1]: Stopped Dynamic System Tuning Daemon.
 
  • NTPD

For virtualbox the time is provided by Vbox Guest Additions, but in order to install Grid Infrastructure we need NTPD to be running or CTSS cluster service configured. I choose to disable NTPD and use CTSS, because I have no time servers configured on the VB Host (my ubuntu machine) and I don’t really want to play with it.

maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status ntpd
Unit ntpd.service could not be found.

Ok ntpd not configured, don’t need to disable it then

 
  • FIREWALL

Firewalld needs to be stopped in order to not to interfer the cluster communication.
There are tutorials how to keep it, but for lab this is not necessary.

maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl stop firewalld
[sudo] password for maciek:
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

lis 17 16:43:43 oel7rac1n1 systemd[1]: Starting firewalld - dynamic firewall daemon...
lis 17 15:43:47 oel7rac1n1 systemd[1]: Started firewalld - dynamic firewall daemon.
lis 17 16:43:26 oel7rac1n1 systemd[1]: Stopping firewalld - dynamic firewall daemon...
lis 17 16:43:26 oel7rac1n1 systemd[1]: Stopped firewalld - dynamic firewall daemon.
 
  • iSCSI

Perform following steps

# Check if iscsi initiantors has been installed
maciek@oel7rac1n1.dba24.pl ~ $ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-35.0.1.el7 (x86_64)
iscsi-initiator-utils-iscsiuio-6.2.0.873-35.0.1.el7 (x86_64)

# check iscsid status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:iscsid(8)
           man:iscsiadm(8)

# enable iscsid and check status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl enable iscsid
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:iscsid(8)
           man:iscsiadm(8)

# start iscsid and check its status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl start iscsid
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running) since czw 2016-11-17 16:48:21 CET; 1s ago
     Docs: man:iscsid(8)
           man:iscsiadm(8)
  Process: 3854 ExecStart=/usr/sbin/iscsid (code=exited, status=0/SUCCESS)
 Main PID: 3856 (iscsid)
   CGroup: /system.slice/iscsid.service
           ├─3855 /usr/sbin/iscsid
           └─3856 /usr/sbin/iscsid

# check again if both iscsid and iscsi have been enabled
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl is-enabled iscsid.service
enabled
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl is-enabled iscsi.service
enabled
 
 
System confguration
 
 
 
  • TRANSPARENT HUGEPAGES

Disable transparent hugepages. First check current settings.


maciek@oel7rac1n1.dba24.pl ~ $ sudo ls -l /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/redhat_transparent_hugepage
ls: cannot access /sys/kernel/mm/transparent_hugepage: No such file or directory
ls: cannot access /sys/kernel/mm/redhat_transparent_hugepage: No such file or directory

“If Transparent HugePages is removed from the kernel then the /sys/kernel/mm/transparent_hugepage or /sys/kernel/mm/redhat_transparent_hugepage files do not exist.”

Docs say also that Transparent HugePages are disabled by default in later releases of Oracle Linux https://docs.oracle.com/database/121/UNXAR/appi_vlm.htm#UNXAR428

 
  • IO SCHEDULER

For best performance for Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler https://docs.oracle.com/database/121/CWLIN/prelinux.htm#CHDCEBCD

Check which of the IO schedulers is set at the moment

maciek@oel7rac1n1.dba24.pl ~ $ cat /sys/block/sda/queue/scheduler
noop deadline [cfq]

CFQ is set by default.

In order to enable deadline scheduler we need to tweak grub. Remember: do not change /etc/grub2.cfg file, because it is automatically generated by grub2-mkconfig using templates from /etc/grub.d and settings from /etc/default/grub. Change /etc/default/grub instead.

Change the line

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never"

into

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never elevator=deadline"

Backup current /etc/grub2.cfg file somewhere. After you have modified /etc/default/grub run grub2-mkconfig.

maciek@oel7rac1n1.dba24.pl ~ $ sudo cp /etc/grub2.cfg /root/grub2.cfg.backup

maciek@oel7rac1n1.dba24.pl ~ $ sudo grub2-mkconfig -o /etc/grub2.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.8.13-118.14.1.el7uek.x86_64
Found initrd image: /boot/initramfs-3.8.13-118.14.1.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-3.8.13-98.7.1.el7uek.x86_64
Found initrd image: /boot/initramfs-3.8.13-98.7.1.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-9d9f28c6023b438bb429d70373246640
Found initrd image: /boot/initramfs-0-rescue-9d9f28c6023b438bb429d70373246640.img
done

Reboot OS and recheck the io scheduler set

maciek@oel7rac1n1.dba24.pl ~ $ cat /sys/block/sda/queue/scheduler
noop [deadline] cfq

Deadline io scheduler has been set successfully.

 
  • SHM FILESYSTEM

The plan is to use HugePages for DB, but for ASM Oracle recommends Automatic Memory Management (memory_* params).

How it looks now

maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep shm
tmpfs                 2972032       0   2972032   0% /dev/shm

Well, I just need 2G not more. Add shm tmpfs filesystem to the /etc/fstab, 2GB of size and remount it

#
# /etc/fstab
# Created by anaconda on Sun Nov 13 18:01:11 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=630aa2e1-384a-4370-8721-cad3a7b92732 /boot                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0

tmpfs                   /dev/shm                tmpfs   defaults,size=2G  0 0

maciek@oel7rac1n1.dba24.pl ~ $ sudo mount -o remount /dev/shm

# and after remounting
maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep shm
tmpfs                 2097152       0   2097152   0% /dev/shm

That should be enough

 
  • SELINUX

To disable SELINUX modify /etc/selinux/config file.

Change the following line:

SELINUX=enforcing

into

SELINUX=disabled

Remember it will take effect after a next reboot, this is why we need to turn the SELINUX temporarily into PERMISSIVE mode if we are not able to restart the box immediately

maciek@oel7rac1n1.dba24.pl ~ $ sudo setenforce 0
maciek@oel7rac1n1.dba24.pl ~ $ sudo getenforce
Permissive

Check SELINUX status

maciek@oel7rac1n1.dba24.pl ~ $ sudo sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          disabled
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

If you are allowed to reboot OS do it asap, then check status again

maciek@oel7rac1n1.dba24.pl ~ $ sudo selinuxenabled &amp;&amp; echo enabled || echo disabled
disabled
maciek@oel7rac1n1.dba24.pl ~ $ getenforce
Disabled
maciek@oel7rac1n1.dba24.pl ~ $ sestatus
SELinux status:                 disabled
 
  • Create and set login information banner file

Create or edit banner file /etc/issue

#######################################################################
#                          Welcome to DBA24 lab
#   Access to this system is permitted only for authorized personel.
#       All actions made on this system are monitored and logged.
#  ***  Disconnect IMMEDIATELY if you are not an authorized user  ***
#######################################################################

Add the following line to the /etc/ssh/sshd_config file if doesn’t exist

Banner /etc/issue

Restart sshd

maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart sshd

Check if banner shows up – try to ssh to the machine

maciek@macieksrv.dba24.pl ~ $ ssh 192.168.1.21
#######################################################################
#                          Welcome to DBA24 lab
#   Access to this system is permitted only for authorized personel.
#       All actions made on this system are monitored and logged.
#  ***  Disconnect IMMEDIATELY if you are not an authorized user  ***
#######################################################################
maciek@192.168.1.21:

It works!! :)

 
  • SYSTEM SETTINGS /etc/sysctl.conf

Remove /etc/sysctl.d/99-sysctl.conf the file was created by oracle-rdbms-server-12cR1-preinstall.x86_64 package

[root@oel7rac1n1 ~]# rm /etc/sysctl.d/99-sysctl.conf

Create configuration file /etc/sysctl.d/98-oracle.conf and put into it settings overriding default ones. We need theme to allow our Oracle software to work properly and efficiently

So first create empty file

maciek@oel7rac1n1.dba24.pl ~ $ sudo touch /etc/sysctl.d/98-oracle.conf

Then put following settings into this file, remember to tweak memory settings accordingly to the memory size of your VM or your needs. My VM has got 6GB of RAM assigned.

# Change IP fingerprint
net.ipv4.tcp_keepalive_time=30
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.tcp_max_tw_buckets=1440000

# Deny source-routed packages
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.all.forwarding=0

# Deny icmp netmask &amp; timestamp broadcast
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.tcp_timestamps=0

# Deny IP redirects
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.all.send_redirects=0

# Bufferbloat
kernel.random.write_wakeup_threshold=1024
vm.swappiness=1
vm.dirty_background_ratio=3
vm.dirty_ratio=80
vm.dirty_expire_centisecs=500
vm.dirty_writeback_centisecs=100
vm.min_free_kbytes=524228
kernel.sem = 1510 193280 1510 128
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048576
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=9
net.ipv4.tcp_retries2=3
net.ipv4.tcp_syn_retries=2
net.ipv4.tcp_rmem=4096 262144 4194304
net.ipv4.tcp_wmem=4096 262144 4194304
fs.file-max=6815744
fs.aio-max-nr=1048576
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.default.log_martians=1
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_ignore_bogus_error_responses=1
kernel.msgmax=8192
kernel.msgmnb=65536

# RAM 64GB
#kernel.shmmax=56371445760
#kernel.shmall=13762560
# RAM 4GB
#kernel.shmmax=3523215360
#kernel.shmall=860160

# RAM 6GB 
kernel.shmmax=5284741120
kernel.shmall=1290220

vm.hugetlb_shm_group=2020
vm.nr_hugepages=1024

To enable above settings please run the following command or reboot your server

maciek@oel7rac1n1.dba24.pl ~ $ sudo sysctl -p /etc/sysctl.d/98-oracle.conf

I have met the opinion, each time you set HugePages you should reboot your server.
I will do it as some really wise guys recommend it.

After the reboot let’s check in example if hugepages has been set properly

maciek@oel7rac1n1.dba24.pl ~ $ cat /proc/meminfo |grep Huge
AnonHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Yup looks as expected. Of course you can check all the params from the Oracle documentation related to the database and grid infrastructure installation. The above settings were tailored to the 12c requirements, but they will be verified later by the cluvfy tool

 
  • SECURITY LIMITS

As it was with system setting let’s create file /etc/security/limits.d/99-oracle.conf

maciek@oel7rac1n1.dba24.pl ~ $ sudo touch /etc/security/limits.d/99-oracle.conf

Add the following content into the file

*               soft    core            0
*               hard    core            0

# Specific setting for oracle users
@oinstall       hard    nofile  65536
@oinstall       soft    nofile  4096
@oinstall       soft    nproc   131072
@oinstall       hard    nproc   131072
@oinstall       soft    stack   32768
@oinstall       hard    stack   32768

# for 6GB servers
@oinstall        soft     memlock  5662310
@oinstall        hard     memlock  5662310

Modify memlock settings accordingly to your RAM size

 
  • GROUPS

Create all the below groups on the Oracle database servers. Please keep the GID numbers, this is important to keep the standard in an environment.

GID Group Name
2001 oinstall
2002 dba
2003 oper
2004 backupdba
2005 dgdba
2006 kmdba
2007 asmadmin
2008 asmoper
2009 asmdba
2020 hugetlb

As you can see groups oinstall and dba have already been created by the oracle-rdbms-server-12cR1-preinstall.x86_64 package.

maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group
.........
oinstall:x:54321:
dba:x:54322:oracle
......

I need to change their gids to fit to my standard

maciek@oel7rac1n1.dba24.pl ~ $ sudo groupmod -g 2001 oinstall
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupmod -g 2002 dba
maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group|grep oin
oinstall:x:2001:
maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group|grep dba
dba:x:2002:oracle

Now let’s add other groups

maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2003 oper
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2004 backupdba
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2005 dgdba
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2006 kmdba
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2007 asmadmin
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2008 asmoper
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2009 asmdba
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2020 hugetlb

As you can see all the groups are in place

maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group |tail -11
oinstall:x:2001:
dba:x:2002:oracle
vboxsf:x:992:
oper:x:2003:
backupdba:x:2004:
dgdba:x:2005:
kmdba:x:2006:
asmadmin:x:2007:
asmoper:x:2008:
asmdba:x:2009:
hugetlb:x:2020:
 
  • USERS

We need two users for our installation – grid for Grid Infrastructure and oracle for Oracle Database sofware. I like this idea of roles separation and have been using it from the time this concept apeared with Oracle 11R2.

As you can see below user oracle was created by oracle-rdbms-server-12cR1-preinstall.x86_64 package.

maciek@oel7rac1n1.dba24.pl ~ $ sudo id -a oracle
uid=54321(oracle) gid=2001(oinstall) groups=2001(oinstall),2002(dba)

I want to change its id accordingly to my requirements and add user oracle to the required groups

Users required:

UID Username Main Group Other Groups
2001 oracle oinstall dba,oper,backupdba,dgdba,kmdba,hugetlb,asmdba
2002 grid oinstall dba,asmadmin,asmoper,asmdba

Modify oracle user

maciek@oel7rac1n1.dba24.pl ~ $ sudo usermod -a -u 2001 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,hugetlb,asmdba oracle
maciek@oel7rac1n1.dba24.pl ~ $ id -a oracle
uid=2001(oracle) gid=2001(oinstall) groups=2001(oinstall),2002(dba),2003(oper),2004(backupdba),2005(dgdba),2006(kmdba),2009(asmdba),2020(hugetlb)

Add new grid user

maciek@oel7rac1n1.dba24.pl ~ $ sudo useradd -u 2002 -m -s /bin/bash -g oinstall -G dba,asmadmin,asmoper,asmdba grid

maciek@oel7rac1n1.dba24.pl ~ $ id -a grid
uid=2002(grid) gid=2001(oinstall) groups=2001(oinstall),2002(dba),2007(asmadmin),2008(asmoper),2009(asmdba)
 

Important: Remember to set passwords for oracle and grid, ssh user equivalency won’t work without it

 
  • ULIMITS

Set ulimits for oracle and grid. Create file /etc/profile.d/oracle-grid.sh with the following settings

#Setting the appropriate ulimits for oracle and grid user
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
                ulimit -u 16384
                ulimit -n 65536
        else
                ulimit -u 16384 -n 65536
        fi
fi

if [ $USER = "grid" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
                ulimit -u 16384
                ulimit -n 65536
        else
                ulimit -u 16384 -n 65536
        fi
fi
 
  • UMASK

Check if umask settings for grid and oracle are set to 0022

maciek@oel7rac1n1.dba24.pl ~ $ sudo runuser -l oracle -c 'umask'
0022
maciek@oel7rac1n1.dba24.pl ~ $ sudo runuser -l grid -c 'umask'
0022

Settings are correct!!

 
  • PAM

Verify the latest version of PAM is loaded, then add or edit the following line in the /etc/pam.d/login file, if it does not already exist:

session required pam_limits.so
 
  • REMOVE IPC

We need RemoveIPC explicitly set to “no” in /etc/systemd/logind.conf. Verify current settings:

maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/systemd/logind.conf |grep Rem
#RemoveIPC=no
RemoveIPC=no

As you can see in 7.2 RemoveIPC is set to now by default. We don’t need to change anything then.

 
 
 
Prepare directories for RAC installation
 
 
  • Run the following script to get the desired directories (as root)
#!/bin/bash

[ -d "/u01/app/grid" ] &amp;&amp; echo "Directory /u01/app/grid exists" || ( mkdir -p /u01/app/grid; chown grid:oinstall /u01/app/grid;chmod 770 /u01/app/grid )
[ -d "/u01/app/oracle" ] &amp;&amp; echo "Directory /u01/app/oracle exists" || ( mkdir -p /u01/app/oracle; chown oracle:oinstall /u01/app/oracle;chmod 770 /u01/app/oracle )

[ -d "/u01/app/grid/grid-software" ] &amp;&amp; echo "Directory /u01/app/grid/grid-software exists" || ( mkdir -p /u01/app/grid/grid-software; chown grid:oinstall /u01/app/grid/grid-software )
[ -d "/u01/app/oracle/oracle-software" ] &amp;&amp; echo "Directory /u01/app/oracle/oracle-software exists" || ( mkdir -p /u01/app/oracle/oracle-software; chown oracle:oinstall /u01/app/oracle/oracle-software )

chown  root:oinstall /u01
chown  root:oinstall /u01/app
chmod  750 /u01
chmod  770 /u01/app

# RAC specific
[ -d "/u01/app/12.1.0/grid" ] &amp;&amp; echo "Directory /u01/app/12.1.0/grid exists" || ( mkdir -p /u01/app/12.1.0/grid; chown grid:oinstall /u01/app/12.1.0/grid )


# link sudo to /usr/local/bin directory – needed for cluvfy to run properly
[ -h "/usr/local/bin/sudo" ] &amp;&amp; echo "Symlink /usr/local/bin/sudo exists" || ln -s /usr/bin/sudo /usr/local/bin/sudo

# dirs for cluvfy
[ -d "/u01/app/grid/cluvfy" ] &amp;&amp; echo "Directory /u01/app/grid/cluvfy exists" || ( mkdir -p /u01/app/grid/cluvfy  )

[ -h "/u01/app/oracle/cluvfy" ] &amp;&amp; echo "Symbolic link /u01/app/oracle/cluvfy exists" || (  ln -s /u01/app/grid/cluvfy /u01/app/oracle/cluvfy )

chown  -R grid:oinstall /u01/app/grid/cluvfy/
chmod  770 /u01/app/grid/cluvfy

#dirs for TFA
[ -d "/u01/app/tfa" ] &amp;&amp; echo "Directory /u01/app/tfa exists" || ( mkdir -p /u01/app/tfa  )
chown  -R root:root /u01/app/tfa/
chmod  755 /u01/app/tfa

# Add sudo entry for ORACHK (part of TFA)
echo '@oinstall ALL=(root) NOPASSWD:/home/oracle/orachk/root_orachk.sh' &gt;&gt; /etc/sudoers
=====================================================

Check the if directories have been created

maciek@oel7rac1n1.dba24.pl ~ $ sudo find  /u01 -type d
/u01
/u01/app
/u01/app/grid
/u01/app/grid/grid-software
/u01/app/grid/cluvfy
/u01/app/oracle
/u01/app/oracle/oracle-software
/u01/app/12.1.0
/u01/app/12.1.0/grid
/u01/app/tfa

OK!

 
 
 
CLUVFY
 
  • Install Cluster Verification Utility

Download the latest cluvfy from
http://www.oracle.com/technetwork/products/clustering/downloads/cvu-download-homepage-099973.html

cluvfyscreen

Install cluvfy

maciek@oel7rac1n1.dba24.pl ~ $ sudo unzip /home/maciek/cvupack_Linux_x86_64.zip -d /u01/app/grid/cluvfy/
maciek@oel7rac1n1.dba24.pl ~ $ sudo chown  -R grid:oinstall /u01/app/grid/cluvfy/
maciek@oel7rac1n1.dba24.pl ~ $ sudo chmod  770 /u01/app/grid/cluvfy

Install cvuqdisk tool from rpm provided by the cluvfy package (as root)

[root@oel7rac1n1 ~]# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP;
[root@oel7rac1n1 ~]# rpm -i /u01/app/grid/cluvfy/cv/remenv/cvuqdisk-1.0.9-1.rpm
 
 
 
 

The OS is ready now for Oracle 12c. To be able to install GI and RAC db we need some iSCSI storage presented by Openfiler servers (openfiler1, openfiler2,openfiler3). Next parts of the series will guide you through the process of openfiler storage configuration, openfiler and rac node VM cloning. I will also show you how to present and attach iSCSI disks for RAC nodes.

 

Stay tuned!!!

 
 
What do you think?? Please post your comments :)
Source: My experience and WorlWideWeb

About the author

 
maciej tokar
Maciej Tokar

An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU

 
View Maciej Tokar's profile on LinkedIn         logoDB4
LinkedIn Auto Publish Powered By : XYZScripts.com