This is sixth part of my RAC lab series. I will show you step by step how to prepare your OEL 7.x OS for GI and RAC installation. VM with OEL 7.2 in part 3 of the series will be used to show this.
RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database
Oracle 12c official documentation
Please refer to the official 12.1 documentation to verify current OS requirements:
- Grid Infrastructure Installation Guide
https://docs.oracle.com/database/121/CWLIN/toc.htm
- Real Application Clusters Installation Guide
https://docs.oracle.com/database/121/RILIN/toc.htm
- Database Installation Guide
https://docs.oracle.com/database/121/LADBI/toc.htm
Oracle Enterprise Linux 7.2 Network Basic Configuration
- If oel72rac1n1 VM is not running please start it using vboxmanage in the cli.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage startvm oel72rac1n1 Waiting for VM "oel72rac1n1" to power on... VM "oel72rac1n1" has been successfully started.
-
Before I do anything with the OS, I need VM’s network to be functional.
Let’s check with vboxmanage showvminfo what kind of interfaces we got for this VM, and what they MAC addresses are.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel72rac1n1"|grep NIC NIC 1: MAC: 0800276A189E, Attachment: Bridged Interface 'wlx98ded00b5b05', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: den y, Bandwidth group: none NIC 2: MAC: 080027DB1740, Attachment: Internal Network 'storage-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: den y, Bandwidth group: none NIC 3: MAC: 080027D59C97, Attachment: Internal Network 'priv1-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none NIC 4: MAC: 080027AE9147, Attachment: Internal Network 'priv2-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none NIC 5: MAC: 08002748771B, Attachment: Internal Network 'apriv1-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny , Bandwidth group: none NIC 6: MAC: 0800271A1EE2, Attachment: Internal Network 'apriv2-internal', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny , Bandwidth group: none
- Let’s clean it up manually a little to get clear output:
NIC 1: MAC: 0800276A189E Bridged Interface 'wlx98ded00b5b05' NIC 2: MAC: 080027DB1740 Internal Network 'storage-internal' NIC 3: MAC: 080027D59C97 Internal Network 'priv1-internal' NIC 4: MAC: 080027AE9147 Internal Network 'priv2-internal' NIC 5: MAC: 08002748771B Internal Network 'apriv1-internal' NIC 6: MAC: 0800271A1EE2 Internal Network 'apriv2-internal'
- Now compare macs to find out NIC vs dev name in the using the “ip addr” command output text (on the VM)
2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:6a:18:9e brd ff:ff:ff:ff:ff:ff
3: enp0s8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:db:17:40 brd ff:ff:ff:ff:ff:ff
4: enp0s9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d5:9c:97 brd ff:ff:ff:ff:ff:ff
5: enp0s10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ae:91:47 brd ff:ff:ff:ff:ff:ff
6: enp0s16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:48:77:1b brd ff:ff:ff:ff:ff:ff
7: enp0s17: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:1a:1e:e2 brd ff:ff:ff:ff:ff:ff
- What gives us after some analysis a list of interface <-> network relationship
NIC 1: MAC: 0800276A189E Bridged Interface 'wlx98ded00b5b05' enp0s3 dhcp NIC 2: MAC: 080027DB1740 Internal Network 'storage-internal' enp0s8 192.168.10.x NIC 3: MAC: 080027D59C97 Internal Network 'priv1-internal' enp0s9 192.168.20.x NIC 4: MAC: 080027AE9147 Internal Network 'priv2-internal' enp0s10 192.168.21.x NIC 5: MAC: 08002748771B Internal Network 'apriv1-internal' enp0s16 192.168.30.x NIC 6: MAC: 0800271A1EE2 Internal Network 'apriv2-internal' enpos17 192.168.31.x
So now as I know that I need to get IP from dhcp for bridged interface I take MAC address of the bridged interface (0800276A189E) and register a static IP for it in my DHCP server (192.168.1.21).
After having done that, I can proceed with configuration of the network interfaces
-
First we need to remove incorrectly configured connections – done by the installer.
Look how it looks now:
[maciek@oel72rac1n1 ~]# sudo nmcli con show NAME UUID TYPE DEVICE enp0s10 528e61d5-c88d-4ebc-bd73-bbece03229f0 802-3-ethernet -- enp0s16 41e299d3-b303-4aef-a7ab-6356166a7af8 802-3-ethernet -- enp0s3 2093693b-13c3-4fbc-99f2-5b4eee6895b5 802-3-ethernet -- enp0s8 4460b313-5b21-4dce-b816-ea780ffc120e 802-3-ethernet -- enp0s9 3e0d7ed2-4109-4682-a79a-7b370b91dd75 802-3-ethernet -- enp0s17 9b951f52-e1bb-4863-a6d5-64b2c7670d60 802-3-ethernet --
- Remove those networks – not sure the reason they were configured like this, but I want to get rid of them and configure networks from scratch
[maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s3 [maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s8 [maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s9 [maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s10 [maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s16 [maciek@oel72rac1n1 ~]# sudo nmcli con del enp0s17
- After having it cleared, let’s add the networks back with proper config
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name bridged ifname enp0s3 type ethernet
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name storage-internal ifname enp0s8 type ethernet ip4 "192.168.10.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name priv1-internal ifname enp0s9 type ethernet ip4 "192.168.20.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name priv2-internal ifname enp0s10 type ethernet ip4 "192.168.21.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name apriv1-internal ifname enp0s16 type ethernet ip4 "192.168.30.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con add con-name apriv2-internal ifname enp0s17 type ethernet ip4 "192.168.31.21/24"
[maciek@oel72rac1n1 ~]# sudo nmcli con show
NAME UUID TYPE DEVICE
priv1-internal 0acddf7c-be94-49ec-8d05-2fc6bf10d25a 802-3-ethernet enp0s9
storage-internal 4d92bf55-a505-405d-b4a2-fd36cfbef270 802-3-ethernet enp0s8
bridged 5851f1af-fb84-4bc1-bc3f-394ab7524714 802-3-ethernet enp0s3
apriv1-internal 87d2f222-d768-4fd6-be6d-96a10d198922 802-3-ethernet enp0s16
apriv2-internal c23f2a48-d4e7-4c4d-8c67-170ef5ae9118 802-3-ethernet enp0s17
priv2-internal 518cd49c-51fb-482f-9bf1-19af3c462a6d 802-3-ethernet enp0s10
[maciek@oel72rac1n1 ~]# sudo ip addr|grep -v inet6|grep -v lft
1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:6a:18:9e brd ff:ff:ff:ff:ff:ff
inet 192.168.1.21/24 brd 192.168.1.255 scope global enp0s3
3: enp0s8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:db:17:40 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.21/24 brd 192.168.10.255 scope global enp0s8
4: enp0s9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d5:9c:97 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.21/24 brd 192.168.20.255 scope global enp0s9
5: enp0s10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ae:91:47 brd ff:ff:ff:ff:ff:ff
inet 192.168.21.21/24 brd 192.168.21.255 scope global enp0s10
6: enp0s16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:48:77:1b brd ff:ff:ff:ff:ff:ff
inet 192.168.30.21/24 brd 192.168.30.255 scope global enp0s16
7: enp0s17: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:1a:1e:e2 brd ff:ff:ff:ff:ff:ff
inet 192.168.31.21/24 brd 192.168.31.255 scope global enp0s17
As you can see all IPs are set, the bridged interface enp0s3 has got it’s IP from the DHCP server
- It is good to perform some basic test now. First let’s try to ping some url on the internet and then openfiler1 interface on the storage network.
# check internet first [maciek@oel72rac1n1 ~]# sudo ping onet.pl PING onet.pl (213.180.141.140) 56(84) bytes of data. 64 bytes from sg1.any.onet.pl (213.180.141.140): icmp_seq=1 ttl=57 time=260 ms # OK!!! # Now check if openfiler1 is visible [maciek@oel72rac1n1 ~]# sudo ping 192.168.10.10 PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data. 64 bytes from 192.168.10.10: icmp_seq=1 ttl=64 time=1.28 ms
Great!!!, the rest of the networks to be checked after the second rac node is up and running
Installation of required missing packages and OEL 7.2 system update
We will get back to the network configuration later, but now let’s add some software to the system. Remember, we have installed just the minimal software pack for the system, we lack some packages required according to the oracle documentation and some others that can be useful.
- Let’s check available software groups including hidden groups
[maciek@oel72rac1n1 ~]$ sudo yum grouplist hidden Loaded plugins: ulninfo There is no installed groups file. Maybe run: yum groups mark convert (see man yum) ol7_UEKR3 | 1.2 kB 00:00:00 ol7_latest | 1.4 kB 00:00:00 (1/5): ol7_UEKR3/x86_64/updateinfo | 69 kB 00:00:00 (2/5): ol7_latest/x86_64/group | 681 kB 00:00:03 (3/5): ol7_latest/x86_64/updateinfo | 1.2 MB 00:00:06 (4/5): ol7_UEKR3/x86_64/primary | 22 MB 00:00:42 (5/5): ol7_latest/x86_64/primary | 21 MB 00:00:40 ol7_UEKR3 525/525 ol7_latest 17829/17829 Available Environment Groups: Minimal Install Infrastructure Server File and Print Server Basic Web Server Virtualization Host Server with GUI Available Groups: Additional Development Anaconda Tools Backup Client Backup Server Base Common NetworkManager submodules Compatibility Libraries Conflicts (Server) Console Internet Tools Core DNS Name Server Debugging Tools Desktop Debugging and Performance Tools Development Tools Dial-up Networking Support Directory Client Directory Server E-mail Server Emacs FTP Server File and Storage Server Fonts GNOME Graphical Administration Tools Graphics Creation Tools Guest Agents Guest Desktop Agents Hardware Monitoring Utilities High Availability Identity Management Server Infiniband Support Input Methods Internet Browser Java Platform KDE Large Systems Performance Legacy UNIX Compatibility Legacy X Window System Compatibility Load Balancer Mainframe Access MariaDB Database Client MariaDB Database Server Multimedia MySQL Database Client MySQL Database Server Network File System Client Network Infrastructure Server Networking Tools PHP Support Performance Tools Perl Support Perl for Web Platform Development PostgreSQL Database Client PostgreSQL Database Server Print Server Printing Client Python Remote Desktop Clients Remote Management for Linux Resilient Storage Ruby Support Scientific Support Security Tools Smart Card Support System Administration Tools System Management Technical Writing Virtualization Client Virtualization Hypervisor Virtualization Platform Virtualization Tools Web Server Web Servlet Engine X Window System Done
- What we generally need to install is the base group – a lot of packages are installed by this group.
[maciek@oel72rac1n1 ~]$ sudo yum groupinstall base Loaded plugins: ulninfo There is no installed groups file. Maybe run: yum groups mark convert (see man yum) Resolving Dependencies --> Running transaction check ---> Package abrt-addon-ccpp.x86_64 0:2.1.11-45.0.1.el7 will be installed --> Processing Dependency: abrt = 2.1.11-45.0.1.el7 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: abrt-libs = 2.1.11-45.0.1.el7 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: gdb >= 7.6.1-63 for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: libreport-python for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: elfutils for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: libsatyr.so.3()(64bit) for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 --> Processing Dependency: libabrt.so.0()(64bit) for package: abrt-addon-ccpp-2.1.11-45.0.1.el7.x86_64 ......... lines removed for better clarity ......................... --> Processing Dependency: iw for package: crda-3.13_2016.02.08-1.el7.x86_64 ---> Package cryptsetup.x86_64 0:1.7.2-1.el7 will be installed --> Processing Dependency: cryptsetup-libs(x86-64) = 1.7.2-1.el7 for package: cryptsetup-1.7.2-1.el7.x86_64 ---> Package cyrus-sasl-plain.x86_64 0:2.1.26-20.el7_2 will be installed --> Processing Dependency: cyrus-sasl-lib(x86-64) = 2.1.26-20.el7_2 for package: cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 ---> Package dmraid.x86_64 0:1.0.0.rc16-28.el7 will be installed --> Processing Dependency: dmraid-events for package: dmraid-1.0.0.rc16-28.el7.x86_64 ---> Package dosfstools.x86_64 0:3.0.20-9.el7 will be installed ---> Package ed.x86_64 0:1.9-4.el7 will be installed ---> Package fprintd-pam.x86_64 0:0.5.0-4.0.el7_0 will be installed --> Processing Dependency: fprintd = 0.5.0-4.0.el7_0 for package: fprintd-pam-0.5.0-4.0.el7_0.x86_64 ---> Package hunspell.x86_64 0:1.3.2-15.el7 will be installed --> Processing Dependency: hunspell-en-US for package: hunspell-1.3.2-15.el7.x86_64 --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================== Package Arch Version Repository Size ==================================================================================================================== Installing for group install "Base": abrt-addon-ccpp x86_64 2.1.11-45.0.1.el7 ol7_latest 190 k abrt-addon-python x86_64 2.1.11-45.0.1.el7 ol7_latest 100 k abrt-cli x86_64 2.1.11-45.0.1.el7 ol7_latest 85 k abrt-console-notification x86_64 2.1.11-45.0.1.el7 ol7_latest 86 k at x86_64 3.1.13-22.el7 ol7_latest 50 k attr x86_64 2.4.46-12.el7 ol7_latest 65 k bash-completion noarch 1:2.1-6.el7 ol7_latest 85 k ......... lines removed for better clarity ......................... yum noarch 3.4.3-150.0.1.el7 ol7_latest 1.2 M Transaction Summary ==================================================================================================================== Install 67 Packages (+101 Dependent packages) Upgrade ( 13 Dependent packages) Total download size: 61 M Is this ok [y/d/N]: Complete!
- Install some additional required packages
[maciek@oel72rac1n1 ~]$ sudo yum install ksh rsh nfs-utils java telnet [maciek@oel72rac1n1 ~]$ sudo yum install xorg-x11-xauth xterm xinetd libXxf86misc libdmx xorg-x11-utils [maciek@oel72rac1n1 ~]$ sudo yum install device-mapper-multipath sg3_utils* iscsi-initiator-utils [maciek@oel72rac1n1 ~]$ sudo yum install compat-libstdc++-33.i686 compat-libstdc++-33.x86_64 [maciek@oel72rac1n1 ~]$ sudo yum install libstdc++-devel.i686 libstdc++-devel.x86_64 [maciek@oel72rac1n1 ~]$ sudo yum install gcc gcc-c++.x86_64 compat-libcap1.x86_64 glibc-devel.i686 [maciek@oel72rac1n1 ~]$ sudo yum install libaio.i686 libaio-devel.i686 libaio-devel.x86_64 [maciek@oel72rac1n1 ~]$ sudo yum install libXext.i686 libXtst.i686 libXp.i686 libXp.x86_64 [maciek@oel72rac1n1 ~]$ sudo yum install elfutils-libelf-devel elfutils-libelf-devel.i686 unixODBC-devel unixODBC-devel.i686 unixODBC [maciek@oel72rac1n1 ~]$ sudo yum install compat-db47 compat-gcc-44 compat-gcc-44-c++ compat-db47.i686 [maciek@oel72rac1n1 ~]$ sudo yum install oracle-rdbms-server-12cR1-preinstall.x86_64
- Install packages required for VboxAdditions
[maciek@oel72rac1n1 ~]$ sudo yum install kernel-uek-devel-*uek.x86_64
-
Update whole OS, the update process will update packages and linux kernel.
Make sure you have ol7_UEKR3 public repository enabled in /etc/yum.repos.d/
[maciek@oel72rac1n1 ~]$ sudo yum update
- Check your kernel version before you restart the server after the update has completed
[maciek@oel72rac1n1 ~]$ uname -a Linux oel72rac1n1.dba24.pl 3.8.13-98.7.1.el7uek.x86_64 #2 SMP Wed Nov 25 13:51:41 PST 2015 x86_64 x86_64 x86_64 GNU/Linux
- After restart you check it again
[maciek@oel72rac1n1 ~]$ uname -a Linux oel72rac1n1.dba24.pl 3.8.13-118.14.1.el7uek.x86_64 #2 SMP Mon Oct 31 17:32:03 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux [root@oel72rac1n1 log]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo)
Wow, during the update process, OS version has been upgraded from 7.2 – 7.3 so the server and VM name is no longer valid. I need to change it maybe to oel7rac1n1 to avoid doubts after any future updates that upgrade my OS to higher version:)
Hostname and VM name change
- Ok so first we change hostname
maciek@oel7rac1n1 ~ $ sudo hostnamectl set-hostname oel7rac1n1
- Then change domain by adding one line to the /etc/hosts for 192.168.1.21 IP. Additionally let’s get rid of ipv6 ::1 entry – IPv6 is going to be disabled in the OS
maciek@oel7rac1n1 ~ $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 # after the tweak maciek@oel7rac1n1 ~ $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 192.168.1.21 oel7rac1n1.dba24.pl oel7rac1n1
- Relog again as maciek and verify
maciek@oel7rac1n1.dba24.pl ~ $ hostname -f oel7rac1n1.dba24.pl maciek@oel7rac1n1.dba24.pl ~ $ hostname oel7rac1n1
We got what we expected.
- Change VM name
maciek@oel7rac1n1.dba24.pl ~ $ sudo shutdown -h now vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage modifyvm "oel72rac1n1" --name "oel7rac1n1"
- Check how the files related to the VM looks like now
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"|grep oel7 Name: oel7rac1n1 Config file: /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.vbox Log folder: /vbox-repo1/metadata/RacLab1/oel7rac1n1/Logs SATA Controller (1, 0): /vbox-repo1/disks/oel72rac1n1_localOSdisk1.vdi (UUID: be004bfa-0069-452c-b9b5-2448a988cd3b) Capture file: /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.webm
Well, vm configuration files’ names has changed, but local disk still bears the old server name
- Change name of the disk (medium)
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage modifymedium /vbox-repo1/disks/oel72rac1n1_localOSdisk1.vdi --move /vbox-repo1/disks/oel7rac1n1_localOSdisk1.vdi 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Move medium with UUID be004bfa-0069-452c-b9b5-2448a988cd3b finished
- Let’s see how it looks now. As you can see Virtualbox copes with changing of the file name automatically. New media file name has been registerd in the VM’s configuration
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"|grep oel7 Name: oel7rac1n1 Config file: /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.vbox Log folder: /vbox-repo1/metadata/RacLab1/oel7rac1n1/Logs SATA Controller (1, 0): /vbox-repo1/disks/oel7rac1n1_localOSdisk1.vdi (UUID: be004bfa-0069-452c-b9b5-2448a988cd3b) Capture file: /vbox-repo1/metadata/RacLab1/oel7rac1n1/oel7rac1n1.webm
- Start up VM again after the modification.
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage startvm oel72rac1n1 Waiting for VM "oel72rac1n1" to power on... VM "oel72rac1n1" has been successfully started.
Install additional software
I would like to have also some additional pkgs installed like rlwrap, Vbox Guest Additions
- RLWRAP Installation From EPEL
In order to configure EPEL yum repository download and install EPEL repository package
maciek@oel7rac1n1.dba24.pl ~ $ sudo wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm maciek@oel7rac1n1.dba24.pl ~ $ sudo rpm -Uvh epel-release-7-8.noarch.rpm
With the repository in place you can install rlwrap using the following command.
maciek@oel7rac1n1.dba24.pl ~ $ sudo yum install rlwrap
Let’s see the installation process in action
maciek@oel7rac1n1.dba24.pl ~ $ wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm --2016-11-15 16:03:58-- http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm Resolving www.mirrorservice.org (www.mirrorservice.org)... 212.219.56.184, 2001:630:341:12::184 Connecting to www.mirrorservice.org (www.mirrorservice.org)|212.219.56.184|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 14612 (14K) [application/x-redhat-package-manager] Saving to: ‘epel-release-7-8.noarch.rpm’ 100%[===================================================================================>] 14.612 --.-K/s in 0,06s 2016-11-15 16:03:59 (240 KB/s) - ‘epel-release-7-8.noarch.rpm’ saved [14612/14612] maciek@oel7rac1n1.dba24.pl ~ $ sudo rpm -Uvh epel-release-7-8.noarch.rpm warning: epel-release-7-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY Preparing... ################################# [100%] Updating / installing... 1:epel-release-7-8 ################################# [100%] maciek@oel7rac1n1.dba24.pl ~ $ sudo yum install rlwrap Loaded plugins: langpacks, ulninfo epel/x86_64/metalink | 21 kB 00:00:00 epel | 4.3 kB 00:00:00 ol7_UEKR4 | 1.2 kB 00:00:00 ol7_latest | 1.4 kB 00:00:00 epel/x86_64/updateinfo FAILED http://ftp.ps.pl/pub/Linux/fedora-epel/7/x86_64/repodata/7eeedbbcd4abfb06739248bf68ee306e4ec49718c86557b1bacc28f456dfd8ba-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. (1/3): epel/x86_64/group_gz | 170 kB 00:00:01 (2/3): epel/x86_64/updateinfo | 673 kB 00:00:05 (3/3): epel/x86_64/primary_db | 4.3 MB 00:00:16 Resolving Dependencies --> Running transaction check ---> Package rlwrap.x86_64 0:0.42-1.el7 will be installed --> Processing Dependency: perl(Data::Dumper) for package: rlwrap-0.42-1.el7.x86_64 --> Running transaction check ---> Package perl-Data-Dumper.x86_64 0:2.145-3.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================= Package Arch Version Repository Size ============================================================================================================================= Installing: rlwrap x86_64 0.42-1.el7 epel 96 k Installing for dependencies: perl-Data-Dumper x86_64 2.145-3.el7 ol7_latest 47 k Transaction Summary ============================================================================================================================= Install 1 Package (+1 Dependent package) Total download size: 143 k Installed size: 306 k Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7Server/epel/packages/rlwrap-0.42-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY Public key for rlwrap-0.42-1.el7.x86_64.rpm is not installed (1/2): rlwrap-0.42-1.el7.x86_64.rpm | 96 kB 00:00:00 (2/2): perl-Data-Dumper-2.145-3.el7.x86_64.rpm | 47 kB 00:00:00 ----------------------------------------------------------------------------------------------------------------------------- Total 260 kB/s | 143 kB 00:00:00 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Importing GPG key 0x352C64E5: Userid : "Fedora EPEL (7) <epel@fedoraproject.org>" Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5 Package : epel-release-7-8.noarch (installed) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Is this ok [y/N]: y Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : perl-Data-Dumper-2.145-3.el7.x86_64 1/2 Installing : rlwrap-0.42-1.el7.x86_64 2/2 Verifying : perl-Data-Dumper-2.145-3.el7.x86_64 1/2 Verifying : rlwrap-0.42-1.el7.x86_64 2/2 Installed: rlwrap.x86_64 0:0.42-1.el7 Dependency Installed: perl-Data-Dumper.x86_64 0:2.145-3.el7 Complete!
Rlwrap has been installed successfully!
- Install VBox Guest Additions
Accordingly to the Virtualbox documentation chapter 4.1
“As mentioned in Section 1.2, “Some terminology”, the Guest Additions are designed to be installed inside a virtual machine after the guest operating system has been installed. They consist of device drivers and system applications that optimize the guest operating system for better performance and usability. Please see Section 3.1, “Supported guest operating systems” for details on what guest operating systems are fully supported with Guest Additions by VirtualBox.
The VirtualBox Guest Additions for all supported guest operating systems are provided as a single CD-ROM image file which is called VBoxGuestAdditions.iso. This image file is located in the installation directory of VirtualBox. To install the Guest Additions for a particular VM, you mount this ISO file in your VM as a virtual CD-ROM and install from there.”
The Guest Additions offer the following features:
- Mouse pointer integration
- Shared folders
- Better video support
- Seamless windows
- Time synchronization
- Shared clipboard
- Automated logons (credentials passing)
For more infor please read documentation:
https://www.virtualbox.org/manual/ch04.html
# Attach VBox Guest Addition ISO to the rac node
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage storageattach "oel7rac1n1" --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /usr/share/virtualbox/VBoxGuestAdditions.iso
# Mount cdrom on host
maciek@oel7rac1n1.dba24.pl ~ $ sudo mount /dev/sr0 /media/
mount: /dev/sr0 is write-protected, mounting read-only
maciek@oel7rac1n1.dba24.pl /media $ cd /media
maciek@oel7rac1n1.dba24.pl /media $ sudo sh VBoxLinuxAdditions.run --nox11
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.8 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Building Guest Additions kernel modules.
vboxadd.sh: Starting the VirtualBox Guest Additions.
# shutdown VM operating system and remove iso file from drive
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage storageattach "oel7rac1n1" --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium emptydrive
# startup the VM again and check VB Guest Addition status
vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage showvminfo "oel7rac1n1"
......
Facility "VirtualBox Base Driver": active/running (last update: 2016/11/17 14:37:46 UTC)
Facility "VirtualBox System Service": active/running (last update: 2016/11/17 14:37:50 UTC)
Facility "Seamless Mode": not active (last update: 2016/11/17 14:37:46 UTC)
Facility "Graphics Mode": not active (last update: 2016/11/17 14:37:46 UTC)
# Graphic and seamless modes not supported since no X11 server installed on the host
# On the OS directly check if service is running
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status vboxadd-service.service
● vboxadd-service.service
Loaded: loaded (/opt/VBoxGuestAdditions-5.1.8/init/vboxadd-service; enabled; vendor preset: disabled)
Active: active (running) since czw 2016-11-17 15:43:45 CET; 47min ago
CGroup: /system.slice/vboxadd-service.service
└─774 /usr/sbin/VBoxService --pidfile /var/run/vboxadd-service.sh
lis 17 16:43:44 oel7rac1n1 systemd[1]: Starting vboxadd-service.service...
lis 17 16:43:44 oel7rac1n1 vboxadd-service[764]: vboxadd-service.sh: Starting VirtualBox Guest Addition service.
lis 17 16:43:44 oel7rac1n1 vboxadd-service.sh[775]: VirtualBox Guest Addition service started.
lis 17 15:43:45 oel7rac1n1 systemd[1]: Started vboxadd-service.service.
# How to check running VMs? It is really simple (taken where only openfiler1 was running)
vboxuser1@macieksrv.dba24.pl ~ $ VBoxManage list runningvms
"openfiler1" {002ac87b-755b-46e5-897d-35ebbd5f70f2}
DNS configuration
Dns names resolving configuration is very important for RAC to work properly.
You need to take the most care of the SCAN addresses – they have to be available from a dns server. As far as I know other addresses can be resolved locally. Grid Installer checks if dns is configured properly so this is a mandatory step.
- Because I know all the IPs in advance I will add them to the /etc/hosts file and then use dnsmasq tool to serve as local dns on RAC each node
# Add to /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 # Public interfaces 192.168.1.21 oel7rac1n1.dba24.pl oel7rac1n1 192.168.1.23 oel7rac1n2.dba24.pl oel7rac1n2 # VIP addresses 192.168.1.22 oel7rac1n1-vip.dba24.pl oel7rac1n1-vip 192.168.1.24 oel7rac1n2-vip.dba24.pl oel7rac1n2-vip # interconnect network for haip subnet1 192.168.20.21 oel7rac1n1-priv1.dba24.pl oel7rac1n1-priv1 192.168.20.23 oel7rac1n2-priv1.dba24.pl oel7rac1n2-priv1 # interconnect network for haip subnet2 192.168.21.21 oel7rac1n1-priv2.dba24.pl oel7rac1n1-priv2 192.168.21.23 oel7rac1n2-priv2.dba24.pl oel7rac1n2-priv2 # asm network subnet1 192.168.30.21 oel7rac1n1-apriv1.dba24.pl oel7rac1n1-apriv1 192.168.30.23 oel7rac1n2-apriv1.dba24.pl oel7rac1n2-apriv1 # asm network subnet2 192.168.31.21 oel7rac1n1-apriv2.dba24.pl oel7rac1n1-apriv2 192.168.31.23 oel7rac1n2-apriv2.dba24.pl oel7rac1n2-apriv2 # storage network 192.168.10.21 oel7rac1n1-spriv.dba24.pl oel7rac1n1-spriv 192.168.10.23 oel7rac1n2-spriv.dba24.pl oel7rac1n2-spriv 192.168.1.101 rac1-scan.dba24.pl rac1-scan 192.168.1.102 rac1-scan.dba24.pl rac1-scan 192.168.1.103 rac1-scan.dba24.pl rac1-scan # Openfilers 192.168.1.10 openfiler1 192.168.1.11 openfiler2 192.168.1.12 openfiler3 192.168.10.10 openfiler1-spriv 192.168.10.11 openfiler2-spriv 192.168.10.12 openfiler3-spriv
- As you can see in /etc/resolv.conf system queries gateway 192.168.1.1 for IP addresses by default
maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/resolv.conf # Generated by NetworkManager search speedport.ip nameserver 192.168.1.1
- For local addresses I want to use dnsmasq that will run on 127.0.0.1. I need to block dns that bridged network gets from dhcp and set DNS ip to localhost only. This local dnsmasq ip will be the only one in the /etc/resolve.conf
maciek@oel7rac1n1.dba24.pl ~ $ sudo nmcli con mod bridged ipv4.ignore-auto-dns yes maciek@oel7rac1n1.dba24.pl ~ $ sudo nmcli con mod bridged ipv4.dns 127.0.0.1 maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart network
- As you can see below now localhost is gonna be asked for any addresses so all the queries will go through dnsmasq daemon
maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/resolv.conf # Generated by NetworkManager nameserver 127.0.0.1
-
Now we need to enable dnsmasq. You can read about this daemon here: http://www.dnsmasq.org/
DNSmasq serves as dns cache running on a local host’s 53rd port, you need to run it on every node in the cluster. DNSmasq serves adresses from /etc/hosts and forwards queries that it cannot resolve locally to the configured DNS servers (reads /etc/resolv.conf to get the list)
First in order to avoid dns queries for addresses in dba24.pl domain, I will add it as local domain that should be resolved from /etc/hosts only. I will change a line in /etc/dnsmasq.conf. Find that part below
# Add local-only domains here, queries in these domains are answered # from /etc/hosts or DHCP only. #local=/localnet/
and change into
# Add local-only domains here, queries in these domains are answered # from /etc/hosts or DHCP only. local=/dba24.pl/
Second I will add file /etc/dnsmasq.d/externalDNS.conf with content
maciek@oel7rac1n4.dba24.pl ~ $ cat /etc/dnsmasq.d/externalDNS.conf server=192.168.1.1
Next change /etc/dnsmasq.conf by adding the line for resolv-file option pointing to the externalDNS file.
# Change this line if you want dns to get its upstream servers from # somewhere other that /etc/resolv.conf #resolv-file= resolv-file=/etc/dnsmasq.d/externalDNS.conf
- Now enable and start dnsmasq
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl enable dnsmasq Created symlink from /etc/systemd/system/multi-user.target.wants/dnsmasq.service to /usr/lib/systemd/system/dnsmasq.service. maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl start dnsmasq
- Check dnsmasq status
maciek@oel7rac1n2.dba24.pl ~ $ sudo systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
Active: active (running) since pią 2016-12-02 13:59:46 CET; 3s ago
Main PID: 11243 (dnsmasq)
CGroup: /system.slice/dnsmasq.service
└─11243 /usr/sbin/dnsmasq -k
gru 02 13:59:46 oel7rac1n2 systemd[1]: Started DNS caching server..
gru 02 13:59:46 oel7rac1n2 systemd[1]: Starting DNS caching server....
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: started, version 2.66 cachesize 150
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: using nameserver 192.168.1.1#53
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: using local addresses only for domain dba24.pl
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: no servers found in /etc/dnsmasq.d/externalDNS.conf, will retry
gru 02 13:59:46 oel7rac1n2 dnsmasq[11243]: read /etc/hosts - 39 addresses
- Test if dns is working properly
# Test DNS # local addresses maciek@oel7rac1n1.dba24.pl ~ $ nslookup oel7rac1n1 Server: 127.0.0.1 Address: 127.0.0.1#53 Name: oel7rac1n1 Address: 192.168.1.21 maciek@oel7rac1n1.dba24.pl ~ $ nslookup oel7rac1n2.dba24.pl Server: 127.0.0.1 Address: 127.0.0.1#53 Name: oel7rac1n2.dba24.pl Address: 192.168.1.23 # internet address maciek@oel7rac1n1.dba24.pl ~ $ nslookup facebook.com Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: facebook.com Address: 31.13.92.36 # And last but not least verify the holy scan name maciek@oel7rac1n1.dba24.pl ~ $ nslookup rac1-scan Server: 127.0.0.1 Address: 127.0.0.1#53 Name: rac1-scan Address: 192.168.1.103 Name: rac1-scan Address: 192.168.1.101 Name: rac1-scan Address: 192.168.1.102 maciek@oel7rac1n1.dba24.pl ~ $ nslookup rac1-scan.dba24.pl Server: 127.0.0.1 Address: 127.0.0.1#53 Name: rac1-scan.dba24.pl Address: 192.168.1.103 Name: rac1-scan.dba24.pl Address: 192.168.1.101 Name: rac1-scan.dba24.pl Address: 192.168.1.102
Beautifully, works like a charm!!!!
Configure services
- SSH
Add two lines at the end of /etc/ssh/sshd_config and restart sshd server
AddressFamily inet UseDNS no
First line property is required in a case ipv6 is to be turned off (I will do it soon)
Second line speeds up login operations to the system via ssh – no reverse dns query to find out the hostname for IP of a client
Now we can restart sshd
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart sshd
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since czw 2016-11-17 16:06:16 CET; 5s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 2801 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 2802 (sshd)
CGroup: /system.slice/sshd.service
└─2802 /usr/sbin/sshd
lis 17 16:06:16 oel7rac1n1 systemd[1]: Starting OpenSSH server daemon...
lis 17 16:06:16 oel7rac1n1 sshd[2802]: Server listening on 0.0.0.0 port 22.
lis 17 16:06:16 oel7rac1n1 systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
lis 17 16:06:16 oel7rac1n1 systemd[1]: Started OpenSSH server daemon.
- TUNED
I want to disable tuned, don’t need to change performance settings on the fly
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl stop tuned [sudo] password for maciek: maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl disable tuned Removed symlink /etc/systemd/system/multi-user.target.wants/tuned.service. maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status tuned ● tuned.service - Dynamic System Tuning Daemon Loaded: loaded (/usr/lib/systemd/system/tuned.service; disabled; vendor preset: enabled) Active: inactive (dead) lis 17 15:43:56 oel7rac1n1 systemd[1]: Starting Dynamic System Tuning Daemon... lis 17 15:43:57 oel7rac1n1 systemd[1]: Started Dynamic System Tuning Daemon. lis 17 16:18:58 oel7rac1n1 systemd[1]: Stopping Dynamic System Tuning Daemon... lis 17 16:18:59 oel7rac1n1 systemd[1]: Stopped Dynamic System Tuning Daemon.
- NTPD
For virtualbox the time is provided by Vbox Guest Additions, but in order to install Grid Infrastructure we need NTPD to be running or CTSS cluster service configured. I choose to disable NTPD and use CTSS, because I have no time servers configured on the VB Host (my ubuntu machine) and I don’t really want to play with it.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status ntpd Unit ntpd.service could not be found.
Ok ntpd not configured, don’t need to disable it then
- FIREWALL
Firewalld needs to be stopped in order to not to interfer the cluster communication.
There are tutorials how to keep it, but for lab this is not necessary.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl stop firewalld
[sudo] password for maciek:
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
lis 17 16:43:43 oel7rac1n1 systemd[1]: Starting firewalld - dynamic firewall daemon...
lis 17 15:43:47 oel7rac1n1 systemd[1]: Started firewalld - dynamic firewall daemon.
lis 17 16:43:26 oel7rac1n1 systemd[1]: Stopping firewalld - dynamic firewall daemon...
lis 17 16:43:26 oel7rac1n1 systemd[1]: Stopped firewalld - dynamic firewall daemon.
- iSCSI
Perform following steps
# Check if iscsi initiantors has been installed
maciek@oel7rac1n1.dba24.pl ~ $ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-35.0.1.el7 (x86_64)
iscsi-initiator-utils-iscsiuio-6.2.0.873-35.0.1.el7 (x86_64)
# check iscsid status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
Loaded: loaded (/usr/lib/systemd/system/iscsid.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:iscsid(8)
man:iscsiadm(8)
# enable iscsid and check status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl enable iscsid
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:iscsid(8)
man:iscsiadm(8)
# start iscsid and check its status
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl start iscsid
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl status iscsid
● iscsid.service - Open-iSCSI
Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
Active: active (running) since czw 2016-11-17 16:48:21 CET; 1s ago
Docs: man:iscsid(8)
man:iscsiadm(8)
Process: 3854 ExecStart=/usr/sbin/iscsid (code=exited, status=0/SUCCESS)
Main PID: 3856 (iscsid)
CGroup: /system.slice/iscsid.service
├─3855 /usr/sbin/iscsid
└─3856 /usr/sbin/iscsid
# check again if both iscsid and iscsi have been enabled
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl is-enabled iscsid.service
enabled
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl is-enabled iscsi.service
enabled
System confguration
- TRANSPARENT HUGEPAGES
Disable transparent hugepages. First check current settings.
maciek@oel7rac1n1.dba24.pl ~ $ sudo ls -l /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/redhat_transparent_hugepage ls: cannot access /sys/kernel/mm/transparent_hugepage: No such file or directory ls: cannot access /sys/kernel/mm/redhat_transparent_hugepage: No such file or directory
“If Transparent HugePages is removed from the kernel then the /sys/kernel/mm/transparent_hugepage or /sys/kernel/mm/redhat_transparent_hugepage files do not exist.”
Docs say also that Transparent HugePages are disabled by default in later releases of Oracle Linux https://docs.oracle.com/database/121/UNXAR/appi_vlm.htm#UNXAR428
- IO SCHEDULER
For best performance for Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler https://docs.oracle.com/database/121/CWLIN/prelinux.htm#CHDCEBCD
Check which of the IO schedulers is set at the moment
maciek@oel7rac1n1.dba24.pl ~ $ cat /sys/block/sda/queue/scheduler noop deadline [cfq]
CFQ is set by default.
In order to enable deadline scheduler we need to tweak grub. Remember: do not change /etc/grub2.cfg file, because it is automatically generated by grub2-mkconfig using templates from /etc/grub.d and settings from /etc/default/grub. Change /etc/default/grub instead.
Change the line
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never"
into
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never elevator=deadline"
Backup current /etc/grub2.cfg file somewhere. After you have modified /etc/default/grub run grub2-mkconfig.
maciek@oel7rac1n1.dba24.pl ~ $ sudo cp /etc/grub2.cfg /root/grub2.cfg.backup maciek@oel7rac1n1.dba24.pl ~ $ sudo grub2-mkconfig -o /etc/grub2.cfg Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img Found linux image: /boot/vmlinuz-3.8.13-118.14.1.el7uek.x86_64 Found initrd image: /boot/initramfs-3.8.13-118.14.1.el7uek.x86_64.img Found linux image: /boot/vmlinuz-3.8.13-98.7.1.el7uek.x86_64 Found initrd image: /boot/initramfs-3.8.13-98.7.1.el7uek.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-9d9f28c6023b438bb429d70373246640 Found initrd image: /boot/initramfs-0-rescue-9d9f28c6023b438bb429d70373246640.img done
Reboot OS and recheck the io scheduler set
maciek@oel7rac1n1.dba24.pl ~ $ cat /sys/block/sda/queue/scheduler noop [deadline] cfq
Deadline io scheduler has been set successfully.
- SHM FILESYSTEM
The plan is to use HugePages for DB, but for ASM Oracle recommends Automatic Memory Management (memory_* params).
How it looks now
maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep shm tmpfs 2972032 0 2972032 0% /dev/shm
Well, I just need 2G not more. Add shm tmpfs filesystem to the /etc/fstab, 2GB of size and remount it
# # /etc/fstab # Created by anaconda on Sun Nov 13 18:01:11 2016 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/ol-root / xfs defaults 0 0 UUID=630aa2e1-384a-4370-8721-cad3a7b92732 /boot xfs defaults 0 0 /dev/mapper/ol-swap swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults,size=2G 0 0 maciek@oel7rac1n1.dba24.pl ~ $ sudo mount -o remount /dev/shm # and after remounting maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep shm tmpfs 2097152 0 2097152 0% /dev/shm
That should be enough
- SELINUX
To disable SELINUX modify /etc/selinux/config file.
Change the following line:
SELINUX=enforcing
into
SELINUX=disabled
Remember it will take effect after a next reboot, this is why we need to turn the SELINUX temporarily into PERMISSIVE mode if we are not able to restart the box immediately
maciek@oel7rac1n1.dba24.pl ~ $ sudo setenforce 0 maciek@oel7rac1n1.dba24.pl ~ $ sudo getenforce Permissive
Check SELINUX status
maciek@oel7rac1n1.dba24.pl ~ $ sudo sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: disabled Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28
If you are allowed to reboot OS do it asap, then check status again
maciek@oel7rac1n1.dba24.pl ~ $ sudo selinuxenabled && echo enabled || echo disabled disabled maciek@oel7rac1n1.dba24.pl ~ $ getenforce Disabled maciek@oel7rac1n1.dba24.pl ~ $ sestatus SELinux status: disabled
- Create and set login information banner file
Create or edit banner file /etc/issue
####################################################################### # Welcome to DBA24 lab # Access to this system is permitted only for authorized personel. # All actions made on this system are monitored and logged. # *** Disconnect IMMEDIATELY if you are not an authorized user *** #######################################################################
Add the following line to the /etc/ssh/sshd_config file if doesn’t exist
Banner /etc/issue
Restart sshd
maciek@oel7rac1n1.dba24.pl ~ $ sudo systemctl restart sshd
Check if banner shows up – try to ssh to the machine
maciek@macieksrv.dba24.pl ~ $ ssh 192.168.1.21 ####################################################################### # Welcome to DBA24 lab # Access to this system is permitted only for authorized personel. # All actions made on this system are monitored and logged. # *** Disconnect IMMEDIATELY if you are not an authorized user *** ####################################################################### maciek@192.168.1.21:
It works!! 
- SYSTEM SETTINGS /etc/sysctl.conf
Remove /etc/sysctl.d/99-sysctl.conf the file was created by oracle-rdbms-server-12cR1-preinstall.x86_64 package
[root@oel7rac1n1 ~]# rm /etc/sysctl.d/99-sysctl.conf
Create configuration file /etc/sysctl.d/98-oracle.conf and put into it settings overriding default ones. We need theme to allow our Oracle software to work properly and efficiently
So first create empty file
maciek@oel7rac1n1.dba24.pl ~ $ sudo touch /etc/sysctl.d/98-oracle.conf
Then put following settings into this file, remember to tweak memory settings accordingly to the memory size of your VM or your needs. My VM has got 6GB of RAM assigned.
# Change IP fingerprint net.ipv4.tcp_keepalive_time=30 net.ipv4.tcp_max_syn_backlog=4096 net.ipv4.tcp_max_tw_buckets=1440000 # Deny source-routed packages net.ipv4.conf.all.accept_source_route=0 net.ipv4.conf.all.forwarding=0 # Deny icmp netmask & timestamp broadcast net.ipv4.icmp_echo_ignore_broadcasts=1 net.ipv4.tcp_timestamps=0 # Deny IP redirects net.ipv4.conf.all.accept_redirects=0 net.ipv4.conf.all.send_redirects=0 # Bufferbloat kernel.random.write_wakeup_threshold=1024 vm.swappiness=1 vm.dirty_background_ratio=3 vm.dirty_ratio=80 vm.dirty_expire_centisecs=500 vm.dirty_writeback_centisecs=100 vm.min_free_kbytes=524228 kernel.sem = 1510 193280 1510 128 net.ipv4.ip_local_port_range=9000 65500 net.core.rmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=1048576 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=9 net.ipv4.tcp_retries2=3 net.ipv4.tcp_syn_retries=2 net.ipv4.tcp_rmem=4096 262144 4194304 net.ipv4.tcp_wmem=4096 262144 4194304 fs.file-max=6815744 fs.aio-max-nr=1048576 net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv4.conf.all.secure_redirects=0 net.ipv4.conf.all.log_martians=1 net.ipv4.conf.default.accept_redirects=0 net.ipv4.conf.default.secure_redirects=0 net.ipv4.conf.default.accept_source_route=0 net.ipv4.conf.default.log_martians=1 net.ipv4.conf.default.send_redirects=0 net.ipv4.conf.all.rp_filter=1 net.ipv4.icmp_ignore_bogus_error_responses=1 kernel.msgmax=8192 kernel.msgmnb=65536 # RAM 64GB #kernel.shmmax=56371445760 #kernel.shmall=13762560 # RAM 4GB #kernel.shmmax=3523215360 #kernel.shmall=860160 # RAM 6GB kernel.shmmax=5284741120 kernel.shmall=1290220 vm.hugetlb_shm_group=2020 vm.nr_hugepages=1024
To enable above settings please run the following command or reboot your server
maciek@oel7rac1n1.dba24.pl ~ $ sudo sysctl -p /etc/sysctl.d/98-oracle.conf
I have met the opinion, each time you set HugePages you should reboot your server.
I will do it as some really wise guys recommend it.
After the reboot let’s check in example if hugepages has been set properly
maciek@oel7rac1n1.dba24.pl ~ $ cat /proc/meminfo |grep Huge AnonHugePages: 0 kB HugePages_Total: 1024 HugePages_Free: 1024 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB
Yup looks as expected. Of course you can check all the params from the Oracle documentation related to the database and grid infrastructure installation. The above settings were tailored to the 12c requirements, but they will be verified later by the cluvfy tool
- SECURITY LIMITS
As it was with system setting let’s create file /etc/security/limits.d/99-oracle.conf
maciek@oel7rac1n1.dba24.pl ~ $ sudo touch /etc/security/limits.d/99-oracle.conf
Add the following content into the file
* soft core 0 * hard core 0 # Specific setting for oracle users @oinstall hard nofile 65536 @oinstall soft nofile 4096 @oinstall soft nproc 131072 @oinstall hard nproc 131072 @oinstall soft stack 32768 @oinstall hard stack 32768 # for 6GB servers @oinstall soft memlock 5662310 @oinstall hard memlock 5662310
Modify memlock settings accordingly to your RAM size
- GROUPS
Create all the below groups on the Oracle database servers. Please keep the GID numbers, this is important to keep the standard in an environment.
| GID | Group Name |
| 2001 | oinstall |
| 2002 | dba |
| 2003 | oper |
| 2004 | backupdba |
| 2005 | dgdba |
| 2006 | kmdba |
| 2007 | asmadmin |
| 2008 | asmoper |
| 2009 | asmdba |
| 2020 | hugetlb |
As you can see groups oinstall and dba have already been created by the oracle-rdbms-server-12cR1-preinstall.x86_64 package.
maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group ......... oinstall:x:54321: dba:x:54322:oracle ......
I need to change their gids to fit to my standard
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupmod -g 2001 oinstall maciek@oel7rac1n1.dba24.pl ~ $ sudo groupmod -g 2002 dba maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group|grep oin oinstall:x:2001: maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group|grep dba dba:x:2002:oracle
Now let’s add other groups
maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2003 oper maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2004 backupdba maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2005 dgdba maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2006 kmdba maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2007 asmadmin maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2008 asmoper maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2009 asmdba maciek@oel7rac1n1.dba24.pl ~ $ sudo groupadd -g 2020 hugetlb
As you can see all the groups are in place
maciek@oel7rac1n1.dba24.pl ~ $ sudo cat /etc/group |tail -11 oinstall:x:2001: dba:x:2002:oracle vboxsf:x:992: oper:x:2003: backupdba:x:2004: dgdba:x:2005: kmdba:x:2006: asmadmin:x:2007: asmoper:x:2008: asmdba:x:2009: hugetlb:x:2020:
- USERS
We need two users for our installation – grid for Grid Infrastructure and oracle for Oracle Database sofware. I like this idea of roles separation and have been using it from the time this concept apeared with Oracle 11R2.
As you can see below user oracle was created by oracle-rdbms-server-12cR1-preinstall.x86_64 package.
maciek@oel7rac1n1.dba24.pl ~ $ sudo id -a oracle uid=54321(oracle) gid=2001(oinstall) groups=2001(oinstall),2002(dba)
I want to change its id accordingly to my requirements and add user oracle to the required groups
Users required:
| UID | Username | Main Group | Other Groups |
| 2001 | oracle | oinstall | dba,oper,backupdba,dgdba,kmdba,hugetlb,asmdba |
| 2002 | grid | oinstall | dba,asmadmin,asmoper,asmdba |
Modify oracle user
maciek@oel7rac1n1.dba24.pl ~ $ sudo usermod -a -u 2001 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,hugetlb,asmdba oracle maciek@oel7rac1n1.dba24.pl ~ $ id -a oracle uid=2001(oracle) gid=2001(oinstall) groups=2001(oinstall),2002(dba),2003(oper),2004(backupdba),2005(dgdba),2006(kmdba),2009(asmdba),2020(hugetlb)
Add new grid user
maciek@oel7rac1n1.dba24.pl ~ $ sudo useradd -u 2002 -m -s /bin/bash -g oinstall -G dba,asmadmin,asmoper,asmdba grid maciek@oel7rac1n1.dba24.pl ~ $ id -a grid uid=2002(grid) gid=2001(oinstall) groups=2001(oinstall),2002(dba),2007(asmadmin),2008(asmoper),2009(asmdba)
Important: Remember to set passwords for oracle and grid, ssh user equivalency won’t work without it
- ULIMITS
Set ulimits for oracle and grid. Create file /etc/profile.d/oracle-grid.sh with the following settings
#Setting the appropriate ulimits for oracle and grid user
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
if [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
- UMASK
Check if umask settings for grid and oracle are set to 0022
maciek@oel7rac1n1.dba24.pl ~ $ sudo runuser -l oracle -c 'umask' 0022 maciek@oel7rac1n1.dba24.pl ~ $ sudo runuser -l grid -c 'umask' 0022
Settings are correct!!
- PAM
Verify the latest version of PAM is loaded, then add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so
- REMOVE IPC
We need RemoveIPC explicitly set to “no” in /etc/systemd/logind.conf. Verify current settings:
maciek@oel7rac1n1.dba24.pl ~ $ cat /etc/systemd/logind.conf |grep Rem #RemoveIPC=no RemoveIPC=no
As you can see in 7.2 RemoveIPC is set to now by default. We don’t need to change anything then.
Prepare directories for RAC installation
- Run the following script to get the desired directories (as root)
#!/bin/bash [ -d "/u01/app/grid" ] && echo "Directory /u01/app/grid exists" || ( mkdir -p /u01/app/grid; chown grid:oinstall /u01/app/grid;chmod 770 /u01/app/grid ) [ -d "/u01/app/oracle" ] && echo "Directory /u01/app/oracle exists" || ( mkdir -p /u01/app/oracle; chown oracle:oinstall /u01/app/oracle;chmod 770 /u01/app/oracle ) [ -d "/u01/app/grid/grid-software" ] && echo "Directory /u01/app/grid/grid-software exists" || ( mkdir -p /u01/app/grid/grid-software; chown grid:oinstall /u01/app/grid/grid-software ) [ -d "/u01/app/oracle/oracle-software" ] && echo "Directory /u01/app/oracle/oracle-software exists" || ( mkdir -p /u01/app/oracle/oracle-software; chown oracle:oinstall /u01/app/oracle/oracle-software ) chown root:oinstall /u01 chown root:oinstall /u01/app chmod 750 /u01 chmod 770 /u01/app # RAC specific [ -d "/u01/app/12.1.0/grid" ] && echo "Directory /u01/app/12.1.0/grid exists" || ( mkdir -p /u01/app/12.1.0/grid; chown grid:oinstall /u01/app/12.1.0/grid ) # link sudo to /usr/local/bin directory – needed for cluvfy to run properly [ -h "/usr/local/bin/sudo" ] && echo "Symlink /usr/local/bin/sudo exists" || ln -s /usr/bin/sudo /usr/local/bin/sudo # dirs for cluvfy [ -d "/u01/app/grid/cluvfy" ] && echo "Directory /u01/app/grid/cluvfy exists" || ( mkdir -p /u01/app/grid/cluvfy ) [ -h "/u01/app/oracle/cluvfy" ] && echo "Symbolic link /u01/app/oracle/cluvfy exists" || ( ln -s /u01/app/grid/cluvfy /u01/app/oracle/cluvfy ) chown -R grid:oinstall /u01/app/grid/cluvfy/ chmod 770 /u01/app/grid/cluvfy #dirs for TFA [ -d "/u01/app/tfa" ] && echo "Directory /u01/app/tfa exists" || ( mkdir -p /u01/app/tfa ) chown -R root:root /u01/app/tfa/ chmod 755 /u01/app/tfa # Add sudo entry for ORACHK (part of TFA) echo '@oinstall ALL=(root) NOPASSWD:/home/oracle/orachk/root_orachk.sh' >> /etc/sudoers =====================================================
Check the if directories have been created
maciek@oel7rac1n1.dba24.pl ~ $ sudo find /u01 -type d /u01 /u01/app /u01/app/grid /u01/app/grid/grid-software /u01/app/grid/cluvfy /u01/app/oracle /u01/app/oracle/oracle-software /u01/app/12.1.0 /u01/app/12.1.0/grid /u01/app/tfa
OK!
CLUVFY
- Install Cluster Verification Utility
Download the latest cluvfy from
http://www.oracle.com/technetwork/products/clustering/downloads/cvu-download-homepage-099973.html
Install cluvfy
maciek@oel7rac1n1.dba24.pl ~ $ sudo unzip /home/maciek/cvupack_Linux_x86_64.zip -d /u01/app/grid/cluvfy/ maciek@oel7rac1n1.dba24.pl ~ $ sudo chown -R grid:oinstall /u01/app/grid/cluvfy/ maciek@oel7rac1n1.dba24.pl ~ $ sudo chmod 770 /u01/app/grid/cluvfy
Install cvuqdisk tool from rpm provided by the cluvfy package (as root)
[root@oel7rac1n1 ~]# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP; [root@oel7rac1n1 ~]# rpm -i /u01/app/grid/cluvfy/cv/remenv/cvuqdisk-1.0.9-1.rpm
The OS is ready now for Oracle 12c. To be able to install GI and RAC db we need some iSCSI storage presented by Openfiler servers (openfiler1, openfiler2,openfiler3). Next parts of the series will guide you through the process of openfiler storage configuration, openfiler and rac node VM cloning. I will also show you how to present and attach iSCSI disks for RAC nodes.
Stay tuned!!!
What do you think?? Please post your comments
Source: My experience and WorlWideWeb
About the author
Maciej Tokar
An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU
