This is eighth part of my RAC lab series. In the last part I showed you how to configure openfilers and present LUNs to other VMs and also how to discover those luns on OEL 7.x VMs. Now I am going to show you how to prepare this storage for Oracle 12c Grid Infrastructure. At that moment we have still just one future rac node oel7rac1n1 and all the steps are to be perfromed on this node only.
oel7rac1n1 will be cloned in Part 9 of the series and all the settings below will be cloned as well in oel7rac1n2 (future second node).
RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database
Documentation and other sources
-
Openfiler official documentation
Documentation regarding installation an administration of Openfiler software can be found at the following link:
http://openfiler.com/community/download -
ISCSI – how to configure and use
http://www.oracle.com/technetwork/articles/hunter-rac11gr2-iscsi-088677.html#11
http://open-source-experiments.blogspot.com/2008/04/birds-eye-view-of-end-configuration.html
https://docs.oracle.com/cd/E52668_01/E54669/html/ol7-s17-storage.html
https://wiki.debian.org/SAN/iSCSI/open-iscsi
List storage luns and create partitions
- List disks using fdisk and lssci. Ignore /dev/sda (local disk) and mapper (LVM volumes on /dev/sda)
[root@oel7rac1n1 ~] fdisk -l|grep Disk Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors Disk label type: dos Disk identifier: 0x000d2def Disk /dev/mapper/ol-root: 32.7 GB, 32744931328 bytes, 63954944 sectors Disk /dev/mapper/ol-swap: 17.7 GB, 17716740096 bytes, 34603008 sectors Disk /dev/sdb: 10.5 GB, 10502537216 bytes, 20512768 sectors Disk /dev/sdd: 10.5 GB, 10502537216 bytes, 20512768 sectors Disk /dev/sdc: 536 MB, 536870912 bytes, 1048576 sectors Disk /dev/sde: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdf: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdg: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdh: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdi: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdj: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdk: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdl: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdm: 2113 MB, 2113929216 bytes, 4128768 sectors Disk /dev/sdn: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdo: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdq: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdp: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdr: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sds: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdt: 1073 MB, 1073741824 bytes, 2097152 sectors
Now try with lssci
[root@oel7rac1n1 ~] lsscsi [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0 [3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda [7:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb [7:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc [7:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sde [7:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh [7:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk [7:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm [7:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdo [7:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdq [7:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdt [9:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd [9:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf [9:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi [9:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj [9:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl [9:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdn [9:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdp [9:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdr [9:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sds [11:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg
Luns no. 0 – CRS
Luns no. 1-4 DATA
Luns no. 5-8 FRA
First digit means respectively: 7-openfiler1, 9-openfiler2, 11-openfiler3
- Create partitions on all physical disk devices that you want to use as disk for ASM
“Oracle recommends that you create a single whole-disk partition on each disk.”
https://docs.oracle.com/database/121/CWLIN/manpreins.htm#CWLIN408
[root@oel7rac1n1 ~] parted /dev/sdb mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdc mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sde mklabel "gpt" mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdh mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdk mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdm mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdo mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdq mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdt mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdd mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdf mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdi mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdj mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdl mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdn mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdp mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdr mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sds mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab. [root@oel7rac1n1 ~] parted /dev/sdg mklabel gpt mkpart primary "1 -1" Information: You may need to update /etc/fstab.
Configure udev for iSCSI storage
Ok, so at the moment we have disks discovered from openfilers and partitions configured on each of them. Now we need to configure udev. Udev rules will allow us to fix unique, persistent across restarts and readable name for the luns’ partitions (future asm disks). Udev rules will also apply proper owner:group and privs to those devices. Partitions are fixed to the same scsi id as the whole disk device they belong to.
- To identifiy each disk in SAN networks one can use WWNs. In iSCSI environment disks have their unique scsi IDs. Use “/usr/lib/udev/scsi_id –whitelisted –replace-whitespace /dev/$DISK_device” to retrieve it
# openfiler1 luns [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdb 14f504e46494c45525562365a4f722d717435652d366a4577 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdc 14f504e46494c4552436e4c4f38792d734f304e2d59786a51 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sde 14f504e46494c45527331363261382d483747562d6f303378 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdh 14f504e46494c45525164325873532d32565a622d69687056 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdk 14f504e46494c45527466705151482d6976764f2d4c496e44 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdm 14f504e46494c45526a4d6f6a32672d416478382d37626c4f [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdo 14f504e46494c4552366847424e482d6d6f7a372d55544279 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdq 14f504e46494c45526341536c44322d514749322d61466978 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdt 14f504e46494c45526871726535642d35466f642d31614474 # openfiler2 luns [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdd 14f504e46494c45525138695241632d477837442d33714b6a [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdf 14f504e46494c4552436e4c4f38792d734f304e2d59786a51 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdi 14f504e46494c45527331363261382d483747562d6f303378 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdj 14f504e46494c45525164325873532d32565a622d69687056 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdl 14f504e46494c45527466705151482d6976764f2d4c496e44 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdn 14f504e46494c45526a4d6f6a32672d416478382d37626c4f [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdp 14f504e46494c4552366847424e482d6d6f7a372d55544279 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdr 14f504e46494c45526341536c44322d514749322d61466978 [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sds 14f504e46494c4552A6871726535642d35466f642d31614474 # openfiler3 luns [root@oel7rac1n1 ~] /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdg 14f504e46494c45526b3863454d502d796f55712d3831536f
- Ok so we got now clear disk list from lssci command and scsi id for each and every disk. Let’s build table of associations between them and required disk aliases.
iqn.2006-01.com.openfiler:op1.rac1.asm
| LUN | Device | Scsi Id | Diskgroup | Udev Alias Name (symlink) |
| 0 | sdb | 14f504e46494c45525562365a4f722d717435652d366a4577 | CRS | asm/op1-rac1-asm-crs-01 |
| 1 | sdc | 14f504e46494c4552436e4c4f38792d734f304e2d59786a51 | DATA | asm/op1-rac1-asm-data-01 |
| 2 | sde | 14f504e46494c45527331363261382d483747562d6f303378 | DATA | asm/op1-rac1-asm-data-02 |
| 3 | sdh | 14f504e46494c45525164325873532d32565a622d69687056 | DATA | asm/op1-rac1-asm-data-03 |
| 4 | sdk | 14f504e46494c45527466705151482d6976764f2d4c496e44 | DATA | asm/op1-rac1-asm-data-04 |
| 5 | sdm | 14f504e46494c45526a4d6f6a32672d416478382d37626c4f | FRA | asm/op1-rac1-asm-fra-01 |
| 6 | sdo | 14f504e46494c4552366847424e482d6d6f7a372d55544279 | FRA | asm/op1-rac1-asm-fra-02 |
| 7 | sdq | 14f504e46494c45526341536c44322d514749322d61466978 | FRA | asm/op1-rac1-asm-fra-03 |
| 8 | sdt | 14f504e46494c45526871726535642d35466f642d31614474 | FRA | asm/op1-rac1-asm-fra-04 |
iqn.2006-01.com.openfiler:op2.rac1.asm
| LUN | Device | Scsi Id | Diskgroup | Udev Alias Name (symlink) |
| 0 | sdd | 14f504e46494c45525138695241632d477837442d33714b6a | CRS | asm/op2-rac1-asm-crs-02 |
| 1 | sdf | 14f504e46494c455266386161776f2d714f324b2d50584769 | DATA | asm/op2-rac1-asm-data-01 |
| 2 | sdi | 14f504e46494c45523667525750722d555945432d6c4c674d | DATA | asm/op2-rac1-asm-data-02 |
| 3 | sdj | 14f504e46494c45524342333579442d6c4d42512d6e376168 | DATA | asm/op2-rac1-asm-data-03 |
| 4 | sdl | 14f504e46494c45527059744f71482d5a304f7a2d6b425177 | DATA | asm/op2-rac1-asm-data-04 |
| 5 | sdn | 14f504e46494c45524478624844752d7a7072692d6a484473 | FRA | asm/op2-rac1-asm-fra-01 |
| 6 | sdp | 14f504e46494c455264634a7a41352d4d437a312d50435946 | FRA | asm/op2-rac1-asm-fra-02 |
| 7 | sdr | 14f504e46494c45523232363131422d646d53412d6e463061 | FRA | asm/op2-rac1-asm-fra-03 |
| 8 | sds | 14f504e46494c45524a415a716a492d694345742d6248556b | FRA | asm/op2-rac1-asm-fra-04 |
iqn.2006-01.com.openfiler:op3.rac1.asm
| LUN | Device | Scsi Id | Diskgroup | Udev Alias Name (symlink) |
| 0 | sdg | 14f504e46494c45526b3863454d502d796f55712d3831536f | CRS | asm/op3-rac1-asm-crs-03 |
- Create Udev rules file /etc/udev/rules.d/90-oracle-asmdevices.rules with content that follow our tables of associations.
[root@oel7rac1n1 ~] cat /etc/udev/rules.d/90-oracle-asmdevices.rules # udev rules for asm disks KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45525562365a4f722d717435652d366a4577", SYMLINK+="asm/op1-rac1-asm-crs-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552436e4c4f38792d734f304e2d59786a51", SYMLINK+="asm/op1-rac1-asm-data-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45527331363261382d483747562d6f303378", SYMLINK+="asm/op1-rac1-asm-data-02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45525164325873532d32565a622d69687056", SYMLINK+="asm/op1-rac1-asm-data-03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45527466705151482d6976764f2d4c496e44", SYMLINK+="asm/op1-rac1-asm-data-04", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526a4d6f6a32672d416478382d37626c4f", SYMLINK+="asm/op1-rac1-asm-fra-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552366847424e482d6d6f7a372d55544279", SYMLINK+="asm/op1-rac1-asm-fra-02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526341536c44322d514749322d61466978", SYMLINK+="asm/op1-rac1-asm-fra-03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526871726535642d35466f642d31614474", SYMLINK+="asm/op1-rac1-asm-fra-04", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45525138695241632d477837442d33714b6a", SYMLINK+="asm/op2-rac1-asm-crs-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455266386161776f2d714f324b2d50584769", SYMLINK+="asm/op2-rac1-asm-data-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45523667525750722d555945432d6c4c674d", SYMLINK+="asm/op2-rac1-asm-data-02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45524342333579442d6c4d42512d6e376168", SYMLINK+="asm/op2-rac1-asm-data-03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45527059744f71482d5a304f7a2d6b425177", SYMLINK+="asm/op2-rac1-asm-data-04", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45524478624844752d7a7072692d6a484473", SYMLINK+="asm/op2-rac1-asm-fra-01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455264634a7a41352d4d437a312d50435946", SYMLINK+="asm/op2-rac1-asm-fra-02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45523232363131422d646d53412d6e463061", SYMLINK+="asm/op2-rac1-asm-fra-03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45524a415a716a492d694345742d6248556b", SYMLINK+="asm/op2-rac1-asm-fra-04", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526b3863454d502d796f55712d3831536f", SYMLINK+="asm/op3-rac1-asm-crs-03", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=”sd*1″ – relates to every device with name /dev/sd*1 – as you can see this matches partitions on disks
RESULT refers to the scsi_id returned by each lun (unique), as you can see to retrieve this scsi_id the PROGRAM is used
- Reload UDEV rules and pull the trigger
[root@oel7rac1n1 ~] udevadm control --reload-rules [root@oel7rac1n1 ~] udevadm trigger
- Check results of this operations
[root@oel7rac1n1 ~] ls -l /dev/asm/op* lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-crs-01 -> ../sdb1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-data-01 -> ../sdc1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-data-02 -> ../sde1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-data-03 -> ../sdh1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-data-04 -> ../sdk1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-fra-01 -> ../sdm1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-fra-02 -> ../sdo1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-fra-03 -> ../sdq1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op1-rac1-asm-fra-04 -> ../sdt1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-crs-02 -> ../sdd1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-data-01 -> ../sdf1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-data-02 -> ../sdi1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-data-03 -> ../sdj1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-data-04 -> ../sdl1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-fra-01 -> ../sdn1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-fra-02 -> ../sdp1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-fra-03 -> ../sdr1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op2-rac1-asm-fra-04 -> ../sds1 lrwxrwxrwx 1 root root 7 11-30 15:54 /dev/asm/op3-rac1-asm-crs-03 -> ../sdg1
Symlinks appeared for each partition
- And the last thing check if correct owner:group and privs have been assigned
[root@oel7rac1n1 ~] ls -l /dev/sdb1 brw-rw---- 1 grid asmadmin 8, 17 11-30 15:54 /dev/sdb1 [root@oel7rac1n1 ~] ls -l /dev/sdc1 brw-rw---- 1 grid asmadmin 8, 33 11-30 15:54 /dev/sdc1 [root@oel7rac1n1 ~] ls -l /dev/sde1 brw-rw---- 1 grid asmadmin 8, 65 11-30 15:54 /dev/sde1 [root@oel7rac1n1 ~] ls -l /dev/sdh1 brw-rw---- 1 grid asmadmin 8, 113 11-30 15:54 /dev/sdh1 [root@oel7rac1n1 ~] ls -l /dev/sdk1 brw-rw---- 1 grid asmadmin 8, 161 11-30 15:54 /dev/sdk1 [root@oel7rac1n1 ~] ls -l /dev/sdm1 brw-rw---- 1 grid asmadmin 8, 193 11-30 15:54 /dev/sdm1 [root@oel7rac1n1 ~] ls -l /dev/sdo1 brw-rw---- 1 grid asmadmin 8, 225 11-30 15:54 /dev/sdo1 [root@oel7rac1n1 ~] ls -l /dev/sdq1 brw-rw---- 1 grid asmadmin 65, 1 11-30 15:54 /dev/sdq1 [root@oel7rac1n1 ~] ls -l /dev/sdt1 brw-rw---- 1 grid asmadmin 65, 49 11-30 15:54 /dev/sdt1 [root@oel7rac1n1 ~] ls -l /dev/sdd1 brw-rw---- 1 grid asmadmin 8, 49 11-30 15:54 /dev/sdd1 [root@oel7rac1n1 ~] ls -l /dev/sdf1 brw-rw---- 1 grid asmadmin 8, 81 11-30 15:54 /dev/sdf1 [root@oel7rac1n1 ~] ls -l /dev/sdi1 brw-rw---- 1 grid asmadmin 8, 129 11-30 15:54 /dev/sdi1 [root@oel7rac1n1 ~] ls -l /dev/sdj1 brw-rw---- 1 grid asmadmin 8, 145 11-30 15:54 /dev/sdj1 [root@oel7rac1n1 ~] ls -l /dev/sdl1 brw-rw---- 1 grid asmadmin 8, 177 11-30 15:54 /dev/sdl1 [root@oel7rac1n1 ~] ls -l /dev/sdn1 brw-rw---- 1 grid asmadmin 8, 209 11-30 15:54 /dev/sdn1 [root@oel7rac1n1 ~] ls -l /dev/sdp1 brw-rw---- 1 grid asmadmin 8, 241 11-30 15:54 /dev/sdp1 [root@oel7rac1n1 ~] ls -l /dev/sdr1 brw-rw---- 1 grid asmadmin 65, 17 11-30 15:54 /dev/sdr1 [root@oel7rac1n1 ~] ls -l /dev/sds1 brw-rw---- 1 grid asmadmin 65, 33 11-30 15:54 /dev/sds1 [root@oel7rac1n1 ~] ls -l /dev/sdg1 brw-rw---- 1 grid asmadmin 8, 97 11-30 15:54 /dev/sdg1
Storage for our RAC nodes has been successfully prepared on the operating system level for oel7rac1n1!! I am going to clone oel7rac1n1 in the next part of the series.
What do you think?? Please post your comments 
Source: My experience and WorlWideWeb
About the author
Maciej Tokar
An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU
