2016 - Oracle RAC 11g2 - prepare
The Slovak original of this document: 2016 - Oracle RAC 11g2 - prepare (slovensky).
================================================================================================================
Installing Oracle DB 11g2 in a RAC configuration using Oracle ASM on SUSE Linux Enterprise Server 11 SP3
================================================================================================================
GRID Infrastructure Software (Clusterware + ASM)
--------------------------------------------------------------------------------------------------------
- Server: All the RAC Nodes
- ORACLE_BASE: /data/u01/app/grid
- ORACLE_HOME: /data/u01/app/grid11204
- Owner: grid (Primary Group: oinstall, Secondary Group: asmadmin, asmdba)
- Permissions: 755
- OCR/Voting Disk Storage Type: ASM
- Oracle Inventory Location: /data/u01/app/oraInventory
--------------------------------------------------------------------------------------------------------
Oracle Database Software (RAC 192.0.2.8)
--------------------------------------------------------------------------------------------------------
- ORACLE_BASE: /data/u01/app/oracle
- ORACLE_HOME: /data/u01/app/oracle/db11204
- Owner: oracle (Primary Group: oinstall, Secondary Group: asmdba, dba)
- Permissions: 755
- Oracle Inventory Location: /data/u01/app/oraInventory
- Database Name: csldb
- Listener: opdb (TCP:1525)
--------------------------------------------------------------------------------------------------------
================================================================================================================
[1] Installing the software packages needed to run an Oracle database
================================================================================================================
4.3 Package Requirements - SUSE Linux Enterprise Server 11
REF: https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCGJCEA
********************************************************************************************************
- binutils-2.19
- gcc-4.3
- gcc-32bit-4.3
- gcc-c++-4.3
- glibc-2.9
- glibc-32bit-2.9
- glibc-devel-2.9
- glibc-devel-32bit-2.9
- ksh-93t
- libaio-0.3.104
- libaio-32bit-0.3.104
- libaio-devel-0.3.104
- libaio-devel-32bit-0.3.104
- libstdc++33-3.3.3
- libstdc++33-32bit-3.3.3
- libstdc++43-4.3.3_20081022
- libstdc++43-32bit-4.3.3_20081022
- libstdc++43-devel-4.3.3_20081022
- libstdc++43-devel-32bit-4.3.3_20081022
- libgcc43-4.3.3_20081022
- libstdc++-devel-4.3
- make-3.81
- sysstat-8.1.5
********************************************************************************************************
[1.1] Installing the basic packages needed to run an Oracle database
--------------------------------------------------------------------------------------------------------
# zypper install binutils
# zypper install gcc
# zypper install gcc-32bit
# zypper install gcc-c++
# zypper install glibc
# zypper install glibc-32bit
# zypper install glibc-devel
# zypper install glibc-devel-32bit
# zypper install ksh
# zypper install libaio
# zypper install libaio-32bit
# zypper install libaio-devel
# zypper install libaio-devel-32bit
# zypper install libstdc++33
# zypper install libstdc++33-32bit
# zypper install libstdc++43
# zypper install libstdc++43-32bit
# zypper install libstdc++43-devel
# zypper install libstdc++43-devel-32bit
# zypper install libgcc43
# zypper install libstdc++-devel
# zypper install make
# zypper install sysstat
--------------------------------------------------------------------------------------------------------
[1.2] Checking that the basic packages the Oracle installation needs are on the system
--------------------------------------------------------------------------------------------------------
# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}(%{ARCH})\n' binutils gcc gcc-32bit gcc-c++ glibc glibc-32bit \
glibc-devel glibc-devel-32bit ksh libaio libaio-32bit libaio-devel libaio-devel-32bit libstdc++33 \
libstdc++33-32bit libstdc++43 libstdc++43-32bit libstdc++43-devel libstdc++43-devel-32bit libgcc43 \
libstdc++-devel make sysstat
4.5.1 Additional Software Requirements - 4.5.1 Oracle ODBC Drivers - SUSE Linux Enterprise Server 11
REF: https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCFGFBH
********************************************************************************************************
- unixODBC-2.2.12 or later
- unixODBC-devel-2.2.12 or later
- unixODBC-32bit-2.2.12 (32-bit) or later
********************************************************************************************************
[1.3] Installing the additional packages needed to run an Oracle database
--------------------------------------------------------------------------------------------------------
# zypper install unixODBC
# zypper install unixODBC-devel
# zypper install unixODBC-32bit
--------------------------------------------------------------------------------------------------------
[1.4] Checking that the additional packages Oracle needs are on the system
--------------------------------------------------------------------------------------------------------
# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}(%{ARCH})\n' unixODBC unixODBC-devel unixODBC-32bit
--------------------------------------------------------------------------------------------------------
[1.5] The installation needs an X Window server (Xorg) and the library libcap1
--------------------------------------------------------------------------------------------------------
# zypper install xorg-x11
# zypper install libcap1
--------------------------------------------------------------------------------------------------------
[1.6] Checking that the X Window server packages the Oracle installation needs are on the system
--------------------------------------------------------------------------------------------------------
# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}(%{ARCH})\n' xorg-x11 libcap1
--------------------------------------------------------------------------------------------------------
================================================================================================================
[2] Creating the local users and groups an Oracle database needs
================================================================================================================
5 Creating Required Operating System Groups and Users
REF: https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCBCFDI
********************************************************************************************************
[2.1] Creating the local accounts (oracle, grid) and putting them into the groups an Oracle database needs
--------------------------------------------------------------------------------------------------------
# groupadd -g 30001 oinstall
# groupadd -g 30002 asmadmin
# groupadd -g 30003 asmdba
# groupadd -g 30004 dba
# useradd -g oinstall -G asmadmin,asmdba -u 30101 grid
# useradd -g oinstall -G dba,asmdba -u 30102 oracle
# passwd oracle
New password: Ora.password.123
# passwd grid
New password: Gri.password.123
--------------------------------------------------------------------------------------------------------
[2.2] Installing the required package "cvuqdisk" (it has to be installed after the group "oinstall" exists).
Balicek sa nachadza na instalacnom mediu Oracle DB.
--------------------------------------------------------------------------------------------------------
# cd /data/install/database/rpm
# rpm -ivh ./cvuqdisk-1.0.9-1.rpm
--------------------------------------------------------------------------------------------------------
[2.3] Installing the other packages Oracle RAC + ASM requires
--------------------------------------------------------------------------------------------------------
REF: ASMLIB -> http://www.oracle.com/technetwork/server-storage/linux/downloads/sles11-099661.html
--------------------------------------------------------------------------------------------------------
# zypper install numactl
# wget http://oss.oracle.com/projects/oracleasm-support/dist/files/RPMS/sles11/amd64/2.1.8/oracleasm-support-2.1.8-1.SLE11.x86_64.rpm
# wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.sle11.x86_64.rpm
# rpm -ivh ./oracleasm-support-2.1.8-1.SLE11.x86_64.rpm
# zypper install oracleasm
# rpm -ivh ./oracleasmlib-2.0.4-1.sle11.x86_64.rpm
================================================================================================================
[3] Konfiguracia prostredia OS pre prevadzku Oracle DB
================================================================================================================
********************************************************************************************************
6 Configuring Kernel Parameters and Resource Limits
REF: https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCCADGD
********************************************************************************************************
/etc/sysctl.conf
********************************************************************************************************
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
********************************************************************************************************
********************************************************************************************************
Optimal SHMMAX for Oracle
http://www.dba-oracle.com/t_optimal_shmmax.htm
********************************************************************************************************
********************************************************************************************************
kernel shared memory calculator (shmsetup.sh)
REF: https://gist.github.com/jodell/1285137
********************************************************************************************************
SHMMAX is the maximum size of a single shared memory segment set in “bytes”.
SHMALL is the total size of Shared Memory Segments System wide set in “pages”.
********************************************************************************************************
[3.1] Run the script that works out the shared-memory parameters (shmall, shmax)
--------------------------------------------------------------------------------------------------------
# ./shmsetup.sh
--------------------------------------------------------------------------------------------------------
# Maximum shared segment size in bytes
kernel.shmmax = 4124028928
# Maximum number of shared memory segments in pages
kernel.shmall = 1006843
--------------------------------------------------------------------------------------------------------
CAREFUL!!!! The installer then warned that "kernel.shmax" is wrong. It says it should be 4124030976.
CAREFUL!!!! The installer then warned that "kernel.shmall" is wrong. It says it should be 2097152.
[3.2] Konfiguracia parametrov jadra pre ucely prevadzky Oracle DB
--------------------------------------------------------------------------------------------------------
# vi /etc/sysctl.conf
--------------------------------------------------------------------------------------------------------
...
# Konfiguracia parametrov jadra pre ucely prevadzky Oracle DB
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4124030976
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
# Let the local group oinstall (30001) create shared memory segments (SUSE Linux only.)
# Instalacia.
vm.hugetlb_shm_group=30001
# Let the local group dba (30004) create shared memory segments (SUSE Linux only.)
# Prevadzka.
vm.hugetlb_shm_group=30004
# Here the adapter "eth3" (eth3:INT) is used as the Oracle interconnect
net.ipv4.conf.eth3.rp_filter = 0
net.ipv4.icmp_echo_ignore_broadcasts = 0
--------------------------------------------------------------------------------------------------------
[3.3] Aktivujeme zmenene parametre jadra z bodu [3.2]
--------------------------------------------------------------------------------------------------------
# sysctl -p
--------------------------------------------------------------------------------------------------------
[3.4] Konfiguracia limitov pre ucely prevadzky Oracle DB
--------------------------------------------------------------------------------------------------------
# vi /etc/security/limits.conf
--------------------------------------------------------------------------------------------------------
...
# Limits (minimum values) for running an Oracle database
# oracle soft nproc 2047
# oracle hard nproc 16384
# oracle soft nofile 1024
# oracle hard nofile 65536
# oracle soft as unlimited
# oracle hard as unlimited
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft as unlimited
oracle hard as unlimited
oracle soft core unlimited
oracle hard core unlimited
oracle soft memlock 3500000
oracle hard memlock 3500000
grid soft nofile 131072
grid hard nofile 131072
grid soft nproc 131072
grid hard nproc 131072
grid soft as unlimited
grid hard as unlimited
grid soft core unlimited
grid hard core unlimited
grid soft memlock 3500000
grid hard memlock 3500000
--------------------------------------------------------------------------------------------------------
[3.5] Upravime globalne systemove (system wide) premenne prostredia (Plati pre interpreter BASH a KSH.)
--------------------------------------------------------------------------------------------------------
# vi /etc/profile.local
--------------------------------------------------------------------------------------------------------
# Konfiguracia globalnych systemovych premennych pre ucely prevadzky Oracle DB (BASH interpreter)
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/bash" ]; then
# I HAD TO COMMENT THIS OUT ulimit -p 131072
ulimit -n 131072
else
ulimit -u 131072 -n 131072
fi
fi
if [ $USER = "grid" ]; then
if [ $SHELL = "/bin/bash" ]; then
# I HAD TO COMMENT THIS OUT ulimit -p 131072
ulimit -n 131072
else
ulimit -u 131072 -n 131072
fi
fi
--------------------------------------------------------------------------------------------------------
[3.5.1] Set the right permissions on the file "/etc/profile.local"
--------------------------------------------------------------------------------------------------------
# chmod 644 /etc/profile.local
--------------------------------------------------------------------------------------------------------
[3.6] Configuring PAM (appended to the end of the file) for running an Oracle database
--------------------------------------------------------------------------------------------------------
# vi /etc/pam.d/login
--------------------------------------------------------------------------------------------------------
...
# Konfiguracia PAM pre ucely prevadzky Oracle DB
session required pam_limits.so
--------------------------------------------------------------------------------------------------------
[3.7] Konfiguracia lokalnych hostname zaznamov
I do not currently use this configuration, since every record is in DNS.
--------------------------------------------------------------------------------------------------------
# vi /etc/hosts
--------------------------------------------------------------------------------------------------------
# The loopback must not carry the host name. The Oracle cluster services are unhappy about it
# and the cluster will not come up, at least not on the second and later nodes.
127.0.0.1 localhost.localdomain localhost
# Oracle RAC - public IP
10.10.10.11 oradb01.domena.sk oradb01
10.10.10.12 oradb02.domena.sk oradb02
# Oracle RAC - virtual IP
10.10.10.21 oradb01-vip.domena.sk oradb01-vip
10.10.10.22 oradb02-vip.domena.sk oradb02-vip
# Oracle RAC - scan IP
10.10.10.31 oradb-scan.domena.sk oradb-scan
10.10.10.32 oradb-scan.domena.sk oradb-scan
10.10.10.33 oradb-scan.domena.sk oradb-scan
# Oracle RAC - interconnect IP
10.10.20.11 oradb01-int.domena.sk oradb01-int
10.10.20.12 oradb02-int.domena.sk oradb02-int
# Oracle RAC - symantec backend
10.10.30.11 oradb01-bck.domena.sk oradb01-bck
10.10.30.12 oradb02-bck.domena.sk oradb02-bck
# Oracle RAC - Symantec backend (in case the Oracle listener also listens on these addresses)
# 10.10.30.21 oradb01-bck-vip.domena.sk oradb01-bck-vip
# 10.10.30.22 oradb02-bck-vip.domena.sk oradb02-bck-vip
# Oracle RAC - OS/DB management
10.10.40.11 oradb01-mng.domena.sk oradb01-mng
10.10.40.12 oradb02-mng.domena.sk oradb02-mng
# Oracle RAC - OS/DB management (in case the Oracle listener also listens on these addresses)
# 10.10.40.21 oradb01-mng-vip.domena.sk oradb01-mng-vip
# 10.10.40.22 oradb02-mng-vip.domena.sk oradb02-mng-vip
--------------------------------------------------------------------------------------------------------
[3.8] Set umask for the accounts "grid" and "oracle"
--------------------------------------------------------------------------------------------------------
# su - grid
# vi .bash_profile
--------------------------------------------------------------------------------------------------------
umask 022
--------------------------------------------------------------------------------------------------------
# su - oracle
# vi .bash_profile
--------------------------------------------------------------------------------------------------------
umask 022
--------------------------------------------------------------------------------------------------------
[3.9] Nastavime spravny DNS server/y
--------------------------------------------------------------------------------------------------------
# vi /etc/resolv.conf
--------------------------------------------------------------------------------------------------------
search domena.sk
nameserver 10.10.40.13
--------------------------------------------------------------------------------------------------------
[3.10] Set up time synchronisation (an NTP server) - done with the SUSE configuration tool "Yast"
-------------------------------------------------------------------------------------------------
# vi /etc/ntp.conf
--------------------------------------------------------------------------------------------------------
...
server 10.10.40.13 iburst
...
--------------------------------------------------------------------------------------------------------
[3.11] Set the NTP server to bring the clock in gradually rather than in jumps (the "-x" switch)
--------------------------------------------------------------------------------------------------------
# vi /etc/sysconfig/ntp
--------------------------------------------------------------------------------------------------------
...
NTPD_OPTIONS="-x -g -u ntp:ntp"
...
--------------------------------------------------------------------------------------------------------
================================================================================================================
[4] Vytvorenie logickej particie "data" v logickom zvazku (skupine) "DATA" -> Oracle Install Destination
================================================================================================================
[4.1] Initialising a disk or a partition for use in LVM
--------------------------------------------------------------------------------------------------------
# pvcreate /dev/sdb
--------------------------------------------------------------------------------------------------------
[4.2] Creating the volume group named "DATA", using the disk sdb
--------------------------------------------------------------------------------------------------------
# vgcreate DATA /dev/sdb
--------------------------------------------------------------------------------------------------------
[4.3] Informacie o logickom zvazku (skupine) s nazvom 'DATA'
--------------------------------------------------------------------------------------------------------
# vgdisplay -v DATA
--------------------------------------------------------------------------------------------------------
[4.4] Vytvorenie 99 GB logickej particie s nazvom 'data' v logickom zvazku 'DATA'
--------------------------------------------------------------------------------------------------------
# lvcreate -L 99G -n data DATA
--------------------------------------------------------------------------------------------------------
[4.5] Creating an ext3 filesystem on the logical partition "/dev/DATA/data
--------------------------------------------------------------------------------------------------------
# mkfs.ext3 /dev/DATA/data
--------------------------------------------------------------------------------------------------------
[4.6] Rozlozenie diskovych particii / automaticke pripajanie po starte OS
--------------------------------------------------------------------------------------------------------
# vi /etc/fstab
--------------------------------------------------------------------------------------------------------
/dev/SYSTEM/swap1 swap swap defaults 0 0
/dev/SYSTEM/root / ext3 acl,user_xattr 1 1
/dev/disk/by-id/scsi-360022480eee5727b536ee6bcb234826e-part1 /boot ext3 acl,user_xattr 1 2
/dev/SYSTEM/home /home ext3 nosuid,nodev,noexec,acl,user_xattr 1 2
/dev/SYSTEM/tmp /tmp ext3 nosuid,nodev,acl,user_xattr 1 2
/dev/SYSTEM/usr /usr ext3 nodev,acl,user_xattr 1 2
/dev/SYSTEM/var /var ext3 nodev,acl,user_xattr 1 2
/dev/SYSTEM/varlog /var/log ext3 nosuid,nodev,noexec,acl,user_xattr 1 2
/dev/SYSTEM/varlogaudit /var/log/audit ext3 nosuid,nodev,noexec,acl,user_xattr 1 2
/dev/SYSTEM/vartmp /var/tmp ext3 nosuid,nodev,noexec,acl,user_xattr 1 2
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
# Oracle Install Destination
/dev/mapper/DATA-data /data ext3 nodev,acl,user_xattr 0 0
--------------------------------------------------------------------------------------------------------
[4.7] Creating the directory for the mount point /data
--------------------------------------------------------------------------------------------------------
# mkdir -p /data
--------------------------------------------------------------------------------------------------------
[4.8] Mounting every mount point named in "/etc/fstab"
--------------------------------------------------------------------------------------------------------
# mount -a
--------------------------------------------------------------------------------------------------------
================================================================================================================
[5] Priprava a vytvorenie zdielanych diskov ASM
================================================================================================================
[5.0] Installing Oracle DB (RAC+ASM) needs these block devices
--------------------------------------------------------------------------------------------------------
Name | Capacity | Mount point | Purpose | ASM DISK
--------------------------------------------------------------------------------------------------------
ORADB-CRS | 1073 MB | NA | Cluster Registry (CRS) | ASMDISK1
ORADB-DATA01 | 107,4 GB | NA | Oracle Data Disk1 (ASM) | ASMDISK2
ORADB-DATA02 | 107,4 GB | NA | Oracle Data Disk2 (ASM) | ASMDISK3
ORADB-FRA | 214,7 GB | NA | Fast Recovery Area (FRA) | ASMDISK4
ORADB-ONTRLG | 21,5 GB | NA | Online Transakcne Logy (ONTRLG) | ASMDISK5
--------------------------------------------------------------------------------------------------------
[5.1] Create a primary Linux (type=83) partition on the block device "/dev/sdc"
--------------------------------------------------------------------------------------------------------
# fdisk /dev/sdc
--------------------------------------------------------------------------------------------------------
Command (m for help): n
Command action (e=extended, p=primary): p
Partition number (1-4, default 1): 1
First sector (2048-2095103, default 2048): ENTER
Last sector, +sectors or +size{K,M,G} (2048-2095103, default 2095103): ENTER
--------------------------------------------------------------------------------------------------------
[5.2] Create the partitions on every shared block device that is to serve as an ASM disk
----------------------------------------------------------------------------------------
# fdisk /dev/sdd
# fdisk /dev/sde
# fdisk /dev/sdf
# fdisk /dev/sdg
--------------------------------------------------------------------------------------------------------
Postup rovnaky ako v bode [6.1].
--------------------------------------------------------------------------------------------------------
[5.3] Na vsetkych Oracle nodoch aktualizujeme informacie jadra o zmeny vykonane v tabulkach particii
--------------------------------------------------------------------------------------------------------
# partprobe
--------------------------------------------------------------------------------------------------------
[5.4] Nakonfigurujeme ASMLIB na vsetkych Oracle nodoch
--------------------------------------------------------------------------------------------------------
# oracleasm configure -i
--------------------------------------------------------------------------------------------------------
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
--------------------------------------------------------------------------------------------------------
[5.5] Nacitame ASMLIB modul na vsetkych Oracle nodoch
--------------------------------------------------------------------------------------------------------
# oracleasm init
--------------------------------------------------------------------------------------------------------
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm
--------------------------------------------------------------------------------------------------------
[5.6] Vytvorime ASM DISKY na jednom z Oracle nodov
--------------------------------------------------------------------------------------------------------
# oracleasm createdisk ASMDISK1 /dev/ORADB-CRS
# oracleasm createdisk ASMDISK2 /dev/ORADB-DATA01
# oracleasm createdisk ASMDISK3 /dev/ORADB-DATA02
# oracleasm createdisk ASMDISK4 /dev/ORADB-FRA
# oracleasm createdisk ASMDISK5 /dev/ORADB-ONTRLG
--------------------------------------------------------------------------------------------------------
[5.7] Vykoname scan ASM DISKOV a vypiseme zoznam vsetkych ASM DISKOV na vsetkych Oracle nodoch
--------------------------------------------------------------------------------------------------------
# oracleasm scandisks
# oracleasm listdisks
--------------------------------------------------------------------------------------------------------
ASMDISK1
ASMDISK2
ASMDISK3
ASMDISK4
ASMDISK5
--------------------------------------------------------------------------------------------------------
[5.8] Check that the new ASM disks will be found by the ASM libraries (Oracle ASMLIB) on every Oracle node during the GRID infrastructure installation
--------------------------------------------------------------------------------------------------------
# oracleasm-discover 'ORCL:*'
--------------------------------------------------------------------------------------------------------
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:ASMDISK1 [2095104 blocks (1072693248 bytes), maxio 128]
Discovered disk: ORCL:ASMDISK2 [419428352 blocks (214747316224 bytes), maxio 128]
Discovered disk: ORCL:ASMDISK3 [209713152 blocks (107373133824 bytes), maxio 128]
Discovered disk: ORCL:ASMDISK4 [209713152 blocks (107373133824 bytes), maxio 128]
Discovered disk: ORCL:ASMDISK5 [41940992 blocks (21473787904 bytes), maxio 128]
--------------------------------------------------------------------------------------------------------
================================================================================================================
[6] Konfiguracia diskoveho subsystemu pre ucely prevadzky Oracle DB
================================================================================================================
[6.0] Installing Oracle DB (RAC + ASM) needs these block devices
--------------------------------------------------------------------------------------------------------
Name | Capacity | Mount point | Purpose | Servers ORADB01,02
--------------------------------------------------------------------------------------------------------
ORADB-INSTALL | 107.4 GB | /data | Oracle install destination | One per server = 2x.
ORADB-CRS | 1073 MB | NA | Cluster Registry (CRS) | Zdielany oboma servermi = 1x.
ORADB-DATA01 | 107,4 GB | NA | Oracle Data Disk1 (ASM) | Zdielany oboma servermi = 1x.
ORADB-DATA02 | 107,4 GB | NA | Oracle Data Disk2 (ASM) | Zdielany oboma servermi = 1x.
ORADB-FRA | 214,7 GB | NA | Fast Recovery Area (FRA) | Zdielany oboma servermi = 1x.
ORADB-ONTRLG | 21,5 GB | NA | Online Transakcne Logy (ONTRLG) | Zdielany oboma servermi = 1x.
--------------------------------------------------------------------------------------------------------
[6.1] Preverime ake blokove zariadenia mame na servery ORADB01 dostupne
--------------------------------------------------------------------------------------------------------
# ls -ld /sys/block/sd*/device
--------------------------------------------------------------------------------------------------------
lrwxrwxrwx 1 root root 0 Sep 26 11:40 /sys/block/sda/device -> ../../../2:0:0:0
lrwxrwxrwx 1 root root 0 Sep 26 11:40 /sys/block/sdb/device -> ../../../3:0:1:0 -> ORADB-INSTALL
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdc/device -> ../../../5:0:0:0 -> ORADB-CRS
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdd/device -> ../../../5:0:0:1 -> ORADB-DATA01
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sde/device -> ../../../5:0:0:2 -> ORADB-DATA02
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdf/device -> ../../../5:0:0:3 -> ORADB-FRA
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdg/device -> ../../../5:0:0:4 -> ORADB-ONTRLG
--------------------------------------------------------------------------------------------------------
[6.2] On server ORADB01, check the unique identifiers of the block devices, which are then
pouziju v UDEV pravidlach
--------------------------------------------------------------------------------------------------------
# /lib/udev/scsi_id -g -u /dev/sdc
14f504e46494c455255443464486a2d4b5775472d584a424d
# /lib/udev/scsi_id -g -u /dev/sdd
14f504e46494c45525757386a6e6a2d454b64722d42326733
# /lib/udev/scsi_id -g -u /dev/sde
14f504e46494c45524a47466644452d584b466b2d45433474
# /lib/udev/scsi_id -g -u /dev/sdf
14f504e46494c45525577536271332d54646c4a2d6a58494a
# /lib/udev/scsi_id -g -u /dev/sdg
14f504e46494c455244386a6e36682d334a6e442d47703477
--------------------------------------------------------------------------------------------------------
[6.3] Preverime ake blokove zariadenia mame na servery ORADB02 dostupne
--------------------------------------------------------------------------------------------------------
# ls -ld /sys/block/sd*/device
--------------------------------------------------------------------------------------------------------
lrwxrwxrwx 1 root root 0 Sep 26 11:40 /sys/block/sda/device -> ../../../2:0:0:0
lrwxrwxrwx 1 root root 0 Sep 26 11:40 /sys/block/sdb/device -> ../../../3:0:1:0 -> ORADB-INSTALL
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdc/device -> ../../../5:0:0:0 -> ORADB-CRS
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdd/device -> ../../../5:0:0:1 -> ORADB-DATA01
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sde/device -> ../../../5:0:0:2 -> ORADB-DATA02
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdf/device -> ../../../5:0:0:3 -> ORADB-FRA
lrwxrwxrwx 1 root root 0 Oct 4 12:22 /sys/block/sdg/device -> ../../../5:0:0:4 -> ORADB-ONTRLG
--------------------------------------------------------------------------------------------------------
[6.4] On server ORADB02, check the unique identifiers of the block devices, which are then
pouziju v UDEV pravidlach.
--------------------------------------------------------------------------------------------------------
# /lib/udev/scsi_id -g -u /dev/sdc
14f504e46494c455255443464486a2d4b5775472d584a424d
# /lib/udev/scsi_id -g -u /dev/sdd
14f504e46494c45525757386a6e6a2d454b64722d42326733
# /lib/udev/scsi_id -g -u /dev/sde
14f504e46494c45524a47466644452d584b466b2d45433474
# /lib/udev/scsi_id -g -u /dev/sdf
14f504e46494c45525577536271332d54646c4a2d6a58494a
# /lib/udev/scsi_id -g -u /dev/sdg
14f504e46494c455244386a6e36682d334a6e442d47703477
--------------------------------------------------------------------------------------------------------
[6.5] Oracle ASM needs the right owner (grid) and group (asmadmin) set on the shared
block devices (iSCSI LUNs) intended for ASM.
--------------------------------------------------------------------------------------------------------
# vi /etc/udev/rules.d/59-persistent-disk.rules
--------------------------------------------------------------------------------------------------------
# The first partition of the block device with ID "14f504e46494c455255443464486a2d4b5775472d584a424d" will be called "ORADB-CRS" and "ASMDISK1".
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455255443464486a2d4b5775472d584a424d", SYMLINK+="ASMDISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455255443464486a2d4b5775472d584a424d", SYMLINK+="ORADB-CRS", OWNER="grid", GROUP="asmadmin", MODE="0660"
# The first partition of the block device with ID "14f504e46494c45525757386a6e6a2d454b64722d42326733" will be called "ORADB-DATA01" and "ASMDISK2".
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45525757386a6e6a2d454b64722d42326733", SYMLINK+="ASMDISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45525757386a6e6a2d454b64722d42326733", SYMLINK+="ORADB-DATA01", OWNER="grid", GROUP="asmadmin", MODE="0660"
# The first partition of the block device with ID "14f504e46494c45524a47466644452d584b466b2d45433474" will be called "ORADB-DATA02" and "ASMDISK3".
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524a47466644452d584b466b2d45433474", SYMLINK+="ASMDISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524a47466644452d584b466b2d45433474", SYMLINK+="ORADB-DATA02", OWNER="grid", GROUP="asmadmin", MODE="0660"
# The first partition of the block device with ID "14f504e46494c45525577536271332d54646c4a2d6a58494a" will be called "ORADB-FRA" and "ASMDISK4".
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45525577536271332d54646c4a2d6a58494a", SYMLINK+="ASMDISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45525577536271332d54646c4a2d6a58494a", SYMLINK+="ORADB-FRA", OWNER="grid", GROUP="asmadmin", MODE="0660"
# The first partition of the block device with ID "14f504e46494c455244386a6e36682d334a6e442d47703477" will be called "ORADB-ONTRLG" and "ASMDISK5".
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455244386a6e36682d334a6e442d47703477", SYMLINK+="ASMDISK5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455244386a6e36682d334a6e442d47703477", SYMLINK+="ORADB-ONTRLG", OWNER="grid", GROUP="asmadmin", MODE="0660"
--------------------------------------------------------------------------------------------------------
[6.6] Were an MPIO layer (DM multipath) in use, the UDEV rules would look like this.
--------------------------------------------------------------------------------------------------------
# vi /etc/udev/rules.d/12-persistent-disk.rules
--------------------------------------------------------------------------------------------------------
...
# Owner (grid) and group (dba) for Oracle on the multipath block devices
ENV{DM_NAME}=="ORADB01-ORA-INSTALL", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-CRS", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-DATA1", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-DATA2", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-DATA3", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-DATA4", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-FRA", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="ORADB-ONTRLG", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
LABEL="dm_end"
--------------------------------------------------------------------------------------------------------
================================================================================================================
[7] Creating the directory structure an Oracle database needs
================================================================================================================
7 Creating Required Directories
REF: https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#CEGEGDBA
********************************************************************************************************
[7.1] Creating the directory structure an Oracle database needs
--------------------------------------------------------------------------------------------------------
# mkdir -p /data/u01/app/grid11204
# mkdir -p /data/u01/app/grid
# chown -R grid:oinstall /data/u01
# mkdir -p /data/u01/app/oracle
# chown oracle:oinstall /data/u01/app/oracle
# chmod -R 775 /data/u01
--------------------------------------------------------------------------------------------------------
================================================================================================================
[8] SSH ekvivalencia pre ucty "grid" a "oracle"
================================================================================================================
[8.1] SSH equivalence is set up during the Oracle Grid Infrastructure installation
--------------------------------------------------------------------------------------------------------
See the document "LH-2016_02_Oracle-11g2_install_GRID.txt"; at this point only create the SSH keys.
--------------------------------------------------------------------------------------------------------
[8.1.1] Creating SSH keys for the user "oracle" on every Oracle node
--------------------------------------------------------------------------------------------------------
ORADB01,02# su - oracle
ORADB01,02# mkdir ~/.ssh
ORADB01,02# chmod 700 ~/.ssh
ORADB01,02# ssh-keygen -t rsa
ORADB01,02# ssh-keygen -t dsa
[8.1.2] Creating SSH keys for the user "grid" on every Oracle node
--------------------------------------------------------------------------------------------------------
ORADB01,02# su - grid
ORADB01,02# mkdir ~/.ssh
ORADB01,02# chmod 700 ~/.ssh
ORADB01,02# ssh-keygen -t rsa
ORADB01,02# ssh-keygen -t dsa
[8.2] Na servery ORADB01 otestujeme funkcionalitu bezhesloveho SSH prihlasovania pre ucet "oracle"
--------------------------------------------------------------------------------------------------------
ORADB01# su - oracle
ORADB01# ssh ORADB02 date
ORADB01# ssh ORADB02-int date
ORADB01# ssh ORADB02.domena.sk date
ORADB01# su - grid
ORADB01# ssh ORADB02 date
ORADB01# ssh ORADB02-int date
ORADB01# ssh ORADB02.domena.sk date
[8.3] Na servery ORADB01 a ORADB02 otestujeme funkcionalitu bezhesloveho SSH prihlasovania
--------------------------------------------------------------------------------------------------------
ORADB02# su - oracle
ORADB02# ssh ORADB01 date
ORADB02# ssh ORADB01-int date
ORADB02# ssh ORADB01.domena.sk date
ORADB02# su - grid
ORADB02# ssh ORADB01 date
ORADB02# ssh ORADB01-int date
ORADB02# ssh ORADB01.domena.sk date