Installation Of Linux RAC.

Oracle Linux 6 installation as you would for a basic server. A general pictorial guide to the installation can be found here. More specifically, it should be a server installation with a minimum of 4G+ swap, firewall and SELinux disabled and the following package groups installed:

• Base System > Base
• Base System > Client management tools
• Base System > Compatibility libraries
• Base System > Hardware monitoring utilities
• Base System > Large Systems Performance
• Base System > Network file system client
• Base System > Performance Tools

• Base System > Perl Support
• Servers > Server Platform
• Servers > System administration tools
• Desktops > Desktop
• Desktops > Desktop Platform
• Desktops > Fonts
• Desktops > General Purpose Desktop
• Desktops > Graphical Administration Tools
• Desktops > Input Methods
• Desktops > X Window System
• Applications > Internet Browser
• Development > Additional Development
• Development > Development Tools

To be consistent with the rest of the article, the following information should be set during the installation:

• hostname: ol6-112-rac1.localdomain
• IP Address eth0: 192.168.0.111 (public address)
• Default Gateway eth0: 192.168.0.1 (public address)
• IP Address eth1: 192.168.1.111 (private address)
• Default Gateway eth1: none

You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article.
Once the basic installation is complete, install the following packages whilst logged in as the root user. This includes the 64-bit and 32-bit versions of some packages. The commented out packages are those already installed if you have followed the suggested package selection. There is no suitable "oracle-validated" package available at this time, so the packages need to be installed manually.

# From Oracle Linux 6 DVD
cd /media/cdrom/Server/Packages
#rpm -Uvh binutils-2.*
#rpm -Uvh compat-libstdc++-33*
#rpm -Uvh elfutils-libelf-0.*
#rpm -Uvh libaio-0.*
rpm -Uvh libaio-devel-0.*
#rpm -Uvh sysstat-9.*
#rpm -Uvh glibc-2.*
#rpm -Uvh glibc-common-2.*
#rpm -Uvh glibc-devel-2.* glibc-headers-2.*
rpm -Uvh ksh-2*
#rpm -Uvh make-3.*
#rpm -Uvh libgcc-4.*
#rpm -Uvh libstdc++-4.*
rpm -Uvh libstdc++-4.*.i686*
#rpm -Uvh libstdc++-devel-4.*
#rpm -Uvh gcc-4.*x86_64*
#rpm -Uvh gcc-c++-4.*x86_64*
#rpm -Uvh --allfiles elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
rpm -Uvh elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
rpm -Uvh libtool-ltdl*i686*
rpm -Uvh ncurses*i686*
rpm -Uvh readline*i686*
rpm -Uvh unixODBC*
cd /
eject


# Install the following package from the Oracle grid media after you've defined groups.
cd /your/path/to/grid/rpm
rpm -Uvh cvuqdisk*

Oracle Installation Prerequisites

Perform the following steps whilst logged into the "rac1" virtual machine as the root user.
If you are not using DNS, the "/etc/hosts" file must contain the following information.

127.0.0.1       localhost.localdomain   localhost
# Public
192.168.0.111   ol6-112-rac1.localdomain        ol6-112-rac1
192.168.0.112   ol6-112-rac2.localdomain        ol6-112-rac2
# Private
192.168.1.111   ol6-112-rac1-priv.localdomain   ol6-112-rac1-priv
192.168.1.112   ol6-112-rac2-priv.localdomain   ol6-112-rac2-priv
# Virtual
192.168.0.113   ol6-112-rac1-vip.localdomain    ol6-112-rac1-vip
192.168.0.114   ol6-112-rac2-vip.localdomain    ol6-112-rac2-vip
# SCAN
192.168.0.115   ol6-112-scan.localdomain ol6-112-scan
192.168.0.116   ol6-112-scan.localdomain ol6-112-scan
192.168.0.117   ol6-112-scan.localdomain ol6-112-scan

Note. The SCAN address should not really be defined in the hosts file. Instead is should be defined on the DNS to round-robin between 3 addresses on the same subnet as the public IPs. For this installation, we will compromise and use the hosts file.
If you are using DNS, then only the first line must be present in the "/etc/hosts" file. The other entries are defined in the DNS, as described here. Having said that, I normally include everything except the SCAN entries when using DNS.
Add or amend the following lines to the "/etc/sysctl.conf" file.

fs.aio-max-nr = 1048576
fs.file-max = 6815744
#kernel.shmall = 2097152 
#kernel.shmmax = 1054504960
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586

Run the following command to change the current kernel parameters.

/sbin/sysctl -p

Add the following lines to the "/etc/security/limits.conf" file.

oracle              soft    nproc   2047
oracle              hard    nproc   16384
oracle              soft    nofile  4096
oracle              hard    nofile  65536
oracle              soft    stack   10240

Add the following lines to the "/etc/pam.d/login" file, if it does not already exist.

session    required     pam_limits.so

Disable secure linux by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.

SELINUX=disabled

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. In this case we will deconfigure NTP.

# service ntpd stop
Shutting down ntpd:                                        [  OK  ]
# chkconfig ntpd off
# mv /etc/ntp.conf /etc/ntp.conf.orig
# rm /var/run/ntpd.pid

If you are using NTP, you must add the "-x" option into the following line in the "/etc/sysconfig/ntpd" file.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

Then restart NTP.

# service ntpd restart

Create the new groups and users.

groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle

Create the directories in which the Oracle software will be installed.

mkdir -p  /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01/

Login as the oracle user and add the following lines at the end of the .bash_profile file.

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=ol6-112-rac1.localdomain; export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/db_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Create a file called "/home/oracle/grid_env" with the following contents.

ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Create a file called "/home/oracle/db_env" with the following contents.

ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Once the "/home/oracle/grid_env" has been run, you will be able to switch between environments as follows.

$ grid_env
$ echo $ORACLE_HOME
/u01/app/11.2.0/grid
$ db_env
$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/db_1
$

Install Guest Additions

Log into the VM as the root user and add the "divider=10" option to the kernel boot options in "/etc/grub.conf" file to reduce the idle CPU load. The entry should look something like this.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_ol6112rac1-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server-uek (2.6.32-100.34.1.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-100.34.1.el6uek.x86_64 ro root=/dev/mapper/vg_ol6112rac1-lv_root rd_LVM_LV=vg_ol6112rac1/lv_root rd_LVM_LV=vg_ol6112rac1/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk  rhgb quiet divider=10
        initrd /initramfs-2.6.32-100.34.1.el6uek.x86_64.img
title Oracle Linux Server (2.6.32-131.0.15.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=/dev/mapper/vg_ol6112rac1-lv_root rd_LVM_LV=vg_ol6112rac1/lv_root rd_LVM_LV=vg_ol6112rac1/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk crashkernel=auto rhgb quiet divider=10
        initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img

Click on the "Devices > Install Guest Additions" menu option at the top of the VM screen, then run the following commands.

cd cd /media/VBOXADDITIONS_4.1.4_74291
export MAKE='/usr/bin/gmake –i'
sh ./VBoxLinuxAdditions.run

The VM will need to be restarted for the additions to be used properly. The next section requires a shutdown so no additional restart is needed at this time.

Create Shared Disks

Shut down the "rac1" virtual machine using the following command.

# shutdown -h now

Create 4 sharable virtual disks and associate them as virtual media using one of the following sets of commands on the host server.

$ mkdir -p /u04/VirtualBox/ol6-112-rac
$ cd /u04/VirtualBox/ol6-112-rac
$
$ # Create the disks and associate them with VirtualBox as virtual media.
$ VBoxManage createhd --filename asm1.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm2.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm3.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm4.vdi --size 5120 --format VDI --variant Fixed
$
$ # Connect them to the VM.
$ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
$ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
$ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
$ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
$
$ # Make shareable.
$ VBoxManage modifyhd asm1.vdi --type shareable
$ VBoxManage modifyhd asm2.vdi --type shareable
$ VBoxManage modifyhd asm3.vdi --type shareable
$ VBoxManage modifyhd asm4.vdi --type shareable

Start the "ol6-112-rac1" virtual machine by clicking the "Start" button on the toolbar. When the server has started, log in as the root user so you can configure the shared disks. The current disks can be seen by issuing the following commands.

# cd /dev
# ls sd*
sda  sda1  sda2  sdb  sdc  sdd  sde
#

Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the expected fdisk output for the sdb disk.

# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x62be91cf.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652): 
Using default value 652

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
#

In each case, the sequence of answers is "n", "p", "1", "Return", "Return" and "w".
Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command.

# cd /dev
# ls sd*
sda  sda1  sda2  sdb  sdb1  sdc  sdc1  sdd  sdd1  sde  sde1
#

Configure your UDEV rules, as shown here.
The SCSI ID of my disks are displayed below.

# /sbin/scsi_id -g -u /dev/sdb1
1ATA_VBOX_HARDDISK_VBbfd37c1c-9b7c9336
# /sbin/scsi_id -g -u /dev/sdc1
1ATA_VBOX_HARDDISK_VB46bd30ca-ef5d5d6d
# /sbin/scsi_id -g -u /dev/sdd1
1ATA_VBOX_HARDDISK_VB5d431382-68a070ad
# /sbin/scsi_id -g -u /dev/sde1
1ATA_VBOX_HARDDISK_VB2ceafd7d-e56ad1d8
#

Using these, the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file contained the following entries.

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBbfd37c1c-9b7c9336", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB46bd30ca-ef5d5d6d", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB5d431382-68a070ad", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB2ceafd7d-e56ad1d8", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"

Load updated block device partition tables.

# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# /sbin/partprobe /dev/sdd1
# /sbin/partprobe /dev/sde1

Test the rules are working as expected.

# /sbin/udevadm test /block/sdb/sdb1

Reload the UDEV rules and start UDEV.

# /sbin/udevadm control --reload-rules
# /sbin/start_udev

The disks should now be visible and have the correct ownership using the following command. If they are not visible, your UDEV configuration is incorrect and must be fixed before you proceed.

# ls -al /dev/asm*
brw-rw---- 1 oracle dba 8, 17 Oct 12 14:39 /dev/asm-disk1
brw-rw---- 1 oracle dba 8, 33 Oct 12 14:38 /dev/asm-disk2
brw-rw---- 1 oracle dba 8, 49 Oct 12 14:39 /dev/asm-disk3
brw-rw---- 1 oracle dba 8, 65 Oct 12 14:39 /dev/asm-disk4
#

The shared disks are now configured for the grid infrastructure.

Clone the Virtual Machine

Later versions of VirtualBox allow you to clone VMs, but these also attempt to clone the shared disks, which is not what we want. Instead we must manually clone the VM.
Shut down the "ol6-112-rac1" virtual machine using the following command.

# shutdown -h now

Manually clone the "ol6-112-rac1.vdi" disk using the following commands on the host server.

• mkdir -p /u03/VirtualBox/ol6-112-rac2

• VBoxManage clonehd /u01/VirtualBox/ol6-112-rac1/ol6-112-rac1.vdi /u03/VirtualBox/ol6-112-rac2/ol6-112-rac2.vdi

 Now create the hardisk in the same way as you did in "ol6-112-rac1", with the exception of using an existing rac2.vdi virtual hard drive.

Remember to add the second network adaptor as you did on the "ol6-112-rac1" VM. When the VM is created, attach the shared disks to this VM.


Next Blog I ll tell you the steps about Installing the Grid Infrastructure