I’ve recently installed Oracle RAC on both RedHat 4 and Suse Enterprise 9. The most recent installation reminded me of an issue which has come up more than once: persistance of raw device permissions. If you remember installing Oracle on SLES8, RHEL2 and RHEL3, then you remember that you would just chmod the /dev files and that was it. But with RHEL4 and SLES9 (and all newer releases) everything breaks when you reboot! It doesn’t take long to figure out that the /dev file permissions were reset to their defaults.
The root cause is a change that was introduced several years ago in the 2.6 Linux kernel called udev. Unlike the 2.4 Linux system, where the device nodes in the /dev directory were a static set of files, udev in 2.6 dynamically creates files for the devices that are actually present on a system. It was also designed from the ground up to support persistent device naming. Naturally udev is nothing new to Linux admins; but if you’re not familiar with it and are interested in a good overview then check out Greg Kroah-Hartman’s talk and paper from the 2003 Linux Symposium.
What does this mean to DBAs installing Oracle RAC on Linux? What is the best and most secure way to get the permissions right for Oracle on your system?
The most common approach is to add a few lines to the local startup script which will chmod the devices at boot-time. On RHEL, /etc/rc.d/rc.local gets run last (after all the startup scripts) and on SLES, /etc/rc.d/boot.local gets run after all the boot scripts (including the udev setup) but before entering a runlevel. Many people use these files just like “autoexec.bat” from the DOS days to start services without having to write System-V compatible startup scripts. In reality that’s a lazy approach: there is a much better way to do it.
The best way to assign permissions to udev devices is by using udev itself. A scheme for setting default permissions is already built into the carefully designed udev model. Whenever the machine boots or a new device is added, the kernel sends a message to a hotplug script (usually the default /sbin/hotplug but configurable through /proc/sys/kernel/hotplug). This script, among other things, passes the information to udev. First, udev reads its RULES configuration file(s) to determine what filename it should create under /dev for the device. Second, udev checks its PERMISSIONS configuration file(s) to determine the owner, group, and permissions for that device file. Setting the proper boot-time permissions to your raw devices for Oracle is no more complicated than adding a few lines to the right configuration file.
First of all, udev is only available in distributions based on a 2.6 or newer kernel – that means RHEL 4+ and SLES 9+. RedHat and Suse take slightly different approaches to udev configuration; we’ll look at both of them.
SLES contains a single file which holds all of the RULES and a single file which holds all of the PERMISSIONS. The filenames are set in /etc/udev/udev.conf and by default /etc/udev/udev.permissions is the PERMISSIONS file. Entries at the top of this file take precedence over those later in the file. Toward the bottom of the file you will see these two lines which establish the default permissions for raw devices:
In the interest of security you should not change the defaults, but instead you should add a line at the top of the file for each raw device with properly restrictive permissions. For example:
# 26-Feb-06 : jschneider : raw device default perms for oracle; 1/2 OCR, 3-5 Vote/Quorum raw/raw1:root:oinstall:640 raw/raw2:root:oinstall:640 raw/raw3:oracle:oinstall:644 raw/raw4:oracle:oinstall:644 raw/raw5:oracle:oinstall:644
One thing to watch out for on SLES: this file is part of the udev rpm; if you upgrade your system then you will need to make sure that your changes don’t get removed.
RHEL takes a slightly cleaner approach: it sets up a directory which can contain multiple configuration files. This means that you can put your configuration in your own file and not worry about interfering with a system upgrade. By default /etc/udev/permissions.d is the directory. You will see that the default raw device permissions in RedHat’s config file 50-udev.permissions are the same as Suse:
# raw devices ram*:root:disk:0660 raw/*:root:disk:0660
But now you can create your own file to contain your Oracle updates:
[root@rh4lab15 ~]# cat <<EOF >/etc/udev/permissions.d/10-oracle.permissions > # 26-Feb-06 : jschneider : raw device default perms for oracle; 1/2 OCR, 3-5 Vote/Quorum > raw/raw1:root:oinstall:640 > raw/raw2:root:oinstall:640 > raw/raw3:oracle:oinstall:644 > raw/raw4:oracle:oinstall:644 > raw/raw5:oracle:oinstall:644 > EOF
Filenames from the PERMISSIONS directory that end with “.permissions” will be read in lexical order and the first matching entry will be used.
So what are the properly restrictive permissions? The RAC FAQ in metalink note 220970.1 claims that the perms are changed during the installation process… but it lists the exact same perms before and after install. Pre-install the permissions should match the install guide. The OCR should be owned by root:oinstall with permissions 640. The install guide gives two different sets of permissions for the voting disks in two places; you should create them according to the first set as [crs user]:oinstall with permissions 644. The crs user can be obtained be running
As one final note, in 10.2 you no longer need to create raw devices for ASM – but of course you’ll still need to use this same method to make the /dev/sd* files owned by oracle:dba for ASM to read them.
That’s it! Harnessing udev is a cleaner, more secure and less error-prone way to set default boot-time permissions on RHEL4+ and SLES9+.
Have you tried create devices under /dev using this rule?
KERNEL=”sd?1″, BUS=”scsi”, PROGRAM=”/sbin/scsi_id”, RESULT=”3600143800005eda0b43b826e900e000f”, NAME=”ocr”
In RHEL 4 U5 does not seem to work, permissions neither
Hey sorry for the slow response – I was on vacation much of last week. :)
Anyway, I don’t have anything earth-shattering to add… I didn’t mess around with the RULES much, just the PERMISSIONS. I’ve always left the default names for raw devices. Have you gotten this to work since you posted the comment last week?
Yea I got it to work, after several reboot it suddently started to work…
I had problems with rules in RHEL4U4 and it does not seem to work as documented for IBM storage DS4800 using the mpp driver, but if i reboot i can see the raw devices created, but if i try udevtest it does not seem to work