The trouble with Linux? No… the trouble with computers in general – is that they keep changing! Solaris 10 comes out, Oracle 11g, Red Hat 5… and everything works different!! It’s a full-time job just trying to keep up with everything.
Almost exactly one year ago I wrote about using udev on 2.6 kernels to set the proper permissions for Oracle RAC. Two weeks after that post (March 14) Red Hat Enterprise Linux 5 was released and changed everything.
In my original post, I demonstrated how to create a PERMISSIONS file that udev would use when creating the device nodes. This worked on RHEL4 and SLES9. However this week I’ve been helping a client deploy 11g RAC on a RHEL5-based cluster – and I remembered that the PERMISSIONS facility was removed from udev in RH5. Seems like I remember reading something about having a single source of configuration for udev, which makes sense… so maybe they picked the RULES. (You’ll remember from my previous post that RULES are processes right before PERMISSIONS.) This is just as well since RULES are actually quite a bit more powerful than PERMISSIONS.
So on RHEL5 and OEL5 – in order to conform to Linux Best Practices – we now have to set correct RAC file permissions using udev RULES. To get started, we need to review how RULES work. The udev manual page gives a good overview of rules processing. But of course there are plenty of great tutorials that go deeper if you’re looking for more.
Block Devices and Raw Devices
Now of course I’m not the first person to notice that there’s no permissions.d directory on RHEL5 and OEL5. Last September, Grégory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block devices. I don’t speak French but I was able to copy the text of his RULES files:
# Oracle Configuration Registry
KERNEL=="sdb[8-9]", OWNER="root", GROUP="oinstall", MODE="640"
# Voting Disks
KERNEL=="sdb1[0-2]", OWNER="oracle", GROUP="oinstall", MODE="640"
# ASM Devices
KERNEL=="sdb[5-7]", OWNER="oracle", GROUP="dba", MODE="660"
Many people are still configuring raw devices for their voting disks and cluster registries even though this is not necessary. This Oracle Forums thread gives an sample of using a RULES file to set permissions for raw devices. It says to create a file called /etc/udev/rules.d/65-raw-permissions.rules with these contents:
# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"
In fact there’s a great Oracle Whitepaper on the udev and multipathing that was published all the way back in June of 2007. It gives another sample configuration for raw devices on Red Hat 5.
Linux Multipath (Device Mapper) Devices
That’s all great. However… I’m doing an implementation on RHEL5 right now and we’re using the device mapper to multipath connections to the SAN. And the Oracle white paper – which goes into marvelous depth about multipath, udev and RH5 – never tells us anything about configuring multipath, udev and RH5 all together! So I have to figure this one out on my own. No harm done; it provided an interesting challenge for the day. :)
Multipath devices are a bit tricky. Usually if you’re using multipath then you don’t want to assume that devices will always be discovered in the same order. (This is the purpose of assigning friendly names by WWN in the multipath.conf file with the alias directive.) In fact, on our cluster the devices were assigned in different orders on different nodes – the ocr was dm-9 on one node and dm-10 on another. How do you write a RULES entry if you don’t know what the name of the device is?
I finally did get the multipath rules file to gave the proper permissions to my OCR and Voting Disks based on alias. However I was only able to change the /dev/dm-* files and not the /dev/mapper/* nodes. (Those nodes are not created by udev.) Therefor I had to use the aliases in /dev/mpath – not the aliases in /dev/mapper – when running the Oracle installer. How does it work? The symlinks in /dev/mpath are created by udev RULES – so all I had to do was piggyback on the udev config that created them and modify the permissions of the underlying devices.
This client used the aliases vote1, vote2, vote3, ocr1 and ocr2. Here’s the config file /etc/udev/rules.d/40-multipath.rules with my changes in bold:
# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | /bin/grep
^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p'
-j %M -m %m"
PROGRAM=="/sbin/dmsetup ls --target multipath --exec /bin/basename -j %M -m %m",
RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", GOTO="check_cluster_devs"
PROGRAM!="/bin/bash -c '/sbin/dmsetup info -c --noheadings -j %M -m %m | /bin/grep
-q .*:.*:.*:.*:.*:.*:.*:part[0-9]*-mpath-'", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup ls --target linear --exec /bin/basename -j %M -m %m", NAME="%k",
RESULT=="?*", SYMLINK="mpath/%c", OPTIONS="last_rule"
GOTO="end_mpath"
LABEL="check_cluster_devs"
RESULT=="ocr*", GROUP="dba", MODE="640"
RESULT=="vote*", OWNER="oracle", GROUP="dba", MODE="640"
OPTIONS="last_rule"
LABEL="end_mpath"
Ardent Performance Computing 
You could use ASMLib, as you are using ASM. This is the “right” way of doing it. This simplifies things, as ASM sets its own (required) permissions according to your predefinitions. Simple and elegant.
You don’t need to define RAW devices, and ASMLib makes use of labels, so life is good even if you have many devices, or if you experience LUN (sd) device renames. Also, God forbid, if both (some of) your servers see different LUN names. You needn’t fight with the system to match the mpath names, permissions, binding, etc. Just label them with ASMLib, and you’re happy.
Ez
LikeLike
Posted by Ez Aton | October 4, 2010, 7:40 amHello all – thanks for all the discussion. Sorry I haven’t been so responsive on this particular post! There are lots of great comments here.
First, @Ez’s recent comment about ASMlib being the “right” way: this article is almost 3 years old, and it’s just an update of an even older article. Before ASMlib was released with Oracle 10.1 this was how we used to do it. ASMlib is not the “right” way but just one choice. I’ve discussed this in another old post about ASMLIB Performance vs Udev. I will say that today, I do think ASMlib is the right choice in most situations.
@Dave and @Scott5[55] and @SST and @Robert – thanks for answering other people’s questions in the thread while I wasn’t around. :)
Lots of discussion here about using multipath.conf – I was aware of that when I wrote the post several years ago, and I’ve used it before. It works fine too, though in this particular case we were using UDEV for a lot more than just multipath devices. They were just one type of device. It was awhile ago though… can’t remember all the reasons I didn’t use multipath.conf… it works too. As several people have mentioned in the comments, generally you shouldn’t need to use both multipath.conf and UDEV – just one or the other should do the job.
Regarding /etc/rc.local – the problem with this approach is that it only takes effect at boot-time. With larger production systems it’s not uncommon at all to attach and detach storage at run-time without rebooting (if possible). UDEV handles this very nicely. (I think that multipath.conf does too.) And of course, ASMlib is probably my preferred solution if there aren’t linux guys around who can handle multipath.conf or UDEV.
@Emmanuel – Off the top of my head, I’m not sure why ohasd was throwing an error. I hope you’ve been able to fix it by now.
Thanks again everybody. :)
-Jeremy
LikeLike
Posted by Jeremy Schneider | October 4, 2010, 11:59 amWith Asmlib, you can directly take advantage of device-mapper-multipath
just configure /etc/sysconfig/oracleasm (RedHat 4/5) :
”
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=”dm-”
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=”sd”
”
/etc/init.d/oracleasm will then scan /dev/dm- device instead of /dev/sd, forcing use of multipath. This will map persistently by ASM label the good major/minor device in /dev/oracleasm/disks
You just need to configure wwid persistence for ocr/voting disk and configure rawdevices like this :
(we have created small partitions on the same volume)
“# This file and interface are deprecated.
# Applications needing raw device access should open regular
# block devices with O_DIRECT.
# raw device bindings
# format:
#
# example: /dev/raw/raw1 /dev/sdb1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/mapper/clup1
/dev/raw/raw2 /dev/mapper/clup2
/dev/raw/raw5 /dev/mapper/clup5
/dev/raw/raw6 /dev/mapper/clup6
/dev/raw/raw7 /dev/mapper/clup7
“
LikeLike
Posted by MP Derelle | November 18, 2010, 11:04 amI know it’s a long time after SST’s post but I’ve used his post today and had the same issue as John. The solution if your going to use /etc/multpath.conf to set the permissions as per SST’s post is to recreate your initrd after updating the mutlipath.conf. It worked fine for me and all Oracle devices now have the correct permissions every time.
E.G
[root@~]# vi /etc/multipath.conf
[root@~]# cd /boot
[root@boot]# uname -r
2.6.18-238.9.1.el5
[root@boot]# mkinitrd -f initrd-2.6.18-238.9.1.el5.img 2.6.18-238.9.1.el5
Modulefile is /etc/modprobe.conf
[root@boot]# reboot
LikeLike
Posted by Alex Hopcroft | May 17, 2011, 11:32 amI never had to rebuild initrd with any systems after RHEL 5.3 without the patch referenced above.
The patch is integrated into later versions of the OS (and more specifically, device-mapper-multipath-0.4.7-34.el5 and later)
If you’re running > RHEL 5.3 (or >= device-mapper-multipath-0.4.7-34.el5), I don’t think you should need to rebuild initrd.
I tested this again today with this system (which is due for an update too, but it illustrates the point):
[root@serv ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
[root@serv ~]# rpm -qa | grep multipath
device-mapper-multipath-0.4.7-34.el5_5.6
[root@serv ~]# ls -al /dev/mapper
total 0
drwxr-xr-x 2 root root 100 May 25 17:06 .
drwxr-xr-x 16 root root 4540 May 25 17:21 ..
crw——- 1 root root 10, 63 Mar 10 12:30 control
brw-r—– 1 oracle dba 253, 0 May 25 17:06 dguard
brw-r—– 1 oracle dba 253, 1 May 25 17:06 dguardp1
This is with the following change being made to /etc/multipath.conf:
multipath {
wwid 3603208b7200a3e264000a30001bc0000
alias dguard
path_grouping_policy group_by_prio
path_selector “round-robin 0”
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
mode 640
uid 54321
gid 54322
}
(oracle is UID 54321, dba is gid 54322 thanks to oracle-validated)
HTH, and I think it’s great that the discussions have continued. I still have this thread bookmarked and refer back to it every now and again.
Thanks!
John
LikeLike
Posted by John | May 25, 2011, 5:37 pm