>
Linux, Oracle, Technical

Oracle Clusterware on RHEL5/OEL5 with udev and multipath

The trouble with Linux? No… the trouble with computers in general – is that they keep changing! Solaris 10 comes out, Oracle 11g, Red Hat 5… and everything works different!! It’s a full-time job just trying to keep up with everything.

Almost exactly one year ago I wrote about using udev on 2.6 kernels to set the proper permissions for Oracle RAC. Two weeks after that post (March 14) Red Hat Enterprise Linux 5 was released and changed everything.

In my original post, I demonstrated how to create a PERMISSIONS file that udev would use when creating the device nodes. This worked on RHEL4 and SLES9. However this week I’ve been helping a client deploy 11g RAC on a RHEL5-based cluster – and I remembered that the PERMISSIONS facility was removed from udev in RH5. Seems like I remember reading something about having a single source of configuration for udev, which makes sense… so maybe they picked the RULES. (You’ll remember from my previous post that RULES are processes right before PERMISSIONS.) This is just as well since RULES are actually quite a bit more powerful than PERMISSIONS.

So on RHEL5 and OEL5 – in order to conform to Linux Best Practices – we now have to set correct RAC file permissions using udev RULES. To get started, we need to review how RULES work. The udev manual page gives a good overview of rules processing. But of course there are plenty of great tutorials that go deeper if you’re looking for more.

Block Devices and Raw Devices

Now of course I’m not the first person to notice that there’s no permissions.d directory on RHEL5 and OEL5. Last September, Grégory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block devices. I don’t speak French but I was able to copy the text of his RULES files:

# Oracle Configuration Registry
KERNEL=="sdb[8-9]", OWNER="root", GROUP="oinstall", MODE="640"
# Voting Disks
KERNEL=="sdb1[0-2]", OWNER="oracle", GROUP="oinstall", MODE="640"
# ASM Devices
KERNEL=="sdb[5-7]", OWNER="oracle", GROUP="dba", MODE="660"

Many people are still configuring raw devices for their voting disks and cluster registries even though this is not necessary. This Oracle Forums thread gives an sample of using a RULES file to set permissions for raw devices. It says to create a file called /etc/udev/rules.d/65-raw-permissions.rules with these contents:

# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

In fact there’s a great Oracle Whitepaper on the udev and multipathing that was published all the way back in June of 2007. It gives another sample configuration for raw devices on Red Hat 5.

Linux Multipath (Device Mapper) Devices

That’s all great. However… I’m doing an implementation on RHEL5 right now and we’re using the device mapper to multipath connections to the SAN. And the Oracle white paper – which goes into marvelous depth about multipath, udev and RH5 – never tells us anything about configuring multipath, udev and RH5 all together! So I have to figure this one out on my own. No harm done; it provided an interesting challenge for the day. :)

Multipath devices are a bit tricky. Usually if you’re using multipath then you don’t want to assume that devices will always be discovered in the same order. (This is the purpose of assigning friendly names by WWN in the multipath.conf file with the alias directive.) In fact, on our cluster the devices were assigned in different orders on different nodes – the ocr was dm-9 on one node and dm-10 on another. How do you write a RULES entry if you don’t know what the name of the device is?

I finally did get the multipath rules file to gave the proper permissions to my OCR and Voting Disks based on alias. However I was only able to change the /dev/dm-* files and not the /dev/mapper/* nodes. (Those nodes are not created by udev.) Therefor I had to use the aliases in /dev/mpath – not the aliases in /dev/mapper – when running the Oracle installer. How does it work? The symlinks in /dev/mpath are created by udev RULES – so all I had to do was piggyback on the udev config that created them and modify the permissions of the underlying devices.

This client used the aliases vote1, vote2, vote3, ocr1 and ocr2. Here’s the config file /etc/udev/rules.d/40-multipath.rules with my changes in bold:

# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | /bin/grep 
    ^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p' 
    -j %M -m %m"
PROGRAM=="/sbin/dmsetup ls --target multipath --exec /bin/basename -j %M -m %m", 
    RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", GOTO="check_cluster_devs"
PROGRAM!="/bin/bash -c '/sbin/dmsetup info -c --noheadings -j %M -m %m | /bin/grep 
    -q .*:.*:.*:.*:.*:.*:.*:part[0-9]*-mpath-'", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup ls --target linear --exec /bin/basename -j %M -m %m", NAME="%k", 
    RESULT=="?*", SYMLINK="mpath/%c", OPTIONS="last_rule"
GOTO="end_mpath"
LABEL="check_cluster_devs"
RESULT=="ocr*", GROUP="dba", MODE="640"
RESULT=="vote*", OWNER="oracle", GROUP="dba", MODE="640"
OPTIONS="last_rule"
LABEL="end_mpath"
About these ads

About Jeremy Schneider

Doing stuff with Oracle Database, Performance, Clusters, Linux. about.me/jeremy_schneider

Discussion

35 thoughts on “Oracle Clusterware on RHEL5/OEL5 with udev and multipath

  1. hi & thx for this article!

    btw. to use the /dev/mapper/-nodes (on SLES10) instead of symlinks like you described here i did a quick/dirty ;) workaround – just added 2 new “event-handlers” triggered by the id of the devices of interest to do, for example, a chown oracle:oinstall && chmod 660 for them:

    —–

    /etc/udev/rules.d/71-multipath.rules :

    ENV{ID_DMTYPE}==”linear|multipath”, IMPORT{program}=”/sbin/mpath_id %M %m” # through mpath_id we get ID_MPATH (=WWN)
    
    […]
    

    ########
    patterns for 2 specific disks / wwns to set oracle-permissions

    # setting owner /dev/mapper/
    
    ENV{ID_DMTYPE}==”multipath”, ENV{ID_MPATH}=="12006048000029010125[4,5]533030313030" 
    
    SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_MPATH}”, PROGRAM=”/bin/chown oracle:oinstall /dev/mapper/$env{ID_MPATH}”
    
    # doing chmod /dev/mapper/
    ENV{ID_DMTYPE}==”multipath”, ENV{ID_MPATH}=="12006048000029010125[4,5]533030313030", 
    
    SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_MPATH}”, PROGRAM=”/bin/chmod 660 /dev/mapper/$env{ID_MPATH}”
    
    -----
    

    Posted by sven hessenmueller | February 26, 2008, 5:13 am
  2. Jeremy,
    How does the client create the aliases ocr1, ocr2, etc.?

    Thanks

    Posted by Pradeep | September 15, 2008, 11:47 am
  3. Pradeep – instead of OWNER/GROUP/MODE you can specify NAME or SYMLINK. Check out the Red Hat tutorial that I linked in the article.

    Posted by Jeremy | September 17, 2008, 7:37 pm
  4. Thanks for your article!

    Can you tell us anything about the difference between using the multipath.conf file and /var/lib/multipath/bindings. Does multipath update the bindings file on boot based on multipath.conf?

    We are using /etc/rc.local to update the permissions on the /dev/mapper nodes and it seems to work fine.

    Posted by Greg | September 24, 2008, 5:14 am
  5. Hi, thanx for arcticle. usefull for me. But I have a strange problem: looks like part between LABEL=”check_cluster_devs”
    and OPTIONS=”last_rule” – ignored. also i tryed insert in this part strings like PROGRAM=”/bin/echo MYTEST” to grep them from /var/log/messages, but they are absent. have you same problems on your practice?

    Posted by Andrey | November 5, 2008, 4:51 am
  6. my /etc/udev/rules.d/40-multipath.rules:

    # multipath wants the devmaps presented as meaninglful device names
    # so name them after their devmap name
    SUBSYSTEM!="block", GOTO="end_mpath"
    KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | /bin/grep ^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m"
    KERNEL!="dm-[0-9]*", GOTO="end_mpath"
    PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
    ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p' -j %M -m %m"
    PROGRAM=="/sbin/dmsetup ls --target multipath --exec /bin/basename -j %M -m %m", RESULT=="?*", NAME="%k", GOTO="check_cl_devs"
    PROGRAM!="/bin/bash -c '/sbin/dmsetup info -c --noheadings -j %M -m %m | /bin/grep -q .*:.*:.*:.*:.*:.*:.*:part[0-9]*-mpath-'", GOTO="end_mpath"
    PROGRAM=="/sbin/dmsetup ls --target linear --exec /bin/basename -j %M -m %m", NAME="%k", RESULT=="?*"#, SYMLINK="mpath/%c" #,OPTIONS="last_rule"
    #GOTO="end_mpath"
    LABEL="check_cl_devs"
    RESULT=="ocr*", GROUP="dba", MODE="640", SYMLINK="mpath/%c"
    RESULT=="voting*", OWNER="oracle", GROUP="dba", MODE="640", SYMLINK="mpath/%c"
    OPTIONS="last_rule"
    LABEL="end_mpath"
    

    i remove SYMLINK from bolded string, remove OPTIONS=”last_rule” and GOTO
    ..so now all works fine

    Posted by Andrey | November 12, 2008, 5:53 am
  7. I went about this at a little different angle.

    I did not want to use /dev/dm-* devices as they may change. I went with the Multipath devices in /dev/mapper. I changed the multipath.conf file to apply the Oracle required permissions against the multipath device which I identified by the WWID. I also used my own device/alias name to help identify the device as an ASM device.

    multipaths {
        multipath {
          wwid        360a98000486e58526c34515944703277
          alias       asm01
          mode        666
          uid         1007
          gid         1102
       }
    }
    

    I then had my DBA use the path /dev/mapper/asm01p1 as the device in the createdisk oracleasm operation used to initialize the device for ASM use.

    Posted by SST | July 10, 2009, 7:58 am
  8. >I went about this at a little different angle.
    >
    >I did not want to use /dev/dm-* devices as they may >change. I went with the Multipath devices in >/dev/mapper. I changed the multipath.conf file to apply >the Oracle required permissions against the multipath >device which I identified by the WWID. I also used my >own device/alias name to help identify the device as an >ASM device.
    >
    >multipaths {
    > multipath {
    > wwid 360a98000486e58526c34515944703277
    > alias asm01
    > mode 666
    > uid 1007
    > gid 1102
    > }
    >}
    >
    >I then had my DBA use the path /dev/mapper/asm01p1 as the >device in the createdisk oracleasm operation used to >initialize the device for ASM use.

    Does the above acutally work? It seems like the best solution but I cannot get it to work on my tests. What version of multipath are you using?

    Posted by john | July 20, 2009, 6:14 pm
  9. SST – I am trying the same approach – setting the alias according to WWID in the multipath.conf file.

    For whatever reason, only SOME of my device files / nodes in /dev/mapper/ are getting set with the correct owner and perms.
    What’s worse, is that without changing multipath.conf, with each reboot, a DIFFERENT SUBSET of my oracle device files in /dev/mapper get set correctly. Rarely do more than 3 or 4 nodes get set correctly. Any ideas what’s going on?

    Posted by Larry Schrof | July 22, 2009, 10:09 am
  10. Red Hat Enterprise Linux Server release 5.3 (Tikanga)
    device-mapper-multipath-0.4.7-23.el5_3.4

    Larry, I am having the same problem, the permissions are intermittently set. I have a case open with support. Unfortunately our RHEL support is through IBM so it may be a while until I get an answer. I sent support an strace and I am waiting to hear back. So if you need a solution today it looks like rules modification is the only sure fire solution.

    Posted by SST | August 18, 2009, 7:25 am
  11. I believe you would need to set the permissions by adding something like following like in your /etc/rc.local file.

    chown oracle:dba /dev/mapper/

    hth,
    Manmohan

    Posted by MXJ | August 25, 2009, 2:28 pm
  12. Correction to my previous response. It should be

    chown oracle:dba /dev/mapper/(your_asm_devices_names*)
    chmod 0640 /dev/mapper/(your_asm_devices_names*)

    hth,

    Posted by MXJ | August 25, 2009, 3:05 pm
  13. Actually the issue with the multipath.conf settings (uid/gid,mode) not applying to the /dev/mapper devices is a confirmed bug https://bugzilla.redhat.com/show_bug.cgi?id=518575
    We are trying to get ASMlib to grab onto the /dev/mapper nodes but it seems to not be able to. Anyone know how to get around this?

    Posted by Dave | August 26, 2009, 1:19 pm
  14. oracleasm don’t have to add in uid and gid in /etc/multipath.conf file for example:

    multipath {
    wwid 360a98000486e58526c34515944703277
    alias asm01
    }

    type command to find the multipath name e.g: ll /dev/mapper/
    then copy the path name e.g: asm01p1
    then type command: /sbin/oracleasm createdisk DISK1 /dev/mapper/asm01p1
    DONE!

    Posted by Fred | September 28, 2009, 9:58 pm
  15. Fred – the problem was not the multipath settings, the problem was the DBA not using pattern match or looking to the right directory for the devices. We applied the latest errata for multipath and it fixed the problems for the most part of not being able to apply mode, uid, gid to new devices. The point is not that it is “required” to set the right mode on /dev/mapper devices, the point is that others have tried and seen intermittent results. The documentation for multipath.conf explicitly says you should be able to set mode/gid/uid among other settings on the fly and using this configuration file. It works for us on RHEL5.3 doing “multipath -f” and “multipath -r” after modifying the file. You can easily set these persistenantly rather than using udev rules, especially if alot of the devices need a custom alias, etc.

    Posted by Dave | September 29, 2009, 7:30 am
  16. @Dave:
    Are you saying you applied the patch in the linked RH Bugzilla bug, and that cleared up the issues?

    We’re trying to get RAC set up today, and I’m hoping that’s the case.

    Posted by John | November 19, 2009, 11:19 am
  17. Yeah, when we patched we now can dynamically bring in new devices, set the uid, mode, gid from within multipath.conf, and it works slick. The DBAs then scan for new disks in /dev/mapper and no problems at all (at least for adding them). Now, the only problem we have seen is removing LUNs on the fly from ASM is easy. But when trying to remove and cleanup the OS, it gets pi$$y and usually requires a reboot. But who shrinks storage? We do, and we discovered this the hard way a year or so back in our production RAC. Sometimes you over allocate and want to remove a couple LUNs and give to another RAC or something. Adding them is a piece of cake. Removing them is a piece of carp.

    Posted by Dave | November 20, 2009, 12:35 pm
  18. Thanks a lot for getting back to me. I thought that was what you were saying, so I went ahead and applied the patch. So far, it hasn’t worked. I still have had to manually chown and chmod the /dev/mapper files in order for certain parts of the RAC install process to continue or pass.

    If they’re defined in /etc/multipath.conf, do I still need to have udev rules? I haven’t set any up because I thought that it was one place or the other, not both.

    I’ve also run into problems after the install with srvctl complaining that it can’t load libpthread.so.0. I’m hoping that’s fixed in the 10.2.0.4 patch (10.2.0.1 might have problems using the i686 version of glibc-2.5 rather than the i386 version it replaced?), found the error when I tried to shut down ASM using srvctl to apply the patch.

    We’re still way ahead of our first RAC on Linux adventure, which threw IA64 into the mix just for fun, about 3.5 years ago. I wasn’t around for that one, but I hear it took nearly 3 months of calls between our DBAs, Oracle, Redhat, and HP before they could even get the installer to complete. Not certain who to blame for that one. :-)

    Thanks again for the great article and comments!

    Posted by John | November 20, 2009, 12:48 pm
  19. What I was told was that you don’t need udev “if you don’t need it” and if multipath.conf does all you need, it is way simpler to administer IMO. udev is pretty ugly to us and we used that in our RHEL4 10 g cluster and it sucks. This cluster is RHEL5.3 and 11g. I think if you are using yum and updated to the latest greatest, it should work for you. Remember to to do a multipath -f and then -r to apply your changes. Oh yeah, and the formatting of that file is touchy, and if the formatting is “off” you won’t get aliases and wierdness will surround you. Good luck.

    Posted by Dave | November 20, 2009, 12:56 pm
  20. OK, got everything running correctly now, and persistent across reboots. I was having my ASM disks get the correct permissions, but the OCR and Voting disks were not. I re-applied the patch from RH, apparently something didn’t go correctly the first time, and now it’s working. Thanks a ton for all the help here, great resource, and specifically to Dave.

    As an aside, the srvctl problem ended up being a line added for a RHEL 3 bug to set LD_ASSUME_KERNEL=2.4.xxx, along with a trailing “/” in an ORACLE_HOME= line that made the path read as /u01/app/oracle/product/10//lib (extra / before lib). Once I found and removed those errors, srvctl found the correct libraries and ran great.

    PEBKAC error once again. :-)

    Posted by John | November 23, 2009, 11:55 am
  21. Heh – a blog post from almost two years ago becomes the best online source for udev/multipath support.

    Seriously, I’ve had a number of RAC clusters set up for 6 months or so now; 11gR1 on OEL 5.3, various types of SAN disk; Symmetrix, Clariion, EVA, and I have stayed away from udev modifications like they smell. (don’t they?) I don’t understand what or why to modify the udev rules. With aliases mapped to wwids in the multipath.conf file I *think* I’m covered, but no matter how much I research this udev stuff, I just do not understand it at all. I get the concept, but knowing whether I’m at risk for devices changing arbitrarily at boot is another story.

    Like what John mentioned on the 20th, it *seems* like you can use one or the other. How to tell if devices are at risk?

    Posted by Scott5 | November 25, 2009, 3:16 pm
  22. You *can* use one or the other, but since you mentioned EMC arrays, look at it this way: If you are using pow”r”path instead of mpath, you almost have to use udev rules of some kind. If you are using mpath the *implied* recommended way to ensure device naming consistency accross reboots is with multipath.conf tying the wwid to some sort of alias, and that subsequent alias which is created under /dev/mapper must be used by the volume manager whatever it is. It’s all there in the man pages and in the RHN articles relating to multipathing. It’s not really that confusing but I agree, there is little consistency out there for concrete configuration info in RAC environments.
    If your storage vendor does not support native multipathing then you have to use what you got, and setting the bits can be done with the rules or by some other sequence of events at boot. With more vendors supporting native multipathing, seems like a no brainer to us and a pretty solid solution too. Just no big red easy button to push.

    Posted by Dave | November 30, 2009, 9:13 am
  23. @Dave –

    Thanks for sharing. You bring up a good point RE: PowerPath. No intention of using it ATM, but will consider the additional R&D we’ll need to go through if it becomes part of the stack.

    Could udev rules supplant printing as the most painful SA responsibility? I was certain it would be PAM configuration. :P

    Posted by Scott5 | November 30, 2009, 10:19 am
  24. With the installation of the following package updates, I was able to get UDEV to work with device-mapper-multipath without modifying any standard UDEV rules. This is for Red Hat 5.3 x86-64 baseline.

    device-mapper-multipath.x86_64 0.4.7-30.el5_4.4
    kpartx.x86_64 0.4.7-30.el5_4.4

    The key for me was to create a rule file that would run before the 40-multipath.rules as the OPTIONS=”last_rule” in the 40-multipath.rules file would ignore all later rules.

    My rule file is /etc/udev/rules.d/39-crs.rules and contains the following. It creates a symbolic link to the dm-X device based upon the UUID. It then modifies the ownership and permission on the dm-X device.


    KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m",
    RESULT=="part8-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_1",
    OWNER="oracle", GROUP="oinstall", MODE="0660"

    KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m",
    RESULT=="part9-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_2",
    OWNER="oracle", GROUP="oinstall", MODE="0660"

    KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m",
    RESULT=="part10-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_3",
    OWNER="oracle", GROUP="oinstall", MODE="0660"

    KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m",
    RESULT=="part11-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="ocr_file_1",
    OWNER="root", GROUP="oinstall", MODE="0640"

    KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m",
    RESULT=="part12-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="ocr_file_2",
    OWNER="root", GROUP="oinstall", MODE="0640"

    Posted by Robert | January 19, 2010, 10:39 am
  25. Hi,

    I am trying to install the 11GR2 Clusterware on RHEL5 servers using multipath.

    I am able to create the ASM disk with oracleasm without any trouble, based on /dev/mapper/ocr_vote_01 which is the alias I specified in my multipath.conf.

    I am also able to create an ext3 partition on that device, and mount it.

    The thing is that I am not successful at the end of the clusterware installation :

    CRS-4124: Oracle High Availability Services startup failed.
    CRS-4000: Command Start failed, or completed with errors.
    ohasd failed to start: Inappropriate ioctl for device
    ohasd failed to start at /u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

    I tried and checked a lot of things. Remember it is 11GR2, so /dev/mapper/ link has to be owned by grid:asmdba . I also tried to disable IPV6, with no result.

    Any idea ?

    Regards,

    Emmanuel

    Posted by Emmanuel | May 28, 2010, 6:33 am
  26. Hi Jeremy,

    here is a trace I found in the ohasd.log, may be It can help :

    2010-05-28 10:49:24.482: [ default][3872035408] OHASD Daemon Starting. Command string :reboot
    2010-05-28 10:49:24.489: [ default][3872035408] Initializing OLR
    2010-05-28 10:49:24.494: [ OCROSD][3872035408]utgdv:2:ocr loc file /etc/oracle/olr.loc cannot be opened. errno 2
    2010-05-28 10:49:24.494: [ OCRRAW][3872035408]proprioo: cannot get current configuration (33)
    2010-05-28 10:49:24.495: [ OCRRAW][3872035408]proprinit: Could not open raw device
    2010-05-28 10:49:24.495: [ OCRAPI][3872035408]a_init:16!: Backend init unsuccessful : [33]
    2010-05-28 10:49:24.496: [ CRSOCR][3872035408] OCR context init failure. Error: PROCL-33: Oracle Local Registry is not configured Storage layer error [Error opening olr.loc file. No such file or directory] [2]
    2010-05-28 10:49:24.496: [ default][3872035408] OLR initalization failured, rc=33
    2010-05-28 10:49:24.499: [ default][3872035408]Created alert : (:OHAS00106:) : Failed to initialize Oracle Local Registry
    2010-05-28 10:49:24.499: [ default][3872035408][PANIC] OHASD exiting; Could not init OLR
    2010-05-28 10:49:24.499: [ default][3872035408] Done.

    It is really weird, because I am successful in create ext3 on my /dev/mapper/ocr_vote_01p1 partition, which is the one I use to create the ASM disk.

    By the way, I do not know why but the OUI does not see my ASM volume created at step 8, so I have to change the discovery path to /dev/mapper and select myself /dev/mapper/ocr_vote_01p1 .

    Regards,

    Emmanuel

    Posted by Emmanuel | May 28, 2010, 7:23 am
  27. @Emmanuel –

    I recently went back and forth with one of our DBAs over this. (I’m an SA, and we handle the systems and storage, but the DBAs handle the ASM config and Oracle/RAC installation.) The understanding we came to was that the Oracle installation was essentially broken in that it wouldn’t use the multipathed (mapper) devices during the installation, and a set of standard SCSI devices had to be presented and set up within UDEV for the installation to auto-configure as ASM storage. I don’t know if that’s a fully accurate interpretation of the new 11GR2 install requirements, but it worked for us.

    I like using the wwid of the LUNs when identifying storage devices, and that’s how I do it in the multipath.conf. For straight-up SCSI devices, though; I created this udev rules file to provide a couple of disks to the Oracle install:

    Part of the criteria included isolating devices to individual HBAs, as I wouldn’t want to present both LUNs down the same path

    # cat /etc/udev/rules.d/45-oracle-asmdevices.rules
    # asm1 device; only use the SCSI devices available to 0000:11:00.1
    KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”
    RESULT==”3600508b4000c39390001100000600000″, BUS==”pci”, ID==”0000:11:00.1″, NAME:=”asm1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”0660″
    # asm2 device; only use the SCSI devices available to 0000:0d:00.0
    KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”
    RESULT==”3600508b4000c39390001100000630000″, BUS==”pci”, ID==”0000:0d:00.0″, NAME:=”asm2″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”0660″

    Keep in mind that we did *NOT* partition these devices or create file systems on them; they’re essentially “raw” SCSI block devices.

    This gives me two SCSI devices; /dev/asm1, and /dev/asm2, and the installation seems to be happy with this.

    Posted by Scott555 | May 28, 2010, 8:11 am
  28. hi Scott555,

    Thank for your answer. I will try and let you know.

    Regards,

    Emmanuel

    Posted by Emmanuel | June 1, 2010, 12:10 am
  29. Hi Jeremy,

    I realize that it is what I did with multipath.conf :

    multipaths {
    multipath {
    wwid 3600a0b80004891dc00000ffd4bea8199
    alias ocr_vote_01
    }
    multipath {
    wwid 3600a0b8000423486000016404bed1ace
    alias dg_data_01
    }

    It works because I get my aliases under /dev/mapper/ . I am also able to mount the partition and format them with mkfs.ext3. I can create the OCR disk with oracleasm create disk successfully. It crashes when ohasd tries to format the partition to create the OCR.

    I do not think that it would change anything with Udev.

    What do you think ?

    Regards,

    Emmanuel

    Posted by Emmanuel | June 1, 2010, 12:32 am
  30. You could use ASMLib, as you are using ASM. This is the “right” way of doing it. This simplifies things, as ASM sets its own (required) permissions according to your predefinitions. Simple and elegant.
    You don’t need to define RAW devices, and ASMLib makes use of labels, so life is good even if you have many devices, or if you experience LUN (sd) device renames. Also, God forbid, if both (some of) your servers see different LUN names. You needn’t fight with the system to match the mpath names, permissions, binding, etc. Just label them with ASMLib, and you’re happy.

    Ez

    Posted by Ez Aton | October 4, 2010, 7:40 am

Trackbacks/Pingbacks

  1. Pingback: Log Buffer #84: a Carnival of the Vanities for DBAs - February 15, 2008

Disclaimer

(a) The views expressed on this website are mine alone and do not necessarily reflect the views of my employer.

about.me

Jeremy Schneider
Follow

Get every new post delivered to your Inbox.

Join 896 other followers

%d bloggers like this: