ASM Mirroring – No Hot Spare Disk

Some time ago, we installed Oracle on a Sun box with 48 local 1TB disks (spread across six controllers). I explained to the storage and system teams that we would use ASM as the volume manager – and as such, it would take care of mirroring. One storage admin asked me which disk was the hot spare. I replied that there was no hot spare.

What?!! You call this a volume manager and it doesn’t support a hot spare disk?! That’s it, we’re going back to filesystems!

Not so fast.

You’ll remember that the purpose of a hot spare disk is to restore protection in the event of a disk failure. When the RAID controller or volume manager detects a failed disk (R.I.P.), it uses the spare and reproduces a copy of the complete now-missing disk (R.I.P.) by reading the mirror or parity data.

Restoring Protection Without A Hot Spare Disk

In fact, ASM does even better. Instead of a hot spare disk, it has hot spare capacity – spread across all of the disks. There are two immediate effects of this: (1) no loss of potential performance from a usable disk head sitting idle and (2) the hot spare capacity is available for an emergency situation (like the archivelog volume filling up).

How is this possible? The key of course is understanding how ASM mirrors data – by extent rather than disk. So if I allocate three extents on disk 1, the first could be mirrored on disk 10, the second on disk 11 and the third on disk 12.

Now if disk 1 fails, I don’t need to rebuild the disk – I only need to ensure that each extent is mirrored. For example: lets copy the first extent to disk 11, the second to disk 12, and the third to disk 10. Once everything has two copies, my data is protected – even though the first disk was never rebuilt!

You’ll also notice that the I/O for the rebuild operation is spread across all of the disks. Faster and better.

So there actually is a “hot spare” of sorts – spare capacity instead of a spare disk. And unlike RAID, this hot spare isn’t optional: ASM will always reserve enough spare capacity to bring the diskgroup back to its proper level of redundancy after a failure.

Failure Groups Mean A Little Less Usable Space

WAIT WAIT WAIT!! I know you’re excited, but before you press enter and execute that migration script, I need to tell you about one more very important detail. How does ASM know that amount of spare capacity to reserve? The size of the largest disk, right? NO!!!

This is, in fact, what failure groups are for!! (You always wondered, right?!) Because ASM doesn’t mirror across disks – it mirrors across failure groups. And accordingly, ASM will always reserve enough spare capacity to restore protection after losing the largest failure group.

Now it’s important to understand how ASM will report this. Lets look quickly at asmcmd’s lsdg command. This command (11gR1) gives us four important numbers: Total_MB, Free_MB, Req_mir_free_MB and Usable_file_MB. The 11gR2 Storage Administrator’s Guide gives good descriptions of the output:

Putting it all together, we see that Req_mir_free_MB should be equal to the largest failure group. Secondly, Usable_file_MB is the most important number — that’s how much space we can actually use. And with normal (2-way) mirroring, it should always be half of the available raw free space. And available raw free space is total raw free space minus Req_mir_free_MB.

Example

So let’s see if we can observe this on a system in the wild. We’ll revisit one that I briefly described last year.

The system had 46 local disks (500M each). The disks were attached to six controllers: two controllers had six disks each, the other four controllers had eight disks each. We configured ASM so that each controller is a failure group.

First lets get the output of lsdg, which we’ll verify step-by-step:

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326          3137688         7451819              0  DATA01/

Now – to begin our verification – lets look at the exact size of a single disk:

# sfdisk -suM /dev/sdav1
401624968

The Linux kernel is reporting KB and we need MB – so we’ll divide by 1024.

401624968 /1024 = 392211.8828125 M

ASM uses “allocation units” of 1M and can’t use a partial AU – so we truncate that to 392211. Now there were 46 disks – so the total amount of space available should be 392211 * 46 = 18041706. And this perfectly matches Total_MB from the lsdg output! So far, so good!

Next, lets look at the largest failure group. The four large failure groups were each the same size with eight disks: 392211 * 8 = 3137688. This perfectly matches Req_mir_free_MB from the lsdg output! Lovely!

Finally, lets see how much space I can actually use for datafiles and such – accounting for mirroring. The available raw free space should be Total_MB minus our calculated Req_mir_free_MB: 18041326 – 3137688 = 14903638. I’m using normal mirroring, so my available/usable free space should be half of that: 14903638 / 2 = 7451819. Perfectly matches Usable_file_MB from the lsdg output. Wow.

The Big Caveat

Sounds amazing, right? Well there is one very important caveat. I alluded to it earlier: Oracle will allow you to use this reserved “hot spare” capacity. This is very different from a traditional “hot spare” disk. What happens if you use the space? (1) The “usable free space” reports a negative number!!! (2) If a disk fails, then you will not lose any data – but ASM cannot automatically restore protection until you replace the disk or give ASM additional storage somewhere else.

That sound serious! What’s worse, you can get into this situation without even realizing it. If you create a datafile that’s bigger than the available space then ASM won’t warn you – it will just create the datafile and you lose your protection from double-failure.

You get alerts when your traditional filesystems are filling up. (How many times has your inbox been flooded because of those stupid archivelogs?!) If you deploy ASM, then MAKE SURE YOU HAVE ADEQUATE ALERTING. Oracle’s management tools can help out, and there are other solutions too. Don’t overlook it!

Conclusion

At this point, it should be evident that anyone with storage background needs to be careful around ASM. It doesn’t work the same as a traditional volume manager or RAID controller. Oracle Corp really does love SUPER EXCITING NEW WORDS (aka marketing buzz)… but this one isn’t just a matter of terminology. There is some real difference in the underlying concepts.

A few key takeaways:

• To automatically restore protection after a failure, ASM reserves spare capacity instead of a spare disk.

• In V$ASM_DISKGROUP and asmcmd’s lsdg command, Total_MB and Free_MB report raw capacity while Usable_file_MB reports usable capacity based on the default mirroring level.*

• ASM will allow you to use the spare capacity, after which it will report negative free space and won’t tolerate a double-failure. Make sure you have adequate reporting so that this is always caught and addressed quickly.

*Note: Various objects can have different mirroring levels in ASM.

Well I think that this should serve as a good primer on ASM and mirroring. Let me know if I’ve left anything out!