RAC Investigation on Low-Memory Linux

Back in the Oracle 9i days, I was one of those people who got on eBay to buy firewire PCI cards and disks that could do non-exclusive login.  Remember that?  The first time a little test cluster could be cheap enough for the home enthusiast?  I still have the parts in my closet.

Of course, we all know what happened after that – virtualization.  It didn’t take long before my home-built test clusters were running on VMware.  (Personally, I think that virtualization really started because of those NES and SNES emulators.  Most great achievements start with a geek who wants to play more video games.)  There are lots of people now who run RAC on virtual environments and it’s easy to find tutorials on the web for many different OS and VM combinations.

Low-Memory Linux

Something I haven’t seen many other people do is RAC with a very small memory configuration.  Like 760M of memory per server. (!)  Of course you’d only do this for a hobby setup – never on a system where you want any kind of support.  But I’m kinda cheap… and running RAC on these small VMs means that I don’t have to go buy an expensive new home computer.  My current gateway laptop with Vista Home does the job quite nicely!

10.2 and 11.1 RAC will install and run on servers with 760M of memory. But things were a little unstable at first. Now I’m the curious type… I like to fiddle with things… so I investigated a little bit.

Basic Unix Investigation

There are two basic investigation scenarios:

In this particular case, I noticed pretty quickly from the top utility that one process was consuming over 30% of the system’s memory!  (Note: in top, you can press the ‘<‘ and ‘>’ keys to move the sort column left and right.  The initial sort column is %CPU.  I moved it one column to the right, sorting by %MEM.)

top - 18:41:13 up  5:28,  3 users,  load average: 0.04, 0.35, 0.60
Tasks: 180 total,   2 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  3.7%sy,  0.0%ni, 89.4%id,  6.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    767020k total,   754296k used,    12724k free,     7084k buffers
Swap:  1540088k total,   654696k used,   885392k free,   361800k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17435 oracle    RT   0  230m 229m  31m S  0.3 30.6   0:26.21 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
 7765 oracle    15   0  464m 111m 102m S  0.0 14.9   0:04.52 ora_smon_RAC1
 7743 oracle    -2   0  445m  90m  83m S  0.0 12.1   0:12.01 ora_lms0_RAC1
 7783 oracle    15   0  440m  70m  66m S  0.0  9.4   0:05.35 ora_mmon_RAC1
20321 oracle    15   0  438m  48m  45m S  0.0  6.5   0:00.88 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8676 oracle    16   0  437m  46m  45m S  0.0  6.2   0:02.69 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8801 oracle    15   0  440m  44m  41m S  0.0  6.0   0:04.57 ora_cjq0_RAC1

This process (ocssd) is Oracle’s Cluster Synchronization Services Daemon.  It’s the process that sends and receives heartbeats from other nodes. Any delays sending or receiving those heartbeats can cause node evictions (a.k.a. server reboots) – so it’s a pretty important process! That’s why it runs with realtime (RT) scheduling priority, as you can see in the above output from top.

I was surprised that CSS uses so much physical memory – usually Linux is very good at memory management. In top, the VIRT column shows how much total memory each process is using, while the RES column shows how much actual physical memory Linux has allocated to it. It’s clear that Linux is pretty actively managing the physical memory for other processes.

A quick glance at vmstat shows that although we are actively swapping, it seems under control. This is about what I’d expect when we’re idle and all of the processes except CSS are sharing only 500M of memory:

collabn1:/home/oracle[RAC1]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 604144  14024  34948 300632    0    0    32    86 1044 1797  1  8 84  7  0
 2  1 603912  11224  34968 303968   87    0   763    81 1083 1902  3 11 56 30  0
 0  0 604840  12660  34912 303904   25    0   253    38 1050 1975  2 11 69 18  0
 0  0 604840  12728  34924 303900    0    0    68   106 1043 1975  1  9 82  8  0
 0  0 604784  14536  34936 304140   19    0    89    71 1043 2042  1  5 81 13  0
 1  0 604752  14168  34944 304496    6    0   119    21 1040 2016  2  8 80 11  0
 0  1 604736  18732  35020 306212    0    0   384    73 1050 2489  1 10 70 19  0
 1  1 604736  11144  35232 311252  104    0  1209   128 1074 2024  2 12 38 47  0
 3  1 607900   8056  30788 307352   77 1500   504  1661 1055 2642  9 16 56 19  0
 0  0 607900   9836  30800 307360    0    0    71    70 1031 1798  1  6 85  8  0
 1  0 607884  10536  30812 307400    8    0    44    93 1031 1832  1  4 87  8  0

The SO column tells us when memory is written to disk (and removed from physical). The SI column tells us when memory is read from disk (and put back in physical). On a side note, remember that on a healthy Unix system the free memory is always small. Sometimes this is confusing at first.

Linux Process Memory Investigation

Nonetheless, I’m not happy that CSS is using 30% of my physical memory in this highly-constrained hobby environment. Why is Linux allowing this? The first clue comes simply from the output of the familiar unix ps utility:

collabn1:/home/oracle[RAC1]$ ps v -C ocssd.bin
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
17435 ?        SLl    0:08      7   588 235331 234840 30.6 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin

On Linux, the “v” flag tells ps to give information relevant to virtual memory. TRS and DRS tell me how much physical (resident) memory is used for machine executable code (text) and data, respectively. But more importantly – the STAT column gives some informative BSD-style flags about the process. That capital-L indicates that CSS has some pages that are locked into physical memory. Bingo.

If I have root access, then I can get a very detailed report on process memory usage with the pmap command. The output was a little long, so I’ve abbreviated it here:

[root@collabn1 ~]# pmap -x 17435
17435:   /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
Address   Kbytes     RSS    Anon  Locked Mode   Mapping
00110000     656       -       -       - r-x--  libhasgen11.so
001b4000       8       -       -       - rwx--  libhasgen11.so
  ... 37 more library blocks
02a0d000     100       -       -       - rwx--    [ anon ]
02a26000       4       -       -       - --x--    [ anon ]
02a27000   10240       -       -       - rwx--    [ anon ]
03427000       4       -       -       - --x--    [ anon ]
03428000   10240       -       -       - rwx--    [ anon ]
  ... 10 more anonymous blocks, half are 10240K
08048000     592       -       -       - r-x--  ocssd.bin
080dc000       4       -       -       - rwx--  ocssd.bin
  ... 30 more anonymous blocks, half are 10240K
bfe40000     148       -       -       - rwx--    [ stack ]
bfe65000       8       -       -       - rw---    [ anon ]
-------- ------- ------- ------- -------
total kB  235920       -       -       -

Interestingly, the linux pmap utility does not indicate any locked memory! I don’t know whether that output column is non-functional or if it refers only to some particular kind of locking. But at any rate, I know something is locked. I couldn’t think of anything better, so the next place I looked was in the Linux /proc pseudo-filesystem.

collabn1:/home/oracle[RAC1]$ grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Now we’re talking. The process has a total of 235920 kB of memory – and it’s ALL locked. On a normal RAC system you’d want this. Generally, important realtime processes should be locked so that they are never delayed by paging or swapping. (Remember how that could cause node reboots?)

But I personally doubt that all of the memory really NEEDS to be locked, and I think that Linux will actually do a decent job of not swapping the most important parts. And my highly constrained hobby environment will probably run much smoother if Linux has more flexibility when managing a measly 760M of memory.

Unlocking Linux Process Memory

But is it actually possible to unlock the process memory? As far as I know, Oracle provides no option to disable CSS memory locking. (For good reason.) There is the system call munlockall() – which unlocks all of a particular processes’ memory. But the CSS process itself would have to call this function. And of course it will not. Or will it?

If you’ve got root, then there’s a hacker-back-door way of doing this. Remember, you’d be crazy to try this anywhere besides a dark closet at home. And if you type too slow then CSS could reboot your machine.

But watch this…

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

[root@collabn1 ~]# gdb -p 17435 < call munlockall()
> quit
> EOF

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 17435
Reading symbols from /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin...done.
  ... 17 more Reading/Loading symbols.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f629f0 (LWP 17435)]
  ... 19 more New Threads.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libnsl.so.1...done.
  ... 13 more Loaded/Reading symbols.
0x008e7402 in __kernel_vsyscall ()
(gdb) $1 = 0
(gdb) The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
Detaching from program: /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin, process 17435

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:         0 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Ha. This is something you won’t find on metalink. After generating some activity on the system, the top utility shows me that Linux has significantly reduced the physical memory used by CSS.

These days I’ve actually scripted this for my home and classroom VM environments. I haven’t done a careful comparison or analysis, but it really has seemed to me that my low-memory Linux systems run noticeably smoother.