Before I started consulting, I was an Oracle engineer in a very large software development organization. The company had a number of major products and the one I worked with was used by hospitals and radiology offices world-wide. (These guys are one of the biggest companies worldwide in the field.) Our product included the hardware and software; you could order it with Sun or HP hardware (Solaris or Linux). It had an Oracle backend and a web-based middle tier built with lots of C++ and Java code.
Any software engineer who has worked on large projects – industry or community – can tell you the importance of solid change control processes. So… since everything for the build had to be checked into clearcase… yes, we checked Oracle 10g into the repo. A 2G tarball. And whenever there was a 1K patch to the oracle install? A brand new 2G tarball. The clearcase guys loved me.
And that was how I started thinking about an automated build process. I don’t need to check B24792-01_1of5.zip into the repository because it’s straight off edelivery. I can skip p7150622_10203_Linux-x86-64.zip since it’s direct from metalink. The only thing missing was a solid, simple, flexible program for automating the oracle install, patchset, CPUs and oneoffs – taking Oracle’s official bits as input.
Anyone else out there who could use a program like this? How about for rapid provisioning of servers? (Like all that grid buzz.) Grid Control does some basic stuff (if you can make it work) and there are advanced kits for the big data centers – but there have to be more people than just me who would love to have this program.
Proposing orainstall
That’s why I’m proposing to write a bit of software called orainstall. It would be script-based and cross-platform and intended for community use. I have some ideas for a design but I’m really interested to hear how you might use something like this and what features might be useful to you. Do you think it’s a good design? I’d also be interested in hearing if you’d be interested to use it or to help test it.
The program would be launched by running the main executable “orainstall”. The syntax could look something like this:
Command line syntax:
./orainstall [-oh ] [-tmp ]
Example:
./orainstall ora10203.sf -oh /u01/app/oracle/product/10.2.0/db_1 -tmp /u01/app/oracle/tmp
Return Value:
0 - success
1 - invalid stepfile
2 - installation error
3 - patching error
The Stepfile
What I’m thinking is that the install process is defined in an input file called the stepfile. The stepfile needs to accomodate a large variety of organizational requirements. Here’s my first draft for what a stepfile would look like:
# This stepfile instructs the oracle auto-installer how to setup the oracle
# home. Each line is executed in order from top to bottom.
#
# There are ten defined directives for this stepfile:
# ORACLE_HOME - required
# This will cause a search-replace of each response file to explicitly
# define the oracle home. It must be defined before any open, install or
# patch directives and it can only be defined once per stepfile.
# Order of precedence:
# 1. command line
# 2. stepfile
# 3. environment variable
# ORAINST_TMP - optional
# Top-level temp directory for extracting contents of Oracle installers
# and patches. Each installer or patch will be extracted into a
# subdirectory of this temp location. It must be defined before any
# open or patch directives. It can be defined multiple times in a
# stepfile.
# Order of precedence:
# 1. command line
# 2. stepfile
# 3. environment variable
# 4. "/tmp/orainstall"
# OPEN /path/to/zip
# Path to a zip file with database installer or major patchset which
# requires use if OUI. The file is unzipped into a temporary location.
# If several INSTALL directives are encountered in a row then they are
# all unzipped into the same location; if there is another directive in
# between then the next INSTALL is unzipped into a new temporary
# location.
# INSTALLPATH Disk1/runInstaller
# INSTALLRSP install.rsp
# "runInstaller -waitForCompletion -silent -responseFile "
# INSTALLPATH is optional - if not specified then the script will search
# for it in the most recently created temp location. If there is more
# than one runInstaller then it will use the first one it finds; search
# order is undefined.
# If no path is specified for the response file then orainstall will
# search for it in this order:
# 1. current working directory (pwd) when orainstall was launched
# 2. same directory as orainstall binary (dirname $0)
# OPATCH/path/p8000.zip
# Path to special "OPatch" update - contents are unzipped directly to
# ORACLE_HOME.
# PATCH /path/p2345.zip
# Path to a patch. Contents are unzipped to a temporary location then
# applied by calling "opatch" from the ORACLE_HOME.
# NPATCH /path/p1234.zip
# Path to a patchset with multiple subpatches which need to be applied
# with "napply" (such as a CPU). Contents are unzipped to temp location
# then applied properly to the ORACLE_HOME.
# CLEAN_TMP
# Deletes all files from the current top-level temp directory.
# REMOVE /path/to/file
# Runs "rm -rf $ORACLE_HOME/path/to/file" - use carefully! Some sites
# may require removal of components for security requirements.
#
# If there are any errors (install or patch) then the process will fail.
#
#oracle_home /u01/app/oracle/product/10.2.0/db_1
orainst_tmp /u01/app/oracle/installers.tmp
open /net/rh5lab12/u10/stage/ora10201/B24792-01_1of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_2of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_3of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_4of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_5of5.zip
installpath database/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/database10201.rsp
installpath companion/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/companion10201.rsp
open /net/rh5lab12/u10/stage/ora10203/p5337014_10203_Linux-x86-64.zip
installrsp /net/rh5lab12/u10/stage/rsp/patchset10203.rsp
clean_tmp
orainst_tmp /u01/app/oracle/patches
opatch /net/rh5lab12/u10/stage/patch10203/p6880880_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p5892355_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p6455161_10203_Linux-x86-64.zip
npatch /net/rh5lab12/u10/stage/patch10203/p7150622_10203_Linux-x86-64.zip
remove /htmldb
Program Flow
The program flow would be pretty basic; it would validate the stepfile then execute it one line at a time. I thought of six things to validate before starting execution:
- can read stepfile
- one oracle_home, defined before open/install/patch, can create/write
- orainst_tmp defined before open/install/patch, can create/write each
- clean_tmp not defined before orainst_tmp
- can read all open/install/patch input files
- free space in oracle_home is greater than size of input files
Feedback
Maybe there aren’t as many people as I think who would use this. After all – most DBAs don’t install the database software very often; they just want to keep it running! But please let me know if it interests you. I’ll factor in the feedback I receive as I’m developing it.
Ardent Performance Computing 
Great idea! Would it just create a responsefile and use that internally when it does the runInstaller? I would also have it specify the Oracle software owner, the groups to create like dba, oinstall. A flag for creating this such as true or false?
I would start with a basic install. Then add platforms such as Linux and Solaris. Then add ASM and / or RAC installs with CRS. Oracle really needs a text based installer. The only way to avoid X-Windows is the response file.
Why not just make an installer text based like the Oracle installer? Ask similar questions. Oh well, start simple and build on it. I am sure other people can help with it.
LikeLike
Posted by Tom | October 22, 2008, 7:22 pmThanks for the input Tom! I wasn’t looking to generate a response file – there are two good ways to do that already: (1) launch runInstaller -record or (2) edit the template files provided with the media. In fact it suddenly occurs to me that ORACLE_HOME doesn’t even need to be an option in the stepfile since I could autodetect it from the first responsefile.
This is meant to be more of a framework around the process of installing and patching. You can create the response file on your own with whatever options you want (enterprise vs standard, rac, etc) and then I will use your response file plus a dozen patches and patchsets to churn out the installations. I don’t think I need to worry about ASM since I’m just doing software installation and not database creation.
You do bring up another good point though – I hadn’t thought about RAC support. I’ll have to check if a CRS or RAC install would have any special requirements. Also, I should validate that we’re not running as root.
LikeLike
Posted by Jeremy | October 22, 2008, 7:45 pmHi Jeremy,
A colleague of mine pointed me to this blog post – I just wanted to let you know that we had the same idea back in 2002 and actually started a company around it – http://www.gridapp.com.
We make a software product called “Clarity” that can automate the complete build of databases, such as install, upgrade, patching, configuration, schema loading, etc. for Oracle, Oracle RAC, SQL Server, Sybase, etc.
It’s not a free software solution, but we’re always interested in creative distribution mechanisms, so if you have any ideas, I’d love to hear them.
Thanks!
Matt Zito
GridApp
LikeLike
Posted by Matthew Zito | October 23, 2008, 7:54 amJeremy,
That’s a good idea but probably unrealistic.
I know the subject pretty well since I’ve done a lot to automate those install/patch set/oneoff/clone with and without RAC. I’m pretty good at that btw. If you take every install as a context, you’ll easily beat any automated installation soon: In speed and quality :-). See for some examples here:
http://www.pythian.com/blogs/1046/oracle-silent-mode-part-8-add-a-node-to-a-102-rac-database
There are many issues to automate the provisioning and patching of the Oracle software. I wouldn’t assume I can do better than the provisioning Pack in that area (and it looks like you and me know what we are talking about):
* You need to download the files from eDelivery AND from Metalink
* You need to check the OS pre-requisites that are very dependent of the platform and may still today require reboot
* You have to use several tools to validate your config CVU or RDA. And sometimes there are pre-requisites available in Metalink notes
* You may have to run pre/post scripts that are in the README files
* The installer syntax can differ from one release to another (with 11.1.0.7, you have to take care of OCM, that wasn’t the case before, the procedure to patch the clusterware with 11.1.0.6 differs from the one to patch with 10.2.0.4 or 11.1.0.7)
* You may have to preform some steps as root (check CPUOCT2008 if you wonder). Depending the way the server are secured, the way to do run those scripts will differ…
* You may need to perform strict sequence of operations on several servers (In the case of rolling patch upgrades with RAC)
* You get tons of patches every week and you’ll have to keep up with them so that your orainstall can manage them all
* If you pass everything you’ll still get weird problems due to some configurations with third party software from storage layers to network specificity or clusterware.
But here is my point… On some Linux Platform, I know how to get to a point where you install a 4 node RAC with the OS and all the prerequisites in 10 min. Add a node in 5 on top of the bare metal. But who would be interested to get that level of speed ? Most projects with Oracle are dozen thousand of USD anyway. Do you think anybody really care about 2 days of work to install a RAC?
If you know somebody who care, I’d be more than interested to work with you and him ;-).
LikeLike
Posted by Gregory | October 23, 2008, 8:02 amMatt – thanks for the comment. Clarity looks like a really solid product – I’d probably put it in the category of “advanced kits for the big data centers” (and perhaps the occasional medium-sized one). In fact it handles a lot of the issues that Gregory just mentioned in his post. But I think I’m looking for something a little smaller in scope.
Gregory – good points, especially about RAC and pre/post scripts. But I think that you’re missing the main idea of the script. It’s not intended to validate your environment or check for patch conflicts – that’s all still your job. It also isn’t intended to register with OCM or deal with upgrades – just copy the bits from B24792-01_1of5.zip to an empty ORACLE_HOME.
Like you, I could get a RAC system up and running by hand in a few hours. (In fact we did it at the RAC Attack IOUG event here in Chicago.) But at some organizations I’ve worked with, Oracle is part of a packaged application that needs an automated build process using only code from CVS or clearcase – and there are nightly builds. At other organizations I’ve worked with, teams of 80 DBA’s manage several thousand oracle databases – so they standardize on heavily tested internal “builds” of Oracle with specific patch levels. The build process doesn’t register with OCM or upgrade anything; those steps are part of installing the build on a server. That build process is the specific use case that I have in mind.
LikeLike
Posted by Jeremy | October 23, 2008, 8:46 am@Jeremy – certainly, we do sell to large and medium-sized companies. But we also sell to small companies – some companies have bought the software to roll out one or two rac clusters.
@gregory – it’s not a question of speed, it’s consistency, and simplicity. What happens when you get used to the prereqs on RHEL4, and all of a sudden your organization decides to start rolling out SLES 10? Or what happens when you have to wait for your SA to set up the storage? Or when you want a junior DBA to set it up? Or you have to prove to a regulatory body that every database has the latest CPU rolled out?
LikeLike
Posted by Matthew Zito | October 23, 2008, 8:58 am@Matt. I don’t know any Senior DBAs are not junior at something. I understand what you are talking about.
@Both. I admire you a lot. Both of you! And I’ve been walking in your steps a few times ;-). Regarding Provisioning or what we could name Gold Images, I’m very sure that’s just the beginning and if you’re looking for alpha/beta testers for anything that would improve the way we manage that, I’m in! As a matter of fact, I wouldn’t be surprised if Oracle does a lot of changes in that area in the coming years.
I’m looking forward to see Clarity manage rolling upgrade patches soon and to use orainstaller.
LikeLike
Posted by Gregory | October 24, 2008, 3:39 amSomewhat embarrassed that I didn’t get here until now, but wanted to contribute that I can absolutely see benefit to this, especially for large(r) organizations that will do all the checks once, then via standards, be assured that the same installation, with the same options, can be performed many times in a row.
Equally, I could see potential benefit for those sites without much knowledge at all using this type of tool just to complete the process since they aren’t familiar with opatch or other such tools.
I’d definitely give it a whirl.
@Greg I’m very interested in how you add a node from bare metal in 5 minutes and build a 4-node cluster in 10 mins. Were you serious, or exaggerating to make a point? I can do those things pretty quickly, but not to those extreme numbers…even if I script it all out with rsp files.
LikeLike
Posted by Dan Norris | November 26, 2008, 12:06 pmDan,
I can create extra nodes in about 10 minutes, meaning, if they clone the Hard Disk over, change the host names and IP addy’s, I then have CRS disabled before they map it to the shared storage from the new IP’s hostnames. Once that is done, it is just renaming some directories and running some srvctl commands to add the nodes. It’s not “exactly” supported but I figured it out. I could share more with you if interested. It works great in my “play” environments.
LikeLike
Posted by Tom | November 29, 2008, 11:53 pmOh, I also do the rootconfig too.
LikeLike
Posted by Tom | November 29, 2008, 11:53 pmHi
I had same issue and prepared a bash script to do oracle installation automatically. Now I have some plans to write such a script for installation of Oracle RAC. I think there are many challenges to do Oracle RAC installation automatically, for example some issues like synchronization between nodes and some reboots which need during installation. I would be glad to have your advices about the issues.
Thanks
LikeLike
Posted by Ali | March 2, 2009, 12:07 am