die Seilerwerks will now be found at http://www.seiler.us.

A few months ago, we had a test instance complaining that it couldn’t write to ASM. This was an 11.1.0.7 single (non-RAC) instance on Oracle Enterprise Linux 5, using ASM for the storage. We first saw these errors in the alert log:

ORA-15032: not all alterations performed
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation
ERROR: error ORA-15032 caught in ASM I/O path

Uh-oh, that doesn’t look good. So I log into the ASM instance and try to see if the disks are OK:

SQL> select path, mount_status from v$asm_disk;
select path, mount_status from v$asm_disk
                               *
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation

I can’t even query that. As Ted would say, “strange things are afoot at the Circle K.” To be safe, I thought I’d try to shutdown the DBMS instance, which also failed without having to abort:

SQL> shutdown immediate
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '+FOOTEST_DATA/footest1_footest_db/control01.ctl'
ORA-15081: failed to submit an I/O operation to a disk
SQL> shutdown abort
ORACLE instance shut down.

We decided to restart the whole DBMS/ASM/CSS stack, but CSS wouldn’t stop either:

-bash-3.2# /etc/init.d/init.cssd stop
Stopping Cluster Synchronization Services.
Unable to communicate with the Cluster Synchronization Services daemon.
Shutdown has begun. The daemons should exit soon.

We ended up booting the server altogether, after which everything came up nicely. We filed an SR with Oracle Support, who directed us to Note 391790.1 (Unable To Connect To Cluster Manager Ora-29701). This note lists the cause, quite simply, as:

The hidden directory ‘/var/tmp/.oracle’ was removed while instances & the CRS stack were up and running. Typically this directory contains a number of “special” socket files that are used by local clients to connect via the IPC protocol (sqlnet) to various Oracle processes including the TNS listener, the CSS, CRS & EVM daemons or even database or ASM instances. These files are created when the “listening” process starts.

The solution is to restart CRS or reboot the machine. Our /var/tmp/.oracle directory looked like this:

[oracle@footest ~]$ ls -la /var/tmp/.oracle
total 12
drwxrwxrwt 2 root   root 4096 May  8 15:03 .
drwxrwxrwt 3 root   root 4096 May 10 07:02 ..
srwxrwxrwx 1 oracle dba     0 May  8 15:03 s#18854.1
srwxrwxrwx 1 oracle dba     0 May  8 15:03 s#18854.2
srwxrwxrwx 1 oracle dba     0 May  8 15:03 sEXTPROC
srwxrwxrwx 1 oracle dba     0 May  8 14:44 sfootestDBG_CSSD
srwxrwxrwx 1 oracle dba     0 May  8 14:44 sOCSSD_LL_footest_
srwxrwxrwx 1 oracle dba     0 May  8 14:44 sOCSSD_LL_footest_localhost
srwxrwxrwx 1 oracle dba     0 May  8 14:44 sOracle_CSS_LclLstnr_localhost_0
srwxrwxrwx 1 oracle dba     0 May  8 15:03 sPNPKEY

I did some sandbox testing, and found that only the Oracle and root OS users could delete that directory, and was able to duplicate the error every time when doing so.

However, I really was dumbstruck that Oracle would have so critical a directory in /var/tmp! I politely note this to Oracle Support, who justified this location with a few solid reasons:

1. It has always been in this location (and still is in 11gR2).
2. /var/tmp/.oracle is a hidden directory, so it probably won’t be noticed by any miscreants looking to cause trouble.

OK, I was being sarcastic, these reasons are awful. The only safeguard they gave was “make sure no one deletes it.” We scoured the server for cron jobs that would automatically clean out /var/tmp but didn’t find any, nor any bash history suggesting malice. The only thing that we could think of was that this test server was in a VM (Citrix Xen), although one would hope that it doesn’t happen at all, regardless. We certainly could not find an explanation, but now we’re aware to not delete /var/tmp/.oracle while the instances are running (even though we never did before).

Surachart Opun has also blogged on this topic.

I recently reformatted my laptop with the latest Ubuntu LTS release, 10.04, aka Lucid Lynx. Since I like to have a native client installation as well as a portable sandbox server, I decided to install the latest version of Oracle EE, 11.2.0.1.

Rather than re-invent the wheel, I’m going to direct you to the previous Oracle-on-Ubuntu post by my colleague Augusto Bott. Many of the directions there hold true here (even with 32-bit vs 64-bit), with a few exceptions.

Download the Software
First and foremost, download the 32-or-64-bit installer files (there are 2 of them) from OTN.

Installing Pre-requisite Packages
Install these software packages on your Ubuntu 10.04 system:

$ sudo apt-get install unzip build-essential x11-utils rpm ksh lsb-rpm libaio1

Ubuntu 10.04 comes with libstdc++6 installed. However, Oracle 11gR2 requires libstdc++5. If you do not install libstdc++5, you will see errors as described in this OTN thread. The fix, as described in that thread, is to download and manually shoehorn the libstdc++5 library files onto Ubuntu 10.04:

$ wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
$ dpkg-deb -x libstdc++5_3.3.6-17ubuntu1_amd64.deb ia64-libs
$ sudo cp ia64-libs/usr/lib/libstdc++.so.5.0.7 /usr/lib64/
$ cd /usr/lib64/
$ sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

$ wget http://security.ubuntu.com/ubuntu/pool/universe/i/ia32-libs/ia32-libs_2.7ubuntu6.1_amd64.deb
$ dpkg-deb -x ia32-libs_2.7ubuntu6.1_amd64.deb ia32-libs
$ sudo cp ia32-libs/usr/lib32/libstdc++.so.5.0.7 /usr/lib32/
$ cd /usr/lib32
$ sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

Now we’re ready to move on.

Create User and Groups
First create the oinstall and dba groups:

$ sudo su -
# addgroup oinstall
# addgroup dba

Then create the oracle user and assign it to those groups:

# useradd -g oinstall -G dba -p password -d /home/oracle -s /bin/bash oracle
# mkdir /home/oracle
# chown -R oracle:dba /home/oracle

Now we create some symbolic links to give the Ubuntu system a more “Red Hat-ish layout,” as Augusto put it:

# ln -s /usr/bin/awk /bin/awk
# ln -s /usr/bin/rpm /bin/rpm
# ln -s /usr/bin/basename /bin/basename
# mkdir /etc/rc.d
# for i in 0 1 2 3 4 5 6 S ; do ln -s /etc/rc$i.d /etc/rc.d/rc$i.d ; done
# mkdir -p /u01/app/oracle
# chown -R oracle:dba /u01

Then we update some sysctl parameters by editing /etc/sysctl.conf. It is wisest to backup this file first and then add these lines to the end of /etc/sysctl.conf:

fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
net.ipv4.ip_local_port_range = 9000 65535

Now we update some limits for the oracle software owner by adding these lines to the end of /etc/security/limits.conf:

oracle soft nproc 2047
oracle hard nproc 16383
oracle soft nofile 1023
oracle hard nofile 65535

Now we activate the new settings from /etc/sysctl.conf with this command:

# sysctl -p

Unpack and Install the Software
From here we go into the directory containing our two 11gR2 database zip files. I’m using the 64-bit edition, so my files are named linux.x64_11gR2_databaseXof2.zip, where X is 1 or 2. I unzip both of these files, which creates a “database” subdirectory:

$ unzip linux.x64_11gR2_database_1of2.zip
$ unzip linux.x64_11gR2_database_2of2.zip

$ ls -lh
total 2.2G
drwxr-xr-x 8 seiler seiler 4.0K 2009-08-20 14:34 database
-rw-r--r-- 1 seiler seiler 1.2G 2010-03-09 11:33 linux.x64_11gR2_database_1of2.zip
-rw-r--r-- 1 seiler seiler 1.1G 2010-03-09 12:38 linux.x64_11gR2_database_2of2.zip

Now you simply cd into the database directory and run the runInstaller program:

$ cd database
$ ./runInstaller

This will launch the Oracle Universal Installer, or OUI, program. The rest is pretty straight forward. Since this is just a sandbox, I chose not to provide any email info and declined to receive updates. I chose to first install the software only, which went perfectly well. OUI will complain about missing packages, since it is checking for RPMs. We can safely ignore these and proceed with installation.

Once installation is done I set these variables in my bash environment:

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=orcl
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export EDITOR=/usr/bin/vi

Then some quick tests to verify installation:

$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 10 23:46:42 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

SQL> quit
$ /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2010-06-10_23-46-53PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2010-06-10_23-46-53PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

I launched dbca to create an Oracle 11gR2 instance. The one problem I encountered here was that, at the end of dbca prompts, the “Confirmation” dialog window was blank. Other blog posts on the internet suggested blindly clicking around until you magically hit the “OK” button, but more often than not I would hit the “Cancel” button. Turns out this is a not-so-uncommon problem with Java Swing and Gnome’s visual effects. The solution for Gnome users is to disable Visual Effects in the Appearance preferences before launching dbca. If you are using a lightweight desktop environment such as LXDE, you shouldn’t have this problem.

And that’s that. The rest is nothing new to those who have done installations before. You can optionally configure TNS names or Listener with the netca tool, or log into your new instance and enjoy.

Special thanks to Augusto Bott for not only authoring the previous Oracle-on-Ubuntu articles but also for suggesting the Gnome Visual Effects conflict!

When news of the April 2010 PSU for Oracle 11.2 came out, I was excited to see it, since it marked the first non-one-off patch release for the 11.2 database software. I happened to have an 11gR2 test system running on 11gR2 ASM via standalone Grid Infrastructure. I applied PSU 9352237 to the DBMS home and fired it up, only to see the folly of my ways when any ASM file operations like disk resizing (or auto-extending) failed with ORA-1653. This was due to the DBMS component now having a higher version number than the ASM component, which ASM does not allow. The Grid Infrastucture PSU would need to be applied to bring the ASM component up to snuff, but that patch (9343627) was, at that time, only “announced” with no ETA. Alas, the patch was rolled back and we continued testing without it.

Then this week I check again and saw that PSU 9343627 was released and gave it a whirl. I was a little confused when the README seemed to contain a lot of instructions that always assumed it to be on a clustered, RAC install. My setup was a single-instance Grid Infrastructure installation just to provide ASM. I soon met problem upon problem when going through first this setup step:

# GRID_HOME/crs/install/rootcrs.pl -unlock
2010-05-03 11:40:42: Parsing the host name
2010-05-03 11:40:42: Checking for super user privileges
2010-05-03 11:40:42: User has super user privileges
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly 
cleanup the processes started by Oracle clusterware
The Oracle Clusterware stack failed to stop.
You should stop the stack with 'crsctl stop crs' and rerun the command

$ GRID_HOME/OPatch/opatch napply -local -oh GRID_HOME -id 9343627,9352237
Invoking OPatch 11.2.0.1.2

... snip ...

Running prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:

Patch 9343627:
Copy Action: Desctination File "GRID_HOME/bin/oradnssd" is not writeable.
'oracle.crs, 11.2.0.1.0': Cannot copy file from 'oradnssd' to 'GRID_HOME/bin/oradnssd'

UtilSession failed: Prerequisite check "CheckApplicable" failed.

OPatch failed with error code 73

I created an SR with Oracle support, and after reviewing my case agreed that the instructions were not right for single-instance Grid Infrastructure installations. They will be updating the README, but in the meantime I’d like to share the revised instructions that got me through two successful GI/DBMS PSU applications today.

Naming Conventions
First a few naming conventions to avoid confusion:

• DBMS_HOME – The Oracle DBMS installation directory.
• GRID_HOME – The Oracle Grid Infrastructure installation directory.

I’ve replaced all references to the directories in my examples with DBMS_HOME and GRID_HOME. Whenever you see those strings, substitute the actual path to your DBMS install or GRID install, whichever the case may be.

Download the PSU and OPatch 11.2
First, you’ll obviously need to download patch 9343627 that contains the PSU for your platform. In my case that’s Linux x86_64. As of this writing, I believe it’s only available for Linux x86 and x86_64. Note that patch 9343627 also includes patch 9352237. You do not have to, and should not, download that PSU separately. Everything you need is in the 9343627 download.

Second, choose a working directory to unzip that file. In my case, this is /home/oracle/software/11gR2/psu/gi, which looked like this after unzipping the PSU zip file:

$ ls
9343627  9352237  p9343627_112010_Linux-x86-64.zip

Unlike normal one-off patches, where you would descend into the individual patch-number-named directories, you’ll stay at this level.

Note also that you’ll need OPatch 11.2 or higher. See MOS Doc ID 6880880 and get the latest version of OPatch for your platform. Then just unzip the file in the DBMS_HOME and GRID_HOME directories, overwriting the old OPatch directory.

Pre-Patch Steps
First I’m assuming that you have shut down all DBMS instances on this server. We will shut down ASM later in this process, so be sure to shutdown the DBMS instances now.

Now we’ll start the pre-patch steps. First we stop the database on this server, saving the state of the database configuration in /tmp/oracle_home.stat:

$ DBMS_HOME/bin/srvctl stop home -o DBMS_HOME -s /tmp/oracle_home.stat

Next we save the database home configuration. This only needs to be done if we’re also going to patch the DBMS home in addition to the GI home. This is done from the directory where we unzipped the PSU:

$ ./9343627/custom/server/9343627/custom/scripts/prepatch.sh -dbhome DBMS_HOME
./9343627/custom/server/9343627/custom/scripts/prepatch.sh completed successfully.

After that, we shut down ASM as one would normally do it:

$ . oraenv
+ASM
$ sqlplus / as sysasm
SQL> shutdown immediate;
SQL> quit

Then, as root, we unlock and shutdown HAS (not CRS, as the original README would have you do):

# GRID_HOME/crs/install/roothas.pl -unlock
2010-05-04 17:14:08: Checking for super user privileges
2010-05-04 17:14:08: User has super user privileges
2010-05-04 17:14:08: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock GRID_HOME

Now we’re ready to do some patching.

Applying PSU 9343627 and 9352237 to Oracle 11.2 Grid Infrastructure
As I mentioned earlier, the PSU contains both 9343627 and 9352237, and it is advised that you apply both. I did so successfully with these commands:

$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9343627
$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9352237

Applying PSU 9343627 and 9352237 to Oracle 11.2 DBMS
Here is something slightly different when applying the 9343627 patch to the DBMS_HOME, we specify a subdirectory after the “napply” parameter:

$ DBMS_HOME/OPatch/opatch napply 9343627/custom/server/ -oh DBMS_HOME -id 9343627

$ DBMS_HOME/OPatch/opatch napply -oh DBMS_HOME -id 9352237

Note that I use OPatch from DBMS_HOME when patching DBMS_HOME, and from GRID_HOME when patching GRID_HOME. I can’t say if it is actually necessary, but I like to keep things tidy. This is why I advised installing the new OPatch into both homes earlier.

As of this moment, both homes are patched, but we aren’t done yet!

Post-Patch Steps
First we run the postpatch.sh script included in patch 9343627 to reset some file permissions on the DBMS_HOME:

$ ./9343627/custom/server/9343627/custom/scripts/postpatch.sh -dbhome DBMS_HOME

Now we re-lock the GRID_HOME and restart the GI stack. As root:

# GRID_HOME/crs/install/roothas.pl -patch
2010-05-04 17:33:41: Checking for super user privileges
2010-05-04 17:33:41: User has super user privileges
2010-05-04 17:33:41: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.

Verify that the services we need are started (again as root):

# crsctl check has
CRS-4638: Oracle High Availability Services is online
# crsctl check css
CRS-4529: Cluster Synchronization Services is online

If you don’t see CSS started, then you aren’t alone. On both of my servers, CSS was not set to auto-start from OHASD. This was fixed by running these commands as the GRID_HOME owner (oracle):

$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"

Then, as root:

# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.

# crsctl start has
CRS-4123: Oracle High Availability Services has been started.

# crsctl check has
CRS-4638: Oracle High Availability Services is online

# crsctl check css
CRS-4529: Cluster Synchronization Services is online

Moving on in the post-patch steps, we start the DBMS_HOME from the state file we created earlier:

$ DBMS_HOME/bin/srvctl start home -o DBMS_HOME -s /tmp/oracle_home.stat 

At this point, we are done. You can start ASM and the DBMS instances. For some affirmation, run this for both GRID_HOME and DBMS_HOME (just using GRID_HOME in this example)

$ GRID_HOME/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : GRID_HOME
Central Inventory : /opt/oracle/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : GRID_HOME/oui
Log file location : GRID_HOME/cfgtoollogs/opatch/opatch2010-05-04_18-36-23PM.log

Patch history file: GRID_HOME/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : GRID_HOME/cfgtoollogs/opatch/lsinv/lsinventory2010-05-04_18-36-23PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  9352237      : applied on Tue May 04 18:22:41 EDT 2010
Unique Patch ID:  12381846
   Created on 25 Mar 2010, 00:05:17 hrs PST8PDT
   Bugs fixed:
     8661168, 8769239, 8898852, 8801119, 9054253, 8706590, 8725286, 8974548
     8778277, 8780372, 8769569, 9027691, 9454036, 9454037, 9454038, 8761974
     7705591, 8496830, 8702892, 8639114, 8723477, 8729793, 8919682, 8818983
     9001453, 8475069, 9328668, 8891929, 8798317, 8820324, 8733749, 8702535
     8565708, 9036013, 8735201, 8684517, 8870559, 8773383, 8933870, 8812705
     8405205, 8822365, 8813366, 8761260, 8790767, 8795418, 8913269, 8897784
     8760714, 8717461, 8671349, 8775569, 8898589, 8861700, 8607693, 8642202
     8780281, 9369797, 8780711, 8784929, 8834636, 9015983, 8891037, 8828328
     8570322, 8832205, 8665189, 8717031, 8685253, 8718952, 8799099, 8633358
     9032717, 9321701, 8588519, 8783738, 8796511, 8782971, 8756598, 9454385
     8856497, 8703064, 9066116, 9007102, 8721315, 8818175, 8674263, 9352237
     8753903, 8720447, 9057443, 8790561, 8733225, 9197917, 8928276, 8991997, 8837736

Patch  9343627      : applied on Tue May 04 18:18:31 EDT 2010
Unique Patch ID:  12381846
   Created on 15 Apr 2010, 11:28:38 hrs PST8PDT
   Bugs fixed:
     9343627, 9262748, 9262722



--------------------------------------------------------------------------------

OPatch succeeded.

As I said, Oracle support will be updating the README for PSU 9343627, but in the meantime I hope this guide helps you as much as it helped me. I’d like to thank my Pythian colleague Alex Gorbachev for his help in diagnosing some of the GI problems after the first broken patching, and also Esteban B. at Oracle Support for working closely with us to get a new single-node action plan.

This does require a ~/.drupalsites file that contains the needed database login info (suggest chmod 600). Like I said, nothing too clever.

#!/bin/bash
DATESTAMP=`date +%Y%m%d`
SITESFILE=${HOME}/.drupalsites
# SITESFILE contents are in the following format:
# SITENAME:DBNAME:DBUSER:DBPASSWD
DRUPALDIR=/path/to/drupal
BACKUPDIR=${HOME}/backups/${DATESTAMP}

mkdir -p ${BACKUPDIR}

# Backup the drupal codebase
echo -n "Backing up ${DRUPALDIR} ... "
cd ${DRUPALDIR}/../
tar -cjpf ${BACKUPDIR}/drupal_${DATESTAMP}.tar.bz2 drupal
echo "Done."

cd ${BACKUPDIR}

# Backup MySQL Databases
cat $SITESFILE | while read line; do
        #echo $line
        line=(${line//:/ })
        echo -n "Backing up MySQL db ${line[1]} ... "
        DUMPFILE=${line[0]}_${DATESTAMP}.sql
        mysqldump -u ${line[2]} -p${line[3]} ${line[1]} > ${DUMPFILE}
        gzip ${DUMPFILE}
        echo "Done."
done

echo "Backup completed, all files are in ${BACKUPDIR}."

This script runs fine from cron, I have it scheduled for a weekly run. I then plan to rsync this to my home server for an offsite copy, even though the hosting service provides backups as well.

I recently performed a migration from Oracle 10gR2 on Solaris to the same version on Linux, immediately followed by an upgrade to 11g. Both platforms were x86-64. Migrating to Linux also included migrating to ASM, whereas we had been using ZFS to hold the datafiles on Solaris. Restoring files into ASM meant we would have to use RMAN (which we would probably choose to use anyway).

As with many databases, the client wanted minimal downtime. It was obvious to us that the most time-consuming operation would be the restore and recovery into the new instance. We were basically doing a restore and recovery from production backups and archived redo logs. It quickly dawned on me that we could start this operation well before the scheduled cutover time and downtime window, chopping at least six hours from the downtime window. The client would only need to keep the new instance in mount mode after the initial restore/recovery finished, periodically re-catalog the source instance’s FRA (which was mounted via NFS), and then re-run the recover database command in RMAN. Once the time comes to cutover, simply archivelog current the original instance and shutdown immediate. Then open the new instance with the RESETLOGS option, and voila! Migration complete!

I’ll try to recreate a simple example here.

On the new instance, assume that you have ASM set up and configured with two disk groups, DATA and REDO. Let’s also assume that you have the FRA directory (or the directory where you hold backups and archivelogs) mounted from the Solaris host to the Linux host under /solfra/prod/. Here’s the command we used:

mount -o hard,bg,tcp,nfsvers=3,suid,rsize=32768,wsize=32768,nointr,noac,ro solhost:/fra /solfra/

We then need to get a pfile and edit it for ASM-ification. You can either restore the spfile from backup or just export a pfile from the current running production instance. To do the former, you’d need to go into RMAN and run an operation similar to this (best to ensure that the ORACLE_HOME, ORACLE_SID, and other Oracle-related environment variables are set properly):

RMAN> startup force

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/home/oracle/app/oracle/product/10.2.0/db_1/dbs/initprod.ora'

starting Oracle instance without parameter file for retrival of spfile
Oracle instance started
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 06/29/2009 13:34:50
ORA-00205: error in identifying control file, check alert log for more info
RMAN>
RMAN> restore spfile to pfile 'initprod.ora' from autobackup recovery area = '/solfra/' db_unique_name = 'PROD';

Starting restore at 29-JUN-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=36 devtype=DISK

recovery area destination: /solfra/
database name (or database unique name) used for search: PROD
channel ORA_DISK_1: autobackup found in the recovery area
channel ORA_DISK_1: autobackup found: /solfra/PROD/autobackup/2009_06_29/o1_mf_s_690813149_54kypcxz_.bkp
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 29-JUN-09

From here you can go into sqlplus and CREATE PFILE FROM SPFILE;.

Now we have our text pfile in $ORACLE_HOME/dbs/initprod.ora, and we want to make these changes specific to the transition to ASM:

*.control_files='+DATA','+REDO'
*.db_create_file_dest='+DATA'
*.db_create_online_log_dest_1='+REDO'
*.db_recovery_file_dest='/path/to/fra'
*.db_recovery_file_dest_size='2000G'

Obviously, the last two parameters about location and size of the FRA should be tailored to your setup.

We then create a new SPFILE with CREATE SPFILE FROM PFILE;.

Now we’re ready to restore the controlfile(s) from backup:

RMAN> startup force;

Oracle instance started
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 08/13/2009 20:20:56
ORA-00205: error in identifying control file, check alert log for more info

RMAN> restore controlfile from autobackup recovery area = '/solfra/' db_unique_name = 'PROD';

Starting restore at 13-AUG-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=650 devtype=DISK

recovery area destination: /solfra/
database name (or database unique name) used for search: PROD
channel ORA_DISK_1: autobackup found in the recovery area
channel ORA_DISK_1: autobackup found: /solfra/PROD/autobackup/2009_08_13/o1_mf_s_694801261_5890xyoj_.bkp
channel ORA_DISK_1: control file restore from autobackup complete
output filename=+DATA/prod/controlfile/current.256.694815695
output filename=+REDO/prod/controlfile/current.256.694815697
Finished restore at 13-AUG-00

Note that if you restored the spfile from backup, you don’t need to specify the recovery area in the restore controlfile from autobackup command. In this case, because I chose to create a pfile from the source instance, I needed to supply the recovery area location. Note that this is the recovery area that holds the source backups, not the recovery area that your new instance will use once it is opened.

With the control files restored, we can mount the new instance:

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

The next step is to catalog the backup files and archivelogs from the remote FRA.

RMAN> CATALOG START WITH '/solfra/PROD' NOPROMPT;

Starting implicit crosscheck backup at 13-AUG-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=646 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=650 devtype=DISK
Crosschecked 565 objects
Finished implicit crosscheck backup at 13-AUG-09

Starting implicit crosscheck copy at 13-AUG-09
using channel ORA_DISK_1
using channel ORA_DISK_2
Finished implicit crosscheck copy at 13-AUG-09

searching for all files in the recovery area
cataloging files...

This will continue listing all the files in the catalog, and some files that were not cataloged (e.g. online redo logs, non-RMAN files, etc.).

Now we’re ready to begin our RESTORE phase. You’ll need an rman command file that looks something like this:

run {
  set newname for datafile X to new;
  # ... (repeat for other datafiles)
  set newname for tempfile Y to new;
  # ... (repeat for other tempfiles)
  restore database;
  switch datafile all;
  switch tempfile all;
}
recover database;

Note that X and Y are the file# of the individual datafile and tempfile, respectively. You will probably want to generate this file if you have a large number of datafiles.

It is important that you read MetaLink Doc 415884.1, if you are performing a migration to a different platform with the same endian. Oracle suggests that you should perform a CONVERT DATABASE, but later admits that you only need to perform a CONVERT on tablespaces containing UNDO segments:

The convert database command is a no operation command for all files that do not contain undo but they are scanned regardless during the conversion. Using the convert tablespace/datafile for those containing undo will suffice and all other source files not containing undo can be moved to the target (same endian) platform without the Recovery Manager conversion. This will greatly reduce the time required to move and database with thousands of files from source to target platform also decreasing the amount of time the database is unavailable.

We had an internal debate over whether or not it is truly necessary even to do this. However, I would advise you to read this document and discuss it with Oracle Support Services. They would most likely try to convince you to do it or threaten to not support your database.

Once the script finishes with the recover database portion, this query should indicate that datafile headers and controlfiles are in sync:

SQL> alter session set nls_date_format='YYYY/MON/DD hh24:mi:ss';

Session altered.

SQL> select checkpoint_time,fuzzy,count(*),status
from ( select checkpoint_time,fuzzy,status
       from v$datafile_header
       union all
       select controlfile_time,'CTL',null from v$database)
group by checkpoint_time,fuzzy,status;

CHECKPOINT_TIME      FUZ   COUNT(*) STATUS
-------------------- --- ---------- -------
2009/AUG/14 02:22:11 CTL	  1
2009/AUG/14 02:22:11 YES       1266 ONLINE

The database is still in mount mode. If we were ready to cut over now, we could open the database with the RESETLOGS option. However, we are still a few hours or a day away from the cutover. What we want to do is apply more logs from the current production instance as they get archived. To minimize the babysitting, we decided to run this in a script once every few hours:

CATALOG START WITH '/solfra/PROD' NOPROMPT;
RECOVER DATABASE;

As you can guess, this re-catalogs the NFS-mounted FRA, ignoring files that were already cataloged. So now the controlfile knows about the new archivelogs. The RECOVER DATABASE command then beautifully applies logs from where it left off until it runs out of logs again. We re-run this periodically to keep our new instance as close to current as possible until it’s time to cut over.

When the cutover time comes, we run this on the current production instance:

SQL> alter system archive log current;

System altered.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

Then, on the new instance, we do one last catalog and recover. This should bring us completely up to date with the original prod instance. Now we just need to open with resetlogs:

SQL> alter database open resetlogs;

Database altered.

In my case, I manually renamed the online and standby redo logs to +REDO (they still had the old filesystem names in v$logfile), but I believe that OPEN RESETLOGS does this automatically anyway.

And so, here we are in Linux on on the same version (10.2.0.2) as Solaris with appropriate one-off patches, using ASM. At this point, I was just under 30 minutes into my downtime window since the original prod instance was shutdown. I don’t think that’s too bad for moving a 1.3T database!

After this, I immediately went into upgrading to 11gR1 (11.1.0.7) and then building a new standby instance, but that is outside the scope of the initial migration that I wanted to share with you today. Cheers!

Full credit for this tale should go to my colleague Christo Kutrovsky for the inspiration and basic plan involved.

We recently migrated a large database from Solaris SPARC to Solaris x86-64. All seemed to go well with the migration, but in the next few weeks, we noticed some I/O issues cropping up. Some research led us to find that the ZFS filesystem used to hold the datafiles was killing us on I/O. The default “recordsize” setting for ZFS was 128k.

$ /usr/sbin/zfs get recordsize zfs-data
NAME      PROPERTY    VALUE     SOURCE
zfs-data  recordsize  128K      default

With ZFS, not only was the throughput much lower, but we used more [than] twice the amount of CPU per transaction, and we are doing 2x times the IO. The disks are also more heavily utilized. We noticed that we were not only reading in more data, but we were also doing more IO operations [than] what is needed.

The fix is to set the ZFS recordsize for a datafile filesystem to match the Oracle instance’s db_block_size. We also read in the ZFS Best Practices Guide that redo logs should be in a separate filesystem with the default ZFS recordsize of 128k. We already had them separate, so we just needed to get our datafiles on a ZFS filesystem with a 16k recordsize.

The first step is to change the ZFS recordsize for the current filesystem. Easy enough, but the change in recordsize will only apply to new files written there. It doesn’t do anything for our existing terabyte-plus of datafiles. This means we need to move the files to a new ZFS filesystem with the correct recordsize already set. That’s where the magic happens.

My first thought was that we’d need whole other slab of disk. Then I learned that we can simply create a new ZFS filesystem on the same pool of disks as the old filesystem. Obviously we won’t be able to hold two complete copies of the datafiles, but this would let us migrate groups of files at a time (for example, a tablespace). When the move is complete, disk space is released from the old filesystem, so the sum of used disk is essentially unchanged.

The rest is just a matter of moving datafiles. Now, we all know you can’t just move or copy Oracle datafiles from a running instance. We also know that clients want as little downtime as possible, so doing a complete shutdown/move/mount/rename/open is out of the question. We could offline a tablespace, move its datafiles, rename them in the database, then online the tablespace again. Here’s a sample scenario, starting with tablespace and datafile creation:

SQL> create tablespace zfstest
  2          datafile '/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf'
  3          size 128m;

Tablespace created.

SQL> alter tablespace zfstest
  2          add datafile '/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf'
  3          size 128m;

Tablespace altered.

SQL> create table zfsobjects tablespace zfstest
  2          as select * from dba_objects;

Table created.

• /u01/app/oracle/oradata/orcl/zfs128k/ is a ZFS filesystem with default 128k recordsize
• /u01/app/oracle/oradata/orcl/zfs16k/ is a ZFS filesystem with a 16k recordsize

So here we have datafiles created on a ZFS filesystem with the default 128k recordsize, and a small table just to test data access. To move the entire tablespace in one go, we do this:

SQL> alter tablespace zfstest offline;

Tablespace altered.

SQL> !mv /u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf
     /u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf

SQL> alter database rename file '/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf' to
'/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf';

Database altered.

SQL> !mv /u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf
/u01/app/oracle/oradata/orcl/zfs16k/zfstest02.dbf

SQL> alter database rename file '/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf' to
'/u01/app/oracle/oradata/orcl/zfs16k/zfstest02.dbf';
Database altered.

SQL> alter tablespace zfstest online;

Tablespace altered.

SQL> select count(*) from zfsobjects;

  COUNT(*)
----------
     68844

Note that this operation could take a very long time in the case of large tablespaces. We could try a somwhat similar approach of taking one datafile offline at a time and moving it. We’d just need to perform a quick recovery before bringing it online. That would look like this:

SQL> alter database datafile '/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf' offline;

Database altered.

SQL> !mv /u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf
/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf

SQL> alter database rename file '/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf' to
'/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf';

Database altered.

SQL> recover datafile '/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf';
Media recovery complete.

SQL> alter database datafile '/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf' online;

Database altered.

SQL> alter database datafile
'/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf' offline;

Database altered.

SQL> !mv /u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf
/u01/app/oracle/oradata/orcl/zfs16k/zfstest02.dbf

SQL> alter database rename file '/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf'
to '/u01/app/oracle/oradata/orcl/zfs16k/zfstest02.dbf';

Database altered.

SQL> recover datafile '/u01/app/oracle/oradata/orcl/zfs16k/zfstest02.dbf';
Media recovery complete.

SQL> alter database datafile '/u01/app/oracle/oradata/orcl/zfs16k
/zfstest02.dbf' online;

Database altered.

SQL> select count(*) from zfsobjects;

  COUNT(*)
----------
     68844

SQL> select count(*) from zfsobjects;
select count(*) from zfsobjects
                     *
ERROR at line 1:
ORA-00376: file 6 cannot be read at this time
ORA-01110: data file 6: '/u01/app/oracle/oradata/orcl/zfs16k/zfstest01.dbf'

First, we want to know the file numbers of our datafiles. This is because the RMAN procedure doesn’t provide for an easy way to preserve datafile names when we backup an entire tablespace (which we’re going to do):

SQL> select file_id, file_name
  2  from dba_data_files
  3  where tablespace_name='ZFSTEST';

   FILE_ID FILE_NAME
---------- -------------------------------------------------------
	 6 /u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf
	 7 /u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf

RMAN> backup as copy
2> 	tablespace zfstest format '/u01/app/oracle/oradata/orcl/zfs16k/%U';
3>
4> sql 'alter database datafile 6 offline';
5> switch datafile 6 to copy;
6> recover datafile 6;
7> sql 'alter database datafile 6 online';
8>
9> sql 'alter database datafile 7 offline';
10> switch datafile 7 to copy;
11> recover datafile 7;
12> sql 'alter database datafile 7 online';
13>
Starting backup at 18-AUG-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=134 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf
output file name=/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-6_01kn0irh tag=TAG20090818T140640 RECID=2 STAMP=695225208
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf
output file name=/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-7_02kn0is0 tag=TAG20090818T140640 RECID=3 STAMP=695225222
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 18-AUG-09

sql statement: alter database datafile 6 offline

datafile 6 switched to datafile copy
"/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-6_01kn0irh"

Starting recover at 18-AUG-09
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 18-AUG-09

sql statement: alter database datafile 6 online

sql statement: alter database datafile 7 offline

datafile 7 switched to datafile copy
"/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-7_02kn0is0"

Starting recover at 18-AUG-09
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 18-AUG-09

sql statement: alter database datafile 7 online

Recovery Manager complete.

SQL> select file_id, file_name
  2  from dba_data_files
  3  where tablespace_name='ZFSTEST';

   FILE_ID
----------
FILE_NAME
--------------------------------------------------------------------------------
	 6
/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-6_01kn0irh

	 7
/u01/app/oracle/oradata/orcl/zfs16k/data_D-ORCL_I-1203425866_TS-ZFSTEST_FNO-7_02kn0is0

SQL> select count(*) from zfsobjects;

  COUNT(*)
----------
     68844

UPDATE: Teammate Marc Billette pointed out that the old datafiles were not removed when doing a datafile copy and switch. I clearly left them on disk, continuing to use space! So, once we are done with all the datafile switches, then our original datafiles become the copies, and we can list and delete those in RMAN like this:

RMAN> list datafilecopy all;

List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time
------- ---- - --------------- ---------- ---------------
4       6    A 18-AUG-09       629982     18-AUG-09
        Name: /u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf

5       7    A 18-AUG-09       630001     18-AUG-09
        Name: /u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf

RMAN> delete datafilecopy all;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=135 device type=DISK
List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time
------- ---- - --------------- ---------- ---------------
4       6    A 18-AUG-09       629982     18-AUG-09
        Name: /u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf

5       7    A 18-AUG-09       630001     18-AUG-09
        Name: /u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf

Do you really want to delete the above objects (enter YES or NO)? YES
deleted datafile copy
datafile copy file name=/u01/app/oracle/oradata/orcl/zfs128k/zfstest01.dbf RECID=4 STAMP=695225227
deleted datafile copy
datafile copy file name=/u01/app/oracle/oradata/orcl/zfs128k/zfstest02.dbf RECID=5 STAMP=695225228
Deleted 2 objects

I hope this post can help other people that run into this problem with Oracle on ZFS (I’ve heard from one on twitter already), or people just wanting to move datafiles. Again, thanks to Pythian DBA Christo Kutrovsky for laying this process out for us!

Note: this process was tested on Oracle 10gR2 (10.2.0.2) and 11gR1 (11.1.0.6).

Having recently performed a test upgrade for a client from Oracle RDBMS 10g to 11g, I can tell you that one of the big changes that will likely require action on your part as DBA is the new fine-grained access control for the packages UTL_SMTP, UTL_TCP, UTL_MAIL, UTL_HTTP and UTL_INADDR. Part of the Oracle 11g pre-upgrade tool will notify you of users that will require new privileges.

Of course, Oracle’s post-upgrade network ACL setup documentation is much more confusing than it needs to be, at least for simple minds like me. A colleague stepped forward with a simple set of commands for a basic setup that even the tired and stressed can understand.

I’ll share that here, with some basic explanation:

BEGIN
  -- Create the new ACL, naming it "netacl.xml", with a description.
  -- Also, provide one starter privilege, granting user FOO
  -- the privilege to connect.
  DBMS_NETWORK_ACL_ADMIN.CREATE_ACL('netacl.xml',
      'Allow usage to the UTL network packages', 'FOO', TRUE, 'connect');

  -- Now grant privilege to resolve DNS names for FOO,
  -- and then grant connect and resolve to user BAR
  DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE('netacl.xml' ,'FOO', TRUE, 'resolve');
  DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE('netacl.xml' ,'BAR', TRUE, 'connect');
  DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE('netacl.xml' ,'BAR', TRUE, 'resolve');

  -- Specify which hosts this ACL applies to,
  -- for simplicity, we're saying all (*)
  -- You might want to specify certain hosts to lock this down.
  DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL('netacl.xml','*');
END;
/

Of course, nothing beats reading the Oracle 11g DBMS_NETWORK_ACL_ADMIN documentation, where you can see some examples of stricter ACL setups.

A couple of weeks ago, we noticed some timeouts in some of our standard Oracle RDBMS health check scripts on a new instance. I had just migrated this instance to bigger, better, badder hardware and so it had been given more SGA to use, namely a bigger buffer cache. The software version was still Oracle 10.2.0.2, as we wanted to introduce as few variables as possible (we were already moving to a new platform with an endian change).

At first the timeouts were infrequent, but over the course of a week started to grow in frequencey until the point where none of the checks were finishing in the allowed timeframe. We ran an AWR report, and tucked far down in the “Latch Activity” section, a colleague noticed this:

                                           Pct    Avg   Wait                 Pct
                                    Get    Get   Slps   Time       NoWait NoWait
Latch Name                     Requests   Miss  /Miss    (s)     Requests   Miss
------------------------ -------------- ------ ------ ------ ------------ ------
...
simulator lru latch          10,032,617    3.3    0.7  44950      336,837    0.3
...
Latch Activity                             DB/Inst: FOO/foo  Snaps: 156-157
-> "Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
   willing-to-wait latch get requests
-> "NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
-> "Pct Misses" for both should be very close to 0.0

                                           Pct    Avg   Wait                 Pct
                                    Get    Get   Slps   Time       NoWait NoWait
Latch Name                     Requests   Miss  /Miss    (s)     Requests   Miss
------------------------ -------------- ------ ------ ------ ------------ ------
transaction branch alloc        112,412    0.0    0.0      0            0    N/A
undo global data                466,321    0.0    0.0      0            0    N/A
user lock                         7,440    0.8    0.4      1            0    N/A
          -------------------------------------------------------------

The “simulator lru latch” event brought us to MetaLink note 5918642.8 and bug 5918642. Affecting 10g and 11g prior to 10.2.0.4 and 11.1.0.7, respectively. The bug is with the database buffer cache advisor, controlled by the parameter db_cache_advice, which defaults to ON (depending on statistics_level). The note simply states:

High simulator lru latch contention can occur when db_cache_advice is set to ON if there is a large buffer cache.

We simply set db_cache_advice to OFF (thankfully it is a dynamic parameter), and pretty quickly our checks were running just fine.

My suggestion is to simply turn this off unless you are actively using the cache advisor to tune an instance. Once you are done tuning, and are no longer using the advisor, turn it off.

NOTE:  As Mladen Gogola pointed out in the comments, turning this off will cause problems if you are using automatic memory management (i.e. sga_target > 0).   Re-pasting his post here:

The problem with that advice is that it will prevent automatic memory management from resizing the buffer cache and the instance will end up with a huge, mostly empty, shared pool and default buffer cache. Automatic memory management is biased toward shared pool even with the cache dvice turned on, without it, buffer cache will be reduced to the minimum size, usually only 64MB. If you disable cache advice, I would also recommend disabling the automatic memory management and configuring SGA manually.

I’m back again with another in what I hope will be a long line of “Quick Tips for Newbies” series.

At The Pythian Group, we have employees all over the globe, from our headquarters in Ottawa to regional offices in Boston, Prague, India and Sydney, and a few scattered remote workers in Seattle, Paris, Kiev, Brazil, South Africa and Wisconsin, among other places. In other words, we are spread across multiple timezones, and since it wasn’t too long ago that everyone was in Ottawa, this is something that still presents little quirks.

One such quirk involved email generated by one of our internal Oracle instances—via a stored procedure that used UTL_SMTP to send the messages—did not have timezone information in the “Date” email header. As a result, they would be stamped with the hour in Eastern timezone (Ottawa time), but the mail clients would think that hour was local. Depending on where you are relative to Ottawa, this could be many hours in the past or future. Of course, this wouldn’t be noticed if you were in Ottawa or even Boston, both in Eastern. For the rest, it was at the very least, an annoyance—but one that is easily fixed.

Looking at the PL/SQL stored procedure that we used to generate email messages, I saw that the “Date” header was being built with this code:

date_hdr := 'Date: '||to_char(sysdate,'dd Mon yy hh24:mi:ss');

The fix is almost trivial—just use SYSTIMESTAMP instead of SYSDATE, and include the timezone in the TO_CHAR function:

date_hdr := 'Date: '||to_char(systimestamp,'dd Mon yy hh24:mi:ss tzhtzm');

Voila! Emails now had a full Date header. And there was much rejoicing from around the world.

Here’s a quick query to highlight the difference:

SQL> select to_char(sysdate,'dd Mon yy hh24:mi:ss') from dual;

TO_CHAR(SYSDATE,'D
------------------
01 Dec 08 18:50:02

SQL> select to_char(systimestamp,'dd Mon yy hh24:mi:ss tzhtzm') from dual;

TO_CHAR(SYSTIMESTAMP,'DD
------------------------
01 Dec 08 18:50:10 -0500

Note: I discovered the fix via this blog post, which seems to be invite-only at the time of this writing.