die Seilerwerks

EDIT: DON’T FOLLOW THESE INSTRUCTIONS IF YOU DON’T WANT TO SCREW THINGS UP FOR YOURSELF.  I’LL BE REVISING THESE INSTRUCTIONS WITH THE PROPER SOLUTION IN THE NEAR FUTURE. 

On my lil’ old Gateway tower, I was running CentOS5 and Oracle 10.2.0.3. I set up an ASM instance pretty easily using the dbca, after using ASMLib to provision the disk partition that I carved out for it.

However when I went to put this practice to some “real” use on our development RHEL4 server connected to a SAN, things didn’t want to play so easily. The SA had provided me with two beautiful 900GB RAID 10 LUNs, and ASMLib seemed to provision them fine via “/etc/init.d/oracleasm createdisk”. However when I tried to create the diskgroup, either via dbca or sqlplus, I was presented with an ORA-15072 error:

SQL> CREATE DISKGROUP dgroup1
2      EXTERNAL REDUNDANCY
3      DISK 'ORCL:VOL01','ORCL:VOL02';
CREATE DISKGROUP dgroup1
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 1 failure groups, discovered only 0

Querying the v$asm_disk view indicated that all was not quite right with what ASM thought of my disks:

SQL> select state,header_status,substr(name,1,12) Name,free_mb,
     substr(path,1,16) PATH from v$asm_disk;
STATE    HEADER_STATU NAME            FREE_MB PATH
-------- ------------ ------------ ---------- ----------------
NORMAL   UNKNOWN                            0 ORCL:VOL02
NORMAL   UNKNOWN                            0 ORCL:VOL01

That “UNKNOWN” header status is obviously why ASM wouldn’t let me use those disks to create my disk group. The “ORCL:VOL0x” path format was no different than I saw in my CentOS sandbox:

SQL> select state,header_status,substr(name,1,12) Name,free_mb,
     substr(path,1,16) PATH from v$asm_disk;
STATE    HEADER_STATU NAME            FREE_MB PATH
-------- ------------ ------------ ---------- ----------------
NORMAL   MEMBER       VOL1               6662 ORCL:VOL1
NORMAL   MEMBER       VOL2               6301 ORCL:VOL2

So I wasn’t sure what the problem might be. Krish Hariharan on the oracle-l list suggested that perhaps the asm_diskstring parameter needed to be set, as my instance had an empty value for it (both instances did, actually). I set asm_diskstring to prefix the actual location of the disk devices created by ASMLib:

SQL> alter system set asm_diskstring='/dev/oracleasm/disks/VOL*' scope=spfile;

And voila. After setting this, the HEADER_STATUS value in v$asm_disk changed to PROVISIONED and the PATH value was the actual file system path, /dev/oracleasm/disks/VOL01 and /dev/oracleasm/disks/VOL02 in my case. I was then able to use those disks in my disk group as I pleased.I’m still not sure why I needed to set asm_diskstring in the one instance but not in the sandbox. The sandbox of course was just a small desktop tower with plain old IDE disk, where was the development server was connected to a Hitachi SAN. If anyone can answer for sure, I’d love to hear it.

In the future, I’ll just make a habit of always setting it.

Not only did I recently wave goodbye to ASMM, I’ve also parted ways with ASM and have gone back to traditional filesystem-based storage.

The reason is basically that it simply was not playing nice with our Hitachi disk. The SA worked with Hitachi to try and get things working but in the end it was still causing ASM to lock up and crash the instances. We’ve had such problems from the beginning. So, needing to provide our developers with something stable to work against, the decision was made to go back to normal file storage. ASM would have been a very nice leap for us, and I was really looking forward to seeing our production database running over it.

So that makes ASM and ASMM down. I think at this point Howard Rogers would put a stake in ASSM’s heart as well!

As I sit here on a Sunday morning waiting for some disk issues to be resolved in my datacenter, I took the time to catch up on some pages I had marked for later reading. One was a post by Christo Kutrovsky from the Pythian Group, about aligning ASM disks on linux, to avoid misalignments due to odd default offsets in linux that might cause I/O for relatively small amounts of data to be inefficiently split across disks.

Also now I can’t help but look at “Christo Kutrovsky” and think of Krusty the Clown, a.k.a. Herschel Krustofski.

I have a small tower running Oracle 10.2.0.3 on CentOS4 under my desk that I use as a personal sandbox for playing with Oracle so I’m not always futzing with developer instances. This was where I first played with ASM, etc. However, I rarely actually use this machine from day-to-day.

Recently, I needed access to a 10.2.0.3 instance to see if a bug still persisted. All of my production and development instances are on 10.2.0.2, so this box was called to duty. However, when I tried to start up cssd (required for ASM), I got errors referring to configuration files missing. (I’ll try to find the exact error if I can dig it up).

The trick is that these files were named based on the server hostname. I had recently renamed this box to fall in line with our internal policy of naming servers after Thomas the Tank Engine characters. So I needed to reset the OCR key to use my new hostname, and all was well again.

Well, all was well after I also installed the updated asmlib RPM for the kernel which I thoughtlessly updated as well.

I received a HOT tip yesterday about the existence of asmcmd. Frankly I’m ashamed that I didn’t know about it sooner, since I’ve been doing an ass-ton of reading about ASM over the past month or so. I’d commit seppuku but I don’t have a trusty second available. Instead, I’ll just share this post with you as my penance.

asmcmd basically provides a command-line interface to the ASM “file system”. Since ASM uses raw disk, traditional OS tools are powerless to view the precious files within. asmcmd lets you navigate the contents of your ASM instance’s disk groups with commands that should be familiar to any Linux user, including wildcard matching.