Category: Technology

Is my RAID working? HP / Compaq SmartArray RAID on Linux

One of our servers is an HP / Compaq 2U machine (DL380 ?) with 6 hard drives and the SmartArray hardware RAID controller. This generally works quite well, but we don’t use one of the handful of Linux distributions that HP supports… so from their site we find a great lack of information on how to administrate or monitor the RAID. It seems silly to me, in the Linux world, to offer such narrow support. Even if only a handful of distributions are fully supported, manufacturers ought to at least offer a page of tips and links for using their hardware with other distributions, offer non-distro-specific tool downloads, human-readable dependency lists, etc.

A while back I got the HP tools to install using “alien”, but these tools seemed overly complex for the simple, ongoing task of monitoring the health of the RAID arrays.

Today I found a simple answer: ArrayProbe, a free / open source tool from:

http://www.strocamp.net/opensource/

for monitoring, among others, SmartArray (CCISS) controllers.

Installation on this Debian system was a not as easy as it could have been. First, the dev tools:

sudo apt-get install gcc libc6-dev

Then I found that it requires the kernel sources (not just headers), in /usr/src/linux, which is not where they landed by default. With a minor “ln-s” workaround, the tool complies and runs fine:

root@servername:~# arrayprobe

OK Arrayprobe
All controllers ok

More details are available with command line options. The simple, one-line output is well suited to automated monitoring and notification – far more useful in many-server environment than a whizbang GUI tool.
Also, a great resource for finding Linux tools for hardware RAID administration and monitor is at http://developer.skolelinux.no/info/prosjektet/delprosjekt/hw-raid-info.html.

Update June 2007: Steve Cameron, maintainer of the CCISS Linux drivers, wrote to point out that the CCISS project now includes (through considerable effort) a cciss_vol_status tool which serves a similar purpose. cciss_vol_status emits somewhat more detailed status data, and might also make it “in the box” in Linux distros eventually, with the core CCISS. I’m unable to try it out myself – we’ve replaced that DL380 with commodity hardware and Linux software RAID.

“Looping” an audio file with Sox, Lame and mkfifo

Today I needed a very long (3 hour) MP3 audio file to use for an experiment; a test file with some music on it. My first thought was to start a MP3 audio recorder, turn on the radio, and leave for 3 hours.

But impatience is among the three great virtues of a programmer, so I turned to Google instead, seeking command line tools for audio manipulation. It turns out that sox and lame will do the job. I installed the tools – here is the Debian / Ubuntu invocation:

apt-get install sox lame

then grabbed an MP3 file of a piece of music (Peter T. Noonan, album “Cafe at Arles”, track 9 “Life’s Old Road”, if you are curious) and repeated it a few times:

sox music.mp3 foo.wav repeat 3

lame foo.wav longmusic.mp3

This worked well, the first time… but consumed a lot of disk space. To get to 3 hours I would need enough disk space for a 3 hour uncompressed WAV file. Unfortunately Sox does not support MP3 output, and I didn’t want to compress to a format it does support, then uncompress and recompress again. So I used a Unix/Linux FIFO pipe instead of a file, with Sox running in the background to fill the pipe with data for Lame:

mkfifo foo.mp3

sox music.mp3 foo.wav repeat 10 &

lame foo.wav longmusic.mp3

a little while later, longmusic.mp3 is a very long MP3 file… but not long enough, because sox fails when the “virtual” WAV file it is writing reaches 2 Gb in size, just as it fails with a real WAV file at that size. That was about 1 hour and 41 minutes – not long enough; so I ended up looking elsewhere:

The Ugly Hack

It turns out that Lame will tolerate an MP3 which consists of several appended MP3 files as its input. It complains but keeps processing when encountering the extra headers in the middle of such a file. So this solution with cat, a pipe, and Lame, worked:

cat 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 1.mp3 | lame – longmusic.mp3

A little while later, I had a 3 hour long, valid MP3 file.

Conference / User Group Member Photographs

Here is an idea I picked up at AYE and again at ETech; I mentioned it at the Ruby UG last month, and am writing it up here to encourage its use.

A common problem at growing and changing groups of people (such as user group members, conference attendees who see each other rarely, etc.), is remembering names and faces. To make it easier for everyone to remember everyone else, it’s very helpful to have a photograph of each person on the group’s web site. Sorting and labelling such photos can be a lot of work, so here is an approach to get it done with very little work.

Ingredients:

  • Digital camera
  • Blank paper
  • Wide-line markers

At a meeting, during a break or at the start/end, all of the willing attendees write their name on a blank sheet of paper, in letters at least 1 inch high. Then they hold up the paper and have a photo taken with the digital camera. A low-res, head-and-shoulders shot works best.

Afterwards, transfer the images to a page on the group’s web site. By taking them low-res, no scaling will be needed. Because each person is holding a sheet of paper with their name, noone needs to sift through a pile of images, associating photos with names. Simply post them as-is, unsorted. This works amply well enough for at least several dozen people, and probably well enough for up to 100.

St. Louis Code Camp – I’ll be speaking, you can too

My friend Brian Button is still looking for St. Louis Code Camp speakers; if you’ve thought about giving a user group talk, this is a great way to get your feet wet.  A “code camp” is an informal event, likely with a lot of group discussion.  You don’t need a large topic or a long speech.  Simply pick a technical topic (perhaps something from your work, or something you’ve been wanting to learn on the side), prepare a few examples and notes, and sign up.

I’m already signed up to give a talk on Lua.

Refactoring to Patterns? No, learn the primitives.

Last night at XPSTL, John Sextro gave a talk on the “Move Embellishment to Decorator” refactoring as described in Joshua Kerievsky’s Refactoring to Patterns book. I greatly enjoyed and benefitted from the original Design Patterns book (from the Gang of Four) which was already old (published 1994) when I heard about it and bought it in 1998. (By the way, when I looked it up on Amazon to put in the link above, Amazon reminded me that: “You purchased this item on February 18, 1998”.)

I enjoyed John’s talk, and I hope he does more of them.  He hit a few rough spots along the way (the usually excellent IntelliJ IDEA IDE failed mysteriously, for example), but worked through it and reached the target of composable decorators. The rough spots led to some interesting diversions also.
I’m not sold on the “refactoring to patterns” idea though; it seems like a distraction from a more important goal: to gain deep experience and understanding of how to use the underlying “primitives” (encapsulation, abstraction, polymorpism, low coupling, high cohesion, etc.). Once you grasp the primitives, the design patterns are useful mainly as a tool for talking about how something works – in other words, write good code, then perhaps notice that it follows one of the “patterns”, if you find that helpful in explaining how the code works.

Several times at XPSTL, we’ve had lengthy conversations about how to choose whether to use “Strategy” or “Command” or “Decorator” or …. I’m not convinced that these conversations are helpful. My answer is that it is silly to look for a list of rules in choosing which pattern to use. Read the patterns, use them to learn good ways to use and combine the primitives. Then do that in your code:

  • Notice that you can benefit from polymorphism, and use it.
  • Notice that you can split a class in to two separately cohesive parts, and do so.
  • Notice that you could get composability by replacing inheritance with aggregation, and so so.

You’ll end up with the right “Pattern” – and you probably won’t care.

Intel Integrated Performance Primitives Bewilderment

I’ve been evaluating Intel’s Integrated Performance Primitives, a set of libraries for low-level signal processing and image manipulation. These appear to be very well engineered at the lowest levels, but the packaging is a mess, as I’ll describe below. The point of these libraries is that they use the various enhanced CPU instruction sets (MMX, SSE, SSE2, SSE3, are there more?) of the various Intel processors. Such enhancement are very useful if you are processing a lot of images in a hurry… such as in a video stream.

Big gripe: nearly all of the CPU enhancements that these libraries use, are also available on AMD hardware; but being Intel libraries, these don’t auto-detect and use these things on AMD CPUs. Therefore an off-the-shelf, simple benchmark will show AMD radically slower. Apparently with the right tweaks, the AMD stuff can be used well also, and works fine. Now I can see why Intel would do this, but it has the side effect of making them look scared. Looking scared is not good marketing. I believe it would have been better for them to support AMD seamlessly, and let the benchmarks show that the Intel stuff is a bit faster… if indeed that is what they show.

Now for the bewildering deployment model: These libraries arrive as a large pile of DLLs, with most of them in several flavors, once for each processor family.  But rather than putting the “which logic to load” code in a DLL, that code is supplied only in a .lib file – so it’s quite amenable to use from C code, and much less so from non-C code.

Next bewilderment: To the “rest of us”, getting a machine with a 64-bit processor (Intel EM64T, AMD64) is no big deal, it’s just another CPU family.  But to Intel the IPP developers, it’s a separate product version, with a separate way to invoke it, and no clear example code to show how to auto-load the right libraries (and of the 32-bit, any of the 64-bit) as needed depending on CPU type.  Apparently it did not occur to them that this could have been seamless.

Next bewilderment: Until a few years ago, the “Intel JPEG Library” was a popular, free, and very fast JPEG compression / decompression library.  It’s not free anymore, it is now part of IPP.  I have no objection to that, IPP is a very good value at $200 per developer.  The bewilderment is that rather than supply the working “Intel JPEG Library” as part of IPP, they instead supply it only as example code.  The deveoper than compiles the sample code to get an IGL DLL that runs on top of IPP, with all of the deployment issues mentioned above.  Again, this could have been trivial, and would meet a lot of developers’ needs, but is instead a mess of accidental complexity.  IPP provides a large number of low-level functions useful to write a JPEG compressor / decompression, but a lot of us (myself and this fellow, for example) don’t want to learn the inner working of JPEG; it seems as though IPP could benefit greatly from some higher level API calls.