gitosis on Ubuntu 9.04 Jaunty

As of April 2009, the gitosis package in Ubuntu 9.04 Jaunty is broken; it fails with an error like so:

pkg_resources.DistributionNotFound: gitosis==0.2

There are quite a few pages and mailing list messages that mention this. I only found one with a good hint toward a solution, which was that it is also a known issue on Debian. Following that lead, I got it working by grabbing newer packages from Debian Unstable:

wget http://ftp.us.debian.org/debian/pool/main/p/python-support/python-support_1.0.2_all.deb
wget http://ftp.us.debian.org/debian/pool/main/g/gitosis/gitosis_0.2+20080825-14_all.deb
sudo dpkg -i python-support_1.0.2_all.deb
sudo dpkg -i gitosis_0.2+20080825-14_all.deb

Use this at your own risk; your mileage may vary.

Sometimes You Need to Compile It Yourself

I posted an earlier version of this on the Puppet mailing list recently; it seemed worth expanding here.

An Ideal World

In an ideal world, for each piece of Linux software I use, a very recent version would be “in the box” in the distribution package repositories, for every distro and distro release I use. The package versions would be updated promptly, and backported to the most recent N distro releases. This would be done in such a way as to avoid unexpected breakage, offering a combination of original (as of the distro release), bug-fix-only, and all-updates.

We don’t live in such a world.

The Real World

I’ve found that, with at least Debian, Ubuntu, Red Hat, and Mandrake, typically only the most popular packages are prone to prompt updates for new upstream versions. Backports to not-the-current-distro are even more rare, for various good reasons.

Therefore, when adopting something less widely used, especially if I need the same (current) package version for various distro versions, I’m resigned to having to either package it myself, or find someone “out there” offering updated packages.

Example #1: Puppet

As I write this, the current Puppet version is 0.24.8. It contains a lot of bug fixes and enhancements relative to even the earlier 0.24.x versions. It would probably be only a slight stretch to say that in the Puppet community,  versions before 0.24.x really aren’t recommended for current use at all. Yet the most recent versions offered in-the-box for the last few releases for Debian / Ubuntu are all old enough to be in that category.

Example #2: udpcast

Udpcast is an extremely useful tool, both for cloning systems en masse, and for totally unrelated uses like the one I describe here; yet the version in the very latest Ubuntu is from 2004, and the same is true for Debian.

Example #3: Zabbix

To get good results with Zabbix, it’s necessary to have approximately the same, approximately current versions, on all machines. The versions in various current and past Ubuntu and Debian releases / backports are not even close.

But Please, Use Your Package System

Please, I ask you… and if you work for me on one of my projects, I require of you… do not take any of this to mean you should ever type “sudo make install”.  It’s a nightmare to untangle a system with a mix of packaged and ad hoc compiled code in /usr/bin and friends. Always install using the package system: for a one-off (one machine, ever) install, simply use checkinstall, it takes only an extra minute or two and make it trivial to back the install out later. For a set of production systems. dpkg and RPM are your friends. They won’t bite. Get to know them.

Bar Camp St. Louis, a stealth micro-unconference?

Today I stumbled across the web evidence of Bar Camp St. Louis (Flickr stream, Facebook group) , a software / social networking flavored “unconference”. It occured a few months ago, on Dec. 13th, 2008.  I’m not sure I’d have attended, but I am surprised that went below the radar of any of the various user groups / mailing lists I follow.

DRBD on Ubuntu 8.04

A while back I wrote about setting up DRBD on Ubuntu.

It is considerably simpler now: you don’t need to build the module anymore, a reasonably recent version is already in the box. You should still read the full directions at the DRBD web site, but to get it going, you only need to configure it and start it; don’t download anything, don’t compile any modules kernels, etc.

Rather, the module is already there; you could load it manually like this:

modprobe drbd

But you really want the utils package:

apt-get install drbd8-utils

… which sets things up the rest of the way. Simply proceed with the configuration file, /etc/drbd.conf. If you want a nice clean config file, remove all the existing resource sections in the drbd.conf installed by the Ub package, and put in something like so:

resource kylesdrbd {
  protocol      C;

  startup { wfc-timeout 60; degr-wfc-timeout  120; }
  disk { on-io-error detach; }
  syncer {
  }
  on x1.kylecordes.com {
    device      /dev/drbd4;
    disk        /dev/sdb5;
    address     192.168.17.61:7791;
    meta-disk   /dev/sdb1[0];
  }
  on x2.kylecordes.com {
    device      /dev/drbd4;
    disk        /dev/sda5;
    address     192.168.17.62:7791;
    meta-disk   /dev/sda3[0];
  }
}

The hostnames here need to match the full, actual hostnames. I show my example disk configuration, you’ll need to do somethign locally appropriate, based on understanding the DRBD docs.

Also adjust the syncer section:

syncer {
rate 40M;  # the rate is in BYTES per second
}

If there was already a filesystem on the metadata partitions you’re trying to put under DRBD, you may need to clear it out:

dd if=/dev/zero of=/dev/sdb1 bs=1M count=1

Now you’re ready to fire it up; on both machines:

drbdadm create-md mwm
drbdadm attach mwm
drbdadm connect mwm

# see status:
cat /proc/drbd

You now are ready to make one primary; do this on only one machine of course:

drbdadm -- --overwrite-data-of-peer primary all

or

drbdadm -- --overwrite-data-of-peer primary kylesdrbd

In my case, I now see a resync starting in the /proc/drbd output.

You don’t need to wait for that to finish; go ahead and create a filesystem on /dev/drbd0.

It’s best to have a dedicated Gigabit-ethernet connection between the nodes; or for busy systems, a pair of bonded GigE. For testing though, running over your existing network is fine.

I found that an adjustment in the inittimeout setting helps to avoid long boot delays, if one of the two systems is down.

Of course I only covered DRBD here; typically in production you’d use it in conjuction with a failover/heartbeat mechanism, so that whatever resource you serve on the DRBD volume (database, NFS, etc.) cuts over to the other machine without intervention; there is plenty to read online about Linux high availability.

Move files into an existing directory structure, on Linux

I recently needed to move a large number of files (millions) in a deep directory structure, into another similar directory structure, “merging” the contents of each directory and creating any missing directories. This task is easily (though slowly) performed on Windows with Control-C Control-V in Explorer, but I could find no obvious way to do it on Linux.

There is quite a bit of discussion about this on the web, including:

  • Suggestions to do with with tar; this is a poor idea because it copies all the file data, taking an enormously long time.
  • Suggestions to do it with “mv -r”… but as far as I can tell, mv does not have a -r option.

After a little thought I came up with the script below. I’d love to have a Linux/Bash guru out there point out how awful it and and send me something better!

A critical feature for me is that it does not overwrite files; if a source file name/path overlaps a destinate file, the source file is left alone, untouched. This can be changed easily to overwrite instead: remove the [[-f]] test.

$ cat ~/bin/move_files_merge.sh
#!/bin/bash

# Move files from dir $1 to dir $2,
# merging in to existing dirs
# Call it like so:
# move_files_merge.sh /FROM/directory /TO/directory

# Lousy error handling:
# Exit if called with missing params.
[ "A" == "A${1}" ] && exit 1
[ "A" == "A${2}" ] && exit 1

echo finding all the source directories
cd $1
find . -type d -not -empty | sort | uniq >$2/dirlist.txt

echo making all the destination directories
cd $2
wc -l dirlist.txt
xargs --no-run-if-empty -a dirlist.txt mkdir -p
rm dirlist.txt

echo Moving the files
cd $1
# There is surely a better way to do this:
find . -type f -printf "[[ -f '$2/%p' ]] || mv '%p' '$2/%h'\n" | bash

echo removing empty source dirs
find $1 -depth -type d -empty -delete

echo done.