source-control – Page 3

Upcoming talk: Intro to Distributed Source Control

Where: SLUUG (though my talk is not listed on the site yet)

When: October 10th, meeting starts at 6:30 PM

I’ll introduce distributed source control tools:

A short tour of the basic use of git, bzr, and hg (Mercurial)
Thoughts on why you’d want to use a distributed source control tool at all, vs. a centralized system like SVN or CVS.
Some differences between these tools (and a few others), with thoughts on how to choose

In response to a question below about slides… most likely there will be no slides. Rather there will be a handout (which will be posted on my site).

Update: The handout notes are here. Sorry, no audio/video/transcript this time.

svnmerge, a tool to manage SVN merges

We use SVN on a project with a lot of small branches, i.e. a branch for almost every non-trivial feature. This is not a particularly pleasant want to use SVN, but it meets another important need for our project: code review on the way in to the trunk (as a “gate”), rather than code review for code already in trunk (“drive-by” code review).

Today on the XPSTL mailing list, Mike Jorgensen pointed me to svnmerge:

“svnmerge.py is a tool for automatic branch management. It allows branch maintainers to merge changes from and to their branch very easily, and automatically records which changes were already merged. This allows displaying an always updated list of changes yet to be merged, and totally prevents merge mistakes (such as merging the same change twice).”

svnmerge looks rough, but should still be a big improvement of SVN alone. It’s a ways short of what’s in the box with git, bzr, etc., but is also a much smaller step for a team using SVN.

Speaking of merging, Mark Shuttlework recently argued merging is the key to software developer collaboration. To me, this is obviously true, and not only for open-source projects, but for closed-source projects also. If it sounds untrue to you, then you and I are probably thinking of different meanings of “merge”.

synsync: another way to remotely backup / svnadmin dump an SVN repository

Last month, I described an approach using SVK to remotely clone and then “svnadmin dump” an SVN repository. It turns out that there is an easier way “in the box” in SVN 1.4: the svnsync tool. Bob Ippolito describes how to do it, here are the minimal steps:

$ MYREPO=/home/me/someplace (do not use ~username, use /home/username)
$ svnadmin create $MYREPO
$ echo “#!/bin/sh” >$MYREPO/hooks/pre-revprop-change
$ chmod +x $MYREPO/hooks/pre-revprop-change
$ svnsync init file://$MYREPO http://SVN.remote.url.goes.here/
$ svnsync sync file://$MYREPO

… then repeat the “svnsync sync” to sync down the changes since last time, perhaps in a cron job.

Paul Querna describes some caveats, and Cheah Chu Yeow offers a longer walkthrough of a more “produciton-grade” svnsync setup.

Like the SVK approach, with synsync you get a local mirror of a remote repo, with no need for shell access to that repo – very helpful if the repo is hosted by, for example, a SVN hosting firm, SourceForge, or Google Code. This local mirror can be used to view project history without network access, to svnadmin dump for backups, to mirror efficiently in another source control tool (I’m doing this with git as a test now, for one of our projects), etc.

Linus Torvalds explains distributed source control

On several occasions over the last year, I’ve pointed out that distributed source control tools are dramatically better than centralized tools. It’s quite hard for me to explain why. This is probably because of sloppy and incomplete thinking on my part, but it doesn’t help that most of the audiences / people I’ve said this to, have never used a distributed tool. (I’ve been trying out SVK, bzr, git, etc.) Fortunately, I no longer need to stumble with attempts at explaining this; instead, the answer is to watch as:

Linus Torvalds explains distributed source control in general, and git in particular, at Google

Here are some of his points, paraphrased. They might not be clear without watching the video.

He hates CVS
He hates SVN too, since it’s “CVS done right”, because you can’t get anywhere good from there.
If you need a commercial tool, BitKeeper is the one you should use.
Distributed source control is much more important than which tool you choose
He looked at a lot of alternatives, and immediately tossed anything not distributed, slow, or which does not guarantee that what goes in, comes out [using secure hashes]
He liked Monotone, but it was too slow
With a distributed tool, no single place is vital to your data
Centralized does not scale to Linux-kernel sized projects
Distributed tools work offline, with full history
Branching is not an esoteric, rare event – everyone branches all the time every time they write a line of code – but most tools don’t understand this, they only understand branching as a Very Big Deal.
Distributed tools serve everyone; everyone has “commit” access to their branches. Everyone has all of the tool’s features, rather than only a handful of those with special access.
Of course noone else will necessarily adopt your changes; using a tool in which every developer has the full feature set, does not imply anarchy.
Merging works like security: as a network of trust
Two systems worth looking at: git and Mercurial. The others with good features are too slow [for large projects].
Lots of people are using git for their own work (to handle merges, for example), inside companies where the main tools is SVN. git can pull changes from (and thus stay in sync with) SVN.
Git is now much easier to use than CVS or other common tools
Git makes it easier to merge than other tools. So the problem of doing more merging, is not much of a problem, not something you need to fear.
Distributed tools are much faster because they don’t have to go over the network very often. Not talking to a server, is tremendously faster, than talking to even a high end server over a fast network.
SVN working directories and repositories are quite large, git equivalents are much smaller
The repository (=project) is the unit of checkout / commit / etc.; don’t put them all in to on repository. The separate repositories can share underlying storage.
Performance is not secondary. It affect everything you do, it affects how you use an application. Faster operations = merging not a big deal = more, smaller changes.
SVN “makes branching really cheap”. Unfortunately, “merging in SVN is a complete disaster”. “It is incredible how stupid these people are”
Distributed source control is mostly inherently safer: no single point of failure, secure hashes to protect from even intentional malicious users.
“I would never trust Google to maintain my source code for me” – with git (and other distributed systems) the whole history is in many places, nearly impossible to lose it.

My own observations:

There are important differences between source control tools. I have heard it said that these are all “just tools” which don’t matter, you simply use whatever the local management felt like buying. That is wrong: making better tool choices will make your project better (cheaper, faster, more fun, etc.), making worse tool choices will make your project worse (more expensive, slower, painful, higher turnover, etc.)

Distributed tools make the “network of trust” more explicit, and thus easier to reason about.

I think there is an impression out there that distributed tools don’t accomodate the level of control that “enterprise” shops often specify in their processes, over how code is managed. This is a needless worry; it is still quite possible to set up any desired process and controls for the the official repositories. The difference is that you can do that, without the collateral damage of taking away features from individual developers.

Git sounds like the leading choice on Linux, but at the moment it is not well supported on Windows, and I don’t see any signs of that changing anytime soon. I will continue working with bzr, SVK, etc.

There is widespread, but mostly invisible demand for better source control. Over the next few years, the hegemony of the legacy (centralized) design will be lessened greatly. One early adopter is Mozilla, which is switching (or has switched?) to Mercurial. Of course, many projects and companies (especially large companies) will hang on for years to come, because of interia and widespread tool integration.

Several of the distributed tools (SVK, bzr, git, probably more) have the ability to pull changes from other systems, including traditional centralized systems like SVN. These make it possible to use a modern, powerful tool for your own work, even when you are working on a project where someone else has chosen such a traditional system for the master repository.

Update: Mark commented that he doesn’t feel like he has really commited a changeset, until it is on a remote repository. I agree. However, this is trivial: with any of the tools you can push your change from your local repository to a remote one (perhaps one for you alone), with one command. With some of them (bzr, at least) you can configure this to happen automatically, so it is zero extra commands. This does not negate the benefits of a local repository, because you can still work offline, and all read operations still happen locally and far more quickly.

Update: I didn’t mention Mercurial initially, but I should have (thanks, Alex). It is another strong contender, and Mozilla recently chose it. Its feature set is similar to git’s, but with more eager support for Windows.

Update: There is another discussion of Linus’s talk over on Codice Software’s blog, which was linked on Slashdot. Since Codice sells a source control product, the Slashdot coverage is a great piece of free publicity.

Update: Mark Shuttleworth pointed out below that bzr is much faster than it used to be, and that it has top-notch renaming support; which he explains in more detail on in a post. Bzr was already on my own short-list, I am using it for a small project here at Oasis Digital.

Update: I’ve started gathering notes about git, bzr, hg, etc., for several upcoming posts (and possibly a talk or two). If you’re interested in these, subscribe to the feed using the links in the upper right of my home page.

Use SVK to remotely “svnadmin dump” an SVN repository

One of the nice things about SVN is how easy it is to carry the complete SVN history from one server to another: “svnadmin dump” produces a single (large) dump file with the complete history, then “svnadmin load” to recreate it on the new machine. However, for a handful of our projects we have an SVN repository hosted on a machine where we don’t have shell access to run svnadmin.

After considerable Google searching, I found that SVN itself offers no way to do this; svnadmin takes these parameters:

svnadmin dump path-to-repos

and only works on local repositories. I found some mailing list discussion about it, but unfortunately it was of this form:

user: “I’d like to do X”

developer: “You don’t need to do X, it works like Y”

Fortunately, this post from Thomas Mølhave explains how to use SVK to accomplish something pretty close (though not quite) to this. It was quite easy on my Ubuntu test machine:

apt-get install svk (if you don’t have SVK yet)

rm -rf ~/.svk (if you have SVK, but don’t care about your current stuff)

mv ~/.svk ~/old.svk (if you have SVK, get your current config out of the way)

then:

svk ls https://your.svn.URL/here/

svk will prompt you for all the bits of info needed to mirror the SVN repository to your local SVK storage, then list the top level files/directories. Next, take advantage of the fact that SVK uses SVN’s underlying storage mechanism, and dump that local SVK mirror, skipping revision 1 which contains SVK metadata:

svnadmin dump -r2:HEAD ~/.svk/local >something.dump

This dump can now be restored elsewhere with “svnadmin load”. Unfortunately the pathnames in it will be mangled, with the original SVN repository hostname and path prepended. For my purposes, this didn’t matter, as I only needed to make it available for occasional reference. You could of course use “svn mv” to clean it up.

On a broader note, SVK looks like a very fine distributed source control tool. On a single developer project I recently droppen SVN in favor of bzr, a distributed source control tool; one of these days I will choose such a tool for a larger project.

Update: I discuss another method for remote SVN backups, svnsync in a later post.

Faster TortoiseSVN

I’ve used SVN and the TortoiseSVN client for most projects recently. The combination works well (and contrary to my initial expectation, I’ve found a shell-integrated source control tool quite usable), but sometimes causes annoying slowdowns in Windows Explorer. But with the help of a post in this anonymous “Professional Blog”, a few minutes of configuration you can speed it up considerably. Read the post for all the details, but the most important bits are:

Use “include paths” and “exclude paths” to tell TortoiseSVN what areas of your hard drive potentially contain workspaces; it will then totally disregard other areas.
Trim your SVN repository – if you have a big pile of ancient tags/branches you don’t need, delete them, so that TortoiseSVN can’t possibly waste any time looking at them. Of course this is only useful advice for projects which don’t need all their historical tags and branches kept around.

In upcoming posts I will point out a better tool for using SVN in Eclipse, and a better approach to source control: distributed source control systems.