Rearranging files in SVN? Use git-svn instead.

From time to time I need to rearrange a set of files in a project, typically while revamping the file / directory layout of a set of source files. The most direct way to do so is by dragging the files around (using a Windows or Linux GUI), but this is quite tedious to do if the files are in a Subversion repo, because SVN does not understand the file moves and renames unless you use “svn mv” or (with TortoiseSVN) right-click-drag the files to new locations.

The answer to all this is simple: use git-svn for your SVN checkout, instead of the SVN command lines tools or Tortoise. You can readily Google for extensive git-svn information to get started.

The workflow with git-svn for file rearrangement is fairly simple, and very fast:

  • “git svn rebase” to get your local git repo up to date with your SVN repo.
  • make new directories, rearrange files, rename files, etc., using any tool you like. This is easy and painless because git (wisely) keeps all the metadata in a top-level .git directory. It is fast because you aren’t running any svn code (or any git code) while doing so.
  • use “git add”, or the git GUI, to add the newly moved / renamed files.
  • “git commit”
  • “git svn dcommit” to push the changes over to SVN.

git / git-svn looks at the content of files, notices (rather than you needing to tell it) that the content has been moved to different filenames in different directories, and generates SVN move/copy operations. It even notices nearly-identical (edited) files and handles them correctly.

One loose end is left hanging, which is that git-svn does not remove any now-empty or removed directories from SVN, a consequence of git not caring about directories. Therefore, if you need the empty directories removed from the SVN repo, use a normal SVN client (command line, Tortoise) to do so.

From the description of above, this does not initially sound like a big win; but with hundred of files to move around, the time for the git commands (instant, except for the first and last) is irrelevant, and the time/hassle saved is enormous.

Perhaps someday SVN itself will gain the ability to “just work” when you move or rename files / directories, rather than having to be told.

Need text? Hire a Writer

To help create the document I mentioned earlier about the merits of custom software development, I hired a subcontract writer. The only typing I did was a bit at the start and perhaps a half-hour of editing at the end; all the rest of my input was in the form of spoken-word audio recordings, which I find fast and easy to create.

A critical factor in making this worthwhile is the categorical difference between a writer and a transcriber; I’ve used transcription services with excellent results, but a transcribed talk is far different in form and polish, than purposefully written text. The process was roughly like so:

  • I posted an ad on Craiglist to find a technical writer
  • I wrote short notes/outline of the topic
  • I recorded an audio ramble of my ideas for the content
  • Writer sent me draft #1, organizing my ramble in to a coherent form
  • I recorded audio feedback
  • Writer sent me draft #2
  • I recorded audio feedback
  • Writer sent me draft #3
  • I made a bunch of edits, and posted it

This worked out well, in several ways. The first, minor payoff is that it took a bit less time than just writing it myself. In retrospect it was only a minor time savings; if I had just sat down and wrote it all myself in one go, I could have finished in a handful of hours.

The second payoff was much larger: although I could have theoretically done it all myself, most likely I would not have done so yet. My personal threshold to get started and keep the process moving was much lower. Engaging assistance transformed an idea for what could happen, in to a process that did happen.

The third payoff is that this process is another useful tool in my toolbox; there is a big difference between know that I could use such a tool, and having actually done so.

Thus, I consider the experiment a success, and will almost certainly use the same process on more projects.

Buy vs. Build

A while back Joel Spolsky wrote about “Five Worlds” of software development. Over at Oasis Digital and elsewhere, I’ve been living in two of them:

  • Custom, internal software that we develop as consultants.
  • Commercial software that we develop at our expense, then sell licenses.

As a result, I’ve been thinking a lot about the relative advantage of each, from the points of view of software developers and customers. A portion of these thoughts, those on the merits of custom development from the customer perspective, are in an article (PDF) on the Oasis Digital web site.

There is a story behind how that document was created, and I’ll tell it in another post.

Start with a Prototype

I have often given the advice to start each project with an end to end working first draft, also known as an evolutionary prototype, and sometimes I have even followed this advice myself. My own projects have been most successful when following it, such as in three projects that come to mind:

  1. A 6+ year project, with a team of developers. In month 3 it was deployed to production, meeting real needs.
  2. A 4+ year project, with a team of developers. In month 2 it was deployed to production, meeting real needs.
  3. A 10+ year project, initially just myself but later with a small team of developers. In month 1 it was deployed to production, meeting real needs.

On the other hand, most failed projects that I am aware of had been slated from the beginning to not deliver a working system until late in the schedule, and ended up not delivering at all.

The traditional software development plan for a typical enterprise forms-database-rules-reports application looks like this:

  • Planning and design
  • Implement schema
  • Implement infrastructure
  • Implement admin/setup features
  • Implement core features (transactional data)
  • Implement reports

This waterfall model is a great way to think about software. Think through your requirements in depth at the beginning, and write them down in a coherent form. However, this is generally a bad manner (specifically, a very poor order) in which to write software.

Thanks to the agility movement, many teams are now using iterative, incremental approaches. Unfortunately, it is common nonetheless to lose some of the benefits of agility by laying out a set of releases with features/work in approximately the traditional order.

The solution, espoused by the Extreme Programming movement but rarely executed, is to start at the very beginning of a project, with building an end to end, first draft, working prototype system. This then grows slowly over the life of the project in to the finished product (i.e., it is not a throwaway prototype). We take this approach at Oasis Digital, and generally require it of our subcontractors also.

Development proceeds in a way that we create a working, end-to-end system with minimal features, very early in the work. For example, on a typical forms-database-rules-reports project, after a few days of work, we have:

  • A little bit of schema, just enough to support a small subset of desired features
  • A few screens, with very few features
  • A report or two, again with very limited features
  • Placeholders for some other screens and report, to allow the beginnings of the desired final application flow to appear

Then as the work proceeds, every week (or less) we can deliver a working system with a few more features present.

But this is impossible on my project!

The question is not whether this approach is possible; rather it is to what extent it can be done on each project, and what can be done to make it possible? The primary tool is imagination, in selecting a small subset of the desired feature set which plausibly works end-to-end.

For example, perhaps you are building a CRM (customer relationship management) system, one with thousands of features, 200 database tables, 450 reports, 85 screens, etc. How could you build a working system in the first week? Start by reducing the system to be its bare essence of companies, contacts, and interaction history.

A second example is a time tracking system with elaborate rules, reports, approval mechanisms. For the first draft, discard most of that and create only the ability to add a user (forgot edits and deletes, forget most of the fields), to enter hours for that user (forget approcals), to see a screen with those hours (with the simplest possible layout), and export a data file with those hours (any data file format, forget XML for now).

A third example is a trouble ticketing system. For the first week, make a way to add a ticket, a way to add notes to a ticket, to see a list of tickets, and to close a ticket. This could readily all occur on one screen, with only a few buttons. Its schema could consist of three tables: user, ticket, note.
A fourth example is a system which will write checks. In the very first week of the project, make it write a check. Dummy-out as much as needed to make it achieve this core goal.

Just Do It

Certainly there are plenty of problem domains where this advise doesn’t readily fit. But don’t be too eager to make that claim. By finding a way to deliver a working system early, we eliminate the risk that the project will never deliver anything, and we make progress and value visible (continuously) to whoever is paying for the project.

On the Job Training

I recently heard from a developer who worked on a project here for about one year, several years back. I won’t name or quote him directly; the essence of his message was:

  • He learned a lot during that year.
  • This learning was a turning point in his career.
  • What he learned has proved lucrative.

Of course it feels great to receive a message like this, but beyond that, it got me thinking about on-the-job training at Oasis Digital. We don’t have a formal training process. Rather, opportunities (and pressure) to learn and improve are woven in to our daily work. Described in coarse terms, a typical workflow for a developer (“You”) starting out on our projects is:

  • You write some code (or other work product).
  • Tech lead and other developers (sometimes including me) complain about it. This takes the form ideally of constructive criticism.
  • You improve the code.
  • More criticism.
  • You improve the code more, resulting in code that essentially works.
  • Tech lead revamps and refactors it mercilessly, explaining what was changed and why. Note though, that at this stage the changes do not require justification.
  • Tech lead sends developer a list of things to further improve.
  • You improve the code.
  • The code is now good enough to go in to the project.

It turns out that this process is a very good learning experience, if you are willing to take feedback. It can be hard to hear feedback, especially if its essence is that that your work product is not all that great. Yet the computer is merciless, it does not care about your feelings. The end customer does not care about your feelings either; both care only about correctness and quality.

I started out writing bad code. Having written many lines of such code, I now, somewhat regularly, write good code… though I am certain there are plenty of better developers out there who could find much fault in it. Writing bad code and accepting criticism of it is on the path to writing good code. Is it controversial to observe that a lot of software is quite bad? If so, that’s fine; I mostly agree with Steve Yegge’s observation:

By way of setting context for today’s blog, I’ll summarize everything I’ve ever written to date in one sentence: I think most software is crap.

By accepting that most software is bad, you can get on a path to make yours better. It turns out that working on our projects gets you a lot of coaching and training; and instead of paying for these things, you get paid. For the right kind of developer, I think this is an important part of our value proposition and compensation.