The Strange Loop crew only recorded video in the two largest venues (out of six), so I made a “bootleg” video of my talk, for your viewing pleasure:

The video/audio sync starts out OK, but drifts off by a second or so by the end. The drift is minor, so it is reasonably viewable all the way through. If you don’t have Flash installed (and thus don’t see the video above), you can download the video (x264); it plays well on most platforms (including an iPad).

The slides are online below, and are also available for PDF download.

Video Hackery

This video recording was an experiment: instead of hiring a video crew (with professional equipment), or using my DV camcorder, I instead used the video recording capability of my family’s consumer-grade Canon digicam. This device has three advantages over my DV camcorder:

1. No tape machinery; no motors; thus no motor noise in the audio.
2. Smaller size, easier to carry in and out.
3. Directly produces a video file, easily copied off its SD card.

As you can see from the results, the video quality is adequate but not great. Still, I learned that if I want to increase the quality of recording, the first step is not to use a better camera or lens! Rather, it is to bring (or persuade the venue to provide) better light. For good video results, the key is light the speaker well, without shining any extra light on the projector screen. With that in place, a better camera make sense.

The audio was a different story. Like nearly all consumer video cameras (and digicams with video), mine doesn’t have an external audio input, so the audio (from ~12 feet away) was awful. As a backup I had used a $75 audio recorder and a $30 lapel microphone, and that audio is very good, certainly worth using instead of the video recording audio track.

To combine the video in file A with the audio in file B, I used the ffmpeg invocation below. I reached the time adjustments below in just a few iterations of trial and error, by watching the drafts in VLC, using “f” and “g” to experiment with the audio/video time sync. I also trimmed off a bit of the bottom of the video, and used “mp4creator.exe -optimize”, which I had handy on a Windows machine, to prepare the file for progressive download viewing.

ffmpeg -y -ss 34.0 -i WS_10001.WMA -ss 34.0 -itsoffset -12.05 -i MVI_4285.AVI -shortest -t 8000 -vcodec libx264 -vpre normal -cropbottom 120 -b 400k -threads 2 -async 200 Cordes-2010-StrangeLoop-Lua.m4v

The remaining bits of technology are FlowPlayer, a WordPress FlowPlayer plugin, and Amazon’s CloudFront CDN.

Warning: Sausage-making Discussion Below

The following has nothing to do with the content of the video.

This is an x.264 video, shown here initially with a Flash-only player (FV WordPress Flowplayer). The video file is served by Amazon’s CloudFront CDN, which is trivial to sign up for without any sales process or minimum service cost. Later I’ll replace this Flash-only widget with one that offers HTML5 video (for iPad use, in particular), when I find one that works sufficiently well.

That’s the easy part, though. Getting this video to you here was an adventure, and not in a good way. Three recordings were made of the talk:

1. We hired a professional videographer to record the talk. When I say professional, I mean it only in the most literal way, i.e. the videographer charged money. They showed up with a nice camera and a wireless lapel mic… but somehow produced a broken video recording (the first 10-15 minutes were intermittant video noise). In addition, the mic gain was turned up way too high and thus the audio is awful.
2. Dave Blankenship recorded the talk on his consumer camcoder; he was not paid for this, and he did a much better job. This video is usable all the way through, but arrived in an oddball format produced mostly by some models of JVC camcorders. The audio was not so hot, because he used the mic built in to the camcorder from the back of the room.
3. I recorded the audio using a $5 microphone plugged in to an iPod Nano, sitting on a table at the front of the room. It’s a bit noisy, but with a few minutes of work with Audacity (Noise Removal and Normalization), the results are much better than either video attempt.

Armed with this, I set about to somehow combine the video from #2 with the audio from #3. I send emails describing this mess to several videographers I found on Craigslist. Most of them didn’t reply at all. I finally got a cost estimate from one, of many hundreds of dollars or more, and not much assurance of results.

Now I’m willing to spend some money to get good results, but spending it without confidence of results is less appealing; so I set about trying myself instead.

First, I cleaned the audio in Audacity as mentioned above.

Second, I watched the video and listened to the audio a few times, to get the approximate starting timestamp in each one of the moment the talk actually started; each recording had a different amount of lead-in time

Third, I grabbed ffmpeg, the swiss army knife of command line video and audio processing. After reading a dozen web pages of ffmpeg advice, and a number of experiments (with short -t settings, to quickly see how well it works without waiting to transcode the whole thing), I ended up with this command to produce the encoded video:

ffmpeg -y -ss 40.0 -i Recording-3-audio-only-clean.wav -ss 95 -i Recording-2-video-ok-audio-bad.mod -shortest -t 18000 -vcodec libx264 -vpre normal -b 700k -threads 2 Cordes-2010-SaaS.m4v


I then noticed that the MacPorts installation of ffmpeg omits the important qt-faststart tool, and found this helpful version of qt-faststart and used it instead, on my Mac; later I switched to a Linux machine with an ffmpeg install including qt-faststart. Without the faststart step, the metadata in the m4v file is arranged in a way that prevent progressive/streaming play-while-downloading.

The results are good but not great:

• The video has some motion/interlace artifacts; these were present in the original recording, and I’m not aware offhand of what to do about them
• The video camera used rectangular pixels; the pixel aspect ratio is 3:2 while it is intended for display at 16:9. I wasn’t able (at least in 20 minutes of learning and experimentation) to get the 16:9 output working correctly, so if you grab the underlying m4v file you can see the aspect ratio a bit off in the shape of the clock on the wall, for example.
• The audio-video sync is adequate (and plenty good enough to follow along) but not perfect. Clearly using the audio track on a video recording is much better than putting them together in post-processing.
• The audio is not as good as if I used a lav or headset mic, though I think it’s quite remarkably good for a $5 mic plugged in to iPod.
• I’ve no idea if ffmpeg complies with any of the relevant copyrights/patents/whatever in video production, though it seems hopefully safe to use for a one-off non-commercial video like this. (Normally I use Apple’s iMovie for my videos, and I assume Apple has taken care of such things.)

A few morals of this story:

• Get some powerful tools, and learn how to use them.
• Be willing to pay for professional work, but be skeptical. Just because you pay, doesn’t mean it will be quality work.
• Have a plan B. If I had assumed that at least one of the two videos would get decent audio, and skipped my own audio recording, I’d not have been able to deliver the acceptable audio here. If Dave had assumed that my professional videographer would produce results, and turned off his camera, we’d have no video here at all.

As usual, the vimeo page offers it for HTML5, non-Flash platforms like the iPad.

Back in 2004 I co-founded Mobile Workforce Management, a vertical market SaaS firm. For the first 6+ months, I was the entire development team, while my co-founder was the entire analysis, support, and customer happiness department. Over the course of a few years, we hired developers, a very-senior developer / leader / general manager, support staff, and more. In spite of these hires, as of 2007 I was still in the loop for numerous critical processes that had to happen every day or week to keep the doors open – not a great situation.

Around that time I was inspired to take a month-long family vacation, far longer than any past vacation. My family made arrangements to spend 3 weeks in a house by the beach, 1000 miles away, in the summer of 2008; these arrangements must be made far in advance, as such houses tend to fill up. I’d be away for approximately an entire month, allowing for travel time and stops along the way.

With that hard date in hand, my notions of ironing out the business processes “someday” were swept aside, and I set about tracking, automating, documenting, and delegating any of the work that involved me and had to happen at least monthly.

• accounting / bookkeeping / payroll
• production sysadmin
• development sysadmin
• system monitoring
• management processes
• customer relationship processes
• vendor relationships
• design and code reviews
• much more

It took months of hard work (by myself and others) to build up our company ability to handle all of these things well in my absence. As of the vacation date, all of this was set up to run smoothly either entirely without me, or with a tiny bit of remote input from me.

This worked, in fact it worked so well that our customers didn’t even notice my absence.

Though I didn’t know it at the time, the work I did then to increase our organizational process maturity was a turning point in the life if the business, enabling its eventual sale. Before that work, I’d have been a bit embarrassed to say “organizational process maturity” in public. Afterward, I have lived (rather than just learned about and talked about) the notions of working on-rather-than-in a business, of building a business with a life separate from that of its owners.

In retrospect I’m calling that trip a Strategic Vacation – a vacation taken both for its own value, and to drive the accomplishment other critical goals. If your business needs you every single day, that’s a problem. Create some pressure on yourself to solve it, by scheduling a strategic vacation, then go make it happen.

I know a little Prolog, which I learned in college – just enough to be dangerous. Armed with that, and some vigorous just-in-time online learning, I used Prolog in a production system a few years ago, with great results. There are two stories about that woven together here; one about the technical reasons for choosing this particular tool, and the other about the business payoff for taking a road less travelled.

In 2004 (or so) I was working on a project for an Oasis Digital customer on a client/server application with SQL Server behind it. This application worked (and still works) very well for the customer, who remains quite happy with it. This is the kind of project where there is an endless series of enhancement and additions, some of them to attack a problem-of-the-moment and some of them to enrich and strengthen the overall application capabilities.

The customer approached us with a very unusual feature request – pardon my generic description here; I don’t want to accidentally reveal any of their business secrets. The feature was described to us declaratively, in terms of a few rules and a bunch of examples of those rules. The wrinkle is that these were not “forward” rules (if X, do Y). Rather, these rules describe scenarios, such that if those scenarios happen, then something else should happen. Moreover, the rules were are on complex transitive/recursive relationships, the sort of thing that SQL is not well suited for.

An initial analysis found that we would need to implement a complex depth/breadth search algorithm either in the client application or in SQL. This wasn’t a straightforward graph search, though, rather that part was just the tip of the iceberg. I’m not afraid of algorithmic programming, Oasis Digital is emphatically not an “OnClick-only” programming shop, so I dug in. After spending a couple of days attacking the problem this way, I concluded that this would be a substantial block of work, at least several person-months to get it working correctly and efficiently. That’s not a lot in the grand scheme of things, but for this particular customer, this would use up their reasonable-but-not-lavish budget for months, even ignoring their other feature needs.

We set this problem aside for a few days, and upon more though I realized that:

• this would be a simple problem to describe in Prolog
• the Prolog runtime would then solve the problem
• the Prolog runtime would be responsible for doing it correctly and efficiently, i.e. our customer would not foot the bill to achieve those things.

We proceeded with the Prolog approach.

….

It actually took one day of work to get it working, integrated, and into testing, then a few hours a few weeks later to deploy it.

The implementation mechanism is pretty rough:

• The rules (the fixed portions of the Prolog solution) are expressed in a prolog source file, a page or two in length.
• A batch process runs every N minutes, on a server with spare capacity for this purpose.
• The batch process executes a set of SQL queries (in stored procs), returning a total of tens or hundreds of thousands of rows of data. SQL is used to format that query output as Prolog terms. These stored procs are executed using SQL Server BCP, making it trivial to save the results in files.
• The batch process run a Prolog interpreter, passing the data and rules (both are code, both are data) as input. This takes up to a few minutes.
• The Prolog rules are set up, with considerable hackery, to emit the output data we needed in the form of CSV data. This output is directed to a file.
• SQL Server BCP imports this output data back in to the production SQL Server database.
• The result of the computation is thus available in SQL tables for the application to use.

This batch process is not an optimal design, but it has the advantage of being quick to implement, and robust in operation. The cycle time is very small compared to the business processes being controlled, so practically speaking it is 95% as good as a continuous calculation mechanism, at much less cost.

There are some great lessons here:

• Declarative >>> Imperative. This is among the most important and broad guidelines to follow in system design.
• Thinking Matters. We cut the cost/time of this implementation by 90% or more, not by coding more quickly, but by thinking more clearly. I am a fan of TDD and incremental design, but you’re quite unlikely to ever make it from a handcoded solution to this simply-add-Prolog solution that way.
• The Right Tool for the Job. Learn a lot of them, don’t be the person who only has a hammer.
• A big problem is a big opportunity. It is quite possible that another firm would not have been able to deliver the functionality our customer needed at a cost they could afford. This problem was an opportunity to win, both for us and for our customer.

That’s all for now; it’s time for LessConf.

This is a Vimeo video, embedded as Flash. You can go over to Vimeo to watch it with HTML5-video devices like an iPad. I expect to offer better quality and more options as I learn more.

estimate ≠ promise

Background: Unpredictability

Around half of my software development and leadership experience has been in enterprise/internal software development, and that is the world I am thinking of as I write this.

Software development, like other endeavors with a significant creative component, is inherently unpredictable. With a good, deep understanding of the development process, you can build a model of the probability distribution of the cost, effort, and elapsed time for software development work. In the large, on average this can be made to work: small and large projects can succeed, within some broad range of predictability.

But notice also how common it is for large complex projects (in software and elsewhere) to be farcically over budget and late. This is not (usually) due to incompetence or fraud. It is because of the inherent unpredictability of the work.

If someone claims that they (or you) can exactly predict software development work, they are:

• mistaken, or
• lying, or
• padding their estimates very substantially, stating a date or cost much later/higher than a neutral median estimate would suggest

As a customer of software development services, and as a provider of such services, I don’t want any of those things.

Blame the Service Trades

I place some of the blame for the confusion of these two wildly different things, on the service trades: it is common for auto repair shops, roof installers, landscapers, and the like to offer something they call an estimate, but which is actually a fixed price quote (a promise).

Sadly, while there are plenty of common good synonyms for promise, there aren’t many for estimate. We’re stuck with using the word estimate, and explaining that we really mean it as defined. Perhaps in a few more decades we will lose the word entirely, much like the word “literally” has come to mean its antonym, “figuratively”, which renders it mostly useless.

Estimates

An estimate is an approximation of an unknown quantity. Typically in the world of software development, it is a prediction of the cost, working hours, or delivery date of a project or milestone. It is not in any sense a commitment, any more than estimating the temperature outside this afternoon is a commitment.

As the word implies, a customer reasonably expects the actual value to vary somewhat, in either direction, from the estimate. In fact, if an estimate turns out exactly match the actual result, there is a good chance the books have been cooked. Moreover, if the work is completed at-or-before the estimate most of the time, this means the estimates (on average) are too high.

An estimate “costs” nothing, other than the time/effort required to create it, which consists of analyzing the work at hand, decomposing it in to parts, and comparing those parts to past work.

Promises

A promise, also called a commitment, deadline, quote, fixed price, etc. is a different beast entirely.

With a promise in hand, a customer should expect with high confidence that the actual value (for cost, hours of work, delivery date) will be less than (before), or equal to, the promised value/date.

Be wary of a promise easily made and freely given: it probably doesn’t mean anything at all. A wise customer (and I aim to count myself in this category) should expect that a casually made commitment will probably be broken; not because the maker is morally defective, but simply because meeting a commitment for complex work requires considerable effort and thought. Without evidence that happens, it would be mere wishful thinking to expect the results delivered as promised.

Likewise, keeping promises often has a cost. If the work underway gets behind the schedule needed to meet the promise, something will have to give:

• Other work may fall behind, as time and effort are diverted to meet the promise.
• Weekends, evenings, and overtime work may be needed. These might appear free, but are not.
• Staff may need to be reassigned, or added
• Additional hardware and software may be needed.

These risks cost real money; thus a wise promise-maker will find that, on average, it costs more to promise feature X by date D, than to delivery feature X by date D without such a promise.

Estimates are Cheaper, so Prefer Estimates

At Oasis Digital, we provide many estimates, but few promises. Most of the time, an estimate is what our customers need; and we can provide at estimate with very little cost. Typically we estimate reasonably well:

• small features usually arrive with a day or two (plus or minus) of the estimated delivery date (and likewise for cost)
• medium items within a week or two of the estimate, likewise
• large items (major new features or subsystems with complex interdependencoes) within a month or so, likewise

The key here is is that with good estimate, commitments (promises) aren’t needed very often, and therefore the cost of promises can be avoided.

But Learn How to Promise Well, Also

Yet occasionally, a customer needs a commitment, most often because a software version needs to be available to match an important business event with a fixed date, such as a presentation, a legal filing, etc. I’ll follow up later (no promise or estimate, as to when) with thoughts on:

• how to credibly make promises (as a service provider)
• how to evaluate promises (as a customer)

Here is a more explanatory video from the same project:

I’m curious how difficult it was to create that behavior. I’m confident that our customers (we build a variety of video applications for our customers at Oasis Digital) would be thrilled by it.

It is amazing to me that Guy, whose is something of a legend in language design, and who thinks so clearly about what makes a good language, was also key in designing Java. Java has been extremely slow to grow in the sense described in this talk, because for many years Sun resisted such growth. Only the rise of C# and the growing popularity of dynamic languages generated enough pressure to get Java unstuck… and in the last few year Java has become somewhat growable in the sense Guy describes.

I found it interesting in light of my own comments on YouTube’s 45 TB a while back.

Here are my notes from his talk, a mix of what he said and my commentary:

In the summer of 2006, they grew from 30 million pages per day to 100 million pages per day, in a 4 month period. (Wow! In most organizations, it takes nearly 4 months to pick out, order, install, and set up a few servers.)

YouTube uses Apache for FastCGI serving. (I wonder if things would have been easier for them had they chosen nginx, which is apparently wonderful for FastCGI and less problematic than Lighttpd)

YouTube is coded mostly in Python. Why? “Development speed critical”.

They use psyco, Python -> C compiler, and also C extensions, for performance critical work.

They use Lighttpd for serving the video itself, for a big improvement over Apache.

Each video hosted by a “mini cluster”, which is a set of machine with the same content. This is a simple way to provide headroom (slack), so that a machine can be taken down for maintenance (or can fail) without affecting users. It also provides a form of backup.

The most popular videos are on a CDN (Content Distribution Network) – they use external CDNs and well as Google’s CDN. Requests to their own machines are therefore tail-heavy (in the “Long Tail” sense), because the head codes to the CDN instead.

Because of the tail-heavy load, random disks seeks are especially important (perhaps more important than caching?).

YouTube uses simple, cheap, commodity Hardware. The more expensive the hardware, the more expensive everything else gets (support, etc.). Maintenance is mostly done with rsync, SSH, other simple, common tools.
The fun is not over: Cuong showed a recent email titled “3 days of video storage left”. There is constant work to keep up with the growth.

Thumbnails turn out to be surprisingly hard to serve efficiently. Because there, on average, 4 thumbnails per video and many thumbnails per pages, the overall number of thumbnails per second is enormous. They use a separate group of machines to serve thumbnails, with extensive caching and OS tuning specific to this load.

YouTube was bit by a “too many files in one dir” limit: at one point they could accept no more uploads (!!) because of this. The first fix was the usual one: split the files across many directories, and switch to another file system better suited for many small files.

Cuong joked about “The Windows approach of scaling: restart everything”

Lighttpd turned out to be poor for serving the thumbnails, because its main loop is a bottleneck to load files from disk; they addressed this by modifying Lighttpd to add worker threads to read from disk. This was good but still not good enough, with one thumbnail per file, because the enormous number of files was terribly slow to work with (imagine tarring up many million files).

Their new solution for thumbnails is to use Google’s BigTable, which provides high performance for a large number of rows, fault tolerance, caching, etc. This is a nice (and rare?) example of actual synergy in an acquisition.

YouTube uses MySQL to store metadata. Early on they hit a Linux kernel issue which prioritized the page cache higher than app data, it swapped out the app data, totally overwhelming the system. They recovered from this by removing the swap partition (while live!). This worked.

YouTube uses Memcached.

To scale out the database, they first used MySQL replication. Like everyone else that goes down this path, they eventually reach a point where replicating the writes to all the DBs, uses up all the capacity of the slaves. They also hit a issue with threading and replication, which they worked around with a very clever “cache primer thread” working a second or so ahead of the replication thread, prefetching the data it would need.

As the replicate-one-DB approach faltered, they resorted to various desperate measures, such as splitting the video watching in to a separate set of replicas, intentionally allowing the non-video-serving parts of YouTube to perform badly so as to focus on serving videos.

Their initial MySQL DB server configuration had 10 disks in a RAID10. This does not work very well, because the DB/OS can’t take advantage of the multiple disks in parallel. They moved to a set of RAID1s, appended together. In my experience, this is better, but still not great. An approach that usually works even better is to intentionally split different data on to different RAIDs: for example, a RAID for the OS / application, a RAID for the DB logs, one or more RAIDs for the DB table (uses “tablespaces” to get your #1 busiest table on separate spindles from your #2 busiest table), one or more RAID for index, etc. Big-iron Oracle installation sometimes take this approach to extremes; the same thing can be done with free DBs on free OSs also.

In spite of all these effort, they reached a point where replication of one large DB was no longer able to keep up. Like everyone else, they figured out that the solution database partitioning in to “shards”. This spread reads and writes in to many different databases (on different servers) that are not all running each other’s writes. The result is a large performance boost, better cache locality, etc. YouTube reduced their total DB hardware by 30% in the process.

It is important to divide users across shards by a controllable lookup mechanism, not only by a hash of the username/ID/whatever, so that you can rebalance shards incrementally.

An interesting DMCA issue: YouTube complies with takedown requests; but sometimes the videos are cached way out on the “edge” of the network (their caches, and other people’s caches), so its hard to get a video to disappear globally right away. This sometimes angers content owners.

Early on, YouTube leased their hardware.

in my code I am building video frames, 720x480x24bit. I have in mind generating a large number of these, as long as a full DVD worth at 30fps, then using ffmpeg (followed by dvdauthor) to encode them in to MPEG2 for DVD usage.

There were a few replies, but no definitive answer. With considerable experimentation, I got it to work. It turns out that (as far as I can tell) ffmpeg does not have the ability to accept piped in RGB frames. It will however accept piped in data in its “yuv4mpegpipe” format. With some searching and reading I found that this is roughly akin to the format of raw DV video; each frame consists of a header something like this:

YUV4MPEG2 W%d H%d F%d:%d Ip A0:0 C420mpeg2 XYSCSS=420MPEG2

… then an LF character, then data for the the Y, U, and V “planes”. The Y data is full resolution, while the U and Y are half-resolution (this is called “420″ in the video world). These planes are uncompressed, one byte per pixel. All of my past work with computer video (going back to Commodore 64s and Apple IIs) has arranged all of the bits for each pixel within a few bytes of each other; this format (with all the Y data for the whole frame, then all the U data, then all the V data) is starkly different.

The essential problem remaining was how to convert RGB to YUV. Happily there are plenty of online references for this. Unhappily there are few fast implementations, and a naive implementation will be very slow. I solved this problem by finding and hiring an expert in low-level data processing with MMX, SSE2, etc. instructions. (I am not in a position to publish that code here.)

In retrospect, though, there are routines included in Intel’s “Integrated Performance Primitives” library which perform this transformation in a highly optimized way. IPP is a bargain: for only a few hundred dollars you get a wealth of high optimized ready-to-use library routines for signal processing.

The ffmpeg piping solution consists, therefore, of:

1. A module which generated frames in RGB format, to contain whatever contents your application requires.
2. A module to very quickly convert these to YUV in yuv4mpegpipe format (write your own, or use routines in IPP, for the RGB->YUV420 part).
3. Pipe this data stream to ffmpeg with stdin; ffmpeg is invoked something like this: ffmpeg -y -f yuv4mpegpipe -i – -i audio.mp3 -target ntsc-dvd -aspect 4:3 foo.mpg

By using a multicore CPU and threads, this whole process can be made to happen in real time or better (i.e., one second of “wall clock” processing time, for one second of finished MPEG2 video). The resulting MPEG2 file can be used with a DVD authoring application to produce a ready-to-burn DVD ISO image.

Update: the data format above is published here as part of the mjpegtools man pages.

It appears to be “A-list” event, meaning that I’m not likely to make the cut anytime soon.

Fortunately, many TED sessions are available to the rest of us online: TEDTalks. Here are some that I bookmarked to watch again; the download links are for QuickTime videos. Many of these are also on Google Video or elsewhere.

Burt Rutan

Barry Schwartz

Malcom Gladwell

Steven Levitt

Nicholas Negroponte

Hans Rosling (One word summary: Wow.)

Ken Robinson – Do schools today kill creativity?

Dan Gilbert – The (misguided) pursuit of happiness

The Secret of Happiness, according to Dan Gilbert:

1. Accrue wealth, power, and prestige. Then lose it.
2. Spend as much of your life in prison as you possibly can.
3. Make someone else really, really rich.
4. Never ever join the Beatles.

(watch the video for the story behind this)

Linus Torvalds explains distributed source control in general, and git in particular, at Google

Here are some of his points, paraphrased. They might not be clear without watching the video.

• He hates CVS
• He hates SVN too, since it’s “CVS done right”, because you can’t get anywhere good from there.
• If you need a commercial tool, BitKeeper is the one you should use.
• Distributed source control is much more important than which tool you choose
• He looked at a lot of alternatives, and immediately tossed anything not distributed, slow, or which does not guarantee that what goes in, comes out [using secure hashes]
• He liked Monotone, but it was too slow
• With a distributed tool, no single place is vital to your data
• Centralized does not scale to Linux-kernel sized projects
• Distributed tools work offline, with full history
• Branching is not an esoteric, rare event – everyone branches all the time every time they write a line of code – but most tools don’t understand this, they only understand branching as a Very Big Deal.
• Distributed tools serve everyone; everyone has “commit” access to their branches. Everyone has all of the tool’s features, rather than only a handful of those with special access.
• Of course noone else will necessarily adopt your changes; using a tool in which every developer has the full feature set, does not imply anarchy.
• Merging works like security: as a network of trust
• Two systems worth looking at: git and Mercurial. The others with good features are too slow [for large projects].
• Lots of people are using git for their own work (to handle merges, for example), inside companies where the main tools is SVN. git can pull changes from (and thus stay in sync with) SVN.
• Git is now much easier to use than CVS or other common tools
• Git makes it easier to merge than other tools. So the problem of doing more merging, is not much of a problem, not something you need to fear.
• Distributed tools are much faster because they don’t have to go over the network very often. Not talking to a server, is tremendously faster, than talking to even a high end server over a fast network.
• SVN working directories and repositories are quite large, git equivalents are much smaller
• The repository (=project) is the unit of checkout / commit / etc.; don’t put them all in to on repository. The separate repositories can share underlying storage.
• Performance is not secondary. It affect everything you do, it affects how you use an application. Faster operations = merging not a big deal = more, smaller changes.
• SVN “makes branching really cheap”. Unfortunately, “merging in SVN is a complete disaster”. “It is incredible how stupid these people are”
• Distributed source control is mostly inherently safer: no single point of failure, secure hashes to protect from even intentional malicious users.
• “I would never trust Google to maintain my source code for me” – with git (and other distributed systems) the whole history is in many places, nearly impossible to lose it.

My own observations:

There are important differences between source control tools. I have heard it said that these are all “just tools” which don’t matter, you simply use whatever the local management felt like buying. That is wrong: making better tool choices will make your project better (cheaper, faster, more fun, etc.), making worse tool choices will make your project worse (more expensive, slower, painful, higher turnover, etc.)

Distributed tools make the “network of trust” more explicit, and thus easier to reason about.

I think there is an impression out there that distributed tools don’t accomodate the level of control that “enterprise” shops often specify in their processes, over how code is managed. This is a needless worry; it is still quite possible to set up any desired process and controls for the the official repositories. The difference is that you can do that, without the collateral damage of taking away features from individual developers.

Git sounds like the leading choice on Linux, but at the moment it is not well supported on Windows, and I don’t see any signs of that changing anytime soon. I will continue working with bzr, SVK, etc.

There is widespread, but mostly invisible demand for better source control. Over the next few years, the hegemony of the legacy (centralized) design will be lessened greatly. One early adopter is Mozilla, which is switching (or has switched?) to Mercurial. Of course, many projects and companies (especially large companies) will hang on for years to come, because of interia and widespread tool integration.

Several of the distributed tools (SVK, bzr, git, probably more) have the ability to pull changes from other systems, including traditional centralized systems like SVN. These make it possible to use a modern, powerful tool for your own work, even when you are working on a project where someone else has chosen such a traditional system for the master repository.

Update: Mark commented that he doesn’t feel like he has really commited a changeset, until it is on a remote repository. I agree. However, this is trivial: with any of the tools you can push your change from your local repository to a remote one (perhaps one for you alone), with one command. With some of them (bzr, at least) you can configure this to happen automatically, so it is zero extra commands. This does not negate the benefits of a local repository, because you can still work offline, and all read operations still happen locally and far more quickly.

Update: I didn’t mention Mercurial initially, but I should have (thanks, Alex). It is another strong contender, and Mozilla recently chose it. Its feature set is similar to git’s, but with more eager support for Windows.

Update: There is another discussion of Linus’s talk over on Codice Software’s blog, which was linked on Slashdot. Since Codice sells a source control product, the Slashdot coverage is a great piece of free publicity.

Update: Mark Shuttleworth pointed out below that bzr is much faster than it used to be, and that it has top-notch renaming support; which he explains in more detail on in a post. Bzr was already on my own short-list, I am using it for a small project here at Oasis Digital.

Update: I’ve started gathering notes about git, bzr, hg, etc., for several upcoming posts (and possibly a talk or two). If you’re interested in these, subscribe to the feed using the links in the upper right of my home page.

Why does JavaScript matter?

1) It is ubiquitous now (in nearly every browser, in Flash as ActionScript, etc.)

2) It is likely to be the default choice for building scriptable Java applications, due to the Rhino JS interpreter “in the box” in Java 1.6

Update: These videos are more conveniently all on one page here.

1. No travel.
2. No need to syncronize schedules.
3. The receiving person can view the recording repeatedly, at their convenience.
4. Customers and develoeprs who join the project team later, can look at old recordings to catch up.

It turns out that I am unusually picky about the quality of such recordings; I’ve written up some technical notes on how to get good results, and posted them: HighQualityScreenRecordings.pdf.

A few highlights:

• A reasonably fast computer can both run application and record screen video at the same time; but if you will be recording the use of an application that generates a lot of disk activity, you must save the video to separate hard drive (internal, external, network server, etc.) from the hard drive you are running your OS and applications from. (For applications that generate little disk activity, a single system hard drive works fine.)

• Use a headset-style microphone, and record in a quiet place: close the door, turn off the music, etc.

• Adjust your audio levels well. Please. This is the most common and most annoying problem with screencast and podcast recordings I find.

• Bytes are cheap; use a sufficiently large window and sufficiently high bitrate.

Many more details are in the PDF linked above.

Here are just a few that’s I’ve enjoyed recently, there are many more worth watching.

How Debian (Ubuntu) packages work

Seth Godin (marketing guru)

Mary Poppendieck (“Lean Software Development” author) – Competing on the basis of speed

A new Way to look at Networking – Fascinating

The Mercurial distributed source control system

Added later:

Fission is the New Fire

Jessica Livingston, talking about “Founders at Work”

Still later:

Subscribe to this feed to find out about each talk

It’s a phone. It’s a PDA. It’s an iPod. It’s a widescreen video iPod. It has zero physical buttons, rather the whole front is a multi-touch-screen. I’ll leave the rest of the raving to the many other sites doing a great job of that.

The real innovation of this new device is the OS. Apple has an answer to PalmOS and Windows Mobile (CE): run their real desktop/server OSX, with Unix inside, on the phone. Today’s handhelds have much more computing power and storage than desktop PCs of a decade ago, and there are enormous benefits to running a real, common OS on such a device. I’d been saying since I bought my first Palm (a Handspring, actually) than within a decade the handheld OS’s would go away.

Apple has gone first. How soon will Microsoft follow? Will Palm and RIM make to the new era at all?

(Update: Yes, the title of this post is slightly in jest. I’m serious about real OSs in handheld devices, and the iPhone looks fantastic, but Apple is very unlikely to dominate the phone market in the way the iPod dominates the tiny-media-player market.)