Kyle Cordes – Page 39

Heading to AYE

I’m heading to the AYE conference shortly, as I did last year. My strategy then was to attend many Weinberg sessions, which has turned out to be wise in retrospect, as Jerry won’t be there this year, he is home recovering from illness. I am eager to see how the rest of the AYE team divvies up the load of his sessions.

If you work in software development and haven’t read Jerry’s books (The Secrets of Consulting, Becoming a Technical Leader, The Psychology of Computer Programming, etc.), I heartily recommend doing so.

Software Development Team = Marketplace?

Some months ago, when deciding how to structure a software development project team, I lamented (offline) that what I’d really like the project to resemble is a marketplace – a community of developers all eager to find their niche. As the market clears, each developer would end up doing what they do best, maximizing their comparative advantage while also maximizing overall productivity. We know this works well on a national scale (capitalism), but in a sense, many companies are (internally) little islands of non-capitalism. By saying a team would work this way, I mean internally, in terms of how the various parts of each project are divided up.

Today I read that TopCoder, in addition to their well known programming contests, does something very close to this:

“TopCoder assesses a client’s needs, breaks the project into 30 or so components, and opens the design and development work to a series of online competitions. The coder with the best finished product wins “prize money,” as does the runner-up, which usually amounts to a few thousand dollars. The small pieces are “sewn” together, usually by TopCoder, and delivered to the client.”

Two issues come to mind though, one negative and one positive:

In the kind of complex “enterprise” projects we sometimes take on at Oasis Digital, binding together the pieces in to a coherent solution, tends to be a major aspect of the work. Sewing together disparate pieces could easily dominate the schedule.

A bunch of separate, talented developers, committing to building a system in this fashion, are likely to produce a highly modular system. It is hard to overstate the benefits of modularity in large projects.

Update… in my next 2 minutes of browsing feeds, I cames across a relevant post about another “industry” far outside my expertise.

Fresh Windows Install, Ouch

A few hours ago I started with a fresh Windows XP Home install for a computer for my family.

I’m still going. I have lost count of how many times I have had to reboot the machine then get a new round of updates to install. It is ludicrous.

Modern Linux distributions are far superior to this. I recently installed Ubuntu on a several machines; the installer downloaded updates during the install process, so the machine was fully updated, on the first boot.

Dreamhost Out, TextDrive In

You might have noticed that this site is much faster than it used to be. The reason? I moved it from DreamHost to TextDrive.

TextDrive costs more, its “control panel” is not as good as DreamHost’s, and its bandwidth/storage limits are lower. But my site is far faster, hasn’t had any downtime or email downtime since the switch (during which DreamHost had an email outage), and TextDrive support responds much sooner.

I have a few TextDrive nitpicks though: there is no built in web-stats system (I’ll need to install one), and they apparently don’t have a backup system working at the moment (!). I’ve set up a nightly rsync to a machine here for backup purposes, but I sure hope they don’t intend this as a long term situation.

Update: Jason at Joyent/Textdrive noticed this post, and added a comment that the backup problem is long fixed.

Update: A complaint without data risks sounding like a whine. So I’ll add some data. Today I noticed that sites I still have on DreamHost are slow. Why? let’s look:

$ date
Fri Sep 8 15:56:14 PDT 2006
$ uptime
15:56:18 up 5:31, 3 users, load average: 103.41, 95.54, 181.86

Update: Some months later, TextDrive has turned out to have approximately at much, or more, downtime as DreamHost. It’s still fast when it’s up, and the TextDrive guys are helpful, friendly, and responsive. But the shared hosting they offer has frequent downtime.

Joel, you have got to be kidding

Joel seems to “play it safe” … then goes off the deep end of irony in his final paragraph:

“FogBugz is written in Wasabi, a very advanced, functional-programming dialect of Basic with closures and lambdas and Rails-like active records that can be compiled down to VBScript, JavaScript, PHP4 or PHP5. Wasabi is a private, in-house language written by one of our best developers that is optimized specifically for developing FogBugz; the Wasabi compiler itself is written in C#.”

Fortunately DHH saved me some minutes of typing about it, with a scathing commentary.

Over at Oasis Digital we use both common tools (.NET, Java, PHP, C, Delphi, etc.) and more unusual ones (Lua, Prolog, Ruby, sorry no Lisp yet), so I believe that puts us in the DHH and Paul Graham camp: If you want to win, you must be willing to do something different from the pack… such as, in an extreme case, creating your own language optimized for the task at hand, whether in the form of Lisp macros or a C# compiler for Wasabi.

YouTube’s 45 Terabytes… no big deal?

Over at the Wall Street Journal and Micro Persuasion and Computers.net and a bunch of other places, a big deal is being made of the YouTube’s estimated 45 Terabytes worth of video. It is “about 5,000 home computers’ worth”. Ouch, 45 Terabytes! Wow!

Or maybe not… consider the mathematics.

45 TB really isn’t all that much data. I’ll assume that each video is stored on 6 hard drives across their systems, for reliability and greater bandwidth, for a total of ~300 TB of hard drives. A 300 GB hard drive costs under $200, and ~1000 will be needed, so this is about $200,000 worth of hard drives, which is not a big deal for a major venture-funded firm like YouTube. (Side note – if you need a lot of read-mostly disk bandwidth, you are much better off buying several cheap hard drives with the same data on them, than one expensive hard drive. It’s not even close.)

The 1000 hard drives might be spread across 250 servers. If their systems is build in the economically smart way (the Google way – lots of commodity servers), each server could cost as little as $3000. Those servers could likely serve the traffic also, as well as (at a lower priority) do any needed video transcoding of newly uploaded video. After all, it is mostly static content, and it’s likely that a small fraction of the videos are a large fraction of the views, so the popular ones stay in RAM cache. Adding other machines for various purposes, network hardware, etc., a YouTube-scale app/storage cluster might cost as little as $2 million, of which a relative small portion (above) is hard drives.

Of course I’ve totally skipped the question of paying for the bandwidth (and power), which must be staggeringly expensive.