commentary – Page 10

Joel, you have got to be kidding

Joel seems to “play it safe” … then goes off the deep end of irony in his final paragraph:

“FogBugz is written in Wasabi, a very advanced, functional-programming dialect of Basic with closures and lambdas and Rails-like active records that can be compiled down to VBScript, JavaScript, PHP4 or PHP5. Wasabi is a private, in-house language written by one of our best developers that is optimized specifically for developing FogBugz; the Wasabi compiler itself is written in C#.”

Fortunately DHH saved me some minutes of typing about it, with a scathing commentary.

Over at Oasis Digital we use both common tools (.NET, Java, PHP, C, Delphi, etc.) and more unusual ones (Lua, Prolog, Ruby, sorry no Lisp yet), so I believe that puts us in the DHH and Paul Graham camp: If you want to win, you must be willing to do something different from the pack… such as, in an extreme case, creating your own language optimized for the task at hand, whether in the form of Lisp macros or a C# compiler for Wasabi.

YouTube’s 45 Terabytes… no big deal?

Over at the Wall Street Journal and Micro Persuasion and Computers.net and a bunch of other places, a big deal is being made of the YouTube’s estimated 45 Terabytes worth of video. It is “about 5,000 home computers’ worth”. Ouch, 45 Terabytes! Wow!

Or maybe not… consider the mathematics.

45 TB really isn’t all that much data. I’ll assume that each video is stored on 6 hard drives across their systems, for reliability and greater bandwidth, for a total of ~300 TB of hard drives. A 300 GB hard drive costs under $200, and ~1000 will be needed, so this is about $200,000 worth of hard drives, which is not a big deal for a major venture-funded firm like YouTube. (Side note – if you need a lot of read-mostly disk bandwidth, you are much better off buying several cheap hard drives with the same data on them, than one expensive hard drive. It’s not even close.)

The 1000 hard drives might be spread across 250 servers. If their systems is build in the economically smart way (the Google way – lots of commodity servers), each server could cost as little as $3000. Those servers could likely serve the traffic also, as well as (at a lower priority) do any needed video transcoding of newly uploaded video. After all, it is mostly static content, and it’s likely that a small fraction of the videos are a large fraction of the views, so the popular ones stay in RAM cache. Adding other machines for various purposes, network hardware, etc., a YouTube-scale app/storage cluster might cost as little as $2 million, of which a relative small portion (above) is hard drives.

Of course I’ve totally skipped the question of paying for the bandwidth (and power), which must be staggeringly expensive.

Aiming for Mainstream

Over on defmacro today, a new article appeared: defmacro – Why Exotic Languages Are Not Mainstream in which the author laments that while there appear to be various choices to use Haskell on Windows, it turns out that all of them are, in some way, not ready for prime time… or even for effective hobbiest use.

I’ve noticed this myself, in my last few forays in to esoteric languages: the illusion of plenty of choices, runs in the the reality of no good choices. This is not a universal problem; I’ve had great results with Ruby, Python, and Lua, all of which are to some extent esoteric. The thing that those languages have in common is that there is at least one (and generally, only one) robust, production grade implementation with a community actively supporting it.

If you want to see your favorite language gain acceptance, spend your time creating / maintaining / vigorously supporting a production-ready implementation.

DreamHost Disasters

You might have noticed that kylecordes.com and my other sites have had several long downtimes recently. I certainly have. The cause, in every case, has been DreamHost, the formerly excellent hostiing firm I use. DreamHost has excellent features and configurability, but two large problems:

Lots of downtime incidents
Slow performance of DB-based web apps, including WordPress

I’ve started the process of moving a site off of DreamHost, so that at worst I won’t have everything down at once. After some live evaluation, I’ll decide whether to move the rest.

I am much impressed, though, by a long explatation of the recent problems over on the Dreamhost Blog:
DreamHost Blog » Anatomy of a(n ongoing) Disaster..

This contains far more detail and honesty than I am accustomed to receiving from vendors, and I appreciate it very much. I still recommend DreamHost, if you are looking for a bargain, rich features, and can tolerate (until they demonstrate otherwise) some “issues”.

Long Rails Stack Trace

I was looking at the RailsConf schedule, and saw the Rails web app that runs it fail with this longish stack trace. A few Rubyists sitting nearby commented that the application was not well configured, that usually Rails apps are configured in deployment to store such traces to a log, rather than spit them at the user. Their mistake is my gain though, as it provided fodder for this post.

This stack trace is considerably shorter than a typical Java web app stack trace. But it is not so good either, and it is an ominous sign that excessive complexity is making its way in to Rails.

How To Do Deployment (Dave Thomas RailsConf Keynote)

I just attended Dave Thomas’s keynote at RailsConf. He had many interesting things to say, most notably that 60%+ of his Java-centric conference-circuit friends, mostly people who have written books on Java, speak regularly on Java, and have lots of experience as Java “architects”, are now making a living with Ruby / Rails.

Dave talked about 3 key areas where Rails should improve. The one I want to focus on is deployment. Rails has Capistrano, a slick tools for deployment automation. Capistrano is great, but it misses the boat in one key way: it is push-based. It assumes a world where the developer sets up, controls, and performs deployment to production servers. That is of course the case at the start of a small operation, but it doesn’t scale; in large organization, or even in small growing ones, sooner or later there is staff dedicated to production deployment and monitoring, separate from development.

One way, to me right way, to handle deployment is push / pull:

For the “push”, a developer uploads a new version to a storage site somewhere. It travels in the form of an archive file (ZIP, tgz, whatever) which contains all of the needed artifacts, as well as metadata about the required libraries, pre- and post-install steps, etc. for the new version. It has a unique name; it lands at a URL. Normal, off the shelf technology (FTP, web servers, etc.) can be used to serve and secure this storage, implement policies about who can upload new versions, etc. This upload happens with some simple one-line command (or as part of an auto-build); it does not actually put a new version in production, but only makes it available for pulling.

Then, on a production machine, someone with software installation rights on that machine (administrated by the normal tools – not by any special feature of the deployment mechanism) runs a command (something like “deploy staging http://ourserver/project1/project1-build-3453.tgz”) which downloads the new build, runs various sanity checks, deploys it to a staging area, and makes it available to try out. Once the staging is verified, a similar command brings it in to live production. These builds refer as needed to configuration data on the deployment machines (such as production DB access credentials); the build archives are generic, not specific to any particular production machine.

The main idea behind this is that each person has the rights they need for the work they need to do, and these rights don’t need any special help or support from the deployment mechanism. Developers don’t need any special rights on deployment machines, nor to deployers need any special rights on development machine

I wrote this in the context of a Rails / Ruby talk, but it’s at least as relevant in Java and Delphi world; in fact at Oasis Digital we need something like this on a Java project and a Delphi project right now.