Webby – Client-side, static content management system

This afternoon I rebuilt OasisDigital.com using Webby, stripping out hand-coded HTML and replacing it with much more maintainable Markdown. The site looks about the same as before (which is to say, mediocre), but under the hood it is much easier to update. We intend to use this new ease, to move forward in improving it. There is a general principle here, which applies broading in software development also:

If you need to make a change, but that change is difficult / tedious / risky to make, first improve the underlying system that makes it so.
(OasisDigital.com is a static web site; we have dynamic contact (issue trackers, etc.) to automate our work together and with our customer, but that content is on another domain.)

Webby is a client-side, simple CMS for generating static web sites, written in Ruby. Why serve a static site (plain old files on a web server) in 2008?

  • It minimizes the moving parts, there is almost nothing to break or maintain.
  • It is very unlikely that any hosting issue will break a static site.
  • It is easy to serve a static site fast (though our current host, TextDrive, sometimes is not all that fast).
  • Security vulnerabilities are very unlikely, in the absence of any executable content.
  • The canonical content (in this case, mostly Markdown) is stored in plain text files, which we track, diff, and merge in git.

In an earlier foray in to Drupal, we found that Drupal has extensive and useful capabilities, as well as a vibrant community, but it also has many moving parts; too many, in my judgment, to make it a good solution for building an essentially static web site.

One URL per page

A few weeks ago I was working on a web site that wasn’t getting as much attention from Google as the site owner expected.  One (of many) issues was that for every “interesting” page (those that had link-worthy material), there were at least three URLs for that same content. Depending on how the user navigated the site, they’d get to a different URL.

I pointed out that this is a Bad Thing, since distributing the incoming links across those duplicates would decrease the Google-worthiness of each… but hadn’t had time to write up the detals.  Today I was thrilled to see this detailed writeup of the phenomenon and this one about how fix it.  I’ve done the same thing in the past, but never written it up.

If your site has this “feature”, and you would prefer it gets more traffic, fix it.

A/B Technique for Web Application Deployment

This description of my “A/B technique for web application deployment” was transcribed from audio, so it less tight, more verbose than my normal prose. I chose to post it in rough form, rather than leave it on the “back burner” until an unknown future date when I have time to rewrite it. I first explained this to a colleague around 1999, 8 years is long enough for an idea to wait.

The Problem / Context

At least a dozen times over the last decade, this scenario has come up at consulting client sites: you have a web application and you want to upgrade it with a new version. You could do so with a brute force cutover (stop the app, swap the code), but that’s not the scenario that I’m talking about. I’m talking about the upgrading to a new version, not quite compatible with the old one, without dropping current active users. For example, in the new version, you might have some different data that goes in the session. You might have some different pages, so you have some different URLs, you may be adding a new field so that once you put in this new version, there’ll be an additional field, and it stores that additional field in the session and in the database and the caching and so on. Yet you want the current users to keep working without interruption.


This technique is not language-specific – it applies equally in PHP, ASP, ASP.NET, Java servlets, JSP, CGI, etc.; with nearly any application or infrastructure.

Have the URL of your application, which I’ll call “/contact” here, be the URL of a proxy application (or “launching pad”). Then have two additional URLs for two specific instances of the actual application. For example, you might have “/contact” as your overall application URL and then “/contact/contactA” or “/contact/A” as one instance where you have that application installed.

At the “/contact” URL, install a simple proxy application, whose job is to take a newly arriving user, present them an intro/login screen, then redirect them to one of the specific instances of the application.

As a user I will point my browser to the “/contact” URL, and I bookmark that. I launch that, I see a login screen, I type in my name and password, I press the button to log in. The “/contact” launching pad redirects me to “/contact/A,”, an instance of real application. I’ll call the second instance “B”, perhaps at the URL “/contact/B”. In the normal steady state of the system, the user will be using A, they login to “/contact” and they end up in the “A” instance.

Then you want to do an upgrade, install a new version. Install the new version as “/contact/B.” Leave the existing application in place and working. (By the way, I’ve assumed you are using a technology where you can deploy and undeploy applications without bringing your web server down, but with mod_proxy you could make this work even without that capability) Deploy the new application version in a new and different path than what your current running users are on. Adjust some setting your proxy/launchpad application (perhaps as simple as a single line in a single config file). So, for example, you might install the new version as “/contact/B” and then you flip a switch (edit the config file) to make it so that new users that come to the “/contact” page don’t land in “/contact/A” anymore, they land in “/contact/B” as they login.

The current users already using the application in “/contact/A” stay there – they don’t know or care that you’ve deployed a new version. New users come in the come in the new version. So you want to have some sort of mechanism (likely provided by your application server if you’re using one, and not hard to build otherwise) for monitoring how many users are using each of these applications. So you might for example notice that you have 1000 users active on /contact/A. You deploy a new version as /contact/B and flip the switch. Then, depending on the usage characteristics of your application – over the next few minutes, next few hours, however it works out, the users, as they log out and log in and such, gradually all make it into the /B application. Some kind of maximum-login-time mechanism will ensure that this cutover happens in finite time.

Once the users have moved to /contact/B, you then declare it as your “current” version, and you take down /A, because no one’s using it and no one can get into it. So that next time you need to do an upgrade, you just do it in reverse – you deploy that new version as /contact/A, flip the switch back to make all new users’ logins land in the A… and again, after however many hours or minutes or whatever, you have all your users on the new version, and you can take down the old version.

You can easily implement with just the tools that already come with your Web application development system. You don’t need any kind of special hardware or special application server or HTTP server support. You don’t need any sort of special way of doing session affinity; you’re doing session affinity by simply handing out the URL of one of these other Web applications.


Someone might bookmark a page of your application. So let’s say that you had directed them to /contact/A, and they were on the page /contact/A/lists.jsp. When they return to this bookmark later (you do want to support bookmarking, right?), you don’t want them to land there; you don’t want them to end up in the A application if you’re currently using the B application. This is actually pretty easy to handle also. You simply use some settings on your Web server to do a redirect, with a few lines of configuration in .htaccess or analogous mechanims. So based on your setting of which one is current, you make it so that if someone comes into the application without having a referrer from inside the application, you just redirect them over to whatever the current instance is. And that takes a little bit of thought, but only a little, and you can make it seamlessly solve that problem of users’ bookmarks working in spite of you switching back and forth between two instances.


You might be deployed on a cluster. Perhaps you are using Websphere with 37 web servers. It turns out that this A/B approach works orthogonally to the clustering features of your Web application server. You could have the A application deployed across all 37 servers; you could deploy the B application, with a few clicks, across all 37 servers; you could flip that switch in some global way to kick people onto the B, and so on.

Override the launchpad for testing

You can permit users to enter a special URL to get to the “other side”. you could have some way of entering a URL that takes you past that launch application straight into the B side, so that you could click around, you could manually verify that the new B application works in the production environment before you flip the switch to make that the deployed production system. This is a very wise and useful type of testing to do, a great final stage of testing because the new code in actually in production. It’s obviously not a replacement for testing in a separate test environment, it’s an adjunct for even greater safety in deploment.


When a system is running in a steady state, its caches are fully populated with relevant data, so many requests can be answered with data from the cache (RAM). But when a system is freshly started, its caches are empty, so more requests require (slower) disk access, during the first few minutes of operations. This is sometimes called the “empty cache” problem, and is responsible for the poor performance sometimes seen in the first few minutes after a busy system is restarted.

The technique described here prevents this problem, because with it you avoid ever shutting down and restarting your whole Web application with your full user population on it. Instead, since the switch only brings newly-logging-in users to the new version – the new instance – you gradually have people start using it, so you never take a big hit all at once in terms of cache population.

Schema Changes

Hani asked, in a comment, about schema changes. A simple answer is that you won’t be able to make a transition like the one described here (where both the old and new code versions run in parallel for a while), if you make schema changes such that the old code no longer works. A more complex answer, which I have used with great results, is that this is a programmable computer and you are a programmer – with some effort, you can make the software tolerate both the old and new schemas. So the process works like this:

  1. Decide on the schema change, but don’t deploy it
  2. Modify your software to tolerate the old or new schema, whichever is present
  3. Deploy the new software, transition all users to it (as described above)
  4. Make the schema change; you may need to momentarily quiesce the software, but hopefully not kill user sessions

(There are a few tools out there to help with the schema-change-in-a-live-app problem. One of then is ChronicDB who wrote me to point this out.)

Of course this is a lot more work than just stopping the server, making your change, and restarting. Whether it’s worth it depends on your situation. If you have an overnight non-usage window, consider using it instead of the long path described here.

I hope this is helpful for someone out there. Comments are welcome.

Lots of TextDrive downtime

I apologize for the site downtime today, to my huge audience of loyal readers (er… both of you). It turns out that my new ISP, TextDrive, has been having a lot of downtime on the shared hosting machine my sites are on… frequent freezes/crashes, exacerbated by long delays in noticing and recovering. Apparently it is most likely some oddball problem with some particular site belonging to one of the many customers on that machine… but unclear which one. Hopefully they will locate the trouble soon and return to the high quality of service they were previously known for.

Update: Another 1+ hour downtime on Jan. 17.

Update: Another ~1 hour downtime on Jan. 25. There were some short downtimes in the last few days also.

Update: What a surprise, another ~1 hour downtime on Jan. 27

Update: Another short downtime on Jan. 28

Update: ~2 hour downtime on Jan. 29. Seeing a pattern yet?

Update: 2.5 hour downtime on Jan. 30. This time with a kernel update.

Update: TextDrive says they have a major update coming soon, that will fix all these issues; but no specific timetable.

Update: ~1.5 hours downtime in the early overnight hours of Feb. 4. Not mentioned on the issue tracker site.

Update: ~1 hour downtime on Feb. 7, during the morning part of the workday. Maybe didn’t really want customer after all.

Update: ~1.5 hour downtime on Feb. 8, during morning part of the workday.

Update: Ah, it seems the TextDrive crew is quite busy pushing the envelope for high-end Rails hosting (via Tim Bray). It is quite impressive, but it’s also disappointing – I’d rather they had spent that brainpower making their existing hosted sites stay up. Plus… a 5-10 minutes downtime while I was trying to post this update. TextDrive considers frequent short downtimes normal, for “Apache restarts”. On several other hosts where I have had shared hosting accounts, I have not experienced this at all; so it seems to be something about how TextDrive works.

Update: After a month of stability (aside from a few short downtimes every day for “apache restarts”), ~1.5 hours downtime on March 12.

Overwhelming blog spam, and Thunderbird vs. POPFile

Kylecordes.com has become popular with blog comment spammers recently; though because I have moderation on, so far only I have seen the spam (in the WordPress admin interface), it hasn’t reached the public site.

In the last week, the quantity has grown enormously, to the extent no longer practical to moderate manually. I’ve installed Akismet, and am eager to see how well it works.

On a related (spam) note, I’ve been very disappointed with the spam filter built in to Thunderbird; it’s real-world performance for me has been awful compared to POPFile. With the latter, I get vanishingly few false positives, and only a small handful of spam messages reach my inbox each day, out of many hundreds that arrive. WIth Thunderbird, even after many, many clicks of training (dutifully identifying both Spam and Not-Spam), it still misclassifies far more often.

Update a few weeks later: Akismet works very well – the comment spam problem is, for the moment, completely solved.