As usual, I’ll zoom through some slides, and concentrate instead on the code. For some background on Lua, you may want to watch the video of my 20-minute Lua talk from last year’s Strange Loop.

Update: slides are available here.

Why is Node worthy of attention?

• JavaScript is a Next Big Language, it is everywhere. It is probably the most widely used programming language ever.
• I know a few things about asyncronous server programming, having done a lot of it in 1990s IVR software; it is very well suited to serving a large user population.
• Node is accumulating libraries at an impressive rate, indicating momentum.
• There are significant advantages in developing a whole application stack (server and client code) in a single language. For example, this makes code and business logic sharing works across tiers. Using Node, a JavaScript-HTML tool, a JavaScript-CSS tool, JSON, etc., it is possible to develop a complex web application using only JavaScript.

Node is not all unicorns and roses though.; my most serious misgiving about it is that it does not (yet) have a great strategy to make straightforward use of many-core servers. We’ll have to see how that develops over time.

Node Knockout

The team at Fortnight Labs is putting together Node Knockout, a 48-hour Node programming contest. I am a fan of such contests. I’ve offered to help out by being a judge, and I’ve also signed up Oasis Digital as a sponsor.

As a judge, I can’t be on a team; I’ve like to see a team or two form here in St. Louis, though.

I know a little Prolog, which I learned in college – just enough to be dangerous. Armed with that, and some vigorous just-in-time online learning, I used Prolog in a production system a few years ago, with great results. There are two stories about that woven together here; one about the technical reasons for choosing this particular tool, and the other about the business payoff for taking a road less travelled.

In 2004 (or so) I was working on a project for an Oasis Digital customer on a client/server application with SQL Server behind it. This application worked (and still works) very well for the customer, who remains quite happy with it. This is the kind of project where there is an endless series of enhancement and additions, some of them to attack a problem-of-the-moment and some of them to enrich and strengthen the overall application capabilities.

The customer approached us with a very unusual feature request – pardon my generic description here; I don’t want to accidentally reveal any of their business secrets. The feature was described to us declaratively, in terms of a few rules and a bunch of examples of those rules. The wrinkle is that these were not “forward” rules (if X, do Y). Rather, these rules describe scenarios, such that if those scenarios happen, then something else should happen. Moreover, the rules were are on complex transitive/recursive relationships, the sort of thing that SQL is not well suited for.

An initial analysis found that we would need to implement a complex depth/breadth search algorithm either in the client application or in SQL. This wasn’t a straightforward graph search, though, rather that part was just the tip of the iceberg. I’m not afraid of algorithmic programming, Oasis Digital is emphatically not an “OnClick-only” programming shop, so I dug in. After spending a couple of days attacking the problem this way, I concluded that this would be a substantial block of work, at least several person-months to get it working correctly and efficiently. That’s not a lot in the grand scheme of things, but for this particular customer, this would use up their reasonable-but-not-lavish budget for months, even ignoring their other feature needs.

We set this problem aside for a few days, and upon more though I realized that:

• this would be a simple problem to describe in Prolog
• the Prolog runtime would then solve the problem
• the Prolog runtime would be responsible for doing it correctly and efficiently, i.e. our customer would not foot the bill to achieve those things.

We proceeded with the Prolog approach.

….

It actually took one day of work to get it working, integrated, and into testing, then a few hours a few weeks later to deploy it.

The implementation mechanism is pretty rough:

• The rules (the fixed portions of the Prolog solution) are expressed in a prolog source file, a page or two in length.
• A batch process runs every N minutes, on a server with spare capacity for this purpose.
• The batch process executes a set of SQL queries (in stored procs), returning a total of tens or hundreds of thousands of rows of data. SQL is used to format that query output as Prolog terms. These stored procs are executed using SQL Server BCP, making it trivial to save the results in files.
• The batch process run a Prolog interpreter, passing the data and rules (both are code, both are data) as input. This takes up to a few minutes.
• The Prolog rules are set up, with considerable hackery, to emit the output data we needed in the form of CSV data. This output is directed to a file.
• SQL Server BCP imports this output data back in to the production SQL Server database.
• The result of the computation is thus available in SQL tables for the application to use.

This batch process is not an optimal design, but it has the advantage of being quick to implement, and robust in operation. The cycle time is very small compared to the business processes being controlled, so practically speaking it is 95% as good as a continuous calculation mechanism, at much less cost.

There are some great lessons here:

• Declarative >>> Imperative. This is among the most important and broad guidelines to follow in system design.
• Thinking Matters. We cut the cost/time of this implementation by 90% or more, not by coding more quickly, but by thinking more clearly. I am a fan of TDD and incremental design, but you’re quite unlikely to ever make it from a handcoded solution to this simply-add-Prolog solution that way.
• The Right Tool for the Job. Learn a lot of them, don’t be the person who only has a hammer.
• A big problem is a big opportunity. It is quite possible that another firm would not have been able to deliver the functionality our customer needed at a cost they could afford. This problem was an opportunity to win, both for us and for our customer.

That’s all for now; it’s time for LessConf.

This is a Vimeo video, embedded as Flash. You can go over to Vimeo to watch it with HTML5-video devices like an iPad. I expect to offer better quality and more options as I learn more.

estimate ≠ promise

Background: Unpredictability

Around half of my software development and leadership experience has been in enterprise/internal software development, and that is the world I am thinking of as I write this.

Software development, like other endeavors with a significant creative component, is inherently unpredictable. With a good, deep understanding of the development process, you can build a model of the probability distribution of the cost, effort, and elapsed time for software development work. In the large, on average this can be made to work: small and large projects can succeed, within some broad range of predictability.

But notice also how common it is for large complex projects (in software and elsewhere) to be farcically over budget and late. This is not (usually) due to incompetence or fraud. It is because of the inherent unpredictability of the work.

If someone claims that they (or you) can exactly predict software development work, they are:

• mistaken, or
• lying, or
• padding their estimates very substantially, stating a date or cost much later/higher than a neutral median estimate would suggest

As a customer of software development services, and as a provider of such services, I don’t want any of those things.

Blame the Service Trades

I place some of the blame for the confusion of these two wildly different things, on the service trades: it is common for auto repair shops, roof installers, landscapers, and the like to offer something they call an estimate, but which is actually a fixed price quote (a promise).

Sadly, while there are plenty of common good synonyms for promise, there aren’t many for estimate. We’re stuck with using the word estimate, and explaining that we really mean it as defined. Perhaps in a few more decades we will lose the word entirely, much like the word “literally” has come to mean its antonym, “figuratively”, which renders it mostly useless.

Estimates

An estimate is an approximation of an unknown quantity. Typically in the world of software development, it is a prediction of the cost, working hours, or delivery date of a project or milestone. It is not in any sense a commitment, any more than estimating the temperature outside this afternoon is a commitment.

As the word implies, a customer reasonably expects the actual value to vary somewhat, in either direction, from the estimate. In fact, if an estimate turns out exactly match the actual result, there is a good chance the books have been cooked. Moreover, if the work is completed at-or-before the estimate most of the time, this means the estimates (on average) are too high.

An estimate “costs” nothing, other than the time/effort required to create it, which consists of analyzing the work at hand, decomposing it in to parts, and comparing those parts to past work.

Promises

A promise, also called a commitment, deadline, quote, fixed price, etc. is a different beast entirely.

With a promise in hand, a customer should expect with high confidence that the actual value (for cost, hours of work, delivery date) will be less than (before), or equal to, the promised value/date.

Be wary of a promise easily made and freely given: it probably doesn’t mean anything at all. A wise customer (and I aim to count myself in this category) should expect that a casually made commitment will probably be broken; not because the maker is morally defective, but simply because meeting a commitment for complex work requires considerable effort and thought. Without evidence that happens, it would be mere wishful thinking to expect the results delivered as promised.

Likewise, keeping promises often has a cost. If the work underway gets behind the schedule needed to meet the promise, something will have to give:

• Other work may fall behind, as time and effort are diverted to meet the promise.
• Weekends, evenings, and overtime work may be needed. These might appear free, but are not.
• Staff may need to be reassigned, or added
• Additional hardware and software may be needed.

These risks cost real money; thus a wise promise-maker will find that, on average, it costs more to promise feature X by date D, than to delivery feature X by date D without such a promise.

Estimates are Cheaper, so Prefer Estimates

At Oasis Digital, we provide many estimates, but few promises. Most of the time, an estimate is what our customers need; and we can provide at estimate with very little cost. Typically we estimate reasonably well:

• small features usually arrive with a day or two (plus or minus) of the estimated delivery date (and likewise for cost)
• medium items within a week or two of the estimate, likewise
• large items (major new features or subsystems with complex interdependencoes) within a month or so, likewise

The key here is is that with good estimate, commitments (promises) aren’t needed very often, and therefore the cost of promises can be avoided.

But Learn How to Promise Well, Also

Yet occasionally, a customer needs a commitment, most often because a software version needs to be available to match an important business event with a fixed date, such as a presentation, a legal filing, etc. I’ll follow up later (no promise or estimate, as to when) with thoughts on:

• how to credibly make promises (as a service provider)
• how to evaluate promises (as a customer)

• Aquamacs
• GNU Emacs 23.1

and set up each on my Macbook Pro (10.6.2). For each Emacs, I also set up SLIME, Swank, and Clojure.

If that sounds like a bunch of garbled nonsense, don’t worry, it is simply a sign that you are a normal, well adjusted person, rather than a Lisp person.

It was a bit irritating to get these tools up and running the first time; there are countless web pages with the details (I won’t make that worse by adding my own here). The challenge is figuring out which instructions to follow, while ignoring the rest. The most tractable approach for me, for a quickly-installed, easily-updating, fewest-steps process, was to install the needed packages from ELPA, without any manual downloading/configuration of SLIME, Swank, etc. at all. I have no command line operations to share with you, because in the process I recommend, there aren’t any.

I used Emacs in school, but not at all in the last 10 years. Since then, my expectations for developer tools have grown – I think it is reasonable for a modern development environment to simultaneously:

1. be easily discoverable, for a fast start
2. cooperate with its environment well
3. make good use of the large high-res display on most PCs, to expose expected status and control surfaces
4. be deeply configurable and highly powerful

My experience, coming in from that point of view, differs from what Ryan and a couple other people suggested: I suggest Aquamacs. It is much more of a good citizen on the Mac platform. Its menus are much more approachable. Printing works in a Mac way. You get tabs showing your buffers, by default. You get menu options to gather them up in to one window(frame) or split them apart, all by default.

It may indeed by worth switching to the non-Aqua version later, if/when you get so deeply in the toolset that the pure emacs experience is preferable… but I estimate you’d need to be pretty far down the path (much further than I).

Something I should try out eventually is the Emacs Starter Kit. Perhaps it would have given a sufficiently “saner set of defaults” to make the non-Aqua version a more reasonable choice. Emacs gurus seem to lean that direction, overall.

Sadly the information there has fallen somewhat out of date due to new CF and VS.NET releases, and I’ve been busy on other platforms and projects (Java and much more) since then. If anyone out there can suggest a handful of the best places to learn now about CF development, please do so in the comments below, for the benefit of others reading this page.

I picked that demo project up again last year and recreated it with Flex 3; you can now try it out online, download the code, or browse it on github. It looks like this:

Click on it to try it “live” in as Flash/Flex.

The interesting part is the drag-drop movement – not my less-then-stellar color and design choices.

Appistry again provided space and pizza – thanks guys! (Appistry is our locally-grown but widely-known cloud infrastructure software maker – they’d been cloudy for years before the term entered wide use.)

The talk appeared to go over well. There were many important things about factor that I’d love to talk about but didn’t have time. Nonetheless, I think 45 minutes is a good talk length, and I think the format (short talks) is a key part to the Lambda Lounge’s success so far.

Speaking of the Lambda Lounge,  next month there is a “language shootout” – if you want to participate, be sure to join the mailing list and look over the participants. (You need to join the list, to be able to get to the wiki with that page.) I might submit an entry myself, using Factor or another language.

“The most profound engine of civilization is the inability of a larger and larger fraction of the population to do the basic things needed to survive.  Many people fail to realize this.”

“Technology’s greatest contribution is to permit people to be incompetent at a larger and larger range of things.  Only by embracing such incompetence is the human race able to progress.”

It’s a good read, with important thoughts on civilization and education.  It is part of the introduction to a book about Mathematica, a truly amazing piece of software. (Mathematica, was amazing when I used it in the 1990s, I can scarcely imagine what it does now.)

There is quite a bit of discussion about this on the web, including:

• Suggestions to do with with tar; this is a poor idea because it copies all the file data, taking an enormously long time.
• Suggestions to do it with “mv -r”… but as far as I can tell, mv does not have a -r option.

After a little thought I came up with the script below. I’d love to have a Linux/Bash guru out there point out how awful it and and send me something better!

A critical feature for me is that it does not overwrite files; if a source file name/path overlaps a destinate file, the source file is left alone, untouched. This can be changed easily to overwrite instead: remove the [[-f]] test.

$ cat ~/bin/move_files_merge.sh
#!/bin/bash

# Move files from dir $1 to dir $2,
# merging in to existing dirs
# Call it like so:
# move_files_merge.sh /FROM/directory /TO/directory

# Lousy error handling:
# Exit if called with missing params.
[ "A" == "A${1}" ] && exit 1
[ "A" == "A${2}" ] && exit 1

echo finding all the source directories
cd $1
find . -type d -not -empty | sort | uniq >$2/dirlist.txt

echo making all the destination directories
cd $2
wc -l dirlist.txt
xargs --no-run-if-empty -a dirlist.txt mkdir -p
rm dirlist.txt

echo Moving the files
cd $1
# There is surely a better way to do this:
find . -type f -printf "[[ -f '$2/%p' ]] || mv '%p' '$2/%h'\n" | bash

echo removing empty source dirs
find $1 -depth -type d -empty -delete

echo done.

Will Factor become popular? With a FORTH-like syntax, I suspect the answer is a firm No, the syntax is too foreign (even compared to Ruby, for example) for mainstream devleopers. But I find it fascinating nonetheless, and I will keep my eyes open for an opporunity to use it on a small, real project.

It would also make a great topic for the Lambda Lounge, a new St. Louis area user group about which I am quite excited.

Based on all that, I set about writing a small test app to see what it’s like to program a Swing app with JS.  After a day or so of work (spread over a few months), I offer my Rhino Swing Test App:

Run RSTA now via Java Web Start

Get the RSTA code (git, on GitHub)

It implements the same “flying boxes” animation demo that I presented a few years ago at the St. Louis JUG, but aside from a generic launcher class, the GUI is implemented entirely in JavaScript. To clarify, this is not web browser JavaScript; it is running in Rhino, in the JVM, using Swing classes.

The documentation for interaction between Java and JS is limited, but sufficient. For simplicity, I used Rhino as an interpreter, I did not compile to java bytecode. Nonetheless, the animation runs about as smoothly in JS as it does in Java, because the heavy lifting is done by the JDK classes.
I used Eclipse (with JavaScript support) to write this code, but of course JS makes much less code completion possible than Java, and I missed that. Typically I mildly prefer IDEs, but am also productive with a text editor. For working with a large API like Swing though, IDE support helps greatly.

Still, I recommend a look at this approach if you are fond of dynamic languages but need to build on the Java platform, and I intend to investigate server-side JS development also.

Yesterday I set up a DokuWiki instance (which stlruby.org might migrate to), and found that its underpinnings (PHP, plain text files) make for ridiculously easy installation:

• wget
• tar xzf
• browse to install page, set a few settings
• delete install page

Yet those underpinnings are very well suited to the task at hand. A typical Wiki does not need a database underneath it. As with many things, this reminded me of a general principle: use the least complex, most readily and commonly available, easiest to administrate technology appropriate for the task at hand.

If you need to make a change, but that change is difficult / tedious / risky to make, first improve the underlying system that makes it so.
(OasisDigital.com is a static web site; we have dynamic contact (issue trackers, etc.) to automate our work together and with our customer, but that content is on another domain.)

Webby is a client-side, simple CMS for generating static web sites, written in Ruby. Why serve a static site (plain old files on a web server) in 2008?

• It minimizes the moving parts, there is almost nothing to break or maintain.
• It is very unlikely that any hosting issue will break a static site.
• It is easy to serve a static site fast (though our current host, TextDrive, sometimes is not all that fast).
• Security vulnerabilities are very unlikely, in the absence of any executable content.
• The canonical content (in this case, mostly Markdown) is stored in plain text files, which we track, diff, and merge in git.

In an earlier foray in to Drupal, we found that Drupal has extensive and useful capabilities, as well as a vibrant community, but it also has many moving parts; too many, in my judgment, to make it a good solution for building an essentially static web site.

This application forms the core of our customer’s business; they sell it at a commercial product to their customers, who use it in high-pressure situations. Over 4 major versions and 4+ years, there been remarkably few bugs or other issues found “in the field”, which I credit to:

• Use of automated testing
• An overall level of care and attention to quality

The second item alone is not enough. To achieve quality you need to do more than have the right intentions and work hard. You need to do the right things.