Tag: programming

Playing to Win, Going Beyond the Call of Duty

I’m writing this up as advice to a rhetorical “you”, but it’s really written to a younger version of me.

Imagine that you are working on a project for a customer (or employer), who has a weekly or monthly budget limit. You are paid by the hour, for up to N hours per week; or perhaps you are paid a salary for full-time work, nominally 40 hrs per week.

You want to work on the project more hours than your customer will pay for. For example, maybe your customer can afford only 20 hours per week, but you have more time available that you’d like to spend on the project.

The most obvious path is to simply not work any more than your customer will pay. Go do something else unrelated instead, perhaps not even computer-related. This approach appeals strongly from a work-life balance perspective, and also has a “tit-for-tat” appeal along the lines of “I refuse to do anything that would benefit my customer unless I am being paid to do so”. However, it is a very bad strategy. Do not let your customers’s budget stop you from growing your skills and building something great.

A second path, at the other extreme, is to work all you want on a project, regardless of how few hours you are paid for. Simply don’t bill for the extra hours. This is an easy answer, and in spite of its appeal from my current perspective (as an employer and customer of software development work), I don’t recommend it. It will give your customer an erroneous indication of your productivity level, causing future problems for you (how will you live up to this apparent but false productivity in the future?) and for your customer (whose estimates and schedules will be fed by false data).

I recommend a path between these two extremes, a strategy by which you can do work that benefits you and your customer, now and in the future. Work the hours you bill, directly on your customer’s project. Then, if you have extra time available (that you haven’t filled with other, paying work), work as much as you care to, past those hours, on related activities that indirectly help the project and help you grow:

  • Read about the development tools and technologies you are using. (Learn by reading.)
  • Read about the project problem domain.
  • If your customer project uses a DBMS, read about and experiment with that DBMS.
  • Try another DBMS; port the customer project to this other DBMS. Even though the customer doesn’t need it, you can nonetheless learn a lot from the experience.
  • Build an additional feature that your customer did not ask for (yet); the offer it to them if they want it. By the way, this is a great time to learn to use a powerful source-control system to manage a long-lived branch with such a feature.
  • Start a second, sample project, which uses some of the same tools as your customer project. Learn by doing. Experiment on this sample project, practice using the tools. Try out options and feature of the tools that you haven’t needed yet on your customer project.

To me, this is a strategy for “playing to win”.

More About the Sample Project

By the way, the notion of a second, sample project is very powerful, for reasons far beyond your current project. Be careful to keep the real project code completely out of this sample project, and you will end up with a very valuable artifact: a working piece of software, built with tools you know well, which you own. You can use this project to help get future work, by offering it to future customers as a work sample, and by posting it on your web site to show (rather than merely proclaim) your expertise.

Please, Use a Web Application Framework

Historically I have not been a fan of “frameworks”, and I have often repeated the following joke:

What’s the difference between an application and a framework?

An application is something a customer actually wants!

However, for some applications, I recommend use of an application framework. For some Oasis Digital projects, I require it:

Please, Use a Web Application Framework

My reasoning here applies specifically to web applications with many CRUD (create-read-update-delete) features, and an underlying database. The advice applies much more widely, and with many nuances and caveats, but this article I am discussing only CRUD-ish web applications. Even within this niche, my reasoning does not apply to web applications which “push the envelope” of what is possible or which attempt to advance the state of the art.

Regardless of the programming language, the application should be built on a framework. More specifically, the framework should popular and mainstream, with a community of developers, and the appearance of momentum for the future. Likewise, the client-side JavaScript used in these applications, should also be based on such a framework. Here are some examples:

Ruby: Ruby on Rails, IOWA

Python: Django, TurboGears, Pylons, TwistedWeb

PHP: Akelos, CakePHP, CodeIgnitor, Symfony, Zend

Java: Struts, Seam, Rife, Tapestry, Stripes, Wicket Spring MVC

JavaScript: Scriptaculous, Prototype, JQuery

This is just a list of some frameworks that I am aware of; I have not evaluated all of these in detail, and I do not endorse them; nor is this an exhaustive list. For Oasis Digital projects, we help evaluate proposed frameworks, then I personally give the go-ahead to use a particular framework for a particular project.

An in-house web application framework does not meet the “community of developers” criteria, except at the very largest firms. Everywhere else, you are better off with an off-the-shelf, popular framework than with an inhouse framework, even if the latter is brilliantly designed.

Justification

My recommendation (and requirement, for some projects) for using an application framework for this kind of application is not based on a fad. Rather it is based on my years of experience as a developer, a team leader, a maintainer, and most importantly, a customer of software development.

The wild success of some frameworks (such as Ruby on Rails) has shown that they can reduce the amount of code and time needed to develop an application. That second factor, the amount of code specific to the application, is at least as important as the development time. Lines of code are not an asset; they are a liability. Only the features that the code provides are an asset. The most valuable software provides a lot of features using the smallest possible amount of application-specific code.

Therefore, even if a developer is so extraordinarily fast that they can create a system very quickly without using an off-the-shelf framework, they still have provided less value by doing so, compared to creating that same system quickly with fewer lines of code.

Another benefit of using a common framework (not a custom, in-house framework) is that this makes an application much easier and faster for other developers are work on in the future. A more maintainable system is more valuable.

Framework Caveats

Vidar Hokstad left a lengthy and excellent comment below, disagreeing with my thesis. It turns out that I mostly agree with Vidar, and it sounds like he and I have been through many of the same experiences with poor application frameworks. There are a lot of things an application framework can do wrong, and sadly, many of them take the opportunity to do so. In-house frameworks created by “architecture astronauts” seems to be especially prone to these defects:

  • All-or-nothing: Some frameworks intentionally or accidentally make it hard to replace a section of the framework. Don’t use these. Use a framework instead that has a “library” philosophy, such that you are readily choose to use some parts but not others.
  • Just Different: There are frameworks which offer an API wrapper around the underlying mechanisms, which isn’t really any better, just different. In this case, different is worse. Writing to (for example) the com.acme.inhouse.servlet API is, all else equal, much worse than writing to the standard Java Servlet API. To be worth its weight, a framework API must be demonstrably and obviously more concise.
  • Lower Abstraction: There are frameworks which, ironically, lower the level of abstraction of the application code, because that code ends up working around the framework features to get the job done.
  • Pile of Pieces: There are frameworks in which it is necessary to shred your application in to a pile of pieces, and then wire those pieces together with configuration files. This is sometimes useful, but often makes the application harder to understand, not easier, especially if there are extensive “XML pushups” involved. (I’m looking at YOU, Struts!) Instead, choose a framework with convention-over-configuration, and one which offers but does not require manual wiring.
  • Keyhole Database Access: If you find you mostly use a frameworks’ DB access features, and as a result you have short, easy to change code, then keep it. But if you find you use extensive SQL to work around lots of framework issues, throw it out. If a framework intentionally makes it hard to reach to the underlying SQL access, throw it out now.
  • No Source: If someone proposes a framework for which you won’t have source code, laugh. Aloud. If this gets you fired, then it has set you on a path to find employment at a more enlightened organization.
  • Exceptionally Bad Exception Handling: Java frameworks are especially prone to issues with exception handling, in which the framework code “eats” exception details.

In summary, pick up a framework and use it to get your application up and running quickly, but don’t be stupid. Do what makes sense locally for your project over time. It is a win to use an application framework to reach “1.0” functionality, even if you end up removing or swapping out parts of it later.

Optimize Hierarchy Queries with a Transitive Closure Table

Last year I posted about the use of a Joe Celko-style nested set hierarchy representation, for fast hierarchy queries. Here I will describe another approach which is simpler to query, but more wasteful of space. I did not invent this transitive closure approach, I learned of it from several directions:

There are two (main) places to put the code for building a closure tables: in your application code, or in your database. The application approach is nice if you are aiming to avoid vendor-specific SQL code, but it is quite simple in SQL, and therefore not a big problem to recode for another RDBMS if the need arises. The SQL approach also avoids round-tripping the relevant data in and out of the database. Therefore, the approach I generally recommend for this is an SQL stored procedure / function.

Here is a simplified PostgreSQL stored procedure to do the job; note that his assumes a “widget” table with a widget_id and parent_id (the “adjacency” representation of a hierarchy), and a widget_closure table with fields (ancestor_id, widget_id, depth):

CREATE OR REPLACE FUNCTION populate_widget_closure()
RETURNS integer AS '
DECLARE
  distance int;
BEGIN
  delete from widget_closure;
  insert into widget_closure
    select widget_id, widget_id, 0 from widget;

  FOR distance in 1..20 LOOP
    insert into widget_closure
    select uc.parent_id, u.widget_id, distance
      from widget_closure uc, widget u
      where uc.child_id=u.reports_to_id
        and uc.distance = distance-1;
  END LOOP;
  RETURN 1;
END;
' LANGUAGE plpgsql;

This sample code assumes a maximum depth of 20, and has no error checking. It will blindly miss greater depths and produce garbage if there is a “loop” in the ancestry. I recommend both arbitrary depth handling and error checking for production use.

Once your transitive closure table is populated, querying it is extremely fast and simple. You can get all the descendants of widget 12345 with the query “select widget_id from widget_closure where ancestor_id=12345”. Of course, this hierarchy representation, while simple to generate, is not simple to incrementally update as the hierarchy changes. The most straightforward way to use it is as a cache, regenerated as needed.

Shoes, a new Ruby GUI toolkit

Last year I gave a talk on Ruby GUI toolkits, and concluded that none of those I looked at were compellingly slick or mature. There is a new player on the field now (thus certainly not mature, but interesting nonetheless): Shoes, from why the lucky stiff.

Shoes creates native applications with a Web feel. It is not built atop GTK, Qt, etc., but rather directly on the relevant native API (Win32 on Windows, for example). Shoes has a lot of potential, which it can realize only if a community accrues around it, leading to lots of polish and a rich widget set.

One thing that surprised me about the Shoes install it that, as far as I can tell, it includes its own Ruby install; I installed Shoes on a machine with no (obvious) Ruby installation and it worked.

Great Developers, Projects That Sound Boring

I’ve been a fan of Joel Spolsky for years, though I haven’t agreed with everything he’s written, and even mocked him a bit. Joel has written at length on his web site and in print about attracting the best developers, and one aspect of that has bothered me:

How do you attract top developers to work on something that sounds rather dull, like a bug tracking application? It mostly shuffles data back and forth between screens and reports and database tables – far too boring for top developers.

Of course that’s an exaggeration, but a relevant one: at Oasis Digital much of our work is on enterprise business process automation, database-centric applications, and could likewise be described casually (though not accurately) as “just” shuffling data between screens and tables. I worry that our work will not sound interesting to prospective hires.

This week at the Business of Software conference I got a chance to ask (confront?) Joel about this. He offered a great, four-part answer, which I present here with my own additions mixed in. I don’t have careful notes about which bits came from Joel, so you are welcome to give him all the credit and me all the blame.

1) There is Interesting Technology Inside

Even in an application which, at first glance, just shuffles data around, there can turn out to be a lot of very interesting work inside. This is true of FogBugz and it is true of our work at Oasis Digital as well. Here are some examples of interesting work here, all of it inside dull-sounding applications:

  • Process metadata and generate code and GUI elements. Top developers certainly are those who solve a family of problems with generic code and metadata, rather than tediously one at a time.
  • Process large hierarchies efficiently using Celko’s nested sets representation technique. Top developers are all about using better data structures.
  • Custom GUI components to provide a drag-drop, direct manipulation approach to visualize and modifying data. The results has both a high “wow” factor, and is genuinely useful – a willing combination for top developers.
  • Integrate a Prolog-based rules mechanism to provide a vital algorithm in one page of code, that would have required countless pages of code and hundreds-to-thousands of hours of work to do otherwise. Using a radically different language to solve a problem with a small fraction of the effort… exactly the sort of thing a top developer wants to do.
  • Generic data replication mechanism: building our own was certainly more interesting work than adoptions one off the shelf.
  • Learn how OLAP works, implement an OLAP ETL process.
  • … I could list many more examples

2) One Level Down the Stack

Fog Creek aims to make compellingly good software because that is how you outcompete established competitors in a commercial shrinkwrap market. Oasis Digital likewise has this aim for a different reason: it is our intended niche. We aim to differentiate ourselves from “yet another outsourced dev company” by building unexpectedly good software. We don’t want customers who will be happy with the results available from the typical development firm; we want customers who are playing to win.

To meet either goal, it is sometimes necessary to work at one level of abstraction lower than would otherwise be necessarily. Joel’s examples were their own data grid and their own AJAX library. Some of our examples are listed above.

This kind of work, further in to the details, is generally more compelling to top developers.

Caveat: Don’t do this very often. If you want to ship software anytime soon, you need to mostly use off the shelf libraries that already work. Don’t build a data-grid, for example, unless you really need something that you can’t find in any off the shelf products. We don’t have our own data-grid; we use (among other things) the excellent grid products from Developer Express.

3) Problem Domain is less important than other factors

Joel has observed that developers aren’t as picky about the problem domain of their project as one might think; rather, other factors are more important: great co-workers, nice working environment, working for a boss who is a developers, etc. Top developers want to work on a high quality end product worthy of taking pride in.

4) All Projects need a lot of Grunt Work

To build a high quality product, in any problem domain, will require spending a great amount of your time on grunt work: tracking down bugs and fixing them, filling in feature “holes”, cleaning up design problems, improving GUI layouts, and the like. This is true for any problem domain. Top developers know this, and just get on with doing that work when needed.

Conclusion

My worry was unjustified. Our work at Oasis Digital is interesting and worthwhile, both for our customers and for our developers. To grow our team, we must focus on making it an increasingly good place to work.

A final anecdote: Joel mentioned that FogBugz 6 was feature-complete in the summer of 2006 – that means they spent around a year polishing it, fixing bugs, filling in holes, etc. That shows a phenomenal amount of dedication and discipline to create quality software.

Growing a Language, by Guy Steele

This is an oldie-but-goodie: Guy Steele’s “Growing a Language” talk from OOPSLA 1998.

It is amazing to me that Guy, whose is something of a legend in language design, and who thinks so clearly about what makes a good language, was also key in designing Java. Java has been extremely slow to grow in the sense described in this talk, because for many years Sun resisted such growth. Only the rise of C# and the growing popularity of dynamic languages generated enough pressure to get Java unstuck… and in the last few year Java has become somewhat growable in the sense Guy describes.