Category: Business

Buy vs. Build

A while back Joel Spolsky wrote about “Five Worlds” of software development. Over at Oasis Digital and elsewhere, I’ve been living in two of them:

  • Custom, internal software that we develop as consultants.
  • Commercial software that we develop at our expense, then sell licenses.

As a result, I’ve been thinking a lot about the relative advantage of each, from the points of view of software developers and customers. A portion of these thoughts, those on the merits of custom development from the customer perspective, are in an article (PDF) on the Oasis Digital web site.

There is a story behind how that document was created, and I’ll tell it in another post.

Start with a Prototype

I have often given the advice to start each project with an end to end working first draft, also known as an evolutionary prototype, and sometimes I have even followed this advice myself. My own projects have been most successful when following it, such as in three projects that come to mind:

  1. A 6+ year project, with a team of developers. In month 3 it was deployed to production, meeting real needs.
  2. A 4+ year project, with a team of developers. In month 2 it was deployed to production, meeting real needs.
  3. A 10+ year project, initially just myself but later with a small team of developers. In month 1 it was deployed to production, meeting real needs.

On the other hand, most failed projects that I am aware of had been slated from the beginning to not deliver a working system until late in the schedule, and ended up not delivering at all.

The traditional software development plan for a typical enterprise forms-database-rules-reports application looks like this:

  • Planning and design
  • Implement schema
  • Implement infrastructure
  • Implement admin/setup features
  • Implement core features (transactional data)
  • Implement reports

This waterfall model is a great way to think about software. Think through your requirements in depth at the beginning, and write them down in a coherent form. However, this is generally a bad manner (specifically, a very poor order) in which to write software.

Thanks to the agility movement, many teams are now using iterative, incremental approaches. Unfortunately, it is common nonetheless to lose some of the benefits of agility by laying out a set of releases with features/work in approximately the traditional order.

The solution, espoused by the Extreme Programming movement but rarely executed, is to start at the very beginning of a project, with building an end to end, first draft, working prototype system. This then grows slowly over the life of the project in to the finished product (i.e., it is not a throwaway prototype). We take this approach at Oasis Digital, and generally require it of our subcontractors also.

Development proceeds in a way that we create a working, end-to-end system with minimal features, very early in the work. For example, on a typical forms-database-rules-reports project, after a few days of work, we have:

  • A little bit of schema, just enough to support a small subset of desired features
  • A few screens, with very few features
  • A report or two, again with very limited features
  • Placeholders for some other screens and report, to allow the beginnings of the desired final application flow to appear

Then as the work proceeds, every week (or less) we can deliver a working system with a few more features present.

But this is impossible on my project!

The question is not whether this approach is possible; rather it is to what extent it can be done on each project, and what can be done to make it possible? The primary tool is imagination, in selecting a small subset of the desired feature set which plausibly works end-to-end.

For example, perhaps you are building a CRM (customer relationship management) system, one with thousands of features, 200 database tables, 450 reports, 85 screens, etc. How could you build a working system in the first week? Start by reducing the system to be its bare essence of companies, contacts, and interaction history.

A second example is a time tracking system with elaborate rules, reports, approval mechanisms. For the first draft, discard most of that and create only the ability to add a user (forgot edits and deletes, forget most of the fields), to enter hours for that user (forget approcals), to see a screen with those hours (with the simplest possible layout), and export a data file with those hours (any data file format, forget XML for now).

A third example is a trouble ticketing system. For the first week, make a way to add a ticket, a way to add notes to a ticket, to see a list of tickets, and to close a ticket. This could readily all occur on one screen, with only a few buttons. Its schema could consist of three tables: user, ticket, note.
A fourth example is a system which will write checks. In the very first week of the project, make it write a check. Dummy-out as much as needed to make it achieve this core goal.

Just Do It

Certainly there are plenty of problem domains where this advise doesn’t readily fit. But don’t be too eager to make that claim. By finding a way to deliver a working system early, we eliminate the risk that the project will never deliver anything, and we make progress and value visible (continuously) to whoever is paying for the project.

On the Job Training

I recently heard from a developer who worked on a project here for about one year, several years back. I won’t name or quote him directly; the essence of his message was:

  • He learned a lot during that year.
  • This learning was a turning point in his career.
  • What he learned has proved lucrative.

Of course it feels great to receive a message like this, but beyond that, it got me thinking about on-the-job training at Oasis Digital. We don’t have a formal training process. Rather, opportunities (and pressure) to learn and improve are woven in to our daily work. Described in coarse terms, a typical workflow for a developer (“You”) starting out on our projects is:

  • You write some code (or other work product).
  • Tech lead and other developers (sometimes including me) complain about it. This takes the form ideally of constructive criticism.
  • You improve the code.
  • More criticism.
  • You improve the code more, resulting in code that essentially works.
  • Tech lead revamps and refactors it mercilessly, explaining what was changed and why. Note though, that at this stage the changes do not require justification.
  • Tech lead sends developer a list of things to further improve.
  • You improve the code.
  • The code is now good enough to go in to the project.

It turns out that this process is a very good learning experience, if you are willing to take feedback. It can be hard to hear feedback, especially if its essence is that that your work product is not all that great. Yet the computer is merciless, it does not care about your feelings. The end customer does not care about your feelings either; both care only about correctness and quality.

I started out writing bad code. Having written many lines of such code, I now, somewhat regularly, write good code… though I am certain there are plenty of better developers out there who could find much fault in it. Writing bad code and accepting criticism of it is on the path to writing good code. Is it controversial to observe that a lot of software is quite bad? If so, that’s fine; I mostly agree with Steve Yegge’s observation:

By way of setting context for today’s blog, I’ll summarize everything I’ve ever written to date in one sentence: I think most software is crap.

By accepting that most software is bad, you can get on a path to make yours better. It turns out that working on our projects gets you a lot of coaching and training; and instead of paying for these things, you get paid. For the right kind of developer, I think this is an important part of our value proposition and compensation.

Playing to Win, Going Beyond the Call of Duty

I’m writing this up as advice to a rhetorical “you”, but it’s really written to a younger version of me.

Imagine that you are working on a project for a customer (or employer), who has a weekly or monthly budget limit. You are paid by the hour, for up to N hours per week; or perhaps you are paid a salary for full-time work, nominally 40 hrs per week.

You want to work on the project more hours than your customer will pay for. For example, maybe your customer can afford only 20 hours per week, but you have more time available that you’d like to spend on the project.

The most obvious path is to simply not work any more than your customer will pay. Go do something else unrelated instead, perhaps not even computer-related. This approach appeals strongly from a work-life balance perspective, and also has a “tit-for-tat” appeal along the lines of “I refuse to do anything that would benefit my customer unless I am being paid to do so”. However, it is a very bad strategy. Do not let your customers’s budget stop you from growing your skills and building something great.

A second path, at the other extreme, is to work all you want on a project, regardless of how few hours you are paid for. Simply don’t bill for the extra hours. This is an easy answer, and in spite of its appeal from my current perspective (as an employer and customer of software development work), I don’t recommend it. It will give your customer an erroneous indication of your productivity level, causing future problems for you (how will you live up to this apparent but false productivity in the future?) and for your customer (whose estimates and schedules will be fed by false data).

I recommend a path between these two extremes, a strategy by which you can do work that benefits you and your customer, now and in the future. Work the hours you bill, directly on your customer’s project. Then, if you have extra time available (that you haven’t filled with other, paying work), work as much as you care to, past those hours, on related activities that indirectly help the project and help you grow:

  • Read about the development tools and technologies you are using. (Learn by reading.)
  • Read about the project problem domain.
  • If your customer project uses a DBMS, read about and experiment with that DBMS.
  • Try another DBMS; port the customer project to this other DBMS. Even though the customer doesn’t need it, you can nonetheless learn a lot from the experience.
  • Build an additional feature that your customer did not ask for (yet); the offer it to them if they want it. By the way, this is a great time to learn to use a powerful source-control system to manage a long-lived branch with such a feature.
  • Start a second, sample project, which uses some of the same tools as your customer project. Learn by doing. Experiment on this sample project, practice using the tools. Try out options and feature of the tools that you haven’t needed yet on your customer project.

To me, this is a strategy for “playing to win”.

More About the Sample Project

By the way, the notion of a second, sample project is very powerful, for reasons far beyond your current project. Be careful to keep the real project code completely out of this sample project, and you will end up with a very valuable artifact: a working piece of software, built with tools you know well, which you own. You can use this project to help get future work, by offering it to future customers as a work sample, and by posting it on your web site to show (rather than merely proclaim) your expertise.

Please, Use a Web Application Framework

Historically I have not been a fan of “frameworks”, and I have often repeated the following joke:

What’s the difference between an application and a framework?

An application is something a customer actually wants!

However, for some applications, I recommend use of an application framework. For some Oasis Digital projects, I require it:

Please, Use a Web Application Framework

My reasoning here applies specifically to web applications with many CRUD (create-read-update-delete) features, and an underlying database. The advice applies much more widely, and with many nuances and caveats, but this article I am discussing only CRUD-ish web applications. Even within this niche, my reasoning does not apply to web applications which “push the envelope” of what is possible or which attempt to advance the state of the art.

Regardless of the programming language, the application should be built on a framework. More specifically, the framework should popular and mainstream, with a community of developers, and the appearance of momentum for the future. Likewise, the client-side JavaScript used in these applications, should also be based on such a framework. Here are some examples:

Ruby: Ruby on Rails, IOWA

Python: Django, TurboGears, Pylons, TwistedWeb

PHP: Akelos, CakePHP, CodeIgnitor, Symfony, Zend

Java: Struts, Seam, Rife, Tapestry, Stripes, Wicket Spring MVC

JavaScript: Scriptaculous, Prototype, JQuery

This is just a list of some frameworks that I am aware of; I have not evaluated all of these in detail, and I do not endorse them; nor is this an exhaustive list. For Oasis Digital projects, we help evaluate proposed frameworks, then I personally give the go-ahead to use a particular framework for a particular project.

An in-house web application framework does not meet the “community of developers” criteria, except at the very largest firms. Everywhere else, you are better off with an off-the-shelf, popular framework than with an inhouse framework, even if the latter is brilliantly designed.

Justification

My recommendation (and requirement, for some projects) for using an application framework for this kind of application is not based on a fad. Rather it is based on my years of experience as a developer, a team leader, a maintainer, and most importantly, a customer of software development.

The wild success of some frameworks (such as Ruby on Rails) has shown that they can reduce the amount of code and time needed to develop an application. That second factor, the amount of code specific to the application, is at least as important as the development time. Lines of code are not an asset; they are a liability. Only the features that the code provides are an asset. The most valuable software provides a lot of features using the smallest possible amount of application-specific code.

Therefore, even if a developer is so extraordinarily fast that they can create a system very quickly without using an off-the-shelf framework, they still have provided less value by doing so, compared to creating that same system quickly with fewer lines of code.

Another benefit of using a common framework (not a custom, in-house framework) is that this makes an application much easier and faster for other developers are work on in the future. A more maintainable system is more valuable.

Framework Caveats

Vidar Hokstad left a lengthy and excellent comment below, disagreeing with my thesis. It turns out that I mostly agree with Vidar, and it sounds like he and I have been through many of the same experiences with poor application frameworks. There are a lot of things an application framework can do wrong, and sadly, many of them take the opportunity to do so. In-house frameworks created by “architecture astronauts” seems to be especially prone to these defects:

  • All-or-nothing: Some frameworks intentionally or accidentally make it hard to replace a section of the framework. Don’t use these. Use a framework instead that has a “library” philosophy, such that you are readily choose to use some parts but not others.
  • Just Different: There are frameworks which offer an API wrapper around the underlying mechanisms, which isn’t really any better, just different. In this case, different is worse. Writing to (for example) the com.acme.inhouse.servlet API is, all else equal, much worse than writing to the standard Java Servlet API. To be worth its weight, a framework API must be demonstrably and obviously more concise.
  • Lower Abstraction: There are frameworks which, ironically, lower the level of abstraction of the application code, because that code ends up working around the framework features to get the job done.
  • Pile of Pieces: There are frameworks in which it is necessary to shred your application in to a pile of pieces, and then wire those pieces together with configuration files. This is sometimes useful, but often makes the application harder to understand, not easier, especially if there are extensive “XML pushups” involved. (I’m looking at YOU, Struts!) Instead, choose a framework with convention-over-configuration, and one which offers but does not require manual wiring.
  • Keyhole Database Access: If you find you mostly use a frameworks’ DB access features, and as a result you have short, easy to change code, then keep it. But if you find you use extensive SQL to work around lots of framework issues, throw it out. If a framework intentionally makes it hard to reach to the underlying SQL access, throw it out now.
  • No Source: If someone proposes a framework for which you won’t have source code, laugh. Aloud. If this gets you fired, then it has set you on a path to find employment at a more enlightened organization.
  • Exceptionally Bad Exception Handling: Java frameworks are especially prone to issues with exception handling, in which the framework code “eats” exception details.

In summary, pick up a framework and use it to get your application up and running quickly, but don’t be stupid. Do what makes sense locally for your project over time. It is a win to use an application framework to reach “1.0” functionality, even if you end up removing or swapping out parts of it later.

Is Delphi Dead? No.

A few months ago Alex Miller pointed me to this Delphi doom article (the site appears to be down at the moment), which reminded me to post about the same topic. Here goes.

Delphi shipped in 1995, and its demise has been declared frequently since 1997 or so. In a sense this demise is true, yet also false. Delphi’s current popularity is very different in form (not only in magnitude) from that of Java, C#, etc. Delphi is used substantially by commercial software vendors, and only rarely by enterprises. An ugly reality of the software industry is that the bulk of software developers nationwide work inside large non-software companies, so this usage pattern most likely does not produce the level of unit sales that Codegear (Borland’s dev-tools subsidiary) would like to see. It does, however, produce an enormous number of Delphi application instances running “in the field”, used by real paying end users, who don’t care (or know) what development tools were used to build the software they buy. Many commercial software products, both those in shrinkwrap at retail stores and those for vertical markets, are written in Delphi and will continue to be, because there are very few other good choices for high quality (polished) native Win32 GUI software. In these markets, shipping a Java or .NET app can be a competitive disadvantage (though to a lesser extent over time), and old-style VB is a sad joke.

I don’t think Delphi is eligible for demise until the dominant desktop operating system ships with a dominant runtime platform “in the box”. For example, if all of this happens at the same time:

  • Microsoft ships Windows with the .NET runtime already installed
  • That version Windows is the commonly deployed version
  • That version of the .NET runtime is the commonly targeted version

At that time, the .NET platform (with the language of your choice) could be a compelling replacement for Delphi in its niche. There is a lot to like about .NET (and Java, and I use them both), but I’m not holding my breath for the above conjunction.

Over at Oasis Digital we have several ongoing Delphi projects in which we develop and extend in-house, enterprise applications. These projects feel notably lonely (very few developers here in the midwest use Delphi), and the Delphi language leaves a lot to be desired (such as garbage collection) – but the resulting software works very well for our customers, especially when we add in a bit of Lua or Prolog (story coming someday…).

Delphi is not dead. It’s not at the top of the popularity charts, and won’t be. It probably shouldn’t be your first choice for a new in-house enterprise application starting today, because of the network effects of Java and .NET popularity. But Delphi is not going away anytime soon, and is a great choice for certain classes of projects.