What’s the difference between an application and a framework?

An application is something a customer actually wants!

However, for some applications, I recommend use of an application framework. For some Oasis Digital projects, I require it:

Please, Use a Web Application Framework

My reasoning here applies specifically to web applications with many CRUD (create-read-update-delete) features, and an underlying database. The advice applies much more widely, and with many nuances and caveats, but this article I am discussing only CRUD-ish web applications. Even within this niche, my reasoning does not apply to web applications which “push the envelope” of what is possible or which attempt to advance the state of the art.

Regardless of the programming language, the application should be built on a framework. More specifically, the framework should popular and mainstream, with a community of developers, and the appearance of momentum for the future. Likewise, the client-side JavaScript used in these applications, should also be based on such a framework. Here are some examples:

Ruby: Ruby on Rails, IOWA

Python: Django, TurboGears, Pylons, TwistedWeb

PHP: Akelos, CakePHP, CodeIgnitor, Symfony, Zend

Java: Struts, Seam, Rife, Tapestry, Stripes, Wicket Spring MVC

JavaScript: Scriptaculous, Prototype, JQuery

This is just a list of some frameworks that I am aware of; I have not evaluated all of these in detail, and I do not endorse them; nor is this an exhaustive list. For Oasis Digital projects, we help evaluate proposed frameworks, then I personally give the go-ahead to use a particular framework for a particular project.

An in-house web application framework does not meet the “community of developers” criteria, except at the very largest firms. Everywhere else, you are better off with an off-the-shelf, popular framework than with an inhouse framework, even if the latter is brilliantly designed.

Justification

My recommendation (and requirement, for some projects) for using an application framework for this kind of application is not based on a fad. Rather it is based on my years of experience as a developer, a team leader, a maintainer, and most importantly, a customer of software development.

The wild success of some frameworks (such as Ruby on Rails) has shown that they can reduce the amount of code and time needed to develop an application. That second factor, the amount of code specific to the application, is at least as important as the development time. Lines of code are not an asset; they are a liability. Only the features that the code provides are an asset. The most valuable software provides a lot of features using the smallest possible amount of application-specific code.

Therefore, even if a developer is so extraordinarily fast that they can create a system very quickly without using an off-the-shelf framework, they still have provided less value by doing so, compared to creating that same system quickly with fewer lines of code.

Another benefit of using a common framework (not a custom, in-house framework) is that this makes an application much easier and faster for other developers are work on in the future. A more maintainable system is more valuable.

For new Oasis Digital projects, we will be using Git rather than SVN for source control. Here are instructions I wrote to help our teams get started. The contents here are 95% generic, only the references to me are Oasis-Digital-specific.

GitHub

Although Git is a fully distributed source control system, it is very convenient to have a set of robust, central repositories. Oasis Digital’s repositories are hosted by GitHub:

http://github.com/

Github offers a useful set of online features to supplement what Git has built in and available locally. As of the spring of 2008, GitHub is certainly a work-in-progress, I’d characterize is as a “beta” level service. Nonetheless it is worthwhile and recommended. There is a lot to learn from the “guides” published here also:

http://github.com/guides

Install and Configure msysGit on Windows

I assume here that you are using Windows, although Git works very well (better, actually) on Linux or Mac. As I write this, the best Windows Git package is msysgit, available here:

http://code.google.com/p/msysgit/

Make sure to follow the download instructions labeled “If you only want to use Git”. As I write this the download is Git-1.5.5-preview20080413.exe, but get the current version available as you read this instead, not that specific version.

Install by running the EXE installer. Accept the default install directory. When you get to the PATH setting screen, I recommend the “Use Git Bash only” setting, because it avoid any risk of PATH conflicts.

By default, Git will be configured to translate text files between Windows CRLFs (in your working copy) and Unix LFs (in repositories). This setting is fine if you like to use an editor on Windows that insists on Windows CRLFs. I generally use an editor that is equally happy to use Unix LFs, so I sometimes use Git in the other (non-translating) mode.

msysgit includes both the git command line, and a usable GUI. The GUI is not on par with more mature products, but it is helpful and good enough for users who are allergic to the command line.

Creating your SSH Key

The first step in using Git is to create your SSH Key. This will be used to secure communications between your machine and other machines, and to identify your source code changes.

In Windows Explorer, pick any convenient directory, right-click, and choose “Git Bash Here”.

Then type this command:

<code>ssh-keygen -C &quot;username@email.com&quot; -t rsa</code>

(with your own email address, of course)

Accept the default key file location. When prompted for a passphrase, make one up and enter it. If you feel confident that your own machine is secure, you can use a blank passphrase, for more convenience and less security. Note where it told you it stored the file. On the machine I tested with, it was stored in “C:\Documents and Settings\Kyle\.ssh\”

Open the file id_rsa.pub with a text editor. The text in there is your “public SSH key”. You will need it to set up your GitHub account, in the next section.

More details on the key process are available here:

http://github.com/guides/providing-your-ssh-key

Set up your GitHub account

Go to http://github.com/ and sign up for a free account. In the sample here I signed up with a made-up alter-ego, harry@kylecordes.com. Use your own, real email address of course.

Don’t worry that the free account says it is for “open source” work; I will later add you as a collaborator to my paid (and therefore private, non-open-source) projects. Make sure to copy-paste in your SSH public key that you created earlier.

Once you have set up your github account, email your github username (not your password) to me, so I can add you to the relevant project. Once Kyle has added you as a collaborator to the relevant project and send back its URL, you can navigate to the URL, which will look like this:

https://github.com/kylecordes/PROJECTNAME/tree/master

On that page, click the “fork” button to create your own workspace for the project. This will take you to your own page for the project, something like this:

https://github.com/YOURNAME/PROJECTNAME/tree/master

Now you can clone this project to your own machine, as discussed below.

Getting Started Locally

First, use the “Git Bash Here” feature described above, to get a command prompt. Tell Git about your name and email address:

git config –global user.email Your.Email@domain.com
git config –global user.name “Your Real Name”

Then you are ready to proceed with getting into a project.

Copy the “Clone URL” from your github project page (your fork, not my repo you forked from). Make a new directory on your machine, to become your working directory. You can then get your local clone by clicking, or by typing:

Approach 1: GUI

In Windows Explorer, Right click the working directory and choose “Git GUI Here”.

Choose “Clone Existing Repository” in the dialog that comes up:

Paste the URL that you copied above from GitHub; note that your browser might add an erroneous mailto: to the URL, which you must remove – Git URLs do not start with “mailto”. Enter the directory where you want your working copy:

Make sure to use real, reasonable values. For example, you are not working “sample1” and you probably don’t keep your working projects in a directory named “GitStuff”, so put in a directory that makes sense for the project you are working on.

Click Clone. You will be prompted for your passphrase if you used one. In a few minutes the Clone will finish, and you have the project available on your machine. I’ve had sporadic problems with this process hanging (growing pains at GitHub are the likely cause), so if you don’t see progress for a few minutes, stop it and start over.

Approach 2: Command Line

I find this easier. In Windows Explorer, right-click on the working directory you want and choose “Gui Bash Here”. Then enter a command like this:

git clone repoURL

Git might prompt you about an SSH key, the first time you do this with github (or any other new server). Answer “yes”.

It’s worth pointing out here, if you didn’t already understand from the various Git web sites, that Git is a distributed source control system. It will pull down the whole project history, so you can browse history and even commit changes without online access. Thus Git works very well if you have an intermittent or poor network connection.

Work Flow

As with all source control, work in the directory where you use source control. Do not copy files back and forth between here and some other working directory, that is a path to endless merge and update problems.

Once you have checked out the software, here is a summary of your work flow. For more details, please read the copies Git documentation online. I suggest reading both the official Git material, as well as other sites and articles about Git.

Get changes from others with “git pull” (or using the GUI)

Commit your changes locally with “git commit” (or using the GUI)

Push your changes up to your GitHub repository with “git push”. This step will make it so others on your project can see your changes. Do this at least once per day, and ideally more often as you collaborate.

Once you have a set of changes (one or more commit) that you think are ready to go in to the main-line of the project, use Github to issue a “pull request”. Your project lead (me, typically, at Oasis Digital) will review your commits and either pull them in to the main-line, or send feedback about changes needed before they can be pulled in.

A key thing to understand about Git is that it makes branching extremely easy and fast, so that very convenient to use branches. If you are accustomed to other source control systems where branching is a big, painful thing, it will be very different for you in Git.

I’ve only scratched the surface in this introduction. You now have Git up and running with project code in it, so pick up a Git tutorial and start learning.

With some work (mostly outsourced to a Linux guru) I now present a source driver package for the 3.08 “beta” driver version:

comtrol-source_3.08_all.deb

Comtrol ships the driver source code under a GPL license, so unless I badly misunderstand, it’s totally OK for me to redistribute it here.

To install this, you follow the usual Debian driver-build-install process. The most obvious place to do so is on the hardware where you want to install it, but you can also use another machine the same distro version and Linux kernel version as your production hardware. Some of these commands must be run as root.

dpkg -i comtrol-source_3.08_all.deb

module-assistant build comtrol

This builds a .deb specific to the running kernel kernel. When I ran it, the binary .deb landed here:

/usr/src/comtrol-module-2.6.22-14-server_3.08+2.6.22-14.52_amd64.deb

Copy to production hardware (if you are not already on it), then install:

dpkg -i /usr/src/comtrol-module-2.6.22-14-server_3.08+2.6.22-14.52_amd64.deb

and verify the module loads:

modprobe rocket

and finds the hardware:

ls /dev/ttyR*

To verify those devices really work (that they talk to the modems on your RocketModem card), Minicom is a conveinet tool:

apt-get install minicom

minicom -s

Rebuilding the source .deb

Lastly, if you care to recreate the source .deb, you can do so by downloading the “source” for the source deb: comtrol-source_3.08.tar.gz then following a process roughly like so:

apt-get install module-assistant debhelper fakeroot

m-a prepare

tar xvf comtrol-source_3.08.tar.gz

cd comtrol-source

dpkg-buildpackage –rfakeroot

The comtrol subdirectory therein contains simply the content of Comtrol’s source driver download, and this is likely to work trivially with newer driver versions (3.09, etc.) when they appear.

Msysgit appears to have solved the git-windows problem, at least well enough for small scale work. If you’ve been holding back on trying git because you use Windows, now is the time to jump in.

Update: I’ve posted details on how to get started with msysgit and GitHub.

About a year ago I posted a simple math drill program, suitable for grade school students to practice basic math facts. I don’t know if anyone else has used it, but my own kids have enjoyed it. Today I published the (very small amount of) code for the math-drill software on gitorious:

http://gitorious.org/projects/math-drill/repos/mainline

This software is written in Delphi, and though there is some chance that I am the only developer anywhere using both Delphi and git, hopefully this will be wrong, and someone will fork and offer their improvements.

Update: I also pushed it over to github: http://github.com/kylecordes/mathdrill/tree/master

Then we got a new card; it looks like the old cards; but according to the rocket driver it does not exist.

We initially suspected a bad card. We were unable to test with the floppy-image diagnostics on the Comtrol web site, because like most of the rest of the world we stopped ordering floppy drives in new machines several years ago. Comtrol support helpfully pointed us at a CD .ISO for the diagnostics, in here:

ftp://ftp.comtrol.com/contribs/rocketport/diagnostics/rocketmodem/

…without mentioning any reason why this information would not be on their support web site in 2008.

The diagnostics showed a 100% functional card. It finally dawned on me that I might need an updated driver for the current card; and this turned out to be true. The current RocketModem IV is not recognized at all by the rocket module in-the-box in Linux.

Getting it going was an adventure; the short version is that to get the current RocketModem IV to work on any vaguely current Linux kernel, you need to poke around the Comtrol web site and find the beta 3.08 driver, in here:

ftp://ftp.comtrol.com/beta/rmodem/drivers/u_pci/linux/

then do the usual apt-get of a build environment, make, sudo make install, /etc/init.d/rocket start.

Production Deployment

Of course the instructions above are for development / test usage; it is a very poor practice to do the above installation on production hardware, because it makes no accommodation for the inevitable stream of upgrades coming through the packaging mechanism, and because in most cases sysadmins prefer to not have a build environment on production hardware at all (ever).

I am no packaging / Debian expert, but I can see two workable paths to good production deployment for kernel modules like this:

1) Use checkinstall or similar mechanism to create a package which installs the binary modules; set things up so that the resulting package depends on the proper kernel package version. As new kernel packages appear, make new package versions to accommodate them, and likewise for each architecture (i386, amd64).

2) recast the raw source .tar file, as a source-deb package suitable for use with the Debian module-assistant mechanism.

I’d love to hear from any Debian/Ubuntu experts on the relative merits of these approaches.

Perhaps in a few years, things will settle back to the previous bliss of a working driver in the Linux “box”.

Update: I’ve posted a .deb for this here.

Delphi shipped in 1995, and its demise has been declared frequently since 1997 or so. In a sense this demise is true, yet also false. Delphi’s current popularity is very different in form (not only in magnitude) from that of Java, C#, etc. Delphi is used substantially by commercial software vendors, and only rarely by enterprises. An ugly reality of the software industry is that the bulk of software developers nationwide work inside large non-software companies, so this usage pattern most likely does not produce the level of unit sales that Codegear (Borland’s dev-tools subsidiary) would like to see. It does, however, produce an enormous number of Delphi application instances running “in the field”, used by real paying end users, who don’t care (or know) what development tools were used to build the software they buy. Many commercial software products, both those in shrinkwrap at retail stores and those for vertical markets, are written in Delphi and will continue to be, because there are very few other good choices for high quality (polished) native Win32 GUI software. In these markets, shipping a Java or .NET app can be a competitive disadvantage (though to a lesser extent over time), and old-style VB is a sad joke.

I don’t think Delphi is eligible for demise until the dominant desktop operating system ships with a dominant runtime platform “in the box”. For example, if all of this happens at the same time:

• Microsoft ships Windows with the .NET runtime already installed
• That version Windows is the commonly deployed version
• That version of the .NET runtime is the commonly targeted version

At that time, the .NET platform (with the language of your choice) could be a compelling replacement for Delphi in its niche. There is a lot to like about .NET (and Java, and I use them both), but I’m not holding my breath for the above conjunction.

Over at Oasis Digital we have several ongoing Delphi projects in which we develop and extend in-house, enterprise applications. These projects feel notably lonely (very few developers here in the midwest use Delphi), and the Delphi language leaves a lot to be desired (such as garbage collection) - but the resulting software works very well for our customers, especially when we add in a bit of Lua or Prolog (story coming someday…).

Delphi is not dead. It’s not at the top of the popularity charts, and won’t be. It probably shouldn’t be your first choice for a new in-house enterprise application starting today, because of the network effects of Java and .NET popularity. But Delphi is not going away anytime soon, and is a great choice for certain classes of projects.

• Celko wrote about it as “Path Enumeration” his 1995 book “SQL for Smarties: Advanced SQL Programming” (now in its third edition)
• I’ve seen it listed in various articles and web sites
• The relevant page of documentation for the Mondrian OLAP server calls it a “transitive closure table”. There is a page on their wiki with a tool to create them, and a forum post with Java code for this purpose.
• Steven Lott’s “Creating a Transitive Closure to Optimize Rollups” article
• Most recently, I saw this called an ‘ancestor table’ at evolt.com.

There are two (main) places to put the code for building a closure tables: in your application code, or in your database. The application approach is nice if you are aiming to avoid vendor-specific SQL code, but it is quite simple in SQL, and therefore not a big problem to recode for another RDBMS if the need arises. The SQL approach also avoids round-tripping the relevant data in and out of the database. Therefore, the approach I generally recommend for this is an SQL stored procedure / function.

Here is a simplified PostgreSQL stored procedure to do the job; note that his assumes a “widget” table with a widget_id and parent_id (the “adjacency” representation of a hierarchy), and a widget_closure table with fields (ancestor_id, widget_id, depth). Wordpress has unhelpfully discarded my indentation:

CREATE OR REPLACE FUNCTION populate_widget_closure()
RETURNS integer AS ‘
DECLARE
distance int;
BEGIN
delete from widget_closure;

insert into widget_closure
select widget_id, widget_id, 0 from widget;

FOR distance in 1..20 LOOP
insert into widget_closure
select uc.parent_id, u.widget_id, distance
from widget_closure uc, widget u
where uc.child_id=u.reports_to_id
and uc.distance = distance-1;
END LOOP;
RETURN 1;
END;
‘ LANGUAGE plpgsql;

This sample code assumes a maximum depth of 20, and has no error checking. It will blindly miss greater depths and produce garbage if there is a “loop” in the ancestry. I recommend both arbitrary depth handling and error checking for production use.

Once your transitive closure table is populated, querying it is extremely fast and simple. You can get all the descendants of widget 12345 with the query “select widget_id from widget_closure where ancestor_id=12345″. Of course, this hierarchy representation, while simple to generate, is not simple to incrementally update as the hierarchy changes. The most straightforward way to use it is as a cache, regenerated as needed.

Shoes creates native applications with a Web feel. It is not built atop GTK, Qt, etc., but rather directly on the relevant native API (Win32 on Windows, for example). Shoes has a lot of potential, which it can realize only if a community accrues around it, leading to lots of polish and a rich widget set.

One thing that surprised me about the Shoes install it that, as far as I can tell, it includes its own Ruby install; I installed Shoes on a machine with no (obvious) Ruby installation and it worked.

How well you speak can have a great impact on  your ability to get customers, to attract employees, to persuade others to adopt your ideas, and much more. But how do you know how well you speak? You can’t tell as you are speaking; and if you’ve never heard yourself speak but simply assume you do well, there is a great risk that you are very, very wrong.

To become a better speaker, you need to review your own “performances”. For the last few years I’ve been recording my talks (at user groups etc.) with a hand-held voice recorder. I use an Olympus WS-100 (probably obsolete by now) and you can readily find an adequate device for $70 or less.

Make the Recording

Don’t actually hold the recorder on your hand. Sit it on the podium if you use one (I don’t care for podiums, I use one only if I am presenting with my computer and the podium is the only place to put the computer); but not to close to your noisy computer. Put it on a table or unused chair in the first row, but not too close to any noisy people or equipment.

Start it at the very beginning, and stop it at the very end. It is easy to edit out extra recording later.

Put in a fresh battery, so it doesn’t run out partway through.

Listen to the Recording

Take the recorder/player in your car, copy it to  your iPod, burn to CD, whatever… but listen to it, end to end. Experience your talk as a member of the audience. Make notes about any bad habits you exhibit, such as “umm”, “ahh”, “like”, “ya-know”, long (unintentional) pauses, pointless extra words, etc. This is your first opportunity to become a better speaker by means of the recording. WARNING! The first few times this might be very, very painful.

Edit the Recording

If you plan to post this audio online (as I usually do),  clean up the recording to trim off any extra starting and ending time; the audio you post should start immediately with your talk content and end crisply. Download Audacity (or another audio editing) and use it for this initial trim.

If you can spare the time (approximately 2x the length of the talk), listen through the whole recording in Audacity, trimming out the “umms”, “ahs”, overly long pauses, etc. This is your second opportunity to become a better speaker… but only for the offline listeners, and in an artificial, one-off way. Still, for an occasional important talk it is worthwhile.

Get a Transcript

Finally, be aware that your (now-pristine) audio is completely unsearchable and invisible to Google. To make it findable, have a transcript made and post that also. You can pay CastingWords or other similar service to create a transcript for around a dollar per minute, or perhaps much more cheaply with some searching. The resulting text can be readily posted online, making your talk both more findable, and more accessible.

An example of a recorded, lightly edited, and posted talk (with transcript) is my Ruby GUI talk earlier this year.

How do you attract top developers to work on something that sounds rather dull, like a bug tracking application? It mostly shuffles data back and forth between screens and reports and database tables – far too boring for top developers.

Of course that’s an exaggeration, but a relevant one: at Oasis Digital much of our work is on enterprise business process automation, database-centric applications, and could likewise be described casually (though not accurately) as “just” shuffling data between screens and tables. I worry that our work will not sound interesting to prospective hires.

This week at the Business of Software conference I got a chance to ask (confront?) Joel about this. He offered a great, four-part answer, which I present here with my own additions mixed in. I don’t have careful notes about which bits came from Joel, so you are welcome to give him all the credit and me all the blame.

1) There is Interesting Technology Inside

Even in an application which, at first glance, just shuffles data around, there can turn out to be a lot of very interesting work inside. This is true of FogBugz and it is true of our work at Oasis Digital as well. Here are some examples of interesting work here, all of it inside dull-sounding applications:

• Process metadata and generate code and GUI elements. Top developers certainly are those who solve a family of problems with generic code and metadata, rather than tediously one at a time.
• Process large hierarchies efficiently using Celko’s nested sets representation technique. Top developers are all about using better data structures.
• Custom GUI components to provide a drag-drop, direct manipulation approach to visualize and modifying data. The results has both a high “wow” factor, and is genuinely useful – a willing combination for top developers.
• Integrate a Prolog-based rules mechanism to provide a vital algorithm in one page of code, that would have required countless pages of code and hundreds-to-thousands of hours of work to do otherwise. Using a radically different language to solve a problem with a small fraction of the effort… exactly the sort of thing a top developer wants to do.
• Generic data replication mechanism: building our own was certainly more interesting work than adoptions one off the shelf.
• Learn how OLAP works, implement an OLAP ETL process.
• … I could list many more examples

2) One Level Down the Stack

Fog Creek aims to make compellingly good software because that is how you outcompete established competitors in a commercial shrinkwrap market. Oasis Digital likewise has this aim for a different reason: it is our intended niche. We aim to differentiate ourselves from “yet another outsourced dev company” by building unexpectedly good software. We don’t want customers who will be happy with the results available from the typical development firm; we want customers who are playing to win.

To meet either goal, it is sometimes necessary to work at one level of abstraction lower than would otherwise be necessarily. Joel’s examples were their own data grid and their own AJAX library. Some of our examples are listed above.

This kind of work, further in to the details, is generally more compelling to top developers.

Caveat: Don’t do this very often. If you want to ship software anytime soon, you need to mostly use off the shelf libraries that already work. Don’t build a data-grid, for example, unless you really need something that you can’t find in any off the shelf products. We don’t have our own data-grid; we use (among other things) the excellent grid products from Developer Express.

3) Problem Domain is less important than other factors

Joel has observed that developers aren’t as picky about the problem domain of their project as one might think; rather, other factors are more important: great co-workers, nice working environment, working for a boss who is a developers, etc. Top developers want to work on a high quality end product worthy of taking pride in.

4) All Projects need a lot of Grunt Work

To build a high quality product, in any problem domain, will require spending a great amount of your time on grunt work: tracking down bugs and fixing them, filling in feature “holes”, cleaning up design problems, improving GUI layouts, and the like. This is true for any problem domain. Top developers know this, and just get on with doing that work when needed.

Conclusion

My worry was unjustified. Our work at Oasis Digital is interesting and worthwhile, both for our customers and for our developers. To grow our team, we must focus on making it an increasingly good place to work.

A final anecdote: Joel mentioned that FogBugz 6 was feature-complete in the summer of 2006 – that means they spent around a year polishing it, fixing bugs, filling in holes, etc. That shows a phenomenal amount of dedication and discipline to create quality software.