Strengths (for us)

• It is a “programmable programming language”, especially good for building abstractions – the usual LISP strength.
• It is biased toward immutability and functional style (though many things are mutable, because of JavaScript underneath); yet can easily escape when pragmatically needed.
• Is it is possible to use the same(-ish) language on the client and server; our projects benefitted from this for years in the boring old days of desktop client-server development.
• Client side ClojureScript is built on JavaScript runtimes and libraries: proven, strong, dominant, multiplatform.
• Server side Clojure is built on Java runtimes and libraries: proven, strong, dominant, multiplatform.
• Although there are not a great number of ClojureScript developers, they tend to be good developers.

Weaknesses

• Although parentheses-first syntax does not bother me, it does bother some developers.
• ClojureScript is different enough from JavaScript that knowledge of the latter doesn’t help much with the former;
• Yet, it is still necessary to know JavaScript well.
• There are at least 100x fewer ClojureScript developers than JavaScript developers; perhaps 1000x.
• Fits well with some JS libraries, less well with others – sometimes that prevents the “shove in an hack” style of development which is JavaScript’s strength and weakness.

1. JIRA CSV import can import attachments. It fetches them from URLs. Beware though that it won’t accept alternate ports. http://host/whatever is OK, http://host:nnn/whatever is broken. Moreover, is broken with a completely incorrect (403) error. (Undocumented.)
2. CSV import can import comments. The format is documented on their site, and arcane, and permanently leaves a trail of extra data in the comment text(!). But it works.
3. CSV import will import events as of date-time as imported. Only one date format is supported for comments; though it date format can be changed for  issue-level fields. The date format does not include a time zone. By trial and error we learned that the datetime is interpreted as the time zone which the JIRA server (at the OS level) is set to. (Undocumented.)
4. JIRA OnDemand servers are set to the US Central time zone, or at least they pretend to be for us. YMMV. (Undocumented.)
5. JIRA CSV import will automatically create-as-needed users if they are mentioned at the issue level (as an issue creator etc.)… but will only generate an error about missing users at the comment level.
6. CSV import can import parent-child relationships, but not other kinds of relationships.
7. The JIRA REST API is tempting – but it can’t assign correct historical date-times to anything, so it’s generally not useful for importing data.

In spite of those rough edges, we were able to get the job done, and are happy with JIRA.

If you don’t have Flash installed (and thus don’t see the video above), you can try this direct link to the video. It plays well on most platforms (including an iPad).

It is also available on YouTube.

Instead I wrote a new tool, in Clojure. The following tools and documentation moved things along quickly:

• Github v3 API
• Clojure
• Clojure API wrapper for the Github v3 API: (Thank you Anthony Grimes)
• Doc for the above
• Clojure CSV library
• Clojure Date Time library
• Doc for the JIRA import format
• Doc for the JIRA comment import
• KB article for comment importing, in particular, ”settings.advanced.mapper.comment” : “com.atlassian.jira.plugins.importer.imports.csv.mappers.PvcsComment”

Download the GHI -> JIRA CSV tool now:

https://github.com/kylecordes/ghijira

A very irritating caveat is that as of June 2012, the resulting file works great with the current downloadable JIRA, but fails with JIRA OnDemand (the SaaS version). JIRA support confirmed the import problem is due to a slightly older version of the import tool used in the SaaS offering – by the time you read this, it might already have been upgraded and fixed.

Update: To help those who aren’t here for the Clojure but just to solve their GHI->JIRA problem, I added more instructions to the README file (see github project link above) explaining how to use the software.

It seems likely that I’ll adapt (add minor improvements to) one of the above to yield yet another solution for this, and perhaps publish it here, so there will then be at least four choices for the next person to find.

A lot of small problems end up in this condition: many semi-solutions, no clarity on which to use. Is that what meet users’ needs best? Certainly not. Perhaps Atlassian will step in eventually with a broad, well-tested set of issue importers for JIRA. JIRA appears to be a market leader, this is an opportunity to step up and earn that role.

UPDATE: I made a new export tool, available now.

Applications are now closed.

It’s almost time for our Summer 2012 internship program. You might want to look back to our 2011 program, or our inspiration, the Fog Creek program. Interns at Oasis Digital work on serious, complex projects, though the work is generally exploratory in nature (rather than part of our enterprise development work for customers). Interns write software, test software, write about software, etc. The purpose of the work is to educate and enlighten the interns, and also create something useful to Oasis Digital and its customers.

Who?

We plan to hire two or three student interns.

Unlike some programs, ours is for high school students, or recent graduates about to enter college.

How Long?

The work will last about 10 weeks, from the first week of June through early August. Interns should expect to work around 30 hrs per week, though only a small portion of that will be on-site in-person. Other than a few fixed meetings per week, your schedule will be flexible.

We can adjust the dates if needed, to accomodate college start dates, family vacations, and so on.

Where?

Interns will meet at the Oasis Digital office two or three times each week. Our office is in Chesterfield, MO. Therefore, interns must live in the St. Louis metro area and have transportation available.

At these office sessions, we’ll all work together, and interns will learn from various Oasis Digital developers. Outside of that, interns will work alone, and meet to work together, for a total of around 30 hours per week.

Does it Pay?

Yes, these are paid internships. Each intern will earn about $2200 for a full summer of work.

In additional, our meetings will be scheduled to include a company-paid lunch a couple of times each week.

What Will We Build?

The project for this summer will be a workflow management system, with web and mobile-app (smartphone, tablet) interfaces.

Interns (with help from Oasis Digital staff) will create an end-to-end system:

1. Database Schema
2. Server / backend services
3. Web client application
4. Mobile (native or HTML5) app
5. Tablet (native or HTML5) app
6. Document and deploy all of the above

The schedule goal is to develop an extremely simple version of the above, and “ship” it in working condition, halfway through the summer. Then with the rest of the summer, add interesting features, platforms, polish, etc. GPS, mapping, photo support, multitouch clever interfaces, anything is possible.

More importantly than what we will build, is what everyone will learn. Expect to learn more in this summer than you would in many classes.

What Language?

The exact toolset will be worked out by the interns, and may include:

• JavaScript
• Node
• JQuery and similar libraries
• Java
• Clojure
• Git / Github
• HTML (including HTML5)
• PostgreSQL or MySQL
• Linux, Windows, OSX
• iOS, Android
• Various other open source tools and libraries

Requirements

• Excellent command of written and spoken English
• Currently enrolled in high school, or just graduated.
• Top grades or a track record of success
• Permanent legal right to work in the United States
• Great (for your age) computer programming skills
• Live in the St. Louis MO metro area
• A computer available to work on, when you’re not at the office

How Do I Apply?

Apply via our job application page, not by email or phone. This is the same mechanism we use for hiring non-interns, so it is perhaps overly complex; we expect intern applicants to have a short, one-page resume listing interests and personal projects, not professional work.

Please apply by May 15, 2012. 

Today’s topic is how numbers should be formatted for tabular display. This comes up quite frequently in data-centric applications. For an example, consider this mockup of a working-hours-per-person display screen, example 1:

It looks quite nice with the numbers formatted trivially as whole numbers. What if someone worked 4.56 hours, though? How would that appear? To accomodate that possibility, you might always show two digits to the right of the decimal point like so, example 2:

… which is OK, but not great – all those extra zeros distract the viewer from the essence of the data. One thing you should never do, though, is something like this:

This is horrible. If you feel tempted to write software that displays data like this intentionally, please step away from the computer now. This breaks the rules of writing numbers that we all learned in primary school, probably before the age of 10: always line up the decimal point. Instead, for that particular set of numbers, the is reasonable:

So is the answer to always display as many decimal digits as the data could possibly have? Perhaps, but only if you are unable to dynamically change the format based on the contents of the data. Examples 2 and 4 show a safe but unimpressive approach.

If you have the tools and skills to create a high quality solution, aim higher: dynamically choose the right number of decimal digits, to fit the specific data in question, then apply that same format uniformly to all the numbers. The output will therefore look like #1 if the data permits, and like #4 if the data requires it, but will not needlessly fill a column with 0s as in #3. This is more work, but is a more polished, professional result.

However, we do something perhaps even more “extreme” than pair programming: we spend a few hours every week programming in larger groups, sometimes as many as three of us in person and a couple more remotely. Why would anyone do that?

• To attack particularly hard problems
• To resolve important detailed design issues
• To share our programming style and culture
• To freshen old skills
• To build new skills
• To efficiently pass knowledge 1->N, rather than 1->1

I don’t quite know what to call this. Group programming? Cluster programming? N-tuple programming?

Regardless, we encounter an unavoidable issue: it is not pleasant for several people to cram in front of a PC monitor, even a large one, closely enough to read it. We’d rather spread out, particularly for sessions lasting a couple of hours straight.

In the past at other firms I’ve solved this by working in a conference room with a projector. This doesn’t work very well. Most projectors have a maximum native resolution of 1024×768, or occasionally a bit higher, and those with reasonably high resolution are quite expensive. The reward for spending that money is continuous fan noise and an exhaust heat plume blowing on someone sitting on the unlucky side of the table.

This time we went a different direction: our group programming lair features:

• 40-inch LED-LCD HDTV, with a native resolution of 1920×1080, which generates no noise or heat, at less cost than a mediocre projector
• A dedicated computer, so noone’e development laptop is occupied
• wireless keyboard and mouse can be easily passed around
• Speakers and a standalone mic, for very clear Skype audio
• Nearby tables to accomodate everyone’s laptops, with 22/23 inch extra displays available

It isn’t pretty:

It is very effective. We can comfortably work together in this area for hours, easily reading the screen from 6-8 feet away. (As I write this, I note that the room needs better chairs, a less beige color scheme, a printer that isn’t older than my teenager, and more Apple hardware.)

This approach isn’t for everyone; it requires a willingness to move furniture and buy non-standard equipment. I’d love to hear from anyone else doing something similar. In particular, I wonder how it compares to Pivotal’s setup.

If you haven’t already done so, now would be a good time
to watch Steve Jobs’ 2005 Stanford commencement address.

Serious Work

When some people hear (or say) the word intern, they imagine someone who makes copies and fetches lunch. Perhaps in some work cultures that is helpful, but here we have in mind serious work. Our interns will work on serious work:

• Projects that are helpful to our company operations
• Exploratory programming, to try out ideas that don’t yet warrant commercial attention
• Open Source programming, such as improvements to tools we use
• Perhaps even a few bits of our real customer projects

The kinds of work could include…

• writing software
• testing software
• reading and writing about software
• marketing work around the software

Twofold Purpose

The purpose and goal of our internships will be:

1. Educate and enlighten the intern
2. Hopefully also create something useful to Oasis Digital and its customers

Paid vs Unpaid Internships

Many companies are eager to get free help from interns. But it’s clear from a few minutes research that a for-profit company with an intern working on potentially valuable project, must pay. Therefore, we will only offer paid internships.

Location

The internship work will be conducted at our at Oasis Digital office; interns must live in the St. Louis metro area.

Can I Apply?

Sorry – No. We’ve already selected an intern for our trial-run 2011 program. If we decide to continue and expand it for 2012, it will be announces in the spring (of 2012).

The problem domain was the storage of check images for banks; the images came out of a check-imaging device, a very specialized camera/scanner capable of photographing many checks per second, front and back. For example, to scan 1000 checks (a smallish run), it generated 2000 images. All of the images from a run were stored in a single archive file, accompanied by index data. OCR/mag-type data was also stored.

I don’t recall the exact numbers (and probably wouldn’t be able to talk about them anyway), so the numbers here are estimates to convey a sense of the scale of the problem in its larger installations:

• Many thousands of images per day.
• Archive files generally between 100 MB and 2 GB
• Hundred, then thousands, of these archive files
• In an era when hard drives were much smaller than they are today

Our client considered various off-the-shelf high-capacity storage systems, but instead worked with us to contruct a solution roughly as follows.

Hardware and Networking

• Multiple servers were purchased and installed, over time.
• Servers were distributed across sites, connected by a WAN.
• Multiple hard drives (of capacity C) were installed in each server, without RAID.
• Each storage drive on each server was made accessible remotely via Windows networking

Software

• To keep the file count managable, the files were kept in the many-image archives.
• A database stored metadata about each image, including what file to find it in.
• The offset of the image data within its archive file was also stored, so that it could be read directly without processing the whole archive.
• Each archive file was written to N different drives, all on different servers, and some at different physical sites.
• To pick where to store a new file, the software could simply look through the list of possibility and check for sufficient free space.
• A database kept track of where (all) each archive file was stored.
• An archive file could be read from any of its locations. Client software would connect to the database, learn of all the locations for a file.

This system was read-mostly, and writes were not urgent. For writes, if N storage drives weren’t available, the operator (of the check-scanning system) would try again later. CAP and other concerns weren’t important for this application.

Helpful Properties

• Even if some servers, sites, or links were down, files remained generally accessible.
• Offline media storage could be added, though I don’t recall if we got very far down that path.
• The system was very insensitive to details like OSs, OS versions, etc. New storage servers and drives could be added with newer OS versions and bigger drive sizes, without upgrading old storage.
• Drives could be made read-only once full, to avoid whole classes of possible corruption.
• By increasing the number of servers, and number of hard drives over time, this basic design could scale quite far (for the era, anyway).

This approach delivered for our client a lot of the benefits of an expensive scalable storage system, at a fraction of the cost and using only commodity equipment.

Why do I describe this as cloud-like? Because from things I’ve read, this is similar (but much less sophisticated, of course) to the approach taken inside of Amazon S3 and other cloud data storage systems/services.

Key Lesson

Assume you are willing to pay to store each piece of data on N disks. You get much better overall uptime (given the right software) if those N disks are in N different machines spread across sites, than you do by putting those N disks in a RAID on the same machine. Likewise, you can read a file much faster from an old slow hard drive in the same building than you can from a RAID-6 SAN across an 2000-era WAN. The tradeoff is software complexity.

As usual, I’ll zoom through some slides, and concentrate instead on the code. For some background on Lua, you may want to watch the video of my 20-minute Lua talk from last year’s Strange Loop.

Update: slides are available here.

I heartily recommend Strange Loop for any software developer interested in learning more about a wide variety of technical topics. Unlike many other events, this one stays close to the technology all the way through – you might see a higher ratio of code-to-text on the slides here, than at any other conference.

(Again this year, my firm Oasis Digital is a sponsor, and I’ll probably submit a talk. I hesitate a bit though, because if I give a talk, I have to miss someone else’s talk in that timeslot.)