Opening up

Lottery Lolly

Twenty years ago, before regular people had the web, I was a 13 year old discovering a hybrid database/scripting application called HyperCard. One of my early projects was a random number generator for the newly-launched National Lottery, which incredibly, made it onto the cover floppy disk (‘3MB of great software!’) one of the early editions of Macformat magazine. I was smart: I didn’t put my home address on it, but I did put my school address in the Read Me. So for several weeks after, they came: little letters in the class register with exotic postmarks, from Scotland, Cyprus, Germany, Australia. They contained patient advice on how to improve my randomisation algorithm (it sometimes generated the same number twice). Versions ported to Pascal, or with a nicer UI, or ideas for new features. Some even included the odd financial donation too. My 13 year old friends were in equal parts bemused and impressed.

I never stopped fiddling with code, though I never properly trained in it either. Fast forward ten years, and I was releasing an open source DIY survey package I’d built because I was annoyed with SurveyMonkey – and the feature requests, support queries and very occasional donations came too, this time by email. Four years ago, I was blogging here, and releasing WordPress themes for consultations, as a salaried civil servant. Still the emails and ideas came: lovely, exotic, motivating, time-consuming, rewarding, emails.

Now I run a micro business and coding is for food as well as for fun. Even more than ten years ago, we depend on open source projects and the altruism of others in forums and blog posts, troubleshooting and extending them for us – making our lives easier and helping us make a profit from their work through the services we offer our clients. So when people have understandably asked if we’ll be releasing the code behind our work for DCMS on the intranet, the memory of twenty years of lovely, exotic, time-consuming emails coming flooding back before I feel ready to say: yes. Yes, of course.

To do otherwise would be hypocritical, and counterproductive. And our strongest ‘competitors’ are well ahead of us on this: the Simons at Code for The People have been contributing their plugins to the WordPress directory and playing an integral part in core releases too for years. Harry at DXW has been submitting patches and worrying about password hashing in WordPress core too. So it’s about time we started sharing more of our code and contributing back a bit more, and building time to handle those lovely emails into our business model. These days, GitHub and other tools make that easy.

I think there’s an important difference between saying ‘you can use our stuff, it’s on GitHub’, and actually helping the writers of those lovely, exotic, time-consuming emails to use the stuff that’s been released in their own work – many of whom aren’t especially expert developers but see the appeal of reusing work from elsewhere. We work mainly in WordPress, a massive community project with a vibrant ecosystem of plugins and themes where people expect to find answers ready-made, reliable, and free (and they’ll complain bitterly, without justification, to altruistic plugin and theme developers when they don’t). So releasing a WordPress thing isn’t just an extra commit, it’s a proper commitment.

Luke’s written more about what we’re doing, but in a nutshell, we’re releasing a tidied up, customisable version of our work for DCMS which anyone can pick up and use on their own intranet-style site. We’re started the process of documenting it better, there’s a demo you can play with, and we’ll do our best to answer questions. Thanks to GitHub, technically minded people can contribute their own improvements, so it hopefully becomes a small community project of its own.

We’ll also be introducing a new section of this site to bring together information about our code releases, where they are, their requirements, current status and more. In some cases, we’ll submit them to the WordPress plugin/theme directory too.

Open sourcing your core outputs feels like a scary thing to do as a micro business when they’re so easily re-usable by others working in the same sectors. But we reckon we’re about more than just code, as Luke’s blogging over the years clearly illustrates.

Make things open: it makes things better, as some wise folk once said.

Whitehall in WordPress

Simon Dickson introducing WordUp Whitehall 2

The clue was in the 150pt slide Simon put up at the start of the day, but actually I think he was just catalysing a conclusion most of the participants in Whitehall’s second WordUp event came to independently: we’re on the verge of a new era of Sharing.

12 months ago, only Number 10 and Defra were running WordPress-based corporate sites. Now the Departments of Health and Transport are too, accompanied by GCN, FCO, DCMS and old hands BIS and DFID all running significant WordPress projects. Whereas last year, just launching a WordPress site in government felt brave and radical, now it’s virtually mainstream – thanks in no small part to Simon’s blog and organisation of WordUps. The environment for WordPress in Whitehall is certainly benign, with the new open source procurement toolkit, and a man in charge who thinks the old ways are, well, unacceptable.

So we’re at a new, and quite challenging frontier now. Whereas a year or two ago, we sought permission to use this little blogging tool for serious things, now we’re asking ourselves how to make it do those serious things better. How to handle a library of 60,000 legacy PDF files, or structure content across a network of sites using an agreed classification. How to run a network of 50 bloggers more efficiently, or offer users great organisation-wide search. How to build vibrant social networks of government communicators, or create tools accessible to the highest standards.

And the answer to many of these is: it’s time to tackle these challenges as a community, just as the founders of the open source project we’re all using originally intended.

Two different perspectives today came from Pete Westwood, one of only 6 WordPress Lead Developers worldwide, who talked about the philosophy and management structure of the WordPress project; and Paul Gibbs, a Core Developer of BuddyPress, who shared his journey from hobbyist to open source project leader. Whereas the rest of us mainly talked about making WordPress jump through hoops, they presented it as a project – a community, and an often personal journey – rather than just software.

And that’s the challenge for WordPress in Whitehall: whether and how to become Whitehall in WordPress.

The plugins, themes, widgets and thinking on show today were impressive, but there’s clearly scope to help solve each others’ problems more quickly, effectively and cheaply by sharing what we’ve learned and created. And if the Whitehall community can benefit, why not the wider WordPress community which has provided us with the tools we use? Collaboration – through submitting bug reports and patches, writing reviews, rating plugins, contributing plugins, creating translations and documentation – is baked into the core of WordPress, and the tools to make it easy, like Trac and the Plugin Directory, are there waiting for our input. The White House has done it, albeit for Drupal.

But it’s not a no-brainer. Money and human resource are tight. Timescales in government are often challenging, and talented developers are in demand. On the outside, those who supply WordPress to government are often small agencies and freelancers themselves, with projects to finish and new business to win.

Contributing to open source projects takes time, patience and commitment. The benefits are often indirect. It’s hard to measure the satisfaction of seeing your code appreciated and reused, and it’s a visionary boss who recognises the value in spending paid time on it.

Of course government stands to save money through greater use of open source, and it’s great to hear repeated commitments to promote reuse across the public sector. But what would be truly visionary would be to embrace the ethos of open source itself and commit the time of civil servants and part of the budget of commissioned projects to giving back to the open source projects which made them possible. That’s what Simon was on about when he caused a bit of a stir.

Sharing code on your own site is a start, and using code-sharing platforms like Github is better, but let’s go the whole hog and contribute back to the projects themselves. I’m making that my early resolution for 2012.


How to open source your stuff

One of the most rewarding things about open sourcing your stuff – for example, some code we developed for an internal project – is seeing what clever people do with it in ways you wouldn’t expect.

At my talk at eDemocracy 08, I put up the chart below to try to illustrate four of the key areas in which open source helps to build community, and why you might want to open source your stuff:

Open Source

  • Support: a group of people who talk about, and can help each other solve problems with, the code
  • Direction: people who contribute ideas about where the code should go in future, what features would be useful and so on
  • Rigour: more eyeballs and fingers exposing the weaknesses and limitations in your code, and gently pointing them out to you (and sometimes fixing them themselves)
  • Marketing: advocates who talk about your code online, and help spread the word – generating great PR for your organisation

In spite of the UK Government’s new action plan, there are some understandable and usually fatal barriers to open sourcing the code and information we produce as civil servants:

  • Ownership and sunk costs: We paid for this. Why give it away for free? Or at least, only give it to other nice public sector people.
  • Licensing: Is it Crown Copyright? What’s Click-Use? Who decides this stuff anyway? Let’s not bother the lawyers with this trivial stuff – they probably wouldn’t know anyway.
  • Publishing: How do you publish open source stuff? Can’t really put it on the corporate site. Don’t you need some complicated version control repository?
  • Documentation & support: I can’t be bothered to take the hard coded values out of the scripts and comment it all. I’ve got a day job – what if people keep asking for help using the code?
  • Lack of incentive: I’ve got more urgent and important things to do. Nobody is chasing me (or paying me) to do this.

That list certainly explains why I never got round to it before. Sure, doing open source really well – a true community endeavour like Linux or WordPress – can take a lot of work. But in my experience open sourcing your stuff can be done fairly easily:

1. Clean up the code (a bit)
Make a copy of your scripts. Go through deleting the obsolete bits, the dead ends and the useless commented-out stuff. Where you can, move your hard-coded variables to a configuration file, or at least move them to the top of each script and comment them clearly. Double check you’ve taken out the hard-coded email addresses, usernames and passwords for your server. But don’t go mad trying to refactor everything to be super-efficient- getting it out there is more important than getting it perfect.

2. Write a Read Me
Create a new text file and call it ‘Read Me’. Describe the code briefly, give it a name and a version number. Make sure you’ve credited other people whose code you’ve used, and link to the original libraries (or include them in your package for redistribution). Describe the steps for installation, any special system requirements, known bugs and any beartraps to watch out for. If you’re releasing an update, briefly add a description of what the new version offers. If you can’t realistically provide support, say so – people will understand. Add a disclaimer so users are clear that liability for the code once they use it is theirs, not yours. But do provide an email address or URL for feedback in any case for people to help you by reporting bugs and requesting features. Worst case, you can always just file the emails until you have time.

3. Decide on your licence
(This is slightly squiffy so tell me in the comments if I’m advising people wrongly on this)

If you wrote the code on your employer’s time, and you’re working for the civil service, it’s probably Crown Copyright. That’s who formally owns the code. But you can decide how that property is used. As a rule, if you want to enable people to be free to re-use your information, the OPSI’s Click Use licence is a fairly simple and very flexible licensing scheme. But if you’re talking about code, it’s a bit of a grey area. Smarter people than me reckon it’s OK to use the permissive GNU Affero licence, which allows people to adapt the code without forcing them to republish modifications themselves, and removes even the small barrier of requiring your users to sign up for a Click Use licence. Make sure you describe and link to the licence you’re using in your Read Me file and preferably within the main parts of the code itself.

4. Set up a static page somewhere, upload your code to it, and promote it
One day, there might be a lovely Sourceforge-like open source repository for government code. In the meantime, you can always just put up a nasty-looking page or a simple blog with a description and a link to a Zip file of your code. Or you could use Sourceforge or Google Code if you don’t mind too much about the branding. Then tell people about the code on your blog or via Twitter.

5. Provide a feedback channel
Ideally, set up a blog or forum where people can come together to discuss your code, report bugs and request features. At the very least, promote a URL, email address or Twitter handle where people can reach you to tell you when it breaks their server (sorry again, Paul).

The point I’m making here is that you don’t have to be purist about open sourcing your code. Feel good about making a contribution, not bad about the mess you think you’re exposing. Feel happy that you’re letting other taxpayers benefit from your time and experience, not anxious about losing control over ‘your work’. And above all, just get version 1.0 out there for people to use, critique and improve.

ConsultationXML goes open source

ConsultationXML screenshot

As Harry’s explained over on The Dextrous Web, we’ve just open sourced the code behind ConsultationXML, the tool we’ve been working on which turns consultation PDF files into XML. It’s a 0.1 release so still full of bugs, but at least it’s out there.

You can download the (documented) code, and/or have a play with the sandbox version first.

Open sourcing our code is the kind of thing the new Open Source Action Plan incites government to do more. But it’s generally low down the list and – frankly – documenting and hosting proper code repositories is beyond the humble skills of most clients (including me) and even a good many developers (but happily, not Harry).

Please help us demonstrate that this is important work worth investing time and effort in, by helping us to improve the tool with your feedback and ideas.