The Traindomiser: a train adventure

We were having a family chat over breakfast one Sunday morning about the kind of holidays we enjoy:

“Somewhere interesting & exciting”
“Somewhere not too bleak”
“Go interrailing”
“Go somewhere on a beach”

As it’s a truth commonly acknowledged that a chap in possession of a weekend must be in need of a coding project, my son Arthur and I set to work to come up with something that might tick these boxes. We were aiming for something that might help us go on a train adventure this half term through the UK, exploring (non-bleak, possibly seaside) places we’ve not been to before and staying wherever Airbnb or might find for us that night.

The result is: The Traindomiser. We found a list of UK towns and cities, and explored the excellent Transport API which provides free access to all kinds of things, from bus routes to train fares, and – most useful for our purposes – helps map a place to a nearby station. We coded up a randomising algorithm that will take a maximum distance you want to travel, and suggest a place to suit, along with the train route to get there from a nearby station. We learned how mobile devices can request your location and pass it to a form, fiddled around with the search results URLs for Airbnb and, and added a bit of suspense with some animation of the results. We learned to curse Javascript until a StackOverflow answer emerged, an important skill for any coder.

We’ve been preparing for our inaugural Traindomiser trip this week. We’ve got our Family & Friends railcard and a vintage map of the rail network (sadly out of print now). We’re travelling light, and seeing how things go. The goal: get to where the Traindomiser picks for us each evening of the trip, assuming we can find a place to stay while on our way. We’ll stop off at the interesting points on the journey, recognising that exciting adventures can come from unexpected places when you’re small. (Hopefully nowhere too bleak.)

We’ve built a little travel blog to track our first adventure with the app, and we’ll post updates there from our trip. And hope the Traindomiser can get us back home before school starts on Monday…

UPDATE: We made it home. 4 random destinations in 4 days. Lots of pics and a few reflections on the trip over here.

Good and bad transparency

I’m trying to work out why I’m uneasy about the recent announcement of the transparency revolution sweeping Whitehall. It’s not as if I’m a civil servant any more with a vested interest to conceal. As a developer, and an SME hoping to win public sector work, and a taxpayer, you’d think I’d be cock-a-hoop that I’ll be able to hold officials and politicians to account more easily from now on.

But I’m uneasy, and I think that’s because I’m troubled by the nagging doubt that the unintended consequences of transparency are often unpleasant and I’m not hearing much about their mitigations in public policy and discourse on transparency at the moment.

Let’s separate out four examples of transparency:

Opening up state-managed, non-personal datasets for wider use at low or no cost is Good Transparency. Too often, officials publish or simply file away useful datasets – lists of postboxes, recipients of New Years’ Honours, public sector vacancies and so on – in formats such as PDF that make it harder to turn that data into useful services for public good or commercial enterprises. We should make this data more readily-accessible, in flexible formats, and happily, that’s exactly what a Cabinet Office team has been doing very successfully for a year. That’s not really transparency per se, it’s just good publishing practice. (Yes, the fight to open up Ordnance Survey maps was a struggle, and is arguably a cross-subsidy by the taxpayer for work previously funded by the users of OS products. But I’d argue the wider social benefits of free mapping data outweigh this).

Opening up government contracts to a wider range of bidders, and so that smaller bidders in particular can win work, and understand how the labyrinthine government procurement machine works, is clearly a positive example of transparency too. It’s more controversial, in that what might on one hand drive down costs, might conceivably make it easier for cartels to develop, or inhibit the willingness of bidding companies to be frank and detailed in their proposals. And without reform to the very process, risk appetite and above all length of government procurement, simply being able to read more of the documents won’t entice more SMEs to get involved. But those problems are fixable, and greater transparency in this field is likely to secure real economic benefits.

Opening up performance data – timesheets, civil service salaries, government spending, MP’s expenses, whatever it is that enables rankings and comparisons in individual or institutional performance to be determined – isn’t such a clear-cut positive. We know that setting targets or criteria, and above all publishing data on performance against them, inevitably focuses effort on those, and risks distorting job performance in other areas (there are better sources on that, but I can’t find them right now).

Let’s take Joe Harley, now infamous for being one of the Whitehall high earners, with an annual salary of £245,000-249,999 for his work as CIO at the Department for Work and Pensions. Nice work if you can get it for buying a few computers, the media will say. Now, I don’t know if Joe does a good job or not (I don’t hear much about DWP IT failures – so he’s doing OK, right?). In fact, I don’t really know what Joe’s job is, or what the challenges are in delivering IT to support 120,000 people and managing a roughly £856m budget. I don’t even know what CIOs in other sectors earn, or how well they do their jobs relative to Joe, or whether someone better would be found to replace him if he were to go elsewhere, given the civil service recruitment freeze. All I know is that Joe works for DWP and earns just under £250k, and that that is quite a bit more than the Prime Minister.

It’s not that opening up performance data is a bad thing per se. But in opening up what will only ever be a selected few datasets which describe characteristics of an individual or institution’s performance, it’s human nature to draw inferences about quality and appropriateness which the data in front of us don’t justify. It’s a lesson MPs and MySociety learned a while ago, which is why they add a spurious statistic – how many times an MP has used alliterative phrases – to MPs’ profiles on TheyWorkForYou, as a gentle reminder that rankings don’t tell the whole story. Next time the civil service top earners list appears, I’d love to see an inside leg measurement data column (though I suspect this might show some remarkable, if somewhat spurious, correlations).

Opening up the decision-making process – ministerial and senior officials’ diaries, campaign contributions, voting records, lobbying, policy documents and briefings – is similarly troubling. Lawrence Lessig, in his essay ‘Against Transparency argues strongly that the ‘naked transparency’ movement risks applying undue cyncism to politicians’ behaviour, ascribing motivations to actions which don’t take account of why politicians make the choices they do:

I have increasingly come to worry that there is an error at the core of this unquestioned goodness. We are not thinking critically enough about where and when transparency works, and where and when it may lead to confusion, or to worse. And I fear that the inevitable success of this movement–if pursued alone, without any sensitivity to the full complexity of the idea of perfect openness–will inspire not reform, but disgust. The “naked transparency movement,” as I will call it here, is not going to inspire change. It will simply push any faith in our political system over the cliff.

On one hand, I don’t like the idea of policy being bought, if it happens. But I’m not sure that knowing which industry groups a politician or civil servant has met is necessarily enlightening in itself. In my experience, a Secretary of State’s chance meetings with ‘real people’ at constituency surgeries and visits influence their thinking pretty strongly too, but the data would be unlikely to show that.

Government is a fascinating study in unintended consequences. Its scale and diversity, and the interplay between ministers and the civil service, the centre and the frontline mean that things often don’t go to plan. Broadly-speaking, government is a risk-averse organisation in many ways, sometimes to the extent that on occasion it has preferred ineffectiveness to perceived impropriety, waste to uncertainty.

The coming wave of transparency could transform this in a hugely positive way, using open data on costs, opportunities and performance to become a much more creative, cost-effective and agile institution, mindful of the money it spends and the results it achieves, and ensuring individuals are accountable for their work.

But it might make things worse, frightening senior managers into becoming more guarded, taking fewer ‘risks’ with even small amounts of money, and focusing on the process to the detriment of the outcome. It may also make public service less attractive not only for those with something to hide, but for effective people who don’t want to spend their time fending off misinterpretations of their decisions and personal value for money in the media. And to mirror Lessig’s point, it may push confidence in public administration over a cliff, in revealing evidence of wrongdoing which in fact is nothing of the sort.

Two cheers then for the revolution sweeping down the corridors of Whitehall. But let’s be mindful of its consequences as yet unseen.

Photo credit: Da Goaty

Public Appointments by RSS

In the words of Directgov:

A public appointment is an appointment to the board of a public body or to a government committee. Around 18,500 men and women hold a public appointment.

The public bodies involved are quite important, including health trusts, museum boards and regulators, some demanding specialist skills in law or social work, but many requiring general common sense and broad experience. So it’s important that the people who fill these posts are of the right calibre and reflect the diversity of our society.

The Cabinet Office has recently revamped its Public Appointments system, and you can now sign up to sophisticated email alerts about public appointments vacancies you might be interested in. As a publisher of vacancies, the central system also has an excellent API, enabling you to extract data feeds from the vacancy database to republish on your own site. There’s even some RDFa in the output should you wish to use that to mark-up the vacancy descriptions.

I’ve just created and added a dead simple RSS feed for the BIS-related public appointments to our homepage. But anyone can grab the code and set it up to generate their own feed, or indeed re-publish the vacancy data far and wide in any format compliant with its licence, in order to help spread the word about the interesting and varied positions available.

Hurrah for open data and APIs, and above all, hurrah to the Cabinet Office for building one in this case. Thanks chaps.

Cui bono? The problem with opening up data

Despairing official

There’s a clear argument for opening up public sector information for reuse. It increases transparency. It’s the raw material for new kinds of services with public or commercial value. It improves measurability and makes possible new kinds of analysis. Therefore, the public sector should open up more of its data, in reusable formats.

Building a catalogue of data held by public sector organisations (like the great one set up in Washington, DC and now the Federal is a logical starting point. Audit what’s held in departments, who’s responsible for it, and publish the list (with links to the relevant datasets) for potential reusers to come and browse. Or even go a step further as they’re doing at Kent County Council with their Pic-and-Mix pilot, and develop a mashup hub too, for end users themselves to develop and discuss applications which use that data.

Richard is now asking for feedback on the formats and structure of the UK’s data catalogue – do go and give him your thoughts. But I’m still troubled by some more fundamental problems than whether we publish the data in JSON or RSS:

  • Which data? For example, I suspect a simple, definitive postcoded list of UK higher education institutions would be useful to a fair number of people developing map-based mashups – though I’m not sure a civil servant would identify just how useful that kind of thing could be. I wonder whether a mechanism a bit like a souped-up, less forlorn version of OPSI’s ‘data unlocking service’ might provide a forum for potential re-users make ‘bids’ for useful data – even if they don’t know where it sits or what form it takes – and develop a community which can assess, prioritise and refine those data specification.
  • Who decides whether to publish? Proactively publishing data almost inevitably increases an organisation’s short-term potential exposure to criticism (even if it reduces it in the long term). It invariably generates tedious work for which the perceived ‘market’ is tiny. To play devil’s advocate: from a civil servant’s perspective, what makes the ‘open data people’ any different from the cranks who’ve always made trouble for bureaucrats by asking vexatious questions? There’s no big queue of citizens asking for data right now, any only a hypothetical end user audience for hypothetical tools based upon it. Ask an IT manager, a press officer and a policy official whether to publish any given dataset and you’re likely to get three radically different answers. We need some pretty clear principles to determine what gets published, to prevent our data catalogues being reduced to the blandest lowest common denominator.
  • Who benefits? The civil service isn’t the machine it’s sometimes portrayed as: ours is a surprisingly small, somewhat stretched, government of humans for whom opening up data is not – to put it mildly – a top priority, even where the data itself are simple and uncontroversial. We can tell these people to do it, but until we can show them where the benefits lie – not just in the social value, but the benefit for their organisation and for them personally – we’re unlikely to get buy-in on a large scale.
  • Who pays? Cleaning up data for publication, documenting it, checking it for errors or personal details, reformating it, uploading it, answering queries about it – there’s a lot of work involved in open data. It’s not in estabished job descriptions – so we’re likely to need more people to do this work, if it’s to happen on a large scale. Now, as taxpayers, we might decide that’s a cost worth paying. But for how many datasets? And at what maximum cost?
  • For how long? As any knowledge manager will tell you, information has a life cycle. Publish it now, and in six months’ or six years’ time, bit rot may have rendered it useless. Who is going to be responsible for maintaining the data when published, and what liability should public bodies accept for its misuse or inaccuracy when used by third parties? If the hospital Mashup A told you was at map co-ordinates X,Y turns out not to be, who are you going to be able shout at about it?

Here’s my thought: open data needs a new breed of data gardeners – not necessarily civil servants, but people who know data, what it means and how to use it, and have a role like the editors of Wikipedia or the mods of a busy forum in keeping it clean and useful for the rest of us. Encourage three or four independent people passionate about, say, transport or secondary education, who know and respect the system enough to know how to extract useful data, without rattling too roughly the cages of the people who will be asked to provide it. They’ll know when the data changes, or what a reasonable request is, or where something can be found because… they just know that area like the back of their hands. Support them with some data groundsmen with heavy-lifting tools and technical skills to organise, format, publish and protect large datasets. And then point the digital mentors at the data garden, to get communities to come and enjoy the flowers in ways that enrich their lives.

Personally, I passionately want to see open data work in the UK. But as with so much on the web, I think the primary challenges will be sociological, not technical.

Photo credit: Omargurnah