UK Open Government License

Version 1.0 of the UK's Open Government License has been released, as developed by Her Majesty’s Stationery Office (good to see a stationery office that thinks beyond paperclips and printer paper!) It is similar to a Creative Commons attribution license with some added bits. Press release here - choice quote:

Commenting on the launch of the new licence, Lord McNally, Minister for The National Archives and Public Sector Information, said: 'The National Archives isn't simply a repository of our nation's history, its task is to bring information to life, make it accessible and enable its re-use. This innovative licence gives everyone the opportunity to create products and services which benefit society.'

update: there's some information from Beth Brook about why the Open Government License was developed, rather than using a Creative Commons license. In summary: they wanted a single license that covered both copyright and database rights, covering all UK jurisdictions, and which was as simple as possible (rather than having a simple summary but complex legal language).

update 2: post on data.gov.uk about the license encouraging use of the license by local authorities.

update 3: analysis from eaves.ca on how this positions the UK.

Private organization, open data

Infochimps has a post about the pros and cons of opening data, from the perspective of a private organization (as opposed to governments, which have been getting a lot of attention recently). Infochimps is a nifty site:

We’ve been working since the start of 2008 to build the world’s most interesting data commons, and since the start of 2009 to build the world’s first data marketplace.

Have your cake and eat it

Does your so-called open data pass the cake test? Can you have a baker put a picture of it on a cake for you? If you can't figure this out in a minute or two (hmm, the baker profits commercially, and redistributes, is that ok?) then the data isn't very open, is it? (Here's the original post, in Spanish)

The open data movement

A short inspirational talk from Tim Berners-Lee at TED: the year open data went worldwide. He gives real-world examples of open data making a difference to people's lives, from avoiding bicycle accidents to recovery in Haiti.

UK address database

Tom Hughes has posted a response to a consultation about releasing Ordnance Survey data in the UK. He gives a very good summary of the current options for licensing open data (we are ourselves trying to figure out how to communicate clearly about this). Along the way, he speaks in favor of a national address register in the UK, for an interesting reason:

I believe that there would be significant benefit to the release of an address product, if only to try and break the current deadlock that apparently exists in government and avoid the current crazy situation where different branches of government appear to be competing with each to maintain their own address databases. Perhaps the craziest product of this infighting is the decision of the Office for National Statistics to create (at a cost of at least 12 million pounds) yet another address database in order to deliver the 2011 census. They then propose to discard that database once the census is complete.

Data often gets stuck in single projects or communities, even with the best of intentions. The Fórum Brasileiro de Economia Solidária, which has done massive well-funded surveys of organizations in Brazil, apparently filters out organizations that don't in the end meet the specific criteria they care about from their searchable, mappable online index. Hopefully the data isn't thrown away, but it certainly isn't easy to get to.

On the one hand, it is great to have a laser-like focus on one's community or project. However it is sad to miss out on the opportunities that collaboration with partially overlapping communities or projects can bring. The benefits are fuzzier and longer-term, but no less real.

However, making data public is hard work, and costly. Lowering that cost is a goal of the DCP. That will make it easier for organizations to justify the decision to divert some resources from achieving their own objectives to helping others achieve theirs, when the benefit to them of doing so is speculative rather than known from precedent.

Making open data useful

Interesting post on the "tragedy of Edmontorcouver open data" from an OpenStreetMap contributor. They argue that municipal efforts at releasing data don't have the impact that they might have, because of the following mistakes (Edmontorcouver seems to be a fictional placename, but this presumably reflects experience with real towns is Edmonton/Toronto/Vancouver):

1. They expected a new community to build itself.
2. They wrote their own license.

Here's the advice given for rescuing such efforts:

1. Curate your data and make it the best data possible.
2. License the data with the PDDL.

The argument is that the municipality shouldn't be in the business of "curating" a community, since they won't be very good at it, and there are better curators already out there (e.g. the OpenStreetMap community). The municipality should focus on allocating resources to getting local experts to curate the local data. Similarly for the license, the municipality shouldn't be in the business of curating a custom license now that there are serviceable options already out there:

... the good folks at the Open Data Commons make it easy to select an appropriate license for your open data. Use the Public Domain Dedication and License if you want protect your interests by disclaiming liability and you want your data to have the widest possible audience. [Hint: you do!]

Best practice for open data, a reading list

It can be hard to convince data-sharers that data being "freely available" on a web-page isn't the end of the argument about whether that data can be reused. Here's a collection of postings I've found helpful in understanding the issues.

• Openness and licensing post by Rufus Pollock
• Best practice for data availability roundup by Cameron Neylon.
• Guide to Open Data Licensing at Open Definition.
• The Panton Principles
• A post discussing the panton principles which has an interesting back-and-forth in the comments.

For comparison purposes, it is worth looking at the history of software repositories. For example, Debian has 20,000+ packages within it (depending how you count), covering every kind of software under the sun (and beyond, stargazers should check out the "stellarium" package). A typical package will depend on a dozen or so other packages, which in turn depend on others. It is a massive work of aggregation. There are huge technical challenges, but underlying the solution is the Debian social contract and their Free Software Guidelines. Here are the guidelines in full:

1. Free Redistribution

   The license of a Debian component may not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. The license may not require a royalty or other fee for such sale.

2. Source Code

   The program must include source code, and must allow distribution in source code as well as compiled form.

3. Derived Works

   The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.

4. Integrity of The Author's Source Code

   The license may restrict source-code from being distributed in modified form _only_ if the license allows the distribution of patch files with the source code for the purpose of modifying the program at build time. The license must explicitly permit distribution of software built from modified source code. The license may require derived works to carry a different name or version number from the original software. (This is a compromise. The Debian group encourages all authors not to restrict any files, source or binary, from being modified.)

5. No Discrimination Against Persons or Groups

   The license must not discriminate against any person or group of persons.

6. No Discrimination Against Fields of Endeavor

   The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

7. Distribution of License

   The rights attached to the program must apply to all to whom the program is redistributed without the need for execution of an additional license by those parties.

8. License Must Not Be Specific to Debian

   The rights attached to the program must not depend on the program's being part of a Debian system. If the program is extracted from Debian and used or distributed without Debian but otherwise within the terms of the program's license, all parties to whom the program is redistributed should have the same rights as those that are granted in conjunction with the Debian system.

9. License Must Not Contaminate Other Software

   The license must not place restrictions on other software that is distributed along with the licensed software. For example, the license must not insist that all other programs distributed on the same medium must be free software.

10. Example Licenses

    The GPL, BSD, and Artistic licenses are examples of licenses that we consider free.

There's a lot going on here, and not all of it applies to data. But the desired outcome could perhaps be summarized as:

• We want stuff than can be collected, mixed, and redistributed to others, who can in turn do the same.
• We only want stuff that expressly permits such activity, and where the owners don't have any requirements that make such activity excessively complicated.
  

While data and software are different, projects like the DCP (and many others) would like to see this same outcome achieved for data. A challenge is that there is so much data out there that is "freely available" on a random website, but doesn't expressly permit aggregation, or state any requirements on allowed use. And yet, almost certainly, there are implicit requirements (no commercial use! no use by organizations my members disapprove of!) which may or may not have legal force but certainly ought to be respected by a well-behaved aggregator, even if the only way to respect them without splitting up the "data commons" too much would be by omitting the data entirely.

datapkg: data/knowledge packaging

Via the Open Knowledge Foundation Blog, we're excited to see datapkg maturing. This is a command-line tool for discovering, installing and sharing data packages. If you are familiar with Linux, the idea is to do for data what apt-get/aptitude/dpkg do for programs, with registries like CKAN playing the role for data that Debian or other distributions do for programs. For example, with datapkg installed, one can do something like this:

$ datapkg search ckan:// economics
stw_thesaurus_for_economics -- STW Thesaurus for Economics
energy-stern-review-economics-climate-change -- The Stern Review -- The Economics of Climate Chanage
repec -- Research Papers in Economics
unstats -- United Nations Statistical Databases
usa_bls_employment -- USA Employment status of the civilian noninstitutional population, 1940 to date
eurostat-gfs -- Eurostat - Government Finance Statistics (GFS) Data
numbrary -- Numbrary
economagic -- Economagic Economic Time Series
pl-budget -- Poland - Ministry of Finance - Budget
econ-alfred -- ALFRED: ArchivaL Federal Reserve Economic Data
ehnet -- Economic History Services Databases
eu-cohesion-beneficiaries-ie -- EU Cohesion Beneficiaries - Ireland
nl-statistics -- Netherlands - Statistics
econ-gdp-historical -- World Population, GDP and Per Capita GDP, 1-2003 AD
esfdb -- European State Finance Database
econ-fraser -- FRASER - Federal Reserve Archival System for Economic Research
fi-budget -- Finland - Valtiovarainministeriö - Budget
econ-fred -- Federal Reserve Economic Data
ceprdata -- CEPR Data

$ datapkg info ckan://econ-gdp-historical
## Package: econ-gdp-historical

name: econ-gdp-historical
title: World Population, GDP and Per Capita GDP, 1-2003 AD
version: None
license: Non-OKD Compliant::Other
author: None
author_email: None
maintainer: None
maintainer_email: None
url: http://www.ggdc.net/maddison/
download_url: http://www.ggdc.net/maddison/Historical_Statistics/horizontal-file_03-2007.xls
notes: ### Author

Angus Maddison

### Openness: Not open

  * No license
  * Plus following statement attached to link to data: "Last update: March 2007, copyright Angus Maddison"

### Format

  * xls (excel)
tags: ['economic', 'history', 'gdp', 'license-not-specified', 'data']
extras: {}

$ datapkg install ckan://econ-gdp-historical .
Registering ...
Created on disk at: ./econ-gdp-historical
Downloading package resources ...
horizontal-file_03-2007.x 100% |=========================| 1.5 MB    00:03

$ ls -R
.:
econ-gdp-historical

./econ-gdp-historical:
horizontal-file_03-2007.xls  metadata.txt

In other words, it gets a whole lot easier to discover and install data in an organized fashion. One exciting thing about that is that this opens the door to propagating changes and updates in a sane way, a topic near to our hearts. Good stuff!

Open Database License, version 1 is out

The Open Data Commons project (no relation) has released version 1.0 of its Open Database License (previously mentioned here). It is great to see someone with the legal savvy to put this together.

This license is an "Attribution and Share-Alike" license, doing for data/databases what the creative commons has done for media.

Why and How: Making Data Open

Our similarly-named-but-unrelated-peers, the Open Data Commons, have posted a guide to making your data open, briefly explaining why you might want to do it, and how to do it right.

Here's their take on why open data is important:

[Open data is] crucial because open data is so much easier to break-up and recombine, to use and reuse.

And their take on the need for clear licensing:

Licensing is important because it removes uncertainty. Without a license you don’t know where you, as a user, stand: when are you allowed to use this data? Are you allowed to give to others? To distribute your own changes, etc?

Big Data

Out of curiosity, I've been looking at how some of the "big" sources of open data out there distribute their data. Wikipedia is perhaps the most famous. All the data on Wikipedia and related sites is available for bulk download. For example, the English section of Wikipedia is available here:

• All saved versions: http://download.wikimedia.org/enwiki/
• Latest files: http://download.wikimedia.org/enwiki/latest/
• The main file is "enwiki-latest-pages-articles.xml.bz2"

In other words, with a click or two you can end up with an XML file holding the basic content of all English Wikipedia pages. There are other XML and SQL files for other bits and pieces.

The DMOZ open directory (like Yahoo's directory, but volunteer created and under a free license) is downloadable in RDF format at http://rdf.dmoz.org/.

Of course, there's lots more data out there, but this does give a sense of one way in which "Big Data" may be distributed. What I like:

• It is really easy to get the free data, just like it is easy to get free software.
• The data is in a good format to use, just like free software source code.
• Rights to the data are granted in a clear and free license, just like free software.

What I don't like:

• There's no equivalent of "patches" in software. Let me explain. If you improve a piece of code someone else wrote, you can automatically generate the "difference" between the original and your revised version, send that difference (called a "patch") to the original author, who can then evaluate it and if they like it merge it automatically with their code (even if they've made their own non-overlapping changes in the meantime). That's patching in software. Now what happens if you improve pages you downloaded from Wikipedia? I guess you go to the site and try typing them in - there's no way I see to submit something like a patch. And without a patching mechanism, there's no basis for distributed development of the data, like happens with free software.

There are distributed databases out there. CouchDB is interesting, for example. But it would also be interesting to have a procedure for patching and merging data that operated on an external representation rather than on live databases.

Update: Nat Torkington has a post called Truly Open Data asking similar questions.

Update (2): I've been doing work on storing our own data in fossil, a distributed version control system. The trick, as I see it, is to bridge the gap between git/bzr/hg/fossil/... and programs like Excel that non-programmers keep their data in.

Open Everything NYC

Open Everything NYC is coming up on April 18, 2009. From http://openeverything.net:

Open Everything is a global conversation about the art, science and spirit of 'open'. It gathers people using openness to create and improve software, education, media, philanthropy, architecture, neighbourhoods, workplaces and the society we live in: everything. It's about thinking, doing and being open.

From johndbritton:

Open Everything NYC will take place on Saturday 18 April 2009 at the UNICEF headquarters in the United Nations Plaza, NYC. The event will run the full day, registration will open at 8:00AM and things will be in full swing by 9:00AM.

The event will be 100% free and open to the public on a first come first serve basis, online pre-registration is required. The main hall can hold up to 250 guests.

The event will consist of two keynote presentations (one opening & one closing) each of about 1 hour in duration. In the time between the two keynotes attendees will be in control of the program (Barcamp style). There will be a number of conference rooms available for individuals to hold talks & discussions on topics they see fit. Past events have included topics such as Open Publishing, Open Education, Government Transparency, Open Access, Open Research Data, Creative Commons, Open Hardware, and more.

Good to see conversations like this happening! It seems like the theme of openness is cropping up more and more in every field, but the opportunities for communicating between those fields are few and far between.