Whatever happened to Swivel

Swivel.com was a popular data visualization site that recently failed as a business. Some information is starting to trickle out about what happened, through an interview with its founders. Some choice quotes. First, expenditure:

We got a Round A funding of two million dollars. By the time I left, we had spent about three to four million dollars on the idea.

Second, income:

[How many paying users did you have in the end?] It was single digits

Ouch. Lessons learned? I'm cherry-picking my favorite here:

I think what we learned, like Roseman is saying, that the interface is not that important, that there are analysts who are really good at tools like R, SAS, etc. and prefer to continue to work in those tools to do powerful things with datasets.

Please read the article for the full context. I just wanted to highlight this point, which I think may be important for the Data Commons Project. I think it argues for letting people maintain their data with the tools they are expert with, rather than expecting them to use a new service. My personal theory is that distributed revision control systems are the way to go for collaborative data projects. Web-based services could be part of such a distributed ecosystem, but would not be its "hub".

FaceBook gives users access to their own data

Good news from FaceBook: users can now download everything they've ever put on the site. This is an important step towards transparency. The move is being welcomed by the DataPortability project, whose mission is to help people to use and protect the data they create on networked services - although they are careful to note that being able to download one's data is not the same as being able to control it (read Alisa Leonard).

CoopMetrics

CoopMetrics is a nifty service for benchmarking a cooperative against its peers, and finding ways to improve. By pooling and comparing financial data cooperatives in a particular sector can find out what works, what doesn't, and good ideas spread faster.

• Details on the process: In summary, a co-op's accounts are mapped onto a standardized chart of accounts, quarterly trial balances are uploaded, and CoopMetrics crunches the data to provide various reports and comparisons. There are videos that give a sense of the steps involved.
• History: CoopMetrics traces its lineage back to 1996, with the CoCoFiSt ("Common Cooperative Financial Statements") program developed by Walden Swanson and Kate Sumberg.

This is a great example of organizations gaining an advantage by pooling information, and extracting insight that would otherwise be elusive.

Hat tip: Jim Johnson

Rooted economy directories

The growing list of rooted economy directories in this blog's sidebar is now a package in CKAN. CKAN is an open registry of data and content packages. You can download the directory list as a CSV file.

• http://ckan.net/package/rooted-economy-directories

We maintain the list using Coopy, an experiment in distributed data collection.

"Have you and a colleague ever ended up with two versions of the same spreadsheet, with a mess of independently-made changes that now need to be merged? Coopy makes that problem go away."

Coopy applies the fossil distributed version control system, normally used for software development, to the problem of data collection.

Private organization, open data

Infochimps has a post about the pros and cons of opening data, from the perspective of a private organization (as opposed to governments, which have been getting a lot of attention recently). Infochimps is a nifty site:

We’ve been working since the start of 2008 to build the world’s most interesting data commons, and since the start of 2009 to build the world’s first data marketplace.

Big Data

Out of curiosity, I've been looking at how some of the "big" sources of open data out there distribute their data. Wikipedia is perhaps the most famous. All the data on Wikipedia and related sites is available for bulk download. For example, the English section of Wikipedia is available here:

• All saved versions: http://download.wikimedia.org/enwiki/
• Latest files: http://download.wikimedia.org/enwiki/latest/
• The main file is "enwiki-latest-pages-articles.xml.bz2"

In other words, with a click or two you can end up with an XML file holding the basic content of all English Wikipedia pages. There are other XML and SQL files for other bits and pieces.

The DMOZ open directory (like Yahoo's directory, but volunteer created and under a free license) is downloadable in RDF format at http://rdf.dmoz.org/.

Of course, there's lots more data out there, but this does give a sense of one way in which "Big Data" may be distributed. What I like:

• It is really easy to get the free data, just like it is easy to get free software.
• The data is in a good format to use, just like free software source code.
• Rights to the data are granted in a clear and free license, just like free software.

What I don't like:

• There's no equivalent of "patches" in software. Let me explain. If you improve a piece of code someone else wrote, you can automatically generate the "difference" between the original and your revised version, send that difference (called a "patch") to the original author, who can then evaluate it and if they like it merge it automatically with their code (even if they've made their own non-overlapping changes in the meantime). That's patching in software. Now what happens if you improve pages you downloaded from Wikipedia? I guess you go to the site and try typing them in - there's no way I see to submit something like a patch. And without a patching mechanism, there's no basis for distributed development of the data, like happens with free software.

There are distributed databases out there. CouchDB is interesting, for example. But it would also be interesting to have a procedure for patching and merging data that operated on an external representation rather than on live databases.

Update: Nat Torkington has a post called Truly Open Data asking similar questions.

Update (2): I've been doing work on storing our own data in fossil, a distributed version control system. The trick, as I see it, is to bridge the gap between git/bzr/hg/fossil/... and programs like Excel that non-programmers keep their data in.