fromCONCENTRATE

ScraperWiki is a collaborative environment for building, running, storing and sharing web screen-scrapers and their resulting bounties of data.  (Here is a handy introduction to ScraperWiki by Paul Bradshaw.)  I was lucky enough to encounter ScraperWiki right at the very beginning when Julian Todd and Aiden McGuire launched the concept at Liverpool’s first Barcamp (at the end of 2008) and last Friday I had the opportunity to take part in the ScraperWiki ‘Hacks and Hackers Hackday‘ at open labs LJMU.

The object of the day was for ‘hacks’ (journalists + activist bloggers and curious citizens) to work with ‘hackers’ (programmers + geeks and persistent meddlers) to collaborate in attempting to dig out publicly available data from over the internet and use this to tell new stories.  Although, ScraperWiki was the proposed tool for doing this, the emphasis throughout was on accessing, understanding, re-interpreting and sharing previously oblique data.

The day began with presentations explaining the power of good data-visualisation and information graphics.  Examples cited were:

• BBC ‘best graph of the election’ – showing convergence of 3 Political Party’s public spending and taxation plans – suggesting very high chance of coalition govt.
• Nick Clegg on They Work for You (data from Public Whip) – showing data-vis need not be graphical (computer generated sentences.)
• Schooloscope (visualising schools in relation to OFSTED reports)
• GapMinder – Hans Rosling (below) – 2d comparisons over time – less divided world than we might think!

The delegation in attendance comprised of roughly three groups:

• professional journo’s (Liverpool Echo thru to Graniud) on ‘professional development’
• hardcore coders (already invested in ScraperWiki)
• & a sizeable contingent of those invaluable (and uncategorisable) people who are becoming harder to ignore these days now they are identified as the long tail.

What impressed me most about the event was the total commitment of all of those present to be involved in the process and deliver a fresh idea.

The value of such analytical approaches to journalism is immense and might well mark a sea-change away from soundbite political media. However, my own interest in screen-scraping is perhaps more inclined towards provocative and experimental data-usage and applications which demand reflection on the unanticipated dormant potentials within our familiar networked technology and databases.

Some more blog posts about the day here:

• Policemen, Judges and Libraries – ScraperWikiBlog
• Liverpool’s ScraperWiki Hacks and Hackers Hack Day – Alison Gow
• Hacks meet Hackers – Mike Nolan
• Using Data to Track Bobbies on the Beat – Ed Walker
• Hacks meet Hackers – LJMU News Update