Now check out the projects:

Fire Bugs – This project scrapes the data from the Central Scotland Fire Service’s Recorded Incidents log, creating an alert when new incidents are logged. It also retrieves historic data.

The team consisted of 1 hack (Chris Sleight, from BBC Scotland) and 2 hackers (Ben Lyons and Paul Miller, from IRISS).

Central Scotland Fire Service put a lot of data on their website but as is usual, it was not in a very useful form. 60 incidents are put on the site but if you dig down you get over 15,000 buried records. For one day’s work, Fire Bugs scraped the records and decided to look at malicious false alarms. Luckily for them the language and structure of the records were consistent. They found that 3.5% of all calls were malicious false alarms. They even made a tree map on ScrpaerWiki using protoviz.  Fire Bugs have clearly opened up a huge amount of potential with this data.

Edinburgh Planning App Map – This is Edinburgh’s first automated map of local planning applications! This is a popular theme for our hack days and on ScraperWiki in general. Open Australia are using ScraperWiki for their planning alerts.

This team consists of 1 hack (Michael MacLeod, beatblogger for Guardian Edinburgh on the right) and one hacker (Robert McWilliam, from Blueflow on the left).

As Michael MacLeod pointed out, people dont’ know how to use the local council website. You can’t just type in your postcode to find applications near you. There’s a map online but it’s truly awful! The team scraped the site and made a map which updates everyday rather than just every week like on the council site. Michael used this new tool to take a closer look at his beat and found a planning application for urban paintball. What he duly noted was that the Facebook page was trying to be secretive about the location! Using the map, he found it was going to be right behind a block of flats. He wil be talking to residents!

Hide by the Clyde – This project creates a map to allow the user to compare exam results in different areas and correlates this against measures of social deprivation.

The team consists of 1 hack (Bruce Munro from BBC Scotland) and 3 hackers (Nicola Osborne from Edina, Sean Carroll from BBC Scotland and Bob Kerr from Open Street Map).

They looked at data from Learning and Teaching Scotland and scraped the search for schools form. Here is the map. From this freed data they were able to make a heat map of free school meals registration in Scotland and compare education statistics between Glasgow and Argyll & Bute for example. A major project would be to put all this information on one site in a user friendly format.

Public Buildings for Sale – is a tool to show all publicly-owned property that is for sale/rent. This will be a Scottish sister to ScraperWiki’s brownfield’s sites map. The project aims to answer the question: How much public land is being sold without our knowledge?

This team consisted of 1 hack (Peter Mackay from BBC Scotland on the right) and 1 hacker (Martyn Inglis from The Guardian on the left).

The data they wanted is on this horrible website on property sales and lettings from Scotland’s public sector. The nested html tables are very difficult to scrape. They managed to scrape this far and plan to remodel the data to make it searchable by postcode. From this, they want to glean more information about council’s buy and sell strategies.

Crash Test Dummies – This project takes three separate approaches to looking at Scottish road accident data.

1. What accidents get reported?
2. Are you safe on the roads?
3. What affect do road safety measures make on your journey

The team consisted of 2 hacks (David Eyre and Brendon Crowther from BBC Scotland) and 2 hackers (Ali Craigmile and Mo McRoberts from BBC Scotland).

In just one day they managed to built a prototype for a “Mind how you go!” BBC Scotland site. They used road traffic accident reports based on 2005-2009 data to create a form that showed how likely you were to survive your journey depending on your age, sex and where you’re going! They built a spreadsheet from even more data so that the site had even more potential to go beyond the records. They also scraped Google searches of reported road traffic accidents and mapped the reports from BBC scotland from 2010.

BME Scotland – This project aims to find out what are the effects of the recession on education! Is education a route to the ghetto? It aims to compare BME educational achievement with unemployment statistics to find out which areas of Scotland are economic no-gos.

The team consisted of 1 hack (Fin Wycherley) and 1 hacker (Paul McNally).

The lesson learnt here was that sometimes there’s not enough data to go around. What they know is that the African population is doing exceedingly well in education in Scotland. However, they also have a relatively high level of unemployment. The result was a call out for better data collection as none of the information fitted in a way that would help answer the question: Why?.

Edinburgh Council – This project searches Edinburgh Councillors gifts and expenses!

The team consists of 2 hacks (Paola Di Maoi and Anand Ramkissoon) and 1 hacker (James Baster).

As it turner out, the Council website is easy to scrape. The structure of the site is consistent and clean. ScraperWiki likes this! And so here is the scraper. As James pointed out, the data needs to be double checked for misspelled entries, etc. But the preliminary data shows that Lothian buses gave the most gifts and Phil Wheeler received the most gifts.

Magners Cider – This project aims to scrape the Magners League Rugby scores. The team consisted of a hack/hacker pair of Paul McNally (again, we love eager hackers!) and Tony Sinclair of BBC Scotland (who had to keep up the day job and so was not around for a picture). Apparently, a graphics operator had to input the information from the site by hand into the graphics system to produce the league tables you see on screen. Seeing as the graphics software can access spreadsheets, Tony thought “Why not automate the process by scraping?”. And this is what they did. So the scores have gone from ScraperWiki to TV!

And the winners are… (drum roll please)

• 1st Prize: Edinburgh Planning Map App
• 2nd Prize: Fire Bugs
• 3rd Prize: Magners Cider
• Best Scraper: Fire Bugs

A big shout out

Our judges, Jon Jacob from BBC College of Journalism, Allan Donald from STV and Huw Owen, Editor of Good Morning Scotland.

Our sponsors BBC Scotland, BBC College of Journalism and The Guardian Open Platform.

Edinburgh planning applications, fire incidents and Rugby scores – you’ve been ScraperWikied!

We got five teams:

Co-Ordnance – This project aimed to be a local business tracker. They wanted to make the London Stock Exchange code into meaningful data, but alas, the stock exchange prevents scraping. So they decided to use company data from registers like the LSE and Companies House to extract business information and structure it for small businesses who need to know best place to set up and for local business activists.

The team consisted of 3 hacks (Steve Fossey, Eva Tallaksen from Intrafish and Gareth Morlais from BBC Cymru) and 3 hackers (Carey Hiles, Craig Marvelley and Warren Seymour, all from Box UK).

It’s a good thing they had some serious hackers as they had a serious hack on their hands. Here’s a scraper they did for the London Stock Exchange ticker. And here’s what they were able to get done in just one day!

This was just a locally hosted site but the map did allow users to search for types of businesses by region, see whether they’d been dissolved and by what date.

Open Senedd – This project aimed to be a Welsh version of TheyWorkforYou. A way for people in Wales to find out how assembly members voted in plenary meetings. It tackles the worthy task of making assembly members voting records accessible and transparent.

The team consisted of 2 hacks (Daniel Grosvenor from CLIConline and Hannah Waldram from Guardian Cardiff) and 2 hackers (Nathan Collins and Matt Dove).

They spent the day hacking away and drew up an outline for www.opensenedd.org.uk. We look forward to the birth of their project! Which may or may not look something like this (left). Minus Coke can and laptop hopefully!

They took on a lot for a one day project but devolution will not stop the ScraperWiki digger!

There’s no such thing as a free school meal – This project aimed to extract information on Welsh schools from inspection reports. This involved getting unstructure Estyn reports on all 2698 Welsh schools into ScraperWiki.

The team consisted of 1 hack (Izzy Kaminski) and 2 astronomer hackers (Edward Gomez and Stuart Lowe from LCOGT).

This small team managed to scrape Welsh schools data (which the next team stole!) and had time to make a heat map of schools in Wales. This was done using some sort of astronomical tool. Their longer term aim is to overlay the map with information on child poverty and school meals. A worthy venture and we wish them well.

Ysgoloscope – This project aimed to be a Welsh version of Schooloscope. Its aim was to make accessible and interactive information about schools for parents to explore. It used Edward’s scraper of horrible PDF Estyn inspection reports. These had different rating methodology to Ofsted (devolution is not good for data journalism!).

The team consisted of 6 hacks (Joni Ayn Alexander, Chris Bolton, Bethan James from the Stroke Association, Paul Byers, Geraldine Nichols and Rachel Howells), 1 hacker (Ben Campbell from Media Standards Trust) and 1 troublemaker (Esko Reinikainen).

Maybe it was a case of too many hacks or just trying to narrow down what area of local government to tackle, but the result was a plan. Here is their presentation and I’m sure parents all over Wales are hoping to see Ysgoloscope up and running.

Blasus – This project aimed to map food hygiene rating over Wales. They wanted to correlate this information with deprivation indices. They noticed that the Food Standards Agency site does not work. Not for this purpose which is most useful.

The team consisted of 4 hacks (Joe Goodden from the BBC, Alyson Fielding, Charlie Duff from HRZone and Sophie Paterson from the ATRiuM) and 1 hacker (Dafydd Vaughan from CF Labs).

As you can see below they created something which they presented on the day. They used this scraper and made an interactive map with food hygiene ratings, symbols and local information. Amazing for just a day’s work!

And the winners are… (drum roll please)

• 1st Prize: Blasus
• 2nd Prize: Open Senedd
• 3rd Prize: Co-Ordnance
• Best Scoop: Blasus for finding  a catering college in Merthyr with a Food Hygiene Standard rating of just 2
• Best Scraper: Co-Ordnance

A big shout out

To our judges Glyn Mottershead from Cardiff School of Journalism, Media and Cultural Studies, Gwawr Hughes from Skillset and Sean Clarke from The Guardian.

And our sponsors Skillset, Guardian Platform, Guardian Local and Cardiff School of Journalism, Media and Cultural Studies.

Schools, businesses and eating place of Wales – you’ve been ScraperWikied!

Blasus winning first prize and Best Scoop award (prizes will be delivered, sealed with a handshake from our sponsor).

Scraperwiki is pleased to announce another hacks & hackers hack day: in Glasgow. BBC Scotland is hosting and sponsoring the one day event, with support from BBC College of Journalism. As with our other UK hack days, Guardian Open Platform is providing the prizes.

Web developers and designers will pair up with journalists and bloggers to produce a number of projects and stories based on public data. It’s completely free (food provided) and open to both BBC and non BBC staff. It will take place at the Viewing Theatre, Pacific Quay, Glasgow on Friday 25 March 2011.

• Find out more details & book tickets here: http://hackshackersglasgow.eventbrite.com/

Any questions? Please email judith@scraperwiki.com.

[Video: courtesy of Cathal Furey and Dave Gunnigan.]

“Dublin can be heaven, at a quarter past eleven and a stroll in Stephens Green, there’s no need to worry, there’s no need to hurry, you’re a king and the lady’s a queen…”

Onwards and downwards we headed towards Dublin, as part of our UK & Ireland Hacks & Hackers tour.

We were received as guests at the Irish Dail and given a tour of Leinster House (see left) which was useful given that our event was all about opening up Government data – thank you to Dermot Keehan (Irish Embassy London) and Patrick Rochford (Private secretary to Conor Lenihan TD).

We attended and spoke at a meeting of one of our sponsors, Dublin Freelance Branch of the National Union of Journalists, by kind invitation of Gerard Cunningham and enjoyed an evening and a few pints in the warm and inviting Buswell’s Hotel opposite Leinster House.

On the HHH day we journeyed through Dublin along the River Liffey to Wood Quay, a site that houses the remains of a Viking city dating back to the 12th Century and which was without doubt our most prestigious venues to date. We were there courtesy of Dublin City Council and Innovation Dublin and we received a fantastic welcome and great support from all their staff especially Maeve White and John Downey. Gavin Sheridan, of the excellent transparency site The Story provided us with invaluable guidance and assistance.  We were also sponsored by Guardian Open Platform and developer Michael Brunton-Spall (@bruntonspall) joined us for the event.

We had a great crowd on the day itself and we were delighted with the variety and scope of the projects.

First prize was given to MonuMental. Martha Rotter (@martharotter), Jane Ruffino (@janeruffino), John Craddon (@johncraddon), Elaine Edwards (@elaineedwards), Paul Barker, Michael Brunton-Spall (@bruntonspall) David Garavin (@newgraphic) and Alison Whelan (@smartdesigns). The project aimed to expose information and the location of archaeological monuments and combine these with planning data to show the danger that exists if there is a lack of awareness on planned public works. The idea was that the project would be sustained and would help local people actively campaign for the preservation of works that were treasured by communities.

The second prize was awarded to eTenders: Follow the Money. Fergal Reid (@fergal_reid), Gavin Sheridan (@gavinsblog), Julian Todd (@goatchurch) and Conor Ryan (@Connie_Zevon). The project was designed to highlight the issues facing people trying to understand how government contracts are distributed and to show patterns and relationships between contracts, organisations and government representatives.

The third prize and much-coveted ScraperWiki mug prize went to the ‘EPA Pollution Licenses and Enforcement’ project by Richard Cyganiak (@cygri). Since 1994 the EPA has been licensing large-scale industrial and agricultural activities. The project looked at the history of the applications for these IPPC licenses and aligned these to enforcement activities. It highlighted which sectors needed most attention for enforcement orders. The data was collated by scraping the EPA’s web based IPPC database and a PDF listing enforcement activities.

Road Safety included team members Gerard Cunningham (@faduda), Phil Mac Giolla Bhain, Cian Ginty (@cianginty) Mary O’Carroll, Alison Spilane (@Alison_Spillane) Trish Morgan and Victor Akujobi (@akujobi). The objective of this project was to show the number of road deaths per county and in parallel show the number of speed cameras and penalty points issued.

Twitter Mood Index, by Antonella Sassu (@misentoscossa), Marco Crosa, Victor Akujobi (@akujobi) and John Muldoon (@John__Muldoon) was a project designed to gauge the mood of Dublin people by sampling and analysing twitter feeds.

‘Fingal County Council is first to market’: We were also delighted to have Dominic Byrne from Fingal County Council who explained how he had set our HHH date as the target date for his team to launch their Open Data initiative. It was a coup for the council and a very promising first. It was great to hear him talking about the value in and the process for the publishing of government data.

A special thanks to our judges Michael Fisher (@fishbelfast), Dominic Byrne (Fingal County Council) and Michael Stubbs (Dublin City Council).

Thank you to bloggers and journalists for the additional coverage. Read more here:

• Politico
• Trish Morgan
• Irish Times

Finally a huge ‘thank you’ to the Woods, the Wheatleys and the McGuires for their generous hospitality during our visit.

Oo, and must not forget the obligatory pizza pic!…

We set sail (in a gale but only force 9 this time!) after a few more pints of Guinness in O’Shea’s on the Quays where Francis and Julian declared that they were up for a sprint in the Aran Islands off the coast of Galway in 2011: I guess that must mean that we are going back next year and that we should really do a #hhhgal!

Roll on 2011!

• Dublin tickets at this link

It’s a chance for web developers and designers to pair up with journalists and bloggers to produce a number of projects and stories based on public data. We’re using the tag ‘#hhhDub‘ to talk about this on Twitter.

It’s the latest confirmed date on the ScraperWiki tour: we’ll be going to Manchester on Friday October 15th and Leeds on Thursday  October 28th. We’re currently negotiating venues and dates in Cardiff, Belfast and Glasgow.

What? Hacks and Hackers Hack Day Dublin

When? Tuesday November 16th 2010 from 9.30 am to 7.30 pm

Where? At Wood Quay, Civic Offices, Dublin 8 (on Google Maps)

Who is it for? We hope to attract ‘hacks’ and ‘hackers’ from all different types of backgrounds: people from big media organisations, as well as individual online publishers and freelancers.

How much? Absolutely free, thanks to our sponsors, Guardian Open Platform and the National Union of Journalists Dublin Freelance Branch and Innovation Dublin for making the event possible.

Refreshments, lunch and evening snacks will be all be provided.

NB: We’re still seeking sponsors to get involved. If you think your organisation might be interested please contact aine [at] scraperwiki.com. Or get in touch with any other questions!

What happens on the day? The aim is to show journalists how to use programming and design techniques to create online news stories and features; and vice versa, to show programmers how to find, develop, and polish stories and features. All sorts of data was scraped and played with at our past events: in Liverpool, projects included mashes of police, libraries and courts data. Birmingham saw lots of health-related projects, as well as scraping of political party donor and leisure centre information.

This four minute video from Liverpool gives you a flavour of the kind of projects that can be developed.

Armed with their laptops and WIFI, journalists and developers will be put into teams of around four to develop their ideas, with the aim of finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group. Overall winners will receive a prize at the end of the day.

• Book your place now, via this page: http://hackdaydublin.eventbrite.com/

With thanks to our sponsors:

It’s the fifth stop on the ScraperWiki tour: we’ll be going to Manchester on Friday October 15th and Dublin on Tuesday 16th November as part of the Innovation Dublin festival. We’re currently negotiating venues and dates in Cardiff, Belfast and Glasgow.

What? Hacks and Hackers Hack Day Leeds

When? Thursday October 28th 2010 from 9.30 am to 7.30 pm

Where? At nti Leeds, Old Broadcasting House, 148 Woodhouse Lane, LS2 9EN Leeds (on Google Maps)

Who is it for? We hope to attract ‘hacks’ and ‘hackers’ from all different types of backgrounds: people from big media organisations, as well as individual online publishers and freelancers.

How much? Absolutely free, thanks to our sponsors, Guardian Open Platform and Guardian Local and Leeds Trinity Centre for Journalism for making the event possible, and our hosts nti Leeds at Old Broadcasting House.

Refreshments, lunch and evening snacks will be all be provided.

NB: We’re still seeking sponsors to get involved. If you think your organisation might be interested please contact aine [at] scraperwiki.com. Or get in touch with any other questions!

What happens on the day? The aim is to show journalists how to use programming and design techniques to create online news stories and features; and vice versa, to show programmers how to find, develop, and polish stories and features. All sorts of data was scraped and played with at our past events: in Liverpool, projects included mashes of police, libraries and courts data. Birmingham saw lots of health-related projects, as well as scraping of political party donor and leisure centre information.

This four minute video from Liverpool gives you a flavour of the kind of projects that can be developed.

Armed with their laptops and WIFI, journalists and developers will be put into teams of around four to develop their ideas, with the aim of finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group. Overall winners will receive a prize at the end of the day.

• Book your place now, via this page: http://hackdayleeds.eventbrite.com/

With thanks to our sponsors:

Want to support us too? Contact aine [at] scraperwiki.com.

ScraperWiki is thrilled to announce its next event for journalists and programmers: Hacks and Hackers Hack Day Manchester, a chance for web developers and designers to pair up with journalists and bloggers to produce a number of projects and stories based on public data.

When? Friday October 15th 2010 from 9.30 am to 7.30 pm

Where? At Vision+Media, Broadway 100, Salford Quays, M50 2UW (nearest tram: Broadway) View on Google Map.

Who is it for? We hope to attract ‘hacks’ and ‘hackers’ from all different types of backgrounds: people from big media organisations, as well as individual online publishers and freelancers.

How much? Absolutely free, thanks to our sponsors, Inside the M60, Guardian Open Platform, the Digital Editors Network, Vision+Media (supported by the European Regional Development Fund and the Northwest Regional Development Agency), Journal Local and MEN Media, for making the event possible. Refreshments, lunch and evening snacks will be all be provided.

What happens on the day? The aim is to show journalists how to use programming and design techniques to create online news stories and features; and vice versa, to show programmers how to find, develop, and polish stories and features. All sorts of data was scraped and played with at our past events: in Liverpool, projects included mashes of police, libraries and courts data. Birmingham saw lots of health-related projects, as well as scraping of political party donor and leisure centre information.

This four minute video from Liverpool gives you a flavour of the kind of projects that can be developed.

Armed with their laptops and WIFI, journalists and developers will be put into teams of around four to develop their ideas, with the aim of finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group. Overall winners will receive a prize at the end of the day.

• Book your place now, via this page: http://manchesterhackday.eventbrite.com/

NB: We will also be running another event in Leeds on Thursday October 28th 2010. More details soon.

With thanks to our sponsors:

NB: We’re still seeking one more sponsor to get involved. If you think your organisation might be interested please contact judith [at] scraperwiki.com. Or get in touch with any other questions!