It’s Christmas come early at ScraperWiki HQ as we deliver—like elves popping boxes under the data digging Christmas tree—a bunch of great new improvements to the ScraperWiki site. We’ve been working on these for a while, so it’s great to finally let you all use them!

First up: a new look for your scrapers

The most obvious change will hit you as soon as you look at a scraper – the overview page now sports a svelte, functional new layout. The roster of changes is as long as Santa’s list, so I’ll just pick out a few…

The blue header at the top of the page is now way more informative. As well as the scraper’s title and creator, you can also see the language it’s written in, the domain it scrapes, the number of records in its datastore, and its privacy status. No more hunting around the page: everything you need is there in one place. Hurrah!

Further down, you’ll notice the history and discussion pages have now been merged into the main page, meaning you’ll spend less time flicking between tabs and more time editing or investigating the scraper.

Meanwhile, the page as a whole is a lot more organised. Everything to do with runs (the current status, the last run, the pages scraped, the schedule) is up in the top left. Everything to do with the datastore (including the data preview and download options) is just below that, and everything to do with the scraper’s relationship to other scrapers (like tags, forks, copies and views) is just below that. Neat.

Speaking of which, the data preview has had some serious attention. It’s now way more interactive: you can sort on any column, alter the number of rows displayed, and page through all of the data in all of the tables, with just a few clicks. And other features like syntax-highlighted table schemas and a nifty drop-down when you have too many tabs to fit on the page, should keep ScraperWiki power-users fast and efficient.

And those are just the headline changes. There have also been a load of great tweaks, like a View Source button so you never have to worry about breaking someone’s scraper when you’re just taking a look, and an easy Share button to get your scrapers on Facebook, Twitter and Google+. So go try out the new page, and as ever, we’d really love your feedback.

Never miss a comment again

As well as moving the scraper discussion (or ‘chat’, as it’s now called) onto the main page to make it more obvious, we’ve also enabled email notifications for comments. Now, when someone comments on your scraper, you’ll get a swish new email showing you who they are, what they said, and how to reply (thanks to ScraperWiki’s new engineer, David Jones, for his input on this!).

If, however, all this conversation is a little too much for you, Ebenezer, then you can disable comment notifications by unchecking the box in your Edit Profile page.

And while we were at it: Messages!

For a while, users have grumbled that it’s far too difficult to contact other users. And quite right too – we never anticipated that our developers would be such social creatures! So, we’ve added a “Send a Message” button to everyone’s profile (kudos to Chris Hannam for helping out!). The messages are sent as emails via feedback@scraperwiki.com, meaning the other user never sees your email address – just your name, your message and a link to your profile. And, as with comment notifications, if you want to disable sending and receiving of user messages, just uncheck the box in your Edit Profile page.

But suddenly, all of that seemed pretty unimportant. In the small hours of Thursday morning, people all over the UK woke up to find that Steve Jobs, one of the greatest and most controversial legends of the technology world, had passed away. The news rocked Twitter — there was pretty much nothing else in my stream all day — flowers and candles were laid outside Apple Stores around the world, people published poems and pictures and stories and bittersweet obituaries. Many of the highest–traffic pages on the web displayed humble banners with his name. Some simply shut down and devoted every pixel to his memory.

And it got me wondering — what does a guy do to cause such a stir? Books will no doubt be written (in fact, they already have) answering that question. They’ll talk about Steve the college dropout, Steve the child of the Sixties, Steve the garage marketeer. They’ll picture him bowtied and bespectacled grinning above the first Macintosh, clad in the inimitable blue and black at one of a hundred keynotes, and worryingly gaunt at the height of his battle with cancer. They’ll talk about how he revolutionised not just the computing industry, not just the software industry and not just the music industry, but also the animation industry, the movie industry, and the technology retail industry. And they’ll be right.

But if you ask me, the real reason why we’re all laying flowers for this guy, writing poems for him, even talking about him at all, is because he put the user at the live, beating heart of everything he did. Steve didn’t invent the mouse, or the GUI or the personal computer, but in a world of green-on-black, of FORTRAN and BASIC, he had the foresight, the passion and the balls to back these weird, unpopular and user-centric technologies, because he knew, once normal people had access to the liberating power of the silicon chip, their lives would change forever. It’s only a matter of time until someone (hopefully us!) does the same with data.

I’m no Steve Jobs (I look terrible in turtle-necks). But if I can do anything here at ScraperWiki, it’s to try and bring some of that user focus to the world of data science. Life is too short to spend it puzzling over dubug console output, or commenting out lines of code one by one. And most of all, life’s too short to be doing all of that alone. I have two goals as the ScraperWiki UX guy: to make the experience of using our services as smooth, as intuitive and as integrated as possible, and also to make it as social as possible—not in a Facebook way, but in a hackday way—so you can all benefit from the wealth of experience, backgrounds and talents around you, right now, on this very site. There’s some amazing work being done by our members, and it’s my job to make sure you can keep on doing it, keep on getting the scoops, informing the public, serving your clients, no matter how hideous the HTML or unstructured the PDF.

Like I said, I’m no Steve Jobs. Who could even try to compete? But like Steve, I have an email address – zarino@scraperwiki.com – and I want to hear from you. Yes, you, right now. And in the future, whenever you have a problem. Whenever you think of something ScraperWiki should be doing for you, or whenever it fails to do something it says it should. Drop me an email and we’ll work on a solution (I promise my responses won’t be as famously acerbic as Steve’s).

And with that, I’ll leave you. Our brilliant new Editor interface isn’t going to design itself, you know. But before I go, I should take one last chance to say thank you. To you amazing ScraperWiki diggers, to Francis and the ScraperWiki team, but most of all, to Steve, for making all of this possible. I hope we can do him proud.

ScraperWiki is a Silicon Valley style startup, but based in the UK. We’re changing the world of open data, and how programming is done together on the Internet.

We’re looking for a web product designer who is…

• Able to make design decisions to launch features by themselves.
• Capable of writing CSS and HTML, and some Javascript.

Other bits…

• Loves to balance colour, size, order and prominence on websites.
• Knows what a web scraper is, and would like to learn to write one.
• Thinks that data can change the world, but only if we use it right.
• Either good at working remotely, or willing to relocate to the North West.
• Desirable – able to make igloos.

To apply – send the following:

• An example of a website you’ve made that you’re proud of
• If you have one, a visualisation you’ve made of some data (any data!)
• Oh, and I guess we’d better see your CV

Along to francis@scraperwiki.com with the word swjob2 in the subject.

ScraperWiki is a Silicon Valley style startup, but in the North West of England, in Liverpool. We’re changing the world of open data, and how programming is done together on the Internet.

We’re looking for a web designer/programmer who is…

• Capable of writing standards compliant CSS and HTML, and some Javascript.
• Loves to balance colour, size, order and prominence on websites.
• Good enough at Photoshop to make any mockups and icons required.
• Likes to talk to and track users, and then do what’s needed to make their experience better.
• Server-side coding (Python) a plus but not essential.
• Knows what a web scraper is, and would like to learn to write one.
• Thinks that data can change the world, but only if we use it right.
• Desirable – able to make igloos.

Some practical things…

• We’re early stage, spending our seed funding. So be aware things will go either way – we’ll crash and burn, or you’ll be a key, senior person in a growing company.
• We’d like this to end up a permanent position, but if you prefer we’re happy to do individual contracts to start with.
• Must be willing to either relocate to Liverpool, or able to work from home and travel here regularly (once a week). So somewhere nearby preferred.

To apply – send the following:

• An example of a website you’ve made that you’re proud of
• If you have one, a visualisation of some data (any data!)

Along to francis@scraperwiki.com with the word swjob1 in the subject.