Creating a sustainable technology company involves keeping up with technology. The thing about technology is that it changes, and we have to look to the future, and invest our time now in things that will be valuable in the future. Or, we could switch to doing SharePoint consultancy for the rest of our lives, but […]
End User Programming at the Office for National Statistics
The Office for National Statistics (ONS) approached us regarding a task which involves transforming data in a spreadsheet. Basically, unpivotting it. Data transformation is quite a general problem, but one with recurring patterns. Marginal variables are usually, well, somewhere in the margin. Cells generally refer to an observation or the name or value of a […]
npm install urchin
Urchin, the shell testing framework for extreme hipster superheroes (I’m not including myself in that group I should add), is now available as an npm package. That means you can install it using npm: sudo npm install -g urchin If you’re not hipster enough to use npm then you can still wget it from github: […]
A sea of data
My friend Simon Holgate of Sea Level Research has recently “cursed” me by introducing me to tides and sea-level data. Now I’m hooked. Why are tides interesting? When you’re trying to navigate a super-tanker into San Francisco Bay and you only have few centimetres of clearance, whether the tide is in or out could be […]
So web scraping is easy?
Journalists, academics and budding open data hackers often praise ScraperWiki for making web scraping easy. And while it’s true our platform and powerful APIs let you get more done, more easily, the statement still creates some head-scratching at ScraperWiki HQ. That’s because, as far as we can tell, scraping is hard, no matter what platform […]