I’m no longer twitter domesticated

It’s a weird time in technology and the web. It feels like a couple of big shifts are happening all at once. One of them is a big rise in “enshittification”, most noticeably for me with the death of twitter.

It’s pretty surprising watching twitter die and, I’ll be honest, it’s upsetting. I allowed twitter to become a bigger part of my online life than I should have done, and now it’s going down the toilet in a shift that feels quite sudden, given that I’ve been on twitter for about 16 years. Throughout 2023 I found myself gradually swapping from twitter to mastodon. Admittedly it wasn’t a very deliberate choice. I still go on both, but these days I think about posting to mastodon first and I’ll copy the post into twitter if I can be bothered.

Why is twitter going down the toilet? (I refuse to call it “X” by the way) It took me a while to notice the problems, and maybe it was only the contrast with mastodon that helped me see it more. People talk in dramatic terms about nazis posts, which I never really saw, but there’s a shift in “the algorithm” of the feed. Mastodon only shows stuff from people I follow (and their re-toots). When I first played with mastodon a couple of years ago, it felt similar to twitter, but now twitter has shifted. It’s not even a subtle shift any more. My twitter timeline has very few posts from people I follow. Instead it’s full of clickbait videos and other “engaging” posts from non-followees. A lot of ads too of course.

This algorithm shift is also noticeable with less engagement on my tweets. I see fewer likes and retweets these days. My tweets are hopefully not too dull, but they’re not exactly carefully engineered to “do big numbers”. They’re in a medium-interesting middle-ground which now seems to mean they go ignored more. And, before you roll your eyes at my dreadful narcissism, I know this is not just me. Others have said the same. If you’re guilty of paying attention to how many likes and retweets you get, then you’re probably noticing fewer of them these days. Meanwhile the tweets we do see, from strangers who we’re not following, do often appear to be engineered for engagement. All the click-bait tricks. This includes posing questions where we don’t get to see the answer until we click. Video tweets usually do the same. We see the intriguing freeze-frame. We don’t get to see what happens next until we click.

How twitter made me an angry cyclist

Then there’s the outrageous statements, or the statements which we know to be false and it’s hard to resist replying (or at least clicking to make sure that somebody has set them straight. Invariably hundreds of people already have). Just another kind of click-bait, but I think this one gives rise to the idea that the algorithm is actively evil. The outrageous statements can be on matters which are innocuous yet somehow engaging, but of course it’s also on serious matters of politics, culture wars, nazis versus black-lives-matter etc. I imagine this is where the “twitter is full of nazis now” idea comes from. I think it depends what you engage with. Also I think a lot of people look at what Elon Musk is interested in and maybe extrapolate that as a statement from twitter. But by even looking at that, the algorithm begins to steer you onto these topics.

From my own perspective I really noticed this, not on nazi topics, but on cycling related tweets. I follow quite a few keen cyclists. It’s not a big obsession for me although admittedly it might sound strange to some people if I say that I am sad to no longer be cycle commuting since I started working from home. The algorithm has really pounced on this. It bombards me with videos of cycling road accidents (because I have to click to see what happens!) and people airing rather extreme pro-cycling or anti-cycling opinions. I do click on this stuff. I can feel the result is that twitter is making me angry, to the point where I feel I’m being recruited as foot-soldier in the war between cyclists and motorists. The really bad thing is that I can feel it effecting my psychology and spilling out into the real world. On the rare occasions when I do get on my bike these days, I feel anxious and angry at all the motorists around me, ready to go out and get into a confrontation with somebody. …until I calm down and remember the truth …the reality …the big cycling secret. It feels unfashionable to say it these days because nobody finds it exciting or anger-inducing but the secret is: most other road users are considerate, and the vast majority of cycle journeys, even in central London, are enjoyable and pass without incident. How boring.

You would never get that from my twitter timeline, and this, more than anything, has made me aware of the nature of “the algorithm”. It cuts across all topics of course. In politics I used to think a big problem was the “echo-chamber” effect. But nowadays we’re presented less with our friends agreeing with us, and more with a weird mixture of strangers agreeing with us, plus strangers with unhinged extreme opposite opinions to us. It’s not mild-mannered strangers who vote the other way. It’s people posting short ridiculous caricatures of political opinions. Extra engaging and extra anger-inducing.

Profit-driven enshittification

I think people tend to attribute too much malice to the algorithm. Nobody at twitter sat down and said “how can we make all our users more angry about stuff?”. They just wrote an algorithm driven by clicks. If we engage with something, we get shown more of it. Maybe the anger that emerges from it, is in some sense an expression of human nature.

But of course this algorithm is controlled by twitter, and it’s a shift from what went before. Previously we saw more posts from friends, and fewer ads. But maybe we still attribute too much malice because after all, it’s just about $$$. Blame and outrage is directed at Elon Musk. I’m not here to defend him (the guy’s clearly a bit of a dick, and I wish he would stick to rockets and electric cars) but I can well imagine all that really happened is, he told the twitter staff to take urgent action on profitability. Sort of fair enough I suppose. They already knew about various levers they could pull to make more money. Musk told them to crank the profit levers up to max, and that’s the effect we see. No big nefarious purpose. Just more ads, and more shovelling of engagement fodder posts instead of the (apparently less engaging) posts from our followees.

We call this effect “enshittification” thanks to Cory Doctorow, who writes excellently on the topic. But this ramping up of ads and other money-making patterns has limits. I’m sure the staff at twitter have worried about whether they can get away with pushing the profit levers, the enshittification, up so high. Will users put up with it? It all depends on the degree of lock-in we feel. In this post, Doctorow explains

“An enshittification strategy only succeeds if it is pursued in measured amounts. Even the most locked-in user eventually reaches a breaking-point and walks away, or gets pushed”

“the strong incentives for product managers, executives, and shareholders to enshittify results to the precise threshold at which users are nearly pissed off enough to leave, but not quite”

From my perspective twitter has crossed that threshold. I know plenty of others who (wisely it turns out) had a lower tolerance and left twitter earlier. And in my free-software loving communities such as OpenStreetMap, I know folks who never even touched twitter in the first place (who no doubt read this kind of blog post with a smug sense of “told you so”). Of course quite a few people didn’t leave, but I guess the fact that it lost 72% of its value is strong evidence that Twitter has done something wrong, and that something is “enshittification”.

The word “enshittification” is taking off as a way to describe the place we’re at with a lot of tech at the moment. Actually it seems like a very clumsy word for it. I do prefer Rohan Kumar’s suggestion of “User Domestication”. Like animals which used to run wild and free, but which gradually become dependant on their masters, users are eventually herded into enclosed online experiences where the tech companies can tame us, pacify us and then exploit us.

Twitter user domestication (Sheep at South Molton Livestock Market)

It’s not a surprise to me that, as a profit-driven company, twitter would start to seek that profit in ways which make things worse for users, or that the lock-in network effect comes to work against us in that circumstance. I was a bit surprised by how dramatically they have yanked on the profit levers to enshittify twitter over a relatively short timespan.

It takes a lot to escape user domestication, but the cattle are bursting out of the twitter enclosure right now and running wild across the prairie. Or at least there’s suddenly a lot of people looking around for alternatives.

Seeking new pastures. The move to mastodon

One way the cattle analogy doesn’t follow, is that there’s no real harm in signing up for other services to try them out. You can join the herd in many new cattle enclosures at once. The only question is whether you allow yourself to become domesticated. For example I have signed up and created my profile on “threads”, the new Facebook/meta twitter alternative, even though this seems like an utterly asinine choice for a new twitter alternative if you think in terms of hoping to avoid future exploitation as a domesticated user. Why on earth would you put down roots on a social network closely tied to Instagram, the very flag-bearer of the engagement algorithms we were trying to get away from? There are alternatives, including some operated by quite small friendly looking tech start-ups. No harm in trying them out, but… there is a better way.

Mastodon is very twitter-like but free and open source software, and decentralised software. A network hosted across many “instances”. I won’t go into the full pitch here, except to say that it’s a smart alternative to the depressing inevitability of enshittification we see on any for-profit hosted service. I recommend it.

However… I think the mastodon community (who seem to do a lot of meta-posting about how wonderful mastodon is) is a bit too self-congratulatory sometimes.

On-boarding of new users is confusing just because you’re immediately faced with this paralysingly important choice: which instance to join? I don’t think it really matters that much, or at least I don’t think it should matter, and people should feel free to post on topics unrelated to the topic focus on their instance (the etiquette is a little unclear but hopefully most folks feel the same). But the choice of instance does matter when it comes to the usual service hosting type questions. It made me laugh when people first started crowing about mastodon saying things like “because we don’t trust twitter with all our micro-blogging content” …the solution is to move to some server our friend’s friend has spun up in their free time? This even pre-dated the ability to migrate accounts between instances, but even with that ability, the reliability of hosting (the idea that my instance might disappear suddenly) is still a worry. Relatedly there’s the question of funding. People donating, or talking about donating, to their mastodon hosts. Coming from the twitter-hosts-millions-of-users-for-free model, this feels a bit fragile, and perhaps unlikely to be able to receive those millions of users.

Many of us moved from twitter and followed eachother initially using tools such as “debirdify” or “fedifinder” to slurp our follow-lists from twitter (This is based on mentioning your mastodon account in your twitter profile, so be sure to do that) This was a good little cheat that meant mastodon didn’t really need to build the community from scratch. Even so, it’s obviously a sub-set of my twitter followees who have made this move. To me it feels like a more tech-savvy sub-set. Mastodon is like in the early days of twitter. That’s nice in a geeky clique sort of way. Mastodon enthusiasts will say we don’t want to attract all of twitter onto the platform. But I would like my less tech savvy friends to be on there, and to me the mastodon experiment hasn’t really succeeded unless/until it can go mainstream enough for that to happen.

Decentralisation offers a lot of solutions, but mastodon is not guaranteed to succeed. I think it’s too early to pat ourselves on the back and say we’ve successfully used mastodon to free everyone from twitter domestication.

But having said all of that, mastodon is better than the enshittified twitter experience of 2023 / 2024. I am feeling a lot less twitter domesticated these days. Time to follow me on mastodon instead!

Job at Vivup


I started working at Vivup (vivupbenefits.co.uk). Actually I started there approximately one year ago, so in order to keep this blog bang up to date as ever… time to tell you about this!

Vivup helps organisations provide benefits to their employees. We have many large government clients such as NHS trusts, local councils, police forces etc, and increasingly bringing in private companies too. The benefits on offer are many and varied. One part of the website is like a shopping site with all kinds of products available as a “salary sacrifice” purchase, paid for over many months. There’s lots of bicycles available via cycle-to-work schemes, and there’s store discount cards, and all sorts of health & wellbeing offerings. If these benefits sound good, you’ll have to arrange to sign your whole company up to Vivup!

As you can imagine, it’s quite a big complicated website, and it’s a rails app! We have a tech team working on various aspects, including people focussed on the mobile apps, and the separate .NET bits. We also have people who are a lot more front-end focussed. They’re migrating more and more of the site to use React on GraphQL. Our sub-team’s focus is on the rails coding. There’s back-end aspects, the database and API integrations, plus a big back-office admin interface built with ActiveAdmin, and lots of interesting workflows and dataflows. To deliver improvements to all of this, we have quite a rapid pace of projects running in parallel, which is taking some getting used to. We also have a bunch of QA people, which is something I’m not used to. Quite a big tech team in total.

But the wider organisation feels big compared to previous jobs. I think in my previous big organisation jobs (e.g. 3mobile, B.P. & Alstom) I was in tech project teams isolated from the rest of the organisation, whereas at Vivup I’ve been involved in occasional bits of collaboration and communication across the organisation. Also it’s small enough that the whole organisation gets together on a quick video call each Friday, and for face-to-face social meet-ups three times per year. These are a lot of fun. I joined Vivup just in time to go along to the 2022 Christmas party in Newcastle, and we’ve just had the 2023 one in Liverpool.

Oomph! Wellness

I was waiting for the right time to blog about my latest job. As it has now come to an end, I guess that’s overdue! So let me tell you about where I spent the past ~8 months: Oomph! Wellness.

They’re a company providing training and content to care homes in the UK. Activities and exercises for old folks! They’re not a tech company, or at least not until recently. For many years they’ve been doing face-to-face service provision, with teams of people going out to care homes to train staff in how to run activities and exercises. Another service involves sending bus drivers to take residents out on trips.

And then there’s a third service “Oomph on-demand” providing activities and exercise related content on the website by subscription. This side of the business has been expanding rapidly, particularly during the pandemic lock-downs. By “content” I mean videos, or just written instructions/inspiration on activities. Also things like puzzles. There’s quite a variety, and they’re constantly building it into an extensive library. The nice thing is that they’re drawing upon all their experience of working in care homes, and it really shows. That kind of grounding in the real world is something that is so often missing in tech up-starts, and of course their past work gave them a foot in the door to sell this web-based service to many existing clients.

So it’s a low-tech company transitioning into a subscription web content company. In fact they’d made that transition and were already selling on-demand subscriptions on a site built within wordpress. That was done by an agency with many wordpress plugins and not much coding. It’s quite impressive how far you can push that approach, but the agency had to say “no” to some of their newer ideas. They were going to need coding, actual data-modelling, and development. They were going to need me!

Obviously looking at a wordpress set-up with a tangled mass of plugins, I was quick to tell them this whole thing was going to need re-building (Ruby On Rails was my suggestion!), but I’ve done a fair bit of wordpress tinkering over the years so I wasn’t afraid to roll up my sleeves and implement some of their feature ideas in PHP.

I added a surveying and reporting feature, which had some enjoyable data aspects to it. I’ve had fun with c3js graphs in the past, so brought that in.

That whole thing turned into quite a rich interface. I probably should’ve taken the chance to learn a rich javascript framework. I did learn the way of wordpress a little bit. For example wordpress developers have a particular way of doing database querying, and AJAX requests. I also learned more than I wanted to know about how to persuade IE to print graphs correctly! Maybe I should blog some more about these learnings.

I was doing these things to meet the clamouring needs of some existing customers, to make the sale to a large new customer, and to lay the foundations for a grant funded project. I gradually settled into a role which can best be described as “the one-man tech team”. I quite enjoyed the challenge of always designing solutions and solving problems single-handedly, and presenting the solutions and options to a less-techy rest of the organisation, but I did miss having fellow developers to bounce ideas off. I was always a little out of my comfort zone in that regard. Enjoyable challenge but… I’m looking forward to re-joining a developer team for my next job I think (although I’m keeping an open mind)

While juggling php features and bugs, I started to look at doing a big website rebuild project for them, and I had got as far as architecting this as a Ruby on Rails project. I planned out how I could do the work single-handedly, however this was going to be lot to take on. The alternative for them, was to bring in an agency to do the rebuild, and I think it was sensible for them to go with that approach in the end. A better way of getting it done in time to meet some grant deliverables, as well as to do some Salesforce work which I certainly didn’t want to go near! It makes a lot of sense, and I wish them well with it. They’re a nice bunch of people and they’re doing good things. So… if you run activities in an old folks home (not sure that’s my core blog readership, but you never know!) be sure to check out Oomph Wellness!

It also makes a lot of sense for me to move onto something different. It’s been an interesting time, but as I look for a new job I’ll probably look to get back into ruby, probably within a larger tech team. That’s more my comfort zone. I have a few other ideas about what I’m looking for, but I’m also pretty flexible and keeping an open mind.

2020 for me

2020, the year of the virus, has been pretty alright for me actually. For me it was never going to be a normal year, with a new daughter arriving in January. By comparison the virus …well it’s a big deal, but impacted me relatively lightly.

The wife planned to be on maternity leave for a full year, expecting lots of quiet mother-daughter bonding time. Instead she’s had to put up with me working from home. During that first lockdown, the 4 year old little terror was at home too. I remember a period of a few weeks where the virus was pretty big news, having clearly arrived in the UK, yet still not quite impacting anyone significantly. Then suddenly things moved fast. My company decided we’d all work from home. At the time it seemed very cautious on the part of my employers (OpenCorporates) and I suppose it was a nice expression of care towards employees. But as it turned out, it was only a week ahead of everyone being told to work from home (first lockdown). It was a relatively easy transition for us, since we’ve always been geared up to do our software development work from home. Just a matter of getting used to the whole team being at home the whole time. We’ve since ditched our office and gone remote-only, as I’m sure many companies have.

I remember some friends laughed off the virus as an overreaction at the early stages, and I did begin asking myself a question which I can’t help keeping in the back of my mind: “Will history remember it as an overreaction?”. There’s a paradox. With a multiplying virus we need to take aggressive measures before they seem necessary. If they work, they won’t seem like they were necessary. I don’t envy decision-makers dealing with that lose-lose choice. My twitter feed is full of people either explicitly advocating stronger measures, or expressing worries about the virus (which implies we should respond more aggressively). I think that’s right, however I can’t help also thinking ahead to a time in the future where we will all look back and say “well that was a whole lot of fuss about nothing”. It’s almost inevitable actually. Surely the smart (cowardly) choice is to hold off being opinionated until we can be wise in hindsight. That said, I do have some opinions.

The first lockdown was the big one and (anecdotally) it felt like people were taking the virus more seriously than they are now, in my corner of North Islington. Myself included if I’m honest. I remember heading out to the supermarket, trying to choose a quiet time, and trying to get in enough supplies to avoid any more shopping trips for a week. The streets were eerily empty in a way I haven’t seen since (No traffic. No people. Crows squawking on a nearby rooftop. End of civilisation kind of vibe).

I followed a few people arguing for wearing masks. In the early days there was concern that demand for masks would cause a shortage for NHS workers, which I guess was the main reason it wasn’t a big recommendation. That seemed like a missed opportunity in the early days of the virus response. There could’ve been more encouragement to make your own mask (seemed quite fun to me!) But masks continue to be a missed opportunity. In this second wave they’re easily available to all, and required when shopping, and yet somehow many people who frequent my local corner shop don’t seem to have got that message. Given what an easy virus solution masks are (compared to other restrictions and shuttering of businesses) it is a shame people don’t seem to be doing it enough. I can only imagine this means people are not washing their hands enough either. Am I arguing for more enforcement? Not sure, but with so much debate and anguish about the tough measures of closing schools/theatres/pubs/whatever, it’s a shame if these relatively painless solutions are getting skipped by many people.

These days the news is all about the different tiers, but (because we’re lucky I suppose) the different tiers make very little difference to our lives. For us the big thing is school closures. Whether the primary school and nursery remain open makes a big difference to our lives and … some news just in … turns out schools are closing again next week. Oh joy.

Educating/entertaining the kids more than anything else, has been the theme for the year. Our son just turned 5 and so he started at school reception class this year. The daughter is turning 1 and has started at nursery. During the first lockdown our son would have been in nursery, but it was closed for his final months there. More recently during his first months at school:

  • 1st They sent the class home for one week due to a covid case
  • 2nd We were all self-isolating awaiting a test result for my wife, which took ages (quite a bizarre experience and a story in itself)
  • 3rd He got chickenpox!

Three weeks at home in all. It feels like more, maybe because school closes earlier than nursery. Coping with this extra childcare has been largely down to my wife to look after both kids while I was working, since she’s on maternity leave. She found that quite tough. I think in general our lad’s at an age where we’re struggling to find ways to avoid just screaming at him for being naughty the whole time!

But it’s also quite an interesting stage he’s at. I thought we were doing ok at introducing him to maths and reading/writing (not just with the games I developed!) and I was imagining he’d be ahead of his class for the first half year or so. But no! At school they’ve been motoring on through to more advanced reading and writing and he’s been coming home with ideas about writing whole sentences. I need to develop some new games! The pace of his learning (and the teaching) is impressive. I think less progress was made when we were officially supposed to be home-schooling. There wasn’t much to the materials his teachers posted onto “google classrooms” (and wow that google sign up process is surprisingly broken). Probably we should’ve done better at adding more structure.

But I do quite enjoy coming up with whacky ideas to entertain him. The xmas cardboard “reindeer project” as a recent example. I’ve been taking him on regular walks to try to burn off his energy. Avoiding unnecessary transport, this means we’re restricted to his walking range from the house around the neighbourhood, so I’m taking it as a challenge to avoid route repetition (That’s for my own entertainment. He’s probably not bothered), and I’ve been getting some OpenStreetMap address detail mapping while I’m about it. Sometimes I can interest him in reading people’s door numbers.

The 1 year old daughter meanwhile, is mostly a smiley bundle of joy. Of course it’s a kind of nappy-filling joy, and she’s less joyful when she head-buts the furniture (which is often), so it is also hard work looking after her. Because of the virus our family (especially the grandparents) are sad to be missing out on seeing all the rapid growing up that babies do in their first year. I had worried that she would be missing them, and generally missing meeting anyone outside the house, but she’s started nursery now, and seemed to take to it reasonably well.

But my wife has had a lot of extra time doing childcare and got a bit fed up of it at various stages. During the first lockdown she was craving an escape from the house, while I felt quite at home staying at home (I’ve been practicing for this moment my whole life). So she went online and booked a summer holiday cottage in Cornwall. Bit expensive as they were booking up fast, but we had a very pleasant week in Portscatho. The week before we were due to go there, the evening news had a piece on how Cornwall was overrun with visitors. Certainly when we went to the bigger town of St Ives, it was packed, but otherwise not so bad. Shopkeepers there were taking the virus very seriously. Strict distancing and mask wearing enforced. On the whole the biggest virus risk felt like the dirty overcrowded service stations on the motorway to get there.

The day job at OpenCorporates has been keeping me very busy. Working from home does make a difference and (this seems to be unusual but) I’m trying to decide if I like it. I’ve always allowed the day job to spread into my evenings but that feels different now that there’s no office time and no cycle home to divide up the day.

My OpenStreetMap obsession has fallen by the wayside in 2020. I think this is mostly because there’s been no OSMLondon pub meet-ups, which in turn makes it easier for me to forget about OpenStreetMap. I’ve neglected the Communications Working Group for example. Sorry about that. Hopefully OSMLondon pub sessions will resume in 2021!

So overall 2020 saw a lot of focus on family. That’s kind of a nice way of saying “focus on coping with childcare” although …it has been good on the whole, and I’m sure this year has been harder for many people. There’s some things to worry about with what’s coming in 2021, but let’s be optimistic. Happy New Year!

Linear time calculator

I keep doing the same calculation over and over again, so I decided to make an online tool specially for it:

>>> Linear time calculator <<<

It’s a simple enough calculation to work out a time estimate for something based on progress so far (assuming progress has a number). Could be used for anything I suppose, but I keep doing this for long-running data processing scripts, where a count is being spat out to a log file e.g. a count of records processed.

Here’s how the tool looks for script I was running recently.


Maybe I should’ve just worked out how to do the time format conversion and arithmetic properly in a spreadsheet, but why use a spreadsheet when you can build a whole dedicated tool for the job?!

There are some similar existing online calculators. You can do the time between two date times on timeanddate.com which I suppose is the error prone part.

Various online calculators do a speed calculation (division!) If you imagine “miles per hour” is “records per hour” it works.

Successfully spawning offspring

Our second baby arrived last week. Here she is. Our new baby girl!

It was a planned C-section, and all went to plan, in contrast to the unexpected twists and turns of our boy’s birth which I blogged about four years ago. She seems partial to sleeping, sometimes for six/seven hours straight, and the breastfeeding is proving to be less stressful this time. Some of the same problems as last time, but this time we know what to look out for, and what we wanted to avoid!

A second child is an important milestone in my life. My chosen mate and I have now successfully spawned offspring in sufficient numbers to replace ourselves and project forth our genetics to a new generation. I do feel like a successful organism now.

But we humans have long lifespans, and we rear our young for several years before they are ready to face the world on their own. I can confirm that even my older offspring, now facing his fourth winter, still requires food and shelter supplied by me, although he has a heightened sense of foraging where chocolate is concerned. By convention (for we are socially advanced creatures with widespread social groupings and strong social conventions) I understand we normally rear our young for over a decade. Eugh! So I definitely can’t declare myself a completely successful organism yet.

But worse, we humans are an advanced intelligent species, capable of not just of globe-spanning social interaction, but of contemplating the overall health and ongoing survival of our species within its environment. We can project population growth and demands on resources, and when we do, we see that humans are on course to overpopulate, exhausting our resources, and polluting our environment. As an intelligent organism, perhaps by reproducing I have failed after all.

But we are a highly intelligent inventive and increasingly technological species. We might find ways to collectively solve the environmental threats we face, and/or we might build a sustainable community on another planet as an insurance policy for the survival of our species. We might, but no guarantees we’ll achieve this in time. We should try.

And by “we” I mean my offspring. Astronaut training begins today! Here she is trying on the jetpack in our mini neutral buoyancy pool:

Open Data Camp

I went to Open Data Camp 7 last weekend.

It’s an event I’ve been hoping to get along to for years now. Always lots of interesting looking twitter chat coming out of it. But this year it happened in London, and I managed to be on the ball getting a (free) ticket before they sold out.

I hadn’t particularly realised it’s the “unconference” format. It reminded me of the London “wherecamp” events I was involved in once upon time, although OpenDataCamp is even more unconferency. I mean the sessions were all very conversational. Didn’t see anyone with prepared slides, and often heard less from the session leader than from other people chiming in.

It was also different from wherecamp because there was a “pitching” session at the beginning to establish all the ideas. Those were then written up as a very busy schedule with four/five parallel tracks! Here’s all the sessions listed. In fact that has links on to more notes on each of the sessions too. Here’s the session choices I made:

  • “Bad data” – Discussing the precision of library visitor stats. I gave the example of bad data in company registries!
  • “Quantifying value of open data” – How to measure/justify the value of open data by tracking usage. The great unsolvable problem.
  • “Sorting out configuration so we can do cool things” – Really boiling down to: software can be hard to install and get working. Another great unsolvable problem.
  • “SPARQL 101” – A very quick intro to linked data querying. SPARQL playground and epimorphics qlassroom.
  • “What is non-commercial?” – Or does non-commercial really count as Open Data, or what other ways are there to avoid giving away all your value? As applied to the NHS!
  • “Open Procurement data for less carbon” – Influencing contract and procurement processes to (be more open and) reduce carbon emissions
  • “Joy Diversion” – An Open Data Manchester idea which I was always curious about. Walk round the block looking out for joyful serdipidousness. Quite fun. Quite similar to OpenStreetMap mapping (not as good!)
  • “OpenStreetMap” – My Session! I tried to introduce OpenStreetMap editing and using. Mentioning tags quickly led to questions and conversational diversions (as documented in the session notes) That made for an interesting different way of presenting it!
  • “Learning from our data mistakes / Open data horror stories” – People confessing failed attempts to open data, or opening data leading to other problems.

Some of the discussions gave me a strong sense of deja vu. We’re still collectively bashing our head against some problems of open data which feel unsolvable (not that I didn’t enjoy discussing them anyway!).

I also enjoyed bringing some OpenStreetMap (hobby) and OpenCorporates (day job) perspectives to discussions. I just realised I have over a decade of open data experience in both hobby/volunteering with OpenStreetMap and in my career spanning CloudMade, TransportAPI and OpenCorporates now, but I don’t always think of these things in abstract open data terms. My overriding impression of Open Data Camp was an enjoyable feeling of being among open data people again!

Think tanks and financial transparency

Last week’s Question Time introduced one of the panelists, Kate Andrews from the “Institute of Economic Affairs“. What an organisation name! It sounds so official. So governmental. So trustworthy

…yeah not so much.

The name sounded familiar to me from some recent reading on dodgy think tank funding transparency. It prompted me to look up their rankings on transparify.org:

I’ve improved this image from one found on the transparify blog. You can read much more about how and why transparify have ranked the think tanks on the website. Transparify have zoomed in on the issue of funding transparency because it’s important. If we are to have these “think tank” organisations dedicating themselves to swaying public opinion one way or another, then isn’t it important to know who is paying them to do it?

Well maybe it depends how influential they are. It occurs to me that “think tanks” exist as part of a fuzzily defined spectrum. An individual can set up a wordpress site and start pumping out blogs and social media posts pushing their political viewpoint. That’s a wonderfully easy thing to do, and I’m glad. But then if you name this website “Institute of X” and pretend to be an organisation not an individual, then you’re well on the way to being a “think tank”. If some wealthy backer wants to pay you to hire a team, and work on pumping the messages full time, then that is a think tank. It’s a natural progression to this situation where it feels like there’s a cost-effective way for wealthy individuals or organisations to buy influence over our politics.

But it’s OK. The key remaining barrier is whether we all grant them acceptance and influence (and whether mainstream media do e.g. by inviting them to be panelist on BBC Question Time!). We should be rejecting organisations who fail to disclose exactly what they are, who they are, and who they are funded by. If we don’t know who is paying the salary of Kate Andrews from the “Institute of Economic Affairs”, isn’t it rather sinister that she has been able to gain such a platform?

If you saw the show then you may remain unconvinced, after all Kate Andrews was talking a lot of sense on there right? She’s intelligent, charismatic, and persuasive. I was finding myself agreeing with her even as I was surfing around the transparify website! I was waiting for her to drop a right-wing free-market help-the-rich-get-richer bombshell. She talked on a lot of topics without touching on this, and when it came it was so subtly slipped in at the end of a debate on climate change, that it passed without any cross-examination.

Maybe that’s just good politics. Don’t try to ram your message down people’s throats. Say things that people agree with, and slip your message in. But I suspect that kind of obfuscation game is a think tank speciality. A while ago I followed a tweet to humanprogress.org, and it’s bothered me ever since, because again I found myself agreeing with so much of it. I enjoyed these optimistic pro-science articles, but look out! Slipped in there you’ll find various dubious messages: “inequality doesn’t matter”, “deforestation isn’t happening”, “fossil fuels are the best energy source”, “be happy with your menial existence and help us get richer”. OK I made that last one up, but the others are all messages in articles on that site, all cunningly tucked between other more enjoyable (and more correct) writing. humanprogress.org is one of many sub-brands of content pumped out by the  American right wing think tank with wealthy backers: Cato Institute.

But I guess that’s just dodgy politics as usual, and I’m criticising Cato Institute based on my own political views. I’m quite happy to do so, but to be fair, they do OK on financial transparency. Middling score from transparify.

What transparify do so well is to focus on just that one aspect of think tanks, financial transparency, in a non-partisan very objectively measured manner. It’s possible to do so, and while it won’t solve all the problems with dodgy think tank politics, it is a great starting point for penetrating the sinister goings on behind these organisations. It should be a clear “untrustworthy” red flag if organisations regardless of political flavour, are failing to disclose details of who is backing them financially.

Which dodgy characters will the BBC be inviting onto Question Time tonight I wonder?

Counting game

I made a little counting quiz game for my 3 year old to practice his counting:

>>> Counting game <<<

I’m hoping some other kids at the “learning to count” stage might find this fun. Just click the link to play (not on a phone)

Little video of him playing it

When we talk about “learning to count”, there’s actually a couple of quite different skills. Speaking the numbers out loud in sequence “One… two… three” …  our lad has been good at this for over a year now, but recently I’ve been trying to get him to look at a set of objects and tell me how many there are. Not the same kind of “learning to count”.

So far he’s quite bad at this. It seems like one type of counting skill should help with the other, but sometimes I’ll point and say “how many?” and he’ll just wave his finger at the objects while counting to ten very quickly. Almost like he’s too good at the other skill, he’s not getting the idea of counting objects!

Naturally my solution to these things is to spend hours coding something. Here it is on github. It needs a bit more work. In particular this is no use at all on mobile at the moment (or tablet). You need a keyboard to press the number keys. This and a few other issues listed here.

Oh and thanks to various people for the open licensed images (credited on page), and thanks to wikimedia commons for helpfully naming them “x white backgound”.

If you missed it, check out my previous kids game Keyboard Letters (+previous blog post)

space4 – OpenStreetMap games and engagement talk

I gave the following talk at “space4” the other day, at an event called “Mapping for change – games and other processes of public engagement“.

My talk was an introduction to OpenStreetMap, which is something I haven’t done in a while. And I also went into more detail on the different ways of contributing, from complicated things like JOSM, right down to the very simple OpenStreetMap Notes.

Showing the OpenStreetMap ecosystem diagram (There was also an appearance for my newer workflows summary diagram) – Photo by Carole Wright

Space4 is a new funky little tech hub place, of which London has many, but exciting for me because it’s half a mile from where I live! It’s nice to see Finsbury Park attaining this badge of trendiness, but also it’s super convenient. I can even cheekily duck out of family commitments to attend an event for an hour as I did on this occasion, although it was I shame I didn’t get to stick around and chat to anyone. I did get to answer questions, and it seemed like an interesting audience. I’ll have to try make it along to more events there, perhaps more in this series titled “Mapping for Social and Political Change“.

Below are the slides and transcript, or at least what I intended to say. You can also download the slides in OpenOffice format (34Mb).

Jump to slide:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24


Slide 1



I’m going to present my introduction to OpenStreetMap and then, in relation to the theme of this evening, I’ll run through some app and gamification ideas and other ways we try to encourage participation in this project.


Slide 2



My day job is another interesting open data initiative at a company called OpenCorporates, but that’s a whole other talk! I found this job via my interest in open data and OpenStreetMap actually, but for OpenStreetMap I’m just a spare-time volunteer.

And that’s typical actually. It’s a very volunteer driven initiative. There is a small company called the OpenStreetMap foundation, but there’s only one admin assistant employee. OpenStreetMap is a not-for-profit project with lots of passionate volunteers, although commercial use of the data is allowed, and a number of companies provide services around OpenStreetMap.


Slide 3



So what is OpenStreetMap? If you’ve only taken a quick look, you’ll perhaps have the idea that OpenStreetMap is a competitor to Google maps. It kind of is that, but that’s really not the whole story.


Slide 4



You only have to zoom in on the map and you’ll start to see that this is a bit different. Here’s where we are at space4. In many areas there’s this kind of very rich detailed data, but you can also get the sense, I think, that this is a very human-crafted map.


Slide 5



OpenStreetMap is a community of now over 1 million people who have made contributions to the map. It’s a “mass collaboration”, and engaging many people in participation has been the name of the game really.

I’m going to talk more about how we do this, but essentially a key aspect is making the map editing process (which is inherently quite complex) as simple as possible.

The project is called “Open”StreetMap, not just because it is open to edit, but because we release all the data openly licensed as raw vector data. I’ll explain that more, but I think when it comes to community engagement, a key motivator is simply seeing an updated map, taking in the changes people make, and showing a new improved map with a rapid turn-around.


Slide 6



Everything in OpenStreetMap is made up of “nodes” and “ways”. This is what I mean by “vector data”. And then attached this we have “tags”. These name=value pairs.

This is a simple data model, helping drive simplicity of the editors, but particularly with the tags, it’s also very flexible and it means you can put your own ideas into OpenStreetMap for things to be represented on the map. The tags are an interesting aspect of OpenStreetMap’s design which really makes editing simple and fun, even if it means the data is less tidy and usable than it might be. That’s a bias which OpenStreetMap has. A focus on simple fun contributing.


Slide 7



What about using OpenStreetMap?

Well you can do the Google maps thing of embedding a map on your website. Also you can build interfaces involving clickable markers etc, with OpenStreetMap as the base-map.



Slide 8



But the raw data from OpenStreetMap has allowed developers to do so much more. Here we see…

Open Source Routing Machine” doing route planning across the road network.

OverpassAPI” doing flexible queries within the rich tagging data (show me all the pubs in London)

Nominatim” search system, finding a street. Here we also see it showing the internals of how exactly that search is matched.

There are endless other examples powerful uses for the low level geo-data.


Slide 9



Of course one main use is to have a view of the map. But what kind of view? With access to raw geo-data we can create a customised “rendering” of our map view, choosing different types of data to emphasise and different ways of colouring and styling it.

Top left – Thunderforest Transport Map

Top middle – Quakers Whiterwater map

Top right – hikeandbikemap.org

Bottom left – OpenSnowmap.org

Bottom right – OpenCycleMap.org


Slide 10



So the raw geo-data flows out from this central OpenStreetMap database, out through an ecosystem of various tools and services which developers are providing and refining (often as open source), out to many many website and mobile apps.

If you’re not a techy developer yourself then this might not seem that exciting, but just think… by contributing to OpenStreetMap you’re not only making a view of a map on openstreetmap.org. The data flows out and gets used in a wonderful variety of different ways.


Slide 11



So I’m going to move on now to talk more about the different ways you can contribute to OpenStreetMap.

I thought I would start with the more complicated ways first, so don’t be put off. I hope to show you that you can contribute in very very simple ways too.

But this is the way I often edit OpenStreetMap. It’s a tool called JOSM. Here we see the photo mapping feature. There’s a photo I took of some shops near here, and little icon here shows roughly where I was when took this.

JOSM is a powerful editor, but actually even if you are just getting started with OpenStreetMap, if you don’t mind learning new software, I recommend diving in and giving JOSM a try, because ultimately it may give the most satisfying editing experience.


Slide 12



Most people though, will begin by trying the editor on the website. On openstreetmap.org if you click this “edit” button here, you can edit directly with this tool called “id”. No need to install anything.

This is simpler than JOSM, though still quite powerful. It’s easier to get started with.


Slide 13



Here’s how you can edit using a smartphone. An app called “Go Map!!” on iPhone and “Vespucci” on android.

These are simplified editing experiences for mobile, however both of these are what I would call “fully featured” OpenStreetMap editors because you can make edits to the “ways” as well as the “nodes” and “tags”. You can edit the geometry of buildings for example.


Slide 14



But if I have to make one app recommendation, it would be this: Maps.ME on iPhone and Android is a really great map viewing app. It looks a bit like google maps, so it will feel familiar. But it’s OpenStreetMap! Install it now!

This also lets you download whole countries for use offline, including satnav style routing. It’s great for using without an internet connection while on holiday. But I use it as a really nice useable map view for getting around all the time.

Maps.ME does also have editing functionality. This is an example of a simplified feature-limited editor. You can use it to add a missing shop for example, but only basic edits. It’s not as powerful as Go Map!! or Vespucci.


Slide 15



So you can use your gadgets, but I want to stress that gathering data can be as simple as using a pen and paper. Of course you’ll still need to input your data later using JOSM or “id”, but data collection can be very simple and low tech.

…although some people get quite advanced and quite complex with their paper notes!


Slide 16



And here’s an interesting tool which bridges the gap: fieldpapers.org

This gives a nice easy way of getting a map printout. It creates an “atlas” of many A4 sheets. Here you see the interface where you’re arranging a grid of A4 sheets to cover an area of interest.

The printouts can then be used for noting down more data which needs to be added to the map. If you want to get clever, these can then be scanned in, and a QR code geolocates the scanned images for use within an editor (but you can easily skip that and… look at the paper!)

Because the data collection is very low-tech pen & paper, this has been used for humanitarian mapping, with local people in Lubumbashi writing down things like street names, and then the paper brought back to London, where we input the data at a fun mapping event.


Slide 17



Speaking of humanitarian mapping (this is a whole other area I didn’t go into) check out the “Humanitarian OpenStreetMap Team”, and a project called “Missing Maps” which is having very popular events every month here in London.

But this is a tool we use to coordinate mapping a lot. The “Tasking Manager” at tasks.hotosm.org. You can see the idea. We divide up an area into a grid. The yellow squares have already been done, and you can acquire a square to work on. It’s a coordination tool which means we don’t accidentally work on the same area even if we have a large number of people mapping.

This is for remote “armchair mapping” where we can sit at home with our laptops and contribute using aerial imagery, to draw around buildings for example.


Slide 18



Getting more towards “gamification” side of things, this is a tool called “MapRoulette”. It assigns random little tasks for you to work on (by making edits in the OpenStreetMap data)

Often the “challenges” are created based on data from external sources. Here for example it’s showing Police stations in London, where we could be adding some address data, taking the addresses from some government dataset. So we spin the roulette wheel, and we’re shown a random London police station, and we click to add the address data and then go again!


Slide 19



I wanted to show you a couple of really simple ways of helping OpenStreetMap which don’t involve learning how to edit.

Mapillary is a thing you can contribute to, particularly if you drive a lot. You simply set your phone going taking photos as you drive, and these form a thing a bit like Google StreetView.

But Mapillary works very closely with OpenStreetMap, and allow us to use the images while editing. If we gather enough of these street-level images, it becomes a very powerful supportive tool for OpenStreetMappers. We can easily use it to double-check our data and directly add things like missing shops.

The app is really easy to use because contributing is very passive. You just sort of set it going, and you can almost forget about it. Later you upload lots of photos.


Slide 20



Finally this is perhaps the simplest way to contribute OpenStreetMap: A thing we call “Notes”. Just go to the website and click this button here to add a note. Place a marker and write a textual description of a thing which is missing or needs to be changed on the map. Easy! It doesn’t even require you to log in.

Like Mapillary this is a slightly indirect way of helping. It still requires people who know how to edit, to then go and make use of this information to edit the map.

But this is very much designed to be a very simple way to contribute. It’s dead easy, although do make sure you zoom in and place the marker precisely, and be as descriptive as you can in the text.


Slide 21



Here’s a summary diagram showing different ways of contributing.

It’s split into “gathering data” and “inputting data”, because often we do things in those separate stages. The mobile mapping approach using Go Map!! or Vespucci will cut across these, because you can make the edit directly while you are out and about seeing things to add. This is perhaps not as useful as it sounds because, tapping away with your finger to add all of data is very slow. If you’re adding lots of things, or even if you’re casually doing some fly-by mapping, I tend to prefer a two stage approach. I snap quick photos on my phone (which are geo-located), and then later at home I view these with JOSM to add the data.

We also talked about Tasking Manager & MapRoulette. These options at the bottom only involve a data input phase without the need for data gathering. But as with the other projects we’ve heard about this evening, OpenStreetMap is a great excuse to go out and look at the world! If you like that sort of thing, you may prefer these other modes of contribution.


Slide 22



Beyond editing tools, we also have a few tools like this which show us stats on our OpenStreetMap editing. This helps make editing fun and engaging, and naturally a little competitive! This is “how did you contribute?”, and on the right a tool on MissingMaps.org which shows edit stats related to humanitarian mapping with the tasking manager.


Slide 23



So that’s OpenStreetMap!

Do check out the @OSMLondon twitter account for news of OpenStreetMap events happening here in London.

Thank you very much!


Slide 24



For my part, I release these slides under the Creative Commons Attribution-ShareAlike 2.0 License

However see the notes on the slides, for other specific attributions.

Any standard OSM map images are cc-by-sa2 OpenStreetMap contributors.




Jump to slide:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24