Yak Shaving and Woolly Jumpers

116_1269
Photo by Glen MacLarty – A woolly jumper being knitted (…probably not from Yak’s wool)

Do all developers know the phrase “yak shaving”? Apparently not. I see it being introduced to a new generation of developers here.

What does it mean?

“yak shaving” …the alternative explanation.

You’re an engineer. You have a problem:  you’re feeling a bit cold.

  • You turn up the heating, but you decide it’s better to find a jumper to put on.
  • You find a nylon sweater, but you want to do this properly. You need a woolly jumper.
  • You go out to the shops to buy a woolly jumper. There’s good options, but nothing in a perfect size and shape for you. You decide the only really kick-ass solution would be to knit yourself a good thick woolly jumper.
  • You look through some wool you already have, but since you’ll be spending quite some time on this knitting project, you decide you want some really premium quality wool.
  • Back to the shops to select the very best wool, but you’re not satisfied. If it’s worth finding good wool, you decide it’s worth going to the source of the supply chain where the wool is produced.
  • You go on a multi-day trek around the top wool-producing sheep farms, but as you learn more about wool you discover that the very best type of wool is from the Himalayan yak.
  • …and so it is that you end up on a hillside in the Himalayas, in the baking hot sun   …shaving a yak.

Origins

I like this explanation, but it is “the alternative explanation”, because I’ve no idea where it came from! Can’t find any reference to it on the internet. I think I was taught this by Andy Allan while I was working at CloudMade. A few years ago I introduced “yak shaving” to workmates at TransportAPI, to much hilarity.

For the real origins of the “yak shaving” phrase, all googles lead to Seth Godin‘s blog, which references, Joi Ito’s blog, which references an O’Reilly Book on productivity, and [oopse. now that’s going a bit recursive], but  no there’s lots of pointers to the real origin, which I find a bit disappointing: A “Yak Shaving Day” segment of a Renn & Stimpy episode.

I suppose it’s disappointing because Ren & Stimpy is so unbearably bizarre I can’t get my head around it, but also I was hoping for the origins to have some better link with our tech community use of the “yak shaving” phrase. Oh well.

Government snooping and breaking encryption. Hurrah!

Bletchley Park Bombe MachineI was just looking at some old photos from this time last year. A year ago the weather was a lot better and I finally made it to Bletchley Park.

It filled me with chest-thumping patriotic pride to think of these code-breakers puzzling over intercepted nazi messages, and inventing brilliant machines to break the code. A good old bit of government snooping and breaking encryption. hurrah!

It was a curious coincidence to return home on the same day, and see on the evening news, the home secretary Amber Rudd declaring that it was “completely unacceptable” that the government could not read messages protected by end-to-end encryption. There had been a small terrorism incident in which a few people were killed by a nutter with a gun at Westmister. Hardly deserving of the ‘T’ word. Apparently he had sent a WhatsApp message. So naturally this was being used as an excuse to enlarge government snooping powers. A good old bit of government snooping and breaking encryption. hurrah!

The enigma code was a different matter though. British code-breakers versus the nazi war machine.

It’s a well known story, but I learned a few things at Bletchley park, particularly while putting my stupid questions to a member of staff. I asked about what came before enigma. Radio signals had been extensively encrypted by various communication networks of the German military for years before the outbreak of war, and Poland had been extensively snooping, mapping out all these networks (a complex challenge in itself), and breaking various levels of encryption in a cat ‘n’ mouse game. They were able to pass on all of this information to the Brits. We were only carrying on with the game.

But with enigma machines the nazis thought they had an unbreakable code. To crack it, Alan Turing built the Bombe. This is often described as a forebear of modern computers.  The star attraction at Bletchley Park is a working (moving!) replica. It moves with spectacular spinning and clunking noises. Watching the way it moves, it’s obvious that the spinning drums are performing a kind of brute force attack, trying every combination to break the encryption. But the replica is not complete, and actually seeing behind the drums where they are still missing, is very revealing. Metal brushes on the back of the rotating drums, brush over these metal contact pads. Behind the scenes they’re wired up so that an electrical current flows and the machine suddenly stops when it hits upon the correct combination.

Bletchley Park Bombe Machine

I was amazed and delighted by the simplicity of that idea. The whole giant contraption is just a great big electrical circuit, with some mechanical movement thrown in. We can see and understand the machine at a very low level, in a way which is much harder with modern computers. In computer science studies we learn about all the lovely layers between applications like this web browser, right down to…  well ultimately an electrical circuit. However I actually deliberately chose a computer science degree course which didn’t involve hardware, because I found that low level stuff uninteresting. I have a vague understanding, but maybe I should try building tetris with nand gates some time!

Although there’s a beautiful electromechanical simplicity to the Bombe, the less low level aspects (the details of the cryptography problem it’s actually trying to solve) are harder to understand. There’s a good numberphile video explaining some of it.

To come up with these solutions in 1940, Turing was surely a genius. He clearly knows the necessary hashtags. We rewarded him with chemical castration as a punishment for being gay, which drove him to suicide. Hurrah Great Britain!

New job at OpenCorporates

A month ago I started a new job. This is a big deal for me because I’d been in at TransportAPI for eight years. But time for a new job in 2018! I’ve just started at OpenCorporates.

OpenCorporates provides company listings data. It has a database of corporate entities, rather like Companies House in the UK, but bringing together this and similar datasets from countries all around the world.

Now “company listings data” may sound rather dull, but there’s a couple of aspects of this which I think are quite exciting: open data and… fighting crime and injustice!

Open Data

At one stage the Companies House website was only open during working hours (Yes. Hilariously the website’s clunky querying interface stopped working after 6pm!). Aside from making itself a target of general ridicule from tech people, companies house also became a target of the emerging “open data” campaign at the time. About a decade ago the guardian “Free Our Data” column was leading the march, campaigning for data releases. I also remember attending a “National Hack the Government” event hosted by the guardian, with all sorts of hijinks of civic hacking tech people who were producing better versions of government websites and, if necessary, taking data by force (brute force scraping) to do so.

From these kinds of mischievous beginnings OpenCorporates was born. I remember bumping into Chris Taggart at various conferences and events around this time, while I was generally running around beating the OpenStreetMap drum. Later I ended up working with TransportAPI, and TransportAPI ended up sharing an office space with OpenCorporates at the Open Data Institute for a while.

Companies House went on to get their act together considerably (spurred on by those early civic hackers no doubt). Meanwhile OpenCorporates continues to campaign for open data releases from other governments around the world. My point is, OpenCorporates is very much historically and currently part of the open data movement. I regard any open data work as a kind of “social good”, but if you think about it, company listings are really a flagship example of the kind of dataset any government absolutely should be making public and easy to use.

Fighting crime and injustice!

This brings me onto the other aspect. A more dramatic kind of “social good”. OpenCorporates, by making company listings data more available, is fighting crime and injustice …if a little indirectly.

A “registered company” might be a high-street shop or some other real-world bricks-and-mortar thing, but often not. Registered companies are legal and financial constructs which get used and abused in all sorts of ways. We see examples of dodgy dealings on any episode of “rogue traders” where it invariably reveals how bad people closed down a company, and re-started under a new name to shake off the authorities. It’s clearly too easy to get away with. Company listings data provides a little bit of an antidote by helping with transparency and knowing who you are doing business with.

But that’s just small time fraud. When I started at OpenCorporates they gave me a copy of the book “Treasure Islands: Tax Havens and the Men who Stole the World” by Nicholas Shaxson. It describes tax avoidance, tax evasion, money-laundering, and flows of criminal funds enabled by offshore “secrecy jurisdictions”. It builds up a picture of this as not just a side-show by a dishonest minority, but widespread deeply embedded system of corrupting influence benefitting the rich and powerful, at the expense of everyone else. The problem feels unsolvable as we realise that the corruption runs deep within all of our governments.

“Rich governments cannot be trusted to do the right thing on tax havens and transparency. Many will demand more transparency and international cooperation even as they work to frustrate both. They will call for reasoned debate as they engage in character assassination  secret deals and worse. They will talk the language of democracy and freedom the better to defend unaccountable, irresponsible power and privilege.”

It’s a pretty eye-opening and pretty horrifying book!

The book details some specific financial tricks and loopholes, but generally the idea for a global company (or crime syndicate, or just a wealthy individual) is to have your accountants and lawyers create a tangled web of different corporate entities registered in different jurisdictions, with maximum secrecy making it almost impossible for tax authorities or financial regulators to pin down your assets, profits, risks, and capital. There are many layers of financial trickery, but it’s clear that corporate registrations are the key vehicle for this kind of skullduggery.

As such, data on registered corporate entities is vital. It doesn’t solve all the problems of financial secrecy. Investigators will need to piece together a lot more about companies, their assets, the true owners and beneficiaries, and their links with other entities, in order to cut through the information void created deliberately to bamboozle them. But it’s a start. Company listings form a kind of information bed-rock upon which other information and investigation can be built. Whether it’s an investigative journalist looking to crack open a high profile case, or a tax authority, or a just somebody looking to establish the trustworthiness of a company, OpenCorporates helps enable this important work. It puts the information out there more publicly and in more useful formats to help shed a little bit of light on this murky world of crime and tax dodging.

Maybe “fighting crime and injustice”, is a super-style exaggeration. Folks like the Tax Justice Network are tackling this big problem more head on, but OpenCorporates helps chip away at the problem with a more bottom-up data-oriented approach.

The new job

So I’m getting my head into this new subject area, and joining OpenCorporates as a software engineer. It’s primarily Ruby on Rails development, but with lots of infrastructure and data processing challenges. Fun stuff! We’re based in the main pointy tower at Canary Wharf. We just had our office warming party in a new place 42 storeys up.

If you fancy joining me there, we are hiring back-end engineers, infrastructure engineers, and data analysts. Check out jobs.opencorporates.com!

Printable Calendar tool

Here’s a little thing I just found (I have lots of silly bits of code like this kicking around) :

Printable Calendar

Generate a calendar in a strip layout, suitable for printing, with lots of space for writing notes next to each date.

I originally did this as a visual basic word macro, then later as this php script.

I used to use this to do big calendar printouts to help with family logistics, usually at this time of year, covering the months around Christmas. That’s back when we four kids were all home for Christmas in Yorkshire, all with our own overlapping plans. Nowadays we might do a shared google calendar instead, but I remember the scrawlings all over the calendar wall used to be quite fun.

A tech manifesto from 2007

I’ve just been tidying some old content on this website, which I’d written just over 10 years ago now. Back then I wrote a kind of tech manifesto, or at least a collection of various broad thoughts of tech and the IT industry I was working in at the time. I haven’t worked as an enterprise integration consultant since 2009, and some of it is out-of-date in various other ways. Some of it feels like it’s noticeably coming from my naive younger self.

  • I hate the word “geek”. I don’t really hate the word of course, and that’s not the point I was trying to make. I stand by the idea that we should always work to close the separation between “geeks” and “normal people”. The march of technological progress does this for us of course. If I think back to 2007, the internet was actually a lot less mainstream. Fewer people with broadband at home (including me!). Fewer people required to use the internet or even computers as part of their work. It used to be that “geeks” were people who knew how to use computers and were super savvy with the internet. Nowadays that’s everyone. My 2 year old son is already getting the hang of it! Nowadays I see an interesting push to get more people from more diverse backgrounds to learn to code.
  • It’s a people thing was a piece complaining that clients should discuss high level requirements rather than skipping ahead to designing a solution. This is an accepted anti-waterfall principle rolled into “agile” these days. Perhaps it goes without saying, …except it’s still a common mistake. I recall a few occasions since writing that, when I’ve worked on projects which jumped to discussing a technical solution before getting a high level view of problems we were solving.

    I also talked about user interface design. I think I had a bit of a bee in my bonnet about the project I was on at the time, but I do remember leaving that particular project with great satisfaction at having implemented some of the UI ideas despite initially having them shot down. Since then there has been a couple of times where I’ve found myself surprised by colleagues’ failure to see obvious UI improvements. It makes me wonder whether user interface design is a talent I’ve not really appreciated within myself. Maybe I should do more of that kind of work.

  • With IT project politics, I was talking in general terms, but really I was bearing my soul about some frustrations with my consulting job. Some of the assignments with Green Hat Consulting involved parachuting into a pretty hostile environment. When I quit that industry and went to work in a more fresh funky start-up I left the politics behind on the whole, but of course you never really escape that kind of thing completely. I guess the golden rule I still have to remind myself of would be: work with people you like (and if you don’t, leave, because life’s too short)
  • Maintainable Software. Maybe I had a fairly simplistic view of what makes good software back then, but I think I was just glossing over the details. Obviously there’s a whole universe of coding best practices which make code maintainable, beyond “comments and meaningful variable names”. In fact comments are bad …sometimes. I think keeping up with recommendations and knowing which wisdom to follow and which to discard may be the real skill. Being an “opinionated coder”, and taking pride in your craft. In any case, I’m sure I was correct in saying that most developers consider their own code to be good code. …and I still didn’t learn to drive!

    I was also ranting about documentation. Again I think this was a bee I had my bonnet related a particular request to document a particular project at the time. But it remains a reasonable point, that documentation can be seen as an afterthought; a project delivery box to tick. I quite like documentation. The interesting challenge of trying to distill the most important hand-over information for a project, without making something which is just too long for anyone to bother reading. Also mechanisms to help ensure docs are kept up-to-date e.g. keeping docs close to the code or part of the code. I like that github have established (or helped reinforce) a convention of supplying a README file for each repo.

On the whole my “tech manifesto” of a decade ago wasn’t too bad, but a lot’s happened in 10 years, and some of these thoughts were starting to feel quite old. So I decided today that they belong in the blog archive.

But what would I write about if I were to pick some points to make about tech and the IT industry in 2017? (Not sure we even call it the “IT industry” so much these days). I don’t think I would try. Clearly such things are destined to go out of date. A single page of thoughts also feels incomplete, but maybe I should add some more deep thinking to the blog category ‘technology’

My SOTM 2017 talk: Diagrams of OpenStreetMap


I gave a talk at State Of the Map 2017, the official OpenStreetMap annual conference, which took place in Japan last month.

“The Diagrams of OpenStreetMap” is a tour of a various diagrams illustrating the project, but focussing more on my own diagramming efforts, including quite a few new diagrams I produced especially for this talk.

I would have mentioned in my talk, but I wasn’t going public with it at the time: I’m going to be available for new work opportunities soon. Maybe I can find something map related? We shall see (suggestions welcome). In any case I’ve been involved in OpenStreetMap, mostly as a volunteer, for over a decade now, and so I’m well placed to try to capture some “big picture” overviews of the project… in diagram form! That’s what I’ve tried to do here.

Watch the video on the conference site

Slides as a LibreOffice .odp file (7.3Mb)

But the following are slides and transcript as one big long blog post. In total there’s a whopping 61 slides (I was talking through them quite quickly). I’ve skipped over some animation step slides to reduce this down a bit, but still a lots of slides:

 

Jump to slide:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61


Slide 1

 

 

This talk is called “Diagrams of OpenStreetMap”. I started thinking about this after my previous 2014 State Of The Map talk.

See the rest of the slides

OpenStreetMap Notes in Organic Maps

I made a tool to scratch an itch I’ve been having recently. I wanted to see OpenStreetMap notes within the Organic Maps app, so I made a thing to fetch and generate…

OpenStreetMap notes as KML

This let’s you grab all the notes in an area and download them as KML. Such a download might be useful for various things, but mostly the idea is to easily load notes into Organic Maps

It works nicely if…
(A) you have the Organic Maps app on your phone, and…
(B) you open the above link on your phone  (Or this QRcode)

The KML download will (after some confirmation prompts) be handled by the Organic Maps app, and the result is…   a shed-load of red bookmark icons:

Organic Maps notes map

It will fetch up to a limit of 1000 notes for an area of your choosing, appearing as a layer of “bookmarks” which you can show and hide (third icon on the bottom there).

On a bookmark, swipe up to see the detailed description and you get the full note description plus any other comments people have added:

Organic Maps Note comments

I have control over how that gets formatted, so it’s something I could tweak/improve

What is Organic Maps?

It’s an app which you can install for iPhone or for Android for viewing maps …specifically OpenStreetMap, but it’s worth having even if you’re not a big OpenStreetMap enthusiast, as an alternative to the “maps” app on your phone. You might use it occasionally e.g. when the maps app doesn’t have the detail you want. Organic Maps requires an initial (quite big) download of an area, and then it all works offline. Great for going on holiday somewhere.
(You may have noticed I previously talked about MAPS.ME on here. Organic Maps is basically the same app, but it was newly created by the original developers of MAPS.ME after that app got sold off and plastered with ads)

What are “notes”?

OpenStreetMap Notes allow people (anyone. without even logging in) to report a map problem or an omission on OpenStreetMap. So if you don’t fancy learning how to edit OpenStreetMap (we try to make it very easy, but …takes a bit of learning) then instead you can effectively ask somebody else to make the edit, by simply leaving a note. However it’s important to position the note precisely, and add as much description as you can. Preferably even link a photo of the thing which is missing/wrong.

As a mapper you will understand the importance of this when you come to try to clear out the open notes in an area. In London it feels like most notes are pretty unclear and poorly described, but of course we’re generally only looking at the notes which are left open. It’s quite common for somebody to write just a few words, seemingly reporting the name of a missing thing, but if that’s the level of effort somebody is willing to go to (not even constructing a sentence) then the data doesn’t feel reliable enough to do anything with, other than to leave open and go survey. And there’s various other types of notes which lead to this same conclusion. Going to take a look on-the-ground is going to be the way to solve the note.

And going to look on-the-ground is really the idea behind this. Because I use Organic Maps a lot while out and about, for everyday navigation, I’m hoping having the notes in there will help remind me to go take a look

Open House rails developer

[Update: They’ve now hired a rails developer. Thanks for the responses! Still opportunities to help I’m sure. I will to try post an update on how to get involved]

I wanted to help promote this Ruby on Rails developer job at OpenCity which I think will be a really interesting thing for somebody, maybe a junior rails developer. Know anyone?

If you’re a londoner you’ve probably heard of “Open House” an event taking place each year where, for one day, you can take a look around lots of interesting buildings, for free, which are often closed to the public the rest of the time.

The Open House event is put together by a not-for-profit organisation called OpenCity, in their office near Aldgate. From there they organise this annual architectural bonanaza, coordinating hundreds of people (volunteers and building owners) with lots of careful planning …and some IT challenges.

opencity-diagram-currentopencity-diagram-1st-stage-rails-app opencity-diagram-big-rails-app

Being an interesting and fun not-for-profit organisation, I have ended up volunteering a bit of my time with them. Me and another volunteer have been planning a rebuild of a database system they’re using internally, which will later ripple up to some cool improvements on their public facing website.

To me this looks like a juicy challenge and a fun organisation to be helping out, so I’m hoping we can find a suitable available developer who feels the same! I’m not available myself, except in my spare time. I plan to be dipping in on this project from time-to-time, so I would be working with this person a little bit.

The project planning is at a very early stage, but I’ve been drawing the above diagrams which show… (left-to-right)

  • Their current set-up
  • An initial milestone introducing a rails app for their buildings database
  • And a final situation with websites and database consolidated

Hopefully in the end we will have managed to consolidate things and four different websites (or four different user journeys/permission levels within the same website) will be served by a lovely new rails app!

OpenCity are very open to our ideas, and one thing we’ve suggested, is to do this whole thing open source on github, as a way of being open to contribution from any other developers who fancy helping this organisation, but this is a pretty big job which will need somebody on it full time. Hopefully quite an interesting challenge for somebody! Please pass this on to anyone who might be interested.

Thanks Chain Reaction Cycles!

Look how happy our baby is with his new little bike:

baby-bike-crc

It arrived in a box yesterday along with a letter from Chain Reaction Cycles:

chain-reaction-baby-letter

“Dear Baby Wood,

We noticed that your daddy got a nice new bike delivered from our website and you were excited by this, so we wanted to send you a bike of your own. Hopefully you will be big enough to start riding it soon.

Chris R – Tech Team Leader”

 

How lovely!

They did all of this in response to a tweet I had sent a few days earlier, which was proving rather popular:

chain-reaction-box-baby-tweet

So that was some good tweeting!

31 “likes” on twitter (so far). That’s after Chain Reaction retweeted it. It seems like the likes came from random customers and cycle fans up and down the country. 31 likes seems to be well above average for @Chain__Reaction‘s twitter stream, and 31 likes is very good going by my standards. My best performing tweet ever in fact! So much for my years of tweeting witty and insightful commentary on tech and maps and things. They all just got beaten by a picture of my baby!

Anyway, what a great surprise to receive! It was addressed to “Baby Wood” which caused some amusement when the courier showed up. I didn’t open it at first in case my wife was planning some sort of surprise I wasn’t supposed to know about. But no! It was a surprise from Chain Reaction Cycles.

As you can see, the baby can be balanced upon the saddle just about, but he’ll need to grow a bit before riding it properly. …Or maybe I should get him training early. See if he can learn to cycle before he learns to walk!

Keyboard Letters Game

I made an online game for my 8 month old baby to play. If you are a baby you might like it too! :

>>> Keyboard Letters Game <<<

Maybe I should come up with a more imaginative name for it, but the idea is, he can bash the keyboard and big colourful letters pop up on screen. It also reads out the letters (put your sound on). Here is a video of him in action:

keyboard letters video

[Update: source code is on github]

Our little boy is 8 months old now. For quite a while he’s been interested in my laptop. He’s interested in any new objects, but I think he’s also seen me typing away on it, and wants a bit of that action. He impatiently tries to reach for my laptop whenever he’s nearby.

I realised he’s a bit young for this still. As you can see in the video, he’s whacking the keyboard quite violently with his whole hand. In fact we have a toy with a just one big button, and he’s still developing the coordination needed to press that, so he definitely doesn’t get the idea of pressing individual keys on the keyboard, but he does seem to be making the link that keyboard whacking makes sound and colourful things happen. Unfortunately he’s also very good at somehow finding weird and wonderful keyboard shortcuts I never knew existed, for dropping out of fullscreen mode, and opening settings etc.

For his age, I’m getting some ideas for simpler games I could develop which just bring light colour and sound based on approximate location on the keyboard he’s whacked.

When he’s a bit older I imagine this might help him understand typing and also letters of the alphabet, particularly as it reads out the letters. Mind you, I’m told they don’t teach the reading of the alphabet in the same way any more in UK schools :-O   Do I need to make a “synthetic phonics” version of this?

I actually made this game for the first time when I was helping out with IT in a junior school. I must have been about 15/16 years old at the time, and I made all kinds of experimental things in BASIC running on Acorn computers (odd things we had in schools at the time). So this is a reimplementation of 20 year old idea!

It’s nice that we can do this kind of thing on the web these days, although doing this with javascript is probably quite a messy bodge-job with nasty browser compatibility considerations compared to doing it in Acorn BASIC. It makes me wonder how I would get on as a 15 year old learning to program in modern times. I don’t remember exactly how I did speech synthesis 20 years ago (system call-out to do a *SAY "HELLO" command perhaps). On the web I didn’t actually know this was possible. On a whim I googled “speech synthesis HTML5” thinking “Nah surely it’s not…” but it is! Speech synthesis is available for calling in javascript on modern browsers! Might have to have some more fun with this!

…if my baby doesn’t destroy my laptop first