Not good odds

Robert McNamara, former US Secretary of Defence, talking about early US bombing missions in WWII:

The U.S. was just beginning to bomb. We were bombing by daylight. The loss rate was very, very high… The loss rate was 4% per sortie, the combat tour was 25 sorties — it didn’t mean that 100% of them were going to be killed but a hell of a lot of them were going to be killed.

4% loss rate per sortie, 25 sorties per tour… so the theoretical chance of dying for those guys was actually 100%.

Not good odds.

EDIT

What were the actual odds of a given serviceman dying on over the course of 25 such sorties?

Thankfully my smart friend Nick Fyson knows the answer: 64%.

1 – chance_of_surviving_every_sortie   =   1  –  ( 1 – 0.04 )^25   =   0.639603283

Still not good odds. Thanks Nick!

+++

From the Errol Morris documentary The Fog Of War: 11 Lessons From The Life Of Robert McNamara. You can find the full transcript of the film on Errol Morris’s website.

Mad Mentors

Richard Condon wrote the novel The Manchurian Candidate in 1959. It was his second book, and it’s still famous today because it was turned into a couple of big movies.

Condon wasn’t always a novelist.

He served in the US Merchant Marines during the war, then went to Hollywood as a publicist, copywriter and agent. He started writing books in 1957 while working at United Artists. He complained to his boss, Max E. Youngstein, that he would much rather do that keep working in Hollywood.

Youngstein was a mentor. He took it upon himself to help. Without Condon’s knowledge, Youngstein deducted money from his salary and then fired him after a year – giving him the amount of money he had deducted in a Mexican bank account and the key to a house overlooking the Mexican ocean.

Youngstein told him to take the money, take the keys, and go write his book.

The Manchurian Candidate featured a dedication to Youngstein. A truly mad mentor.

I’m meeting one of my mentors tonight. Mexican Ocean sounds good, so long suckers!

+++

This post originally appeared in a slightly different form on The Media Student Handbook. If you’re a student, and if you’re a Media student in particular, you should check out that site.

Oasis – Cigarettes & Alcohol – 20 years old today

Cigarettes & Alcohol was released on 10 October 1994 – so the best song ever turns 20 years old today.

I’ve tried a few approaches in drafting this post but I can’t take myself seriously as a music critic, so here goes:

THIS SONG FUCKING ROCKS.

I used to actually stop myself from listening to Oasis on the morning of exams when I was at school, because I lost motivation if I had lines like: “Is it worth the aggravation / To find yourself a job when there’s nothing worth working for” going round my head.

Twenty years later it has a different meaning for me. Now I hear “You gotta make it happen” on repeat after listening to it, and this weekend when it came on my headphones as I got close to a half marathon finish, I got a proper rush and burned up the hill to the line.

To celebrate the song’s birthday in my own little way, I wanted to share a side project with you.

I’ve been working on a directory site for all the Oasis B-sides, because I really believe that some of the stuff that didn’t make the early albums is 10x better than most of the top-charting singles from the same era.

I’ve only added the Definitely Maybe B-sides so far, but here are three absolute tunes to start off with:

Cloudburst – moody rocker, builds like a thunderstorm

D’Yer Wanna Be A Spaceman? – acoustic sing-song with a wistful message

I Will Believe (Live) – simple simple simple demo-style tune, great riff, very early recording

And finally – here’s the link to the prototype site:

oasisbsides.com (URL headshot baby!)

It’s still in alpha, so please do send me your thoughts. And more importantly: discover and enjoy!

~ Todd

How to get a bargain on a new iPhone (and a free cost calculator)

Which of these costs more?

  1. iPhone 5C, 8GB
  2. iPhone 6, 16GB

Trick question: they cost the same amount (£1107 over 2 years) if you choose the right purchase option.

That’s one of the surprises I discovered when I compared the costs of all models, storage sizes, and purchase types (direct / 12m contract / 24m contract).

No shady stuff, no gimmicks or trade-ins – below I set out what I found just by building a little tool to compare all the usual options.

My trusty 4S borked last night after 2.5 years’ service, so I’m wondering (1) whether to stick with an iPhone and (2) if so, which to buy.

Assuming that I want to get another iPhone, there are three main variables:

  1. Model (5C, 5S, 6, 6+)
  2. Storage (8GB, 16GB, 32GB, 64GB, 128GB)
  3. Purchase option (direct purchase, 12-month contract, 24-month contract)

I built a basic cost calculator to help me decide which to buy. You can use it too – link below.

Remarkably, there are some relative bargains available.

Example bargains:

  • iPhone 6, 16GB for the same price as 5C, 8GB (the example given at the top)
  • iPhone 5S, 16GB cheaper than 5C, 8GB
  • iPhone 6, 64GB cheaper than iPhone 5S with 16/32GB
  • iPhone 6 Plus, 128GB cheaper than iPhone 5S, 64GB

Also – in all cases, buying direct from Apple is cheaper than spreading the cost across a contract – so if you have the cash, buy direct.

Here’s the key part of the spreadsheet, and here’s the link to download the calculator (.xlsx file): iPhone comparison calculator – 04-10-14

I haven’t made my mind up yet – but this will stop me overpaying if I do decide on an iPhone.

Enjoy!

~ Todd

History students, unite!

I’m sketching out a new project that helps History students.

I’m curious about how History departments decide whom to admit, how History students can study most effectively, and where History students go once they’ve graduated. I would love to have had that information 15 years ago when I was studying.

I’ll post more here if the project starts to develop. But if you know something about such things, please do drop me a line.

Monster Month

Starts on Sunday. Not quite perfect preparation: been off work sick the last two days…

Please donate to Mind, the mental health charity, if you can: https://www.justgiving.com/Todds-Monster-Month

Thanks! ~ Todd

Going up?

Short video in a lift in Madrid, 2012. Recently found it and tidied it up. I loved the change that hit you halfway up to the top floor and wanted to capture it.

What is B? What The Data Can Never Tell You

The 2010 Flash Crash was a United States stock market crash on Thursday May 6, 2010 in which the Dow Jones Industrial Average plunged about 1000 points (about 9%) only to recover those losses within minutes. It was the second largest point swing, 1,010.14 points, and the biggest one-day point decline, 998.5 points, on an intraday basis in Dow Jones Industrial Average history.

The Flash Crash was caused by human error.

The [official] report said that this was an unusually large position and that the computer algorithm the trader used to trade the position was set to “target an execution rate set to 9% of the trading volume calculated over the previous minute, but without regard to price or time”.

And that original error was magnified by a sequence of automated knock-on effects:

The New York Times [wrote]: “Automatic computerized traders on the stock market shut down as they detected the sharp rise in buying and selling.” As computerized high-frequency traders exited the stock market, the resulting lack of liquidity “…caused shares of some prominent companies like Procter & Gamble and Accenture to trade down as low as a penny or as high as $100,000.”

Remarkably, the problem self-corrected after a few minutes. But it was not an isolated incident:

The growth of computerized and high-frequency trading in commodities and currencies has coincided with a series of ‘flash crashes’ in those markets. The role of human market makers, who can match buyers and sellers and provide liquidity to the market, is now more and more played by computer programs. If those program traders pull back from the market, then big “buy” or “sell” orders can lead to sudden, big swings. It increases the probability of surprise distortions… In February 2011, the sugar market took a dive of 6% in just one second. On March 1, Cocoa-futures prices dropped 13% in less than a minute on the IntercontinentalExchange. Cocoa plunged $450 to a low of $3,217 a metric ton before rebounding quickly. The U.S. dollar tumbled against the yen on March 16, falling 5% in minutes, one of its biggest moves ever. According to a former cocoa trader: “The electronic platform is too fast; it doesn’t slow things down” like humans would.

We have so much data, and so many smart tools for managing and manipulating it. These are tools so smart that they work automatically, without human direction or intervention. It’s like when your hand touches the cooker: you pull it away before the pain message even reaches your brain, because your nerves respond automatically, much faster than your thoughts.

But with all this data, and all these smart tools – are we cutting ourselves out of the loop too fast?

An analogy from games: dozens of companies are running thousands of A/B tests on millions of data points to try to figure out how to optimise their products.

But even though examining the data might tell you what you’re doing wrong, it cannot tell you how to put it right.

An A/B test divides players into two groups: A is the control, the normal version. B is the test, the new version. For example – you could run an A/B test which changes the way a new type of archer in Age Of Empires is introduced to the player in a tutorial (is that game still going? classic!). The games guys run versions A and B alongside each other and compare the results – checking which group used the new archer type more, were more likely to return to the game the following day, or were more likely to do more of whatever else they were looking to improve.

But if A is what you have now – the current version – then what is B?

B must be defined, built, designed by humans.

It can’t be automated. So you have to invent it yourself.

Using big data and smart tools is an art as well as a science.

+++

P.S.s…

Age of Empires is still going! Info on the series can be found here.

All the quotes above are from the Wikipedia article on the 2010 Flash Crash. The best stories are the true ones.

For more posts like this, follow me on Twitter – I’m @toddmgreen – or sign up for posts by email:

My German TV debut (with, bizarrely, 3.1m views and counting)

I was on German TV in a police comedy sketch show, and the clip now has over 3 million views on YouTube!

The show was called Alles In Ordnung? (Is Everything In Order?), and it ran on ProSieben from 2005-06. It was pretty cool, a very dry comedy in which useless police officers would bumble through serious and not so serious incidents, from shoot-outs to health & safety infringements.

The clip went viral because it featured the game Counterstrike. Two police officers had been called to a block of flats where residents had heard shots being fired. The officers busted into a flat, only to find three students playing Counterstrike, a multiplayer shoot-em-up game, at full volume.

I was one of the students. I’d only just arrived in Germany, so my language skills were, well… still developing. It’s pretty clear at times in the clip that I don’t understand what the thickly-accented officers are saying to me – e.g. at 0:40 in the video when I look extremely confused by what people are saying to me.

The students’ flat was a mess. Curtains drawn, covered in empty beer cans and pizza boxes, and stinking because we were meant to look like we hadn’t had a shower for days. One of the original guys didn’t turn up, so I was asked to stand in. I went to the costume/make-up lady and asked: “What do we need to do to make me look right for the part?“. She looked me up and down: “No need to do anything, you’re good to go on.”

That was during my first attempt to grow a beard. There’s still some work to do there, but back in 2006 I had no idea that beard trimmers even existed – I had clumps missing where I’d been over-zealous with the nail scissors.

I’ve only just passed 20k views on this blog, so it may be some time before I match the 3.1m views on that YouTube video. But I wanted to write this post to say thanks for reading. I really appreciate it.

~ Todd

Football vs. Data Analysis

Rooney vs Montenegro 2nd half (Wired)

Wayne Rooney’s movement – England vs. Montenegro, second half (Wired)

It’s the day of the World Cup final, huzzah!

The tournament has produced a blizzard of social media-friendly infographics. Even the good old Beeb has got in on the act, and the Press Association has hired two talented footy data/visualisation chaps (cf Matchstory) to produce things like this:

Data ist sexy, ja?

I work with data every day (analysing games), and I’m season ticket holder at Luton Town – winners of last year’s (Blue Square Bet Conference) Premier League – so I was curious about how games and football data analysis compare.

0101010101010101010101010101010101

In games there is a ridiculous amount of data, millions and billions of data points – level success rates, clicks on a given button, use of boosters, and much much much more. Perhaps surprisingly, it’s the same in football. At German nouveau-riche TSG Hoffenheim, sensors are attached to players, cones and goalposts, and inserted into the footballs. With ten players training for just ten minutes with three balls, those sensors will track more than 7m data points.

Reading the Matrix

When you have lots of data, it needs analysis and interpretation. I’m lucky to work with a super-talented team who can identify and understand the patterns in our data. Top football clubs work the same way. Manchester City now have eleven people crunching the numbers, so they could field a team made up entirely of analysts.

Action Men

The aim of the analysis in both fields is to come up with ideas for action – now that we know x, we should do y. We do this in games the whole time, working with the production team to decide on new AB tests and feature priorities. Same in football: Manchester City hadn’t scored from a corner for 22 games, but by switching from out-swinging to in-swinging corners (as suggested by their data team), they scored nine goals from corners in 12 games, and won the title when Vincent Kompany headed in from a corner against Manchester United.

Art + Science

Data alone will not give you the answers. I wrote about this here: What is B?. When you find a pattern or a problem in the data, and want to make a change, what do you change it to? Should we add a new booster? / Should Liverpool switch to a different shape in central midfield? Data can help you identify a problem, but it doesn’t always provide the solution. So using data is an art as well as a science.

Too much love will kill you

There is an absolute avalanche of data available in games, like in sports. So it’s easy to have too much data. Bolton’s Head of Analytic Development admitted that since their goalkeeper had started studying opposition penalty takers, he had actually saved fewer penalties – not the intended result. Sports players at their best operate in a state of flow, so over-thinking is a real risk to performance.

Ignore the ignorables

That sounds like an Ian Holloway quote, but I just made it up. In The Name Of The Rose, wise old William of Baskerville says that:

Learning does not consist only of knowing what we must or we can do, but also of knowing what we could do and perhaps should not do.

How true. With millions/billions of data points, you can’t look at everything – so you have to use instinct and experience to filter out the stuff that is of less importance or which will have less impact. Prioritisation, a constant battle! Simon Kuper, football data guy and writer, is convinced that football is in the very early stages of understanding how to use data to improve performance, because it’s pretty new and because knowing the right things to look at is not easy in a dynamic, unpredictable environment like a football match.

Measuring the wrong things

You have to look at the right things, because the wrong things can lead you astray. Alex Ferguson, grumpy erstwhile Manchester United manager, sold defender Jaap Stam in 2001 because Stam’s number of tackles was decreasing. Ferguson thought Stam was in decline – but he went on to play successfully at big clubs for several more years. It turns out that tackles are not a good yardstick for the defender’s value. Kuper points out that great defenders like Paolo Maldini actually don’t need to tackle that much, because their positional skills alone reduce opportunities for the opposing side.

The limits of data

Football is not like baseball (Moneyball) – it’s more dynamic, less structured, more anarchic; that makes it harder to apply analysis to improving performance. There’s a lot of stuff that cannot be measured or understood in a quantitative way. What does a player think or feel when they’re playing one a mobile game? That’s tough to see in the data (though there are many qualitative ways to learn the answers). If you can only manage what you measure, then the limitations of your measurements are crucial. The same applies to football: there are many limitations, even when it comes to one of the most structured elements of the game – the penalty kick. Again per Kuper, some players are predictable – e.g. at one stage, Diego Forlan alternated which side he would hit his penalties, left-right-left-right. So tracking his penalty taking would have been helpful. But other players are unpredictable: Franck Ribery mixes up his penalties seemingly at random. Unpredictability is rewarded, because it’s harder to combat – and unpredictability also reduces the usefulness of data analysis. You can’t see a pattern that’s not there.

So there you go. Several similarities – rather more in fact than I thought there would be when I started researching football data yesterday. Worthwhile further reading if you’re interested: New StatesmanThe GuardianBBC, and Wired.