Robert McNamara, former US Secretary of Defence, talking about early US bombing missions in WWII:
The U.S. was just beginning to bomb. We were bombing by daylight. The loss rate was very, very high… The loss rate was 4% per sortie, the combat tour was 25 sorties — it didn’t mean that 100% of them were going to be killed but a hell of a lot of them were going to be killed.
4% loss rate per sortie, 25 sorties per tour… so the theoretical chance of dying for those guys was actually 100%.
Not good odds.
What were the actual odds of a given serviceman dying on over the course of 25 such sorties?
Thankfully my smart friend Nick Fyson knows the answer: 64%.
Richard Condon wrote the novel The Manchurian Candidate in 1959. It was his second book, and it’s still famous today because it was turned into a couple of big movies.
Condon wasn’t always a novelist.
He served in the US Merchant Marines during the war, then went to Hollywood as a publicist, copywriter and agent. He started writing books in 1957 while working at United Artists. He complained to his boss, Max E. Youngstein, that he would much rather do that keep working in Hollywood.
Youngstein was a mentor. He took it upon himself to help. Without Condon’s knowledge, Youngstein deducted money from his salary and then fired him after a year – giving him the amount of money he had deducted in a Mexican bank account and the key to a house overlooking the Mexican ocean.
Youngstein told him to take the money, take the keys, and go write his book.
The Manchurian Candidate featured a dedication to Youngstein. A truly mad mentor.
I’m meeting one of my mentors tonight. Mexican Ocean sounds good, so long suckers!
This post originally appeared in a slightly different form on The Media Student Handbook. If you’re a student, and if you’re a Media student in particular, you should check out that site.
Cigarettes & Alcohol was released on 10 October 1994 – so the best song ever turns 20 years old today.
I’ve tried a few approaches in drafting this post but I can’t take myself seriously as a music critic, so here goes:
THIS SONG FUCKING ROCKS.
I used to actually stop myself from listening to Oasis on the morning of exams when I was at school, because I lost motivation if I had lines like: “Is it worth the aggravation / To find yourself a job when there’s nothing worth working for” going round my head.
Twenty years later it has a different meaning for me. Now I hear “You gotta make it happen” on repeat after listening to it, and this weekend when it came on my headphones as I got close to a half marathon finish, I got a proper rush and burned up the hill to the line.
To celebrate the song’s birthday in my own little way, I wanted to share a side project with you.
I’ve been working on a directory site for all the Oasis B-sides, because I really believe that some of the stuff that didn’t make the early albums is 10x better than most of the top-charting singles from the same era.
I’ve only added the Definitely Maybe B-sides so far, but here are three absolute tunes to start off with:
Cloudburst – moody rocker, builds like a thunderstorm
I’m sketching out a new project that helps History students.
I’m curious about how History departments decide whom to admit, how History students can study most effectively, and where History students go once they’ve graduated. I would love to have had that information 15 years ago when I was studying.
I’ll post more here if the project starts to develop. But if you know something about such things, please do drop me a line.
The 2010 Flash Crash was a United States stock market crash on Thursday May 6, 2010 in which the Dow Jones Industrial Average plunged about 1000 points (about 9%) only to recover those losses within minutes. It was the second largest point swing, 1,010.14 points, and the biggest one-day point decline, 998.5 points, on an intraday basis in Dow Jones Industrial Average history.
The Flash Crash was caused by human error.
The [official] report said that this was an unusually large position and that the computer algorithm the trader used to trade the position was set to “target an execution rate set to 9% of the trading volume calculated over the previous minute, but without regard to price or time”.
And that original error was magnified by a sequence of automated knock-on effects:
The New York Times [wrote]: “Automatic computerized traders on the stock market shut down as they detected the sharp rise in buying and selling.” As computerized high-frequency traders exited the stock market, the resulting lack of liquidity “…caused shares of some prominent companies like Procter & Gamble and Accenture to trade down as low as a penny or as high as $100,000.”
Remarkably, the problem self-corrected after a few minutes. But it was not an isolated incident:
The growth of computerized and high-frequency trading in commodities and currencies has coincided with a series of ‘flash crashes’ in those markets. The role of human market makers, who can match buyers and sellers and provide liquidity to the market, is now more and more played by computer programs. If those program traders pull back from the market, then big “buy” or “sell” orders can lead to sudden, big swings. It increases the probability of surprise distortions… In February 2011, the sugar market took a dive of 6% in just one second. On March 1, Cocoa-futures prices dropped 13% in less than a minute on the IntercontinentalExchange. Cocoa plunged $450 to a low of $3,217 a metric ton before rebounding quickly. The U.S. dollar tumbled against the yen on March 16, falling 5% in minutes, one of its biggest moves ever. According to a former cocoa trader: “The electronic platform is too fast; it doesn’t slow things down” like humans would.
We have so much data, and so many smart tools for managing and manipulating it. These are tools so smart that they work automatically, without human direction or intervention. It’s like when your hand touches the cooker: you pull it away before the pain message even reaches your brain, because your nerves respond automatically, much faster than your thoughts.
But with all this data, and all these smart tools – are we cutting ourselves out of the loop too fast?
An analogy from games: dozens of companies are running thousands of A/B tests on millions of data points to try to figure out how to optimise their products.
But even though examining the data might tell you what you’re doing wrong, it cannot tell you how to put it right.
An A/B test divides players into two groups: A is the control, the normal version. B is the test, the new version. For example – you could run an A/B test which changes the way a new type of archer in Age Of Empires is introduced to the player in a tutorial (is that game still going? classic!). The games guys run versions A and B alongside each other and compare the results – checking which group used the new archer type more, were more likely to return to the game the following day, or were more likely to do more of whatever else they were looking to improve.
But if A is what you have now – the current version – then what is B?
B must be defined, built, designed by humans.
It can’t be automated. So you have to invent it yourself.
Using big data and smart tools is an art as well as a science.
Age of Empires is still going! Info on the series can be found here.
All the quotes above are from the Wikipedia article on the 2010 Flash Crash. The best stories are the true ones.
For more posts like this, follow me on Twitter – I’m @toddmgreen – or sign up for posts by email:
I was on German TV in a police comedy sketch show, and the clip now has over 3 million views on YouTube!
The show was called Alles In Ordnung? (Is Everything In Order?), and it ran on ProSieben from 2005-06. It was pretty cool, a very dry comedy in which useless police officers would bumble through serious and not so serious incidents, from shoot-outs to health & safety infringements.
The clip went viral because it featured the game Counterstrike. Two police officers had been called to a block of flats where residents had heard shots being fired. The officers busted into a flat, only to find three students playing Counterstrike, a multiplayer shoot-em-up game, at full volume.
I was one of the students. I’d only just arrived in Germany, so my language skills were, well… still developing. It’s pretty clear at times in the clip that I don’t understand what the thickly-accented officers are saying to me – e.g. at 0:40 in the video when I look extremely confused by what people are saying to me.
The students’ flat was a mess. Curtains drawn, covered in empty beer cans and pizza boxes, and stinking because we were meant to look like we hadn’t had a shower for days. One of the original guys didn’t turn up, so I was asked to stand in. I went to the costume/make-up lady and asked: “What do we need to do to make me look right for the part?“. She looked me up and down: “No need to do anything, you’re good to go on.”
That was during my first attempt to grow a beard. There’s still some work to do there, but back in 2006 I had no idea that beard trimmers even existed – I had clumps missing where I’d been over-zealous with the nail scissors.
I’ve only just passed 20k views on this blog, so it may be some time before I match the 3.1m views on that YouTube video. But I wanted to write this post to say thanks for reading. I really appreciate it.
Wayne Rooney’s movement – England vs. Montenegro, second half (Wired)
It’s the day of the World Cup final, huzzah!
The tournament has produced a blizzard of social media-friendly infographics. Even the good old Beeb has got in on the act, and the Press Association has hired two talented footy data/visualisation chaps (cf Matchstory) to produce things like this:
Data ist sexy, ja?
I work with data every day (analysing games), and I’m season ticket holder at Luton Town – winners of last year’s (Blue Square Bet Conference) Premier League – so I was curious about how games and football data analysis compare.
In games there is a ridiculous amount of data, millions and billions of data points – level success rates, clicks on a given button, use of boosters, and much much much more. Perhaps surprisingly, it’s the same in football. At German nouveau-riche TSG Hoffenheim, sensors are attached to players, cones and goalposts, and inserted into the footballs. With ten players training for just ten minutes with three balls, those sensors will track more than 7m data points.
Reading the Matrix
When you have lots of data, it needs analysis and interpretation. I’m lucky to work with a super-talented team who can identify and understand the patterns in our data. Top football clubs work the same way. Manchester City now have eleven people crunching the numbers, so they could field a team made up entirely of analysts.
The aim of the analysis in both fields is to come up with ideas for action – now that we know x, we should do y. We do this in games the whole time, working with the production team to decide on new AB tests and feature priorities. Same in football: Manchester City hadn’t scored from a corner for 22 games, but by switching from out-swinging to in-swinging corners (as suggested by their data team), they scored nine goals from corners in 12 games, and won the title when Vincent Kompany headed in from a corner against Manchester United.
Art + Science
Data alone will not give you the answers. I wrote about this here: What is B?. When you find a pattern or a problem in the data, and want to make a change, what do you change it to? Should we add a new booster? / Should Liverpool switch to a different shape in central midfield? Data can help you identify a problem, but it doesn’t always provide the solution. So using data is an art as well as a science.
Too much love will kill you
There is an absolute avalanche of data available in games, like in sports. So it’s easy to have too much data. Bolton’s Head of Analytic Development admitted that since their goalkeeper had started studying opposition penalty takers, he had actually saved fewer penalties – not the intended result. Sports players at their best operate in a state of flow, so over-thinking is a real risk to performance.
Ignore the ignorables
That sounds like an Ian Holloway quote, but I just made it up. In The Name Of The Rose, wise old William of Baskerville says that:
Learning does not consist only of knowing what we must or we can do, but also of knowing what we could do and perhaps should not do.
How true. With millions/billions of data points, you can’t look at everything – so you have to use instinct and experience to filter out the stuff that is of less importance or which will have less impact. Prioritisation, a constant battle! Simon Kuper, football data guy and writer, is convinced that football is in the very early stages of understanding how to use data to improve performance, because it’s pretty new and because knowing the right things to look at is not easy in a dynamic, unpredictable environment like a football match.
Measuring the wrong things
You have to look at the right things, because the wrong things can lead you astray. Alex Ferguson, grumpy erstwhile Manchester United manager, sold defender Jaap Stam in 2001 because Stam’s number of tackles was decreasing. Ferguson thought Stam was in decline – but he went on to play successfully at big clubs for several more years. It turns out that tackles are not a good yardstick for the defender’s value. Kuper points out that great defenders like Paolo Maldini actually don’t need to tackle that much, because their positional skills alone reduce opportunities for the opposing side.
The limits of data
Football is not like baseball (Moneyball) – it’s more dynamic, less structured, more anarchic; that makes it harder to apply analysis to improving performance. There’s a lot of stuff that cannot be measured or understood in a quantitative way. What does a player think or feel when they’re playing one a mobile game? That’s tough to see in the data (though there are many qualitative ways to learn the answers). If you can only manage what you measure, then the limitations of your measurements are crucial. The same applies to football: there are many limitations, even when it comes to one of the most structured elements of the game – the penalty kick. Again per Kuper, some players are predictable – e.g. at one stage, Diego Forlan alternated which side he would hit his penalties, left-right-left-right. So tracking his penalty taking would have been helpful. But other players are unpredictable: Franck Ribery mixes up his penalties seemingly at random. Unpredictability is rewarded, because it’s harder to combat – and unpredictability also reduces the usefulness of data analysis. You can’t see a pattern that’s not there.
So there you go. Several similarities – rather more in fact than I thought there would be when I started researching football data yesterday. Worthwhile further reading if you’re interested: New Statesman, The Guardian, BBC, and Wired.