Expected Goals

Why the scoreboard is a terrible teacher

On the afternoon of March 18, 1950, a middle-aged RAF officer sat in the stands at the County Ground in Swindon, watching the home side struggle through a first half against Bristol Rovers. He was irritated. Swindon were slow, their attacks kept dying in traffic, and he had opinions about why. At halftime he reached into his jacket, took out a pencil and a notebook, and began recording what happened to every possession on the pitch.

His name was Charles Reep. He was an accountant by training and a wing commander by rank, and by the end of Swindon’s 1-0 win he had logged 147 attacking moves. He never stopped. Over the next four decades he notated more than two thousand matches by hand, sometimes wearing a miner’s helmet so the lamp would light his notebook during evenings at Plymouth. His conclusions rested on more than 600,000 recorded passing moves.

If you are a soccer fan, you may know what he concluded. Most goals, he found, came from moves of three passes or fewer. A few English managers built careers on his analysis. Graham Taylor took Watford from the Fourth Division to second in the First, and hired Reep along the way. Dave Bassett’s Wimbledon, working from the same doctrine at second hand, finished sixth. Reep was validated by his tables and vilified by everyone else, because the English game heard his numbers as a defense of hoofing the ball forward and many never forgave him.

In 1968, Reep published a paper with the statistician Bernard Benjamin in the Journal of the Royal Statistical Society. Neil Lanham, the analyst who carried Reep’s methods forward after him, calls it the most important document in football: “It tells you firstly that football is controlled by chance.” Reep’s most durable discovery was that goals arrive at random inside a team’s framework of probability.

The world took the tactic and threw away the chance part. We do the same thing in business.

The Glow

Phil Rosenzweig applied the mechanism to business performance in his 2007 book The Halo Effect. He argues that we reason backward, from results to causes, without noticing we have done it.

His central case is Cisco. In the late 1990s the company could do nothing wrong. Wired described a workforce of shiny, happy people. BusinessWeek credited the run to flawless management and a scorching series of acquisitions. In March 2000, Cisco was briefly the most valuable company on earth. Then the tech market broke. The stock fell from roughly eighty dollars a share to fourteen inside a year, and the same publications that had canonized John Chambers turned on him. Now the strategy was muddled, the acquisitions were clumsy, the culture was out of control.

Rosenzweig asked what changed at Cisco between 2000 and 2001. Demand for the product changed. The strategy, the CEO and the culture stayed where they were. Financial performance is the one variable observers can see, and once they see it, they read strategy, leadership, people and culture through it. That is the halo.

Rosenzweig catalogs nine of these delusions. The most useful one for leaders is what he calls the delusion of absolute performance. Company performance should be measured relative to your competitors. You can do everything better than you did last year and still finish lower, because the other guys got better faster.

Most management books ask what produces high performance. Rosenzweig asks why high performance is so hard to understand at all. His verdict on the genre: for all its claims of rigor, much of it operates at the level of storytelling.

The Invisible Denominator

Reep’s critics found their weapon in the arithmetic. In 2005, Mike Hughes and Ian Franks reanalyzed World Cup data and normalized the results against how frequently each length of passing sequence occurred. Longer sequences generated more shots per possession and, among successful teams in their 1990 World Cup sample, more goals per possession. Direct play retained a better goals-per-shot rate. Reep counted the goals correctly. The raw totals hid how many chances of each kind there had been.

Reep’s defenders have a point worth making. Richard Pollard, a statistician who co-authored with him, argues that Hughes and Franks never set out to discredit Reep and that later writers turned their findings into a weapon. Pollard goes further, passing sequence data cannot settle the direct-versus-possession argument, because a three-pass move might be successful direct play, failed direct play, or a possession move that collapsed early. The data cannot tell them apart. The argument is still alive and well. Study only the successful outcomes and you will find something that feels like a cause and is a base rate.

That is what Rosenzweig accuses the business bestsellers of doing. Gather the winners. Interview them. Notice what they have in common. Publish. Nobody assembles the companies that did all the same things and lost, because those companies are boring and their executives do not return the call. Nobody counts the losers, and the losers are the denominator.

Grading the Shot

The metric that cracked soccer deserves some of our attention. Ryan O’Hanlon’s Net Gains is an excellent account of why the sport resisted analytics for so long. Baseball is a sequence of discrete, countable events. Soccer is twenty-two people moving continuously without restriction, where much of what decides a match happens away from the ball and outside any manager’s control. It is low scoring and fluid, and the final result tells you less about the process than in any other major sport.

The breakthrough was expected goals, or xG. Instead of asking whether the ball went in, xG asks how likely a shot of that type, from that spot, in that situation, was to go in. It grades the chance.

One scene in O’Hanlon’s book captures it. Paul Power once showed Gordon Strachan and two of his assistants a video of Roy Hodgson defending England after a match in which they had recorded only two shots on target. Hodgson pointed to blocked attempts and a Sturridge lob that had gone over. Strachan agreed that the shot count did not reflect the performance. Power let him finish and then said, “That’s expected goals.”

xG resists the halo. The value assigned to a shot does not change because the shot happened to go in. It grades the chance using what was knowable before the outcome. It is the closest thing I have seen in any competitive field to a built-in defense against reading backward.

The Ceiling

Analytics clearly did not solve soccer. Stefan Szymanski’s work is the standing rebuke to anyone who thinks otherwise. Compare Premier League and Championship clubs’ average league position from 2003 through 2012 against their wage spending relative to the rest of the league, and the R squared is just over 90%. Money explains most of it.

Szymanski adds the point that in the short run, luck makes that relationship far less reliable, because players get injured and go through inexplicable runs of form. Over time the luck cancels out and you get what you paid for. The short run is where most of us manage, and the short run is where the noise lives.

Expected goals will not let a poor club buy a title. It will let you tell a good process from a good week. Where wages account for more than 90% of the long-run variation, the remaining variation still matters, especially in the short run where process and luck are hardest to separate. I argued in The Pep Effect that systems beat stars. Measurement beats memory for the same reason.

The Army Got There First

The Army solved this before anyone had data good enough to do it properly. I used to joke that the Army could not march a company across a road without stopping to run an after action review. It was a joke with real affection in it, because the AAR is one of the most valuable things I was taught. Learn from your actions.

The After Action Review grew out of the Army’s reckoning with Vietnam, and the National Training Center at Fort Irwin, activated in 1980 and running rotations by 1981, is where the Army made its doctrine. In the early days the reviews were what you would expect from a hierarchy: a senior observer dissecting what the unit had done wrong and handing over a checklist. It put leaders on the defensive and it did not work. Soldiers learned what to do and never learned why.

The Army changed the design. The observer’s job became facilitating learning instead of supplying criticism. Reviews started at the platoon level and worked their way up the chain, so that leaders evaluated their own performance instead of receiving a verdict on it. And the central comparison became the commander’s intent, stated before the mission, measured against what actually happened afterward. You write down what you expect before you find out, then you compare.

An AAR is all about future improvements, and it is run by the same people who will act on it, in time for it to matter. Every strategic plan is a working hypothesis about what will produce success. A hypothesis you did not write down is difficult to test but it certainly can be rationalized.

Your Organization’s Expected Goals

Write the hypothesis down before the result arrives. Not the goal, the reasoning. What you expect to happen, why you expect it, what would tell you that you were wrong, and what you knew at the time. It is the only way I know to stop your future self from rewriting your past self’s thinking to match whatever happened. A reader made this point back to me in the comments on From Metrics to Meaning: teams reconstruct what they expected after the fact, to fit the story they now want to tell.

Put process measures on the dashboard beside the outcome measures. Outcome numbers lag, and luck contaminates them. Leading indicators of decision quality do neither. Track both, and when they diverge, treat it as information instead of an accusation.

Grade decisions on inputs, not results. Somewhere in your organization there is a leader who made a well-reasoned bet that did not land, and another whose sloppy bet happened to work. If you can only see the scoreboard, you will promote the second one and sideline the first, and everyone watching will learn that the way to succeed here is to get lucky and take credit. Two years later you will wonder why nobody takes intelligent risks anymore.

Interrupt the halo before it forms. Red-Teaming Your Strategy and Challenge Network both come at this. Each forces a competing explanation into the room while the outcome is still unknown, the only window in which the exercise stays honest.

And when results do repeat, stop hunting for someone to blame. That is the argument in Perfectly Designed for These Results. Randomness produces scatter. Systems produce consistency. If the same failure keeps arriving, you are not unlucky. You are correctly designed to experience that failure.

Putting This Into Practice

Pick the last significant decision your team made that worked, one with money or people or reputation attached to it. Write down what you knew at the moment you made it and what you expected to happen. Grade the decision on that alone, as though the result had not yet arrived.

Then do the same thing for a decision that failed.

Compare the two grades against the two outcomes. If they line up, you have a process worth repeating. If they do not, and I suspect for at least one of them they will not, you have found something more useful than a win. You have found the place where your organization has been learning the wrong lesson for years.

The scoreboard will tell you who won. It will not tell you which decisions are worth repeating. That is the question a sustainably winning organization has to answer.