The Moneyball Approach to Engineering

Does Engineering = Baseball?

You might not recognize the name but you’ll no doubt recognize his work. Billy Beane is a former professional baseball player who is best known for his work as the general manager of the Oakland Athletics. Beane is credited with revolutionizing the way that Major League Baseball teams evaluate and acquire players, through the use of advanced analytics and data-driven decision making. This approach, known as "moneyball," was popularized in the book and film of the same name, and has been widely adopted by other teams in the league. Despite having one of the lowest payrolls in the league, the Athletics under Bean’s management were consistently competitive, thanks in large part to his innovative approach to building a team.

Beane’s approach was rooted in something called Sabermetrics, which is the use of advanced statistical analysis to evaluate and compare the performance of individual players and teams in baseball. The term is derived from the acronym SABR, which stands for the Society for American Baseball Research, a group of baseball enthusiasts who pioneered the use of statistical analysis in the sport in 1971. Sabermetrics involves the use of a wide range of statistics, including traditional measures like batting average and earned run average, as well as more advanced metrics like runs created and win probability added. These statistics are used to evaluate players and teams, identify patterns and trends, and make predictions about future performance. Beane took this to a whole new level as he favored statistics over the traditional instincts of talent scouts and he sought out players who demonstrated consistency rather than flash. 

Stumbling across moneyball again recently got me thinking about how this has been applied to engineering. There is a lot already written about approaching engineering from a moneyball perspective, from semiconductor engineering to software engineering. There are even companies like Code Climate, that are in the business of moneyballing (I think I just made that term up) software engineers. On their blog they suggest that managers focus on metrics such as the number of commits logged in a given period, or the length of time a pull request stays open before they are picked up for review. They state, “The true measure of your team’s success will be their ability to deliver value in a predictable, reliable manner. To track that, you’ll need to look past success metrics and dig into health metrics, measurements that can help you assess your team’s progress earlier in the process.” 

Consistency is definitely important for many reasons. For starters, it’s a principle of the Agile Manifesto, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Secondly, a nice sustainable, consistent pace helps teams from burning out. However, there is nothing mentioned about measuring teams by the outcomes that benefit customers. Marty Cagan, of SVPG, is known for pushing to measure product teams by the value to customers they create. You could argue that the engineers and engineering managers are responsible for delivery while the product manager is responsible for directing the delivery to the correct outcomes that impact customers. This would be the wrong approach. These three (engineers, PMs, and EMs) as well as designers and analyst, if they are on the team, are all responsible for both the delivery velocity and the outcomes. 

Even if we use both customer impacting metrics, such as conversion rate on an ecommerce site, and delivery metrics, commits and time of open PRs, there is still the issue of comparing different teams. Using Kent Beck’s 3X framework, how do you compare a team that is extracting value from existing functionality vs a team that is exploring brand new functionality? Everything about these teams might be different in terms of metrics. In baseball or any other sport for that matter, the game is fairly consistent from week-to-week. The fields, the level of play by the competitors, the referees, and of course the rules are all similar from game to game and team to team. In software engineering, the goal line is different from team to team. The rules are different from stack to stack. 

I’m very data driven and find a lot of value in metrics, even ones that are used to measure productivity for engineering teams, but using them to rate one team as performing better or worse than another is fraught with problems.