Scaling Organizations III

Part 3 - Managing complexity in organizations

Complexity can neither be created nor destroyed...at least within an application according to Tesler’s law of the conservation of complexity. The idea is that every application has a certain amount of complexity that must be dealt with, either by the software engineer adding complexity to the code or by the user through their interactions with the application. This is interesting from a software development perspective but I’d like to extend this idea to organizations. I posit that complexity in an organization can also neither be created nor destroyed but rather it must be managed by either people or process. 

Before we discuss the management of complexity in an organization, let’s first discuss complexity. While I stated that it can’t be created, of course that’s not completely true. The creation of complexity within an organization happens with more employees and more customers. But, at a certain point in time with X employees and Y customers, I do think the amount of complexity is fixed. But what is complexity? 

First off, complex is different from complicated. Complicated problems are hard to solve, but they typically have rules or algorithms to solve them. Complex problems have too many unknowns and interrelated components to reduce to a set of rules. Fixing a car is complicated, disrupting the automotive industry is complex. 

While there are dictionary definitions of complexity, the scholarly formulation of complexity is not completely agreed upon. Herbert Simon in his 1962 article, The Architecture of Complexity, discusses complexity as having properties that include: distinct components, that in turn have their own components thus a hierarchical and repeating structure (think Mandelbrot), these components interact with each other that results in phenomena that cannot be trivially redacted to the properties of the components (thus the sum is greater than the parts). Using this definition we can see that organizations meet these criteria. Organizations are very much a hierarchical and repeating structure. 

If we can borrow this notion that complexity can neither be created nor destroyed from software, we should also be able to borrow management techniques. In András Tilcsik and Chris Clearfield’s book Meltdown: Why Our Systems Fail and What We Can Do About It, the authors outline a number of strategies to manage complexity including designing loosely coupled systems. Loosely coupled systems help prevent cascading failures. It is not uncommon in system failures for the symptoms to not be proximate to the cause. For example, when databases get busy with long running queries, often the way this manifests itself is with errors on web servers. Tightly coupled systems cause cascading failures and are hard to diagnose. 

Organizations are similar. Teams that are heavily dependent on one another can have cascading failures. For example, if an infrastructure team doesn’t complete one of their projects on time, and a product team is counting on that infrastructure to support their feature development, we see cascading delays. While loosely coupled teams have benefits, just like our systems, there are tradeoffs. Without these dependencies we don’t get economies of scale. If every product team had to build their own infrastructure, they’d likely not get many product features completed. The balance between tightly and loosely coupled is difficult to perfect and changes over time.

There are a number of other strategies outlined in Meltdown and other articles for managing complexity including building diverse teams, learning from past incidents, encouraging dissent, and learning from outsiders. You can easily see how these can be applied to organizations as well as systems. As we grow our organizations, the complexity is going to increase. We need to continue to find solutions to help us manage this complexity and leveraging solutions from other disciplines is one approach.