Engineering Maturity Model

Layering

Beau Miles is this eclectic Australian filmmaker, who describes himself as, “award winning filmmaker, poly-jobist, speaker, writer, odd.” I really appreciate him for not only his adventures  such as planting a tree a minute for 24 hours but also for his storytelling skills. My love of stories and raconteurs is something that I’ve talked about a lot as I think stories are a key part of leadership. More about the importance of stories in a future post. Today, I want to focus on something Beau Miles said in one of his films that resonated with me in terms of engineering organizations. He was talking about why he could build a house and why someone else might not be able to do so. He stated, “It’s all about layering. The reason I can build a house is because I know what goes first, second, third, and fourth…so on and so forth.” I think this is the same thing with great engineering organizations, it’s all about layering, knowing what goes first, second, third, and fourth. Kind of a maturity model.

There are of course already a number of maturity models out there, the most widely known being the Capability Maturity Model (CMM) from Carnegie Mellon’s Software Engineering Institute (SEI). These models, and the CMM in particular, are tools that assess the current state of an organization and help figure out what capabilities the team needs to acquire next in order to continue to improve their performance. As Martin Fowler points out, “Most people I know in the software world treat maturity models with an inherent feeling of disdain…The first problem was the CMM was very much associated with a document-heavy, plan-driven culture…But the more serious problem with the CMM was the corruption of its core value by certification.” I’m certainly not advocating heavy documentation nor certification. I think these layers should be used only as guidelines. They also aren’t stages in that you’re never really finished with any of them but you do need to have the ones at the base stronger and more developed than the ones further up or else you are certain to run into problems.

The base of the layers is infrastructure and your deployment pipeline. You need to be able to quickly and securely deploy code into a stable environment. Preferably this is a continuous integration/continuous delivery/deployment (CI/CD) pipeline. This is pretty self-evident but what might not be is that you need to keep investing in this area as you grow and mature. You will eventually limit your ability to improve in areas above this if you don’t keep investing. However, you also shouldn’t over invest in this area too early. Doing so is a waste of energy and money. 

The next layer is observability and telemetry. I call these out separately because to me they are different but equally important. Telemetry is the ability to collect data such as logs and traces. Great telemetry tooling offers robust data collection and standardization. Observability is the ability to understand or to make meaning from this data through analysis and visualization. After you have the ability to quickly deploy code into a secure and stable infrastructure, you need to be able to detect if and what you broke with your latest deployment. It is shocking how many companies can go weeks or months without knowing if they broke something for their customers with the latest changes that they or a third party made. 

The next layer is experimentation. As I wrote a few weeks ago, research shows that it is not uncommon for experiment success rates to be less than 50% at companies like Amazon and Microsoft. If your organization is as good as some of these very sophisticated product development teams, you are making things better for your customers only about half the time. Without a solid experimentation layer, you have no idea which 50% is making things better and which 50% is making things worse.  

The last layer is product development. This is the layer that you need to focus on building empowered, cross-functional teams that have all the skills they need to ideate, design, test, deploy, analyze, and support product features. Stick to principles over process in this layer. The product teams should be assigned customer outcomes that they are responsible for improving and given the boundaries and constraints within which they must operate. These include constraints like headcount and even parts of the site. 

While I do think of this kind of like a maturity model, they are not stages that one achieves and moves on from. These are areas that one must keep returning to and keep investing in, always from the bottom up. Getting over your skis and investing too much in the top, which is very tempting for startups, is fraught. Too many product development teams without continued investment in the infrastructure or deployment pipeline can slow everyone down, proving Brooks’s Law. The important task for Engineering leaders is to determine when and how much investment gets made into each of these layers.