To me, and apparently quite a few others, Twitter seems very broken lately. As reported in a NYT article, “In February alone, Twitter experienced at least four widespread outages, compared with nine in all of 2022, according to NetBlocks, an organization that tracks internet outages.” This is not schadenfreude, despite not agreeing with most of the management decisions Elon Musk has made since taking over Twitter in late October of last year, I use the platform quite a bit and generally enjoy the wide array of opinions I can find on there. What I think of as “broken” are things like the feeds taking full seconds to load, posts of people I follow not showing up in my feed, my own posts not showing up in my history, the list goes on and on.
Things could have been this way before and I just didn’t notice, or it could all be related to actions like Musk gleefully unplugging racks of servers. It might be related to Twitter engineers no longer having access to Slack or most of the institutional knowledge walking out the door. Really the possibilities of Musk’s actions being the source of these issues are almost endless and very likely to be highly contributory. But, another rant about Musk’s management style/practices is not the topic of this article. Rather what I wanted to talk about is all of the engineering that happens, often behind the scenes, and isn’t noticed by most business folks. And, arguably Musk is a business person and not a software engineer or engineering manager despite playing one for the press. Why would I say this? Because, his actions don’t demonstrate an understanding of how modern software development takes place in large organizations. If you are about to race to the comments to defend Musk, hold up. This is not uncommon for business folks in tech companies, I see it all the time. Let me explain.
Lots of business folks from CEOs to marketers to member services interact with product engineering. They hear about the product roadmap. They provide input on upcoming features. They get a heads up when new features are released. Product engineering is the lens by which most business folks view engineering. Even in operational meetings like quarterly or monthly metrics meetings, product is usually the one presenting on behalf of the product engineering teams that are comprised of product, design, research, analytics, and engineering. Normally this is all fine. Much of the work engineering does, greater than 50% in most organizations, is product related. However, that isn’t all that engineering does. These other parts are referred to as infrastructure, devops, platform engineering, data engineering, site reliability engineering, etc. and much, much of this work goes unnoticed by the business.
At a previous organization, engineers received, on average, over 400 pages per month. These ranged from jobs that needed to be restarted to third parties whose services we used having problems. I don’t think this is excessive but we did track off hours pages trying to minimize disturbing engineers outside of business hours. Month in and month out, this work went completely unnoticed by our business partners. This is the way we wanted it to be. We built, integrated, spun up, triaged, and generally managed dozens and dozens of systems all without our business partners needing to know anything about it. These quiet, invisible engineering efforts are what kept the business running, whether it was the IT teams working on the office network or an SRE team relaunching a service. As I’ve stated before, availability is the most important feature. Too often we as engineering leaders don’t do a good enough job highlighting the effort that it takes to just keep the systems running. We don’t do this intentionally, we do it because this is part of our jobs. We take pride in doing this very important work and doing it quietly in the background. This is why most of our business partners don’t know that this is even happening. I don’t think Musk understands how much of this work actually goes on across large engineering organizations that are managing very complex systems. But he’s not alone, most leaders who have not run engineering organizations, maybe ever but certainly not in the past few years, don’t have a good understanding about the importance or magnitude of this work.
While I’m highlighting the invisible engineering work that takes place I’m sure we’re not alone. I imagine there are all types of invisible work that is done by marketing, human resources, and especially executive admins. Not to get too recursive but even within the hidden engineering work there is other hidden work that we sometimes refer to as glue-work. Households even have this work. One person might do all the grocery shopping and it generally goes unnoticed or unappreciated by the other members of the household.
We as leaders need to get better at highlighting this important work. Our lives and systems couldn’t really run without it. I think Musk is learning about some of this now but leaders around him in the past should have made sure he knew the importance of this hidden work. I don’t doubt that there are inefficiencies in all of our invisible work but throwing it all away as a means of figuring out what is most important seems reckless to me and not the way an engineer or engineering manager might approach the task.
Good post. I deal with this a lot in small startups. The hidden work is always lurking behind the product/functional work, and the issue is that the business has no appreciation of the cost of the work until it becomes an obstacle. I try to make that work transparent by expressing it as part of estimations (and illustrating the work in Jira etc etc) but it is still a challenge to express that "necessary evil" as a cost attached to decision making.
A tangible example is security and compliance. It is commonly regarded as nearly-free or a "process thing" when the reality is that there are both and explicit (e.g. tooling) costs and implicit (people/process) costs. Yet by the time those costs are factored into planning, strategic decisions are already made.
Have you observed this, and how have you addressed it in your experience?
I love the pivot from looking at engineering teams to recognising the invisible efforts going on in other functions and even our personal lives. Notice others, really notice others, be grateful and give them praise and recognition for all the essential invisible work they do.