Product Debt

Paying down technical debt

I wrote a couple weeks ago about re-platforming or rewriting systems but I intentionally left out a somewhat related “re-” that is refactoring. The reason is that while it’s often very difficult to achieve a positive ROI from a re-platform or rewrite, refactoring is essential and almost always has a great payback even if you can’t measure it. We usually think of refactoring as “cleaning up” code, where we change the code to be more easily understood, perform better (faster or more efficiently), or follow current conventions/standards. The goal of refactoring is to change the code without changing its functionality; it should continue to pass all unit and functional tests. 

Refactoring is very important as we almost always are incurring some as we program. I think of this akin to what it is like building or remodeling a house. After the project is done, we almost always have things we would have done differently if we had either thought of them at the time or had known something more when we started. Most of us can relate to a notion like, “had I known how much laundry I was going to be doing, I would have put the washing machine closer to the bedrooms” or some other such tidbit of wisdom we learned too late. In code, we can and should go back and move that figurative washing machine closer to the bedrooms. When it’s me walking through my house six times per day with laundry, it’s not a huge deal, but when it’s your code executing 42 million times per day, that little inefficiency adds up in a hurry. This build up of inefficiencies is known as technical debt. 

Technical debt is not just limited to our code, it can occur when we don’t properly document the current state of a system or when we haven’t updated or patched some third party software that our code runs on or utilizes. When we take actions to eliminate this we call it paying down technical debt, of which refactoring is one method of doing so specifically for our own code. The challenge isn’t identifying the built up debt but rather getting the buy in from product partners and business partners to spend the time tackling this work. Take the recent Southwest airlines scheduling system issues that caused the cancellation of thousands of flights. In March, the flight attendants union for Southwest posted an open letter asking for the modernization of the antiquated scheduling system even above its demands for increased pay. If everyone was aware of the need to “modernize” or pay down technical debt, why wasn’t it addressed? I would argue in part because not everyone sees this debt. Unlike a credit card statement where you see your debt increasing every month if you don’t pay it down, tech debt, except to people who have to interact with the system directly, goes unseen. In Southwest’s case, just 30 days before the major shutdown, their Chief Operating Officer Andrew Watterson, responding to allegations that Southwest has been slow to adopt new technology, stated, “That’s a leftover image from decades ago—I don’t think it’s true...Our scheduling system is the best in the world.”

The other reason that companies fail to adequately address technical debt until it smacks them in the face is back to the last sentence in my opening paragraph, “The goal of refactoring is to change the code without changing its functionality.” When we don’t change functionality, it’s not improving the customers’ experience, it’s not increasing sales, it’s not cutting costs…it’s not doing anything that we can brag to our shareholders about. Whether you are a startup worried about raising your next round of financing, or whether you are a public company worrying about reporting the next quarter’s results, company executives are motivated to push for more features and functionality. I blame this short termed mindset on the SEC ruling in 1970 that mandated quarterly reporting. Many politicians including Hillary Clinton and Donald Trump have suggested changing this back to semi-annual. Former Pepsi CEO Indra Nooyi, reflected that quarterly reporting regulations made her pay undue attention on producing short-term results. While the SEC has yet to make any changes, a study from Brown University, hosted on the SEC website, suggests a bifurcated reporting system based on the size of the company. The authors suggest that smaller companies (less than $1B in gross revenue) adopt a semi-annual earnings guidance, stating, 

Like reporting, quarterly earnings guidance serves to increase transparency and aid investors in identifying attractive investment opportunities. Yet, there is strong evidence indicating that guidance has perpetuated short-term thinking. An overwhelming majority of empirical evidence finds that the issuance of guidance, and the subsequent pressure to surpass expectations, blinds management from seeing the bigger picture

Whether we are a startup or public company with quarterly reporting requirements and guidance expectations, we have to play the hand we are dealt. So, how do we ensure we are paying down technical debt when there is so much pressure to ignore it until things really break? I think one part of the answer is to use a different term. Instead of tech debt, which implies it is the responsibility of the tech team, let’s call it product debt. This does a couple things. First, it labels it more correctly since we incur most of this debt as we build the product. Secondly, it implies that both engineering and product need to together own this. While everyone on a product team brings different skill sets, they all should have joint ownership of ideas, implementation, and maintenance. 

Labeling this correctly as product debt is a great first step but then comes the more difficult part of figuring out how to fit this work into an already packed schedule. My rule of thumb is that teams should be spending 20% of their effort in paying down product debt. This also includes platform, enablement, and infrastructure teams, any team that writes code, as they all incur this debt. How teams choose to schedule this should be left up to them. Some teams find dedicating one engineer per sprint effective, while others prefer to dedicate one full sprint each quarter. In ecommerce, sometimes teams prefer to push this work out until late in the year after they’ve shipped all of their features needed for the holiday shopping season. 

No matter how you convince executives, product partners, or even other engineers of the importance of this work, paying down product debt is an essential process for maintaining high availability and engineering efficiency. One of my favorite sayings on this topic is that “availability is the most important feature.” In any decent size company, there is no feature your product teams will launch in a year that will yield more revenue than just keeping the app or site running. As we’ve seen recently with the Southwest scheduling system, not paying down this debt can eventually lead to catastrophic downtime.