Managing Technical Debt

Most of us have experienced occasions where we've been required to take short-cuts to make delivery deadlines. These short-cuts are seen as bad by many prominent people in the software industry. In fact, Robert C. Martin puts it this way [ Martin ]:

The next time you say to yourself: "I don't have time to do it right." - stop! Rethink the issue. You don't have time not to do it right!

Nonetheless, short term hacks continue to proliferate and many developers are under a lot of pressure to make them. Given that the long term by definition lasts for longer than the short term, this is a problem.

In fact, it's a big old warty problem with attitude and bad breath. Taking short-cuts generally means that the next time the software is touched, it needs to be fixed before any further work can be done. So the second piece of work is even more expensive to do correctly than the original piece of work, and you can be sure that the deadlines are just as tight as they were the first time. Worse, developers generally prefer to play it safe - if someone has left them a dodgy-looking piece of code, they prefer to leave well enough alone. So, unless there are strong individuals present who are really dedicated to good engineering, the team takes another short-cut and works around the code affected by the previous short-cut1. The third change involves working around the first two short-cuts, and so on.

If one follows the trend to its logical conclusion, and in my experience many teams do, one finds that the code complexity grows at an increasing rate. After several changes to the software, it reaches the point where nothing can be changed without significant time and risk. Usually at some point, the team begins to realise that they need to fix the things they've broken. But by then it's too late because they are spending all their time just keeping a fragile system running and have no spare capacity to fix the code. They've painted themselves into a classic Catch 22 situation.

This article is not primarily about escalating long term costs, so I won't labour the point. We'll assume the above argument is sufficiently convincing. The trouble is that in many cases, a good counter-argument can be made for the short term benefits.

Consider a small company in a niche market - its entire future generally rests upon being first to market. Longer term software problems simply aren't important in this case because they are not visible to the customers until it's too late. And they are certainly less important to the software company for whom the long term won't exist unless it makes the sale in the short term.

Or take the financial industry. When new ideas come to the market, there is generally a small window during which very large amounts of money can be made. In this case, a company which spends too long writing the software might as well not have bothered in the first place.

Similar short term pressures occur on both large and small scale every day for any number of reasons.

As any weatherman will tell you, the short term is always clearer than the long term. So while short term costs might not be as great, they are easier to quantify. It can be very difficult, therefore, to determine just what the relative costs and benefits are when deciding whether to "do it the right way" or to take a "short-cut".

Technical debt

The phenomenon mentioned above of costs growing at an increasing rate led Ward Cunningham to liken poorly-engineered software to debt [ Cunningham ]. This turns out to be an extremely good analogy. We've all heard of people who've completely lost control of their credit cards, and who spend all the money they earn just servicing their debts. A company in the Catch 22 situation above is just like someone whose credit card debt has become out of hand.

Formalising the analogy, any time a software team follows bad engineering practices they incur two kinds of cost. First, there is the cost of repaying the "capital", i.e. undoing bad code and replacing it with well-engineered code. Second, there is the "interest", the ongoing increased cost of supporting, maintaining and enhancing the software.

It is generally fairly manageable to take on a small amount of technical debt, but if one doesn't pay the interest cost, or insists on taking on ever more and more technical debt, one soon loses control.

So the analogy serves as a great example because it allows us to draw on the commonly known human experience of taking on debt.

Managing technical debt

Having set the scene, we can now get to the real point of this article - how to manage technical debt. I observed above that sometimes it is vital to create software in the shortest possible time, regardless of quality. At other times it is not vital, but there is a lot of pressure nonetheless. At all times, it is hard to quantify what the long term benefits and costs are.

Human beings have thousands of years experience with managing financial debt. Can this experience teach us anything about managing technical debt? A little googling will confirm that many people, some of them quite prominent, think the answer is "yes".

Here are some key strategies for managing financial debt:

1. Only ever enter into debt if the benefits outweigh the costs. So, for example, a student loan, or a mortgage is generally considered good debt. In the long term the benefits of education or owning your own home outweigh the cost of the debt. Most credit card debt, on the other hand, is not sensible because the exorbitant interest costs outweigh any benefit.
2. Know how you will repay the debt when you enter into it. This is pretty much mandatory with a mortgage, because the banks make it a prerequisite. Credit card debt, once again, often doesn't have this sort of planning associated with it.
3. Keep track of your debt. Ensure you are paying it off. Understand which debt is the most costly and pay that off first. These are the first principles that are discussed with people who've lost control of their financial debt.

If we adapt these to technical debt we get the following strategies:

This isn't as easy as it sounds because it is hard to quantify the cost. But we can work to educate our customers, and help them to see that there are generally significant long term costs caused by rushing in the short term.

Another practice, which seems obvious but is often neglected, is to understand why the customer thinks something is really urgent. Often software users give unreasonable deadlines out of ignorance and would be quite happy to allow more time if the software team talked to them. Other times, users ask for something they think will solve their problem, when a simpler and technically better alternative would do just as well.

Another idea is to break down the requirements into something that can be delivered quickly followed by something that will take a little longer. If you can get one or two high priority features in front of the users quickly, they will often be happy to wait for the lower priority ones.

The key points here are to communicate with the users and to be creative.

Never take on technical debt without first spending some time thinking about a more strategic approach. Too often "tactical fixes" are at complete odds with any potential "strategic solutions" even though there exists an alternative short term approach which would go at least part way towards a strategic solution. With some thought you may be able to find a better tactical fix and reduce the debt straight away.

Having spent, say, half an hour working out a reasonable strategic direction, think about how and when the strategic solution will be implemented. Assign a value for the "capital", i.e. how many man days it will take to repay the debt.

Assign the debt an "interest rate". That is to say, get an idea of the increase in running costs caused by the debt - you might want to use several categories, like high, medium and low.

Finally, and this is really key, raise a change request ticket and add this information to your project plan.

If you follow the advice above about adding technical debt to your project plan, then suddenly you'll be able to keep track of your level of debt. At any point, you can say, "I have X man weeks of outstanding high, medium or low interest technical debt". Then you can start setting thresholds, for example you may say that you won't accumulate more than 2 months of high interest debt, 4 months of medium interest debt or 6 months of low interest technical debt.

As debt increases, you can see it happening and you can therefore manage it. This will mean increasing resourcing or pushing back on some new work. But now you can justify the increased cost because you have real data about what needs to be fixed. This is far better than a vague, unspecified concern about poor engineering practice, or a system that just keeps failing unpredictably.

Keeping track of your debt also means re-assessing it from time to time because the capital cost of technical debt increases over time. There are at least two factors at work here. In the short term, the longer you wait before repaying the debt, the less you remember about how to repay it. This is particularly true when team members leave, taking the knowledge with them. Over the longer term, the technology industry moves on and it becomes harder to find people who know how to work with the technology in which the debt was incurred. For example, it's not as easy as it once was to find someone who can fix code written in FORTRAN or COBOL. Even C is moving (many would say has moved) into this camp.

So you need to keep this inflation effect in mind. Re-evaluate existing debt occasionally and be reticent about open-ended technical debt.

Does it really work?

Of course, there is no silver bullet. However, I have used some of the ideas above extensively and found they do help a lot. I have only fairly recently begun to follow the last of the suggestions labelled "keep track of your debt". So far it looks promising.

Googling confirms that I'm not the only one to find these ideas helpful.

In the long run we're all dead [ Keynes ]

One debt management strategy that hasn't been mentioned so far, but is frankly a very real for many people, is to leave the company (or even the country). How does one answer the team member who says, "What's in it for me, I won't be the guy who sees the long term benefits or costs"?

A less than motivational answer is that this is what we get paid for. So it's a matter of moral obligation.

For a more motivation answer we can appeal to the sense of a job well done.

But perhaps the best answer is a pragmatic one. If all software teams control technical debt well, then everyone's lot is improved. This introduces a whole new topic which deserves an article of its own, the Tragedy of the Commons. This is an observation that in many situations the best outcome for society as a whole only occurs when all individuals choose a less than best outcome for themselves personally.

The Tragedy of the Commons is related to a problem from Game Theory called the Prisoner's Dilemma. This is a game in which two players may either "co-operate" or "defect". The pay-offs are cleverly structured so that:

1. Each individual player is most motivated to defect.
2. The best pay-offs occur when both players co-operate.

According to Wikipedia [ Wikipedia ], Robert Axelrod [ Axelrod ] performed an experiment with automated agents which played an extension of the Prisoner's Dilemma. He found that greedy agents tended to fare worse than more altruistic agents in the long run.

So with a bit of hand waving and some smoke and mirrors, we can make a case for the pragmatic answer.

So what should you do?

The ideas in this article need buy-in from the entire software development team. Very importantly, they need buy-in from the team leader. So you'd think this article is aimed at software team leaders, letting the rest of us off the hook. Not so. In practice it is often the techies at the coal-face who have to exert influence over managers to get the point across.

Given that most of us are in a team of more than one, the first step will be to act as an influencer. Introduce the technical debt analogy. I've found that people tend to take to the basic idea quite quickly. From there, it is a matter of gently extending the analogy.

Finally, the ideas above are just a starting point. For example, at least one web article [ ThinkBox ] extends the analogy to concepts like taking payment holidays and making lump-sum repayments.

Conclusion

Let us assume that a software engineer's job is to create the most profitable software for the least cost. Then it is the responsibility of the software engineering team to take into account the short term benefits and costs as well as the long term benefits and costs. This is hard because we work with complex systems and incomplete data, Kevlin Henney would say that we lack visibility [ Henney ]. Thinking in term of managing technical debt like we'd manage financial debt gives us ways of collecting data about the long term and therefore increasing visibility.

References

[ Martin ] "The Tortoise and the Hare", Robert C. Martin, 2004, http://www.artima.com/weblogs/viewpost.jsp?thread=51769

[ HuntThomas ] Andrew Hunt and David Thomas, The Pragmatic Programmer: From Journeyman to Master , Addison-Wesley, 1999

[ Cunningham ] Ward Cunningham, The WyCash Portfolio Management System, 1992, http://c2.com/doc/oopsla92.html

[ Keynes ] John Maynard Keynes, A Tract on Monetary Reform ,1923

[ Wikipedia ] Prisoner's dilemma , http://en.wikipedia.org/wiki/Prisoner's_dilemma

[ Axelrod ] Robert Axelrod, The Evolution of Cooperation , 1985

[ ThinkBox ] Repaying technical debt , November 2005, http://www.think-box.co.uk/blog/2005/11/repaying-technical-debt.html

[ Henney ] Kevlin Henney, Five Considerations in Practice , ACCU conference 2006