Technical Debt: Where it Comes From and How to Manage it

Technical debt arises when the development team is forced to speed up the process of product delivery or upgrade, postponing refactoring for the future (which is usually unknown). According to Steve McConnell’s theory — the famous author of "Code Complete", "Rapid Development", and other books — there are usually two global causes of technical debt.

1. Intentional technical debt. In the pursuit of reduced time-to-market and business colonization goals, teams are developing the product at a breakneck pace, postponing basic engineering improvements, and focusing solely on delivery speed (release to web/production). At the same time, teams sacrifice quality: the cost of quality assurance increases significantly during internal and external improvements. Whereas at the beginning of the cycle the cost is included in the development process with allocated time, effort, and money.

2. Unintentional technical debt. The code might require improvement after a certain period of time, and teams learn about it from their own bitter experience when the product after release receives negative feedback from users. As a result of the RCA (root cause analysis), the teams find that either the dependencies in the modules have been poorly refactored, or the product code itself is outdated and needs to be extensively updated.

Despite the clear binary distribution, modern industry distinguishes 13 types of technical debt. Among them, the debt for the product code quality occupies an equivalent place among others.

What are the technical debts?

Martin Fowler, the co-author of Manifesto for Agile Software Development, refined Steve McConnell's theory, adding another feature to the "intentional/unintentional" dichotomy. It is distributed by the principle of "reasonable/reckless" in the context of determining technical debt. Accordingly, a matrix of 4 quadrants is drawn reflecting the types of technical debt:

• Reckless/intentional.

• Reckless/unintentional.

• Prudent/intentional.

• Prudent/unintentional.

As we learned, the difference between intentional and unintentional debt is the reverse approach to the work and the degree of the team’s prudence when the code base needs to be debugged. The difference between prudent and reckless is based on:

• The teams' full awareness of the fact of debt accumulation: "We know that we have sacrificed refactoring in this iteration/task/feature and we have to pay it off as soon as possible";

• Unawareness: "Have we accumulated debt? Didn't we refactor the code enough? And is refactoring included in our DoD (definition of done) at all?”

That is, with prudent decision-making, technical debt begins to accumulate for a number of reasons. However, the team and the business quite consciously take a "loan", reducing the quality and increasing the cost curve at the end of the project. In the case of reckless accumulation, the emergence of technical debt is due to the trivial loss of vigilance of the team, the lack of frequent revision of DoD, or intense brainstorming inside teams. This stage of team development is characterized by a large number of conflicts that can turn into personal confrontations instead of constructive and normal interaction.

In any case, each team will face a certain technical debt in the product. Engaging exclusively in engineering in the name of excessive product quality, without giving value to either business or users, is not cost-effective. The main difference will be in the amount of technical debt. Some will fall into the agreed percentage defined in the quality metrics. Others will be forced to engage in very frequent "cleaning" — refining and repaying the accumulated interest to generally accepted quality standards.

Approaches to technical debt management

Debt management strategies depend on the chosen project management process and the terms of the outsourcing contract between the customer and the contractor. In flexible frameworks and methodologies, particularly in Scrum, technical debt management is transparent with the involvement of a sufficient number of stakeholders with varying degrees of responsibility for the work product. Other stakeholders are involved in the development team, depending on the organizational structure, for example, project sponsor, business owner, etc. In my previous projects, where I used Scrum/SAFe, the following approaches to debt management were employed:

1. Prioritizing technical debt transparently in Refinement or Grooming meetings with the product owner, working on each iteration (percentage varied in 10-20% depending on the purpose of the iteration or its proximity to release).

2. Establishing refactoring as a separate item in DoD (product readiness criteria, product approval, and development team). Frequent DoD adjustment — we basically reviewed and adjusted it every 2-3 months before the new (program increment).

3. Tracking the percentage of technical debt in the backlog of the product in comparison with the tasks and risks; standardizing the distribution of the number of such tasks in proportion to the risks of iterations.

4. Taking architectural and infrastructural enablers in sufficient quantity, ensuring product stability in each program increment (PI).

5. Constantly explaining the risks associated with insufficient planning, working on non-functional requirements of the decision in cash equivalent before and during the project development (when the product owner submits preliminary total cost of ownership for consideration by the project sponsor).

6. Applying technical debt accumulation tracking metrics and reviewing them daily, before the backlog audit and during a retrospective. For example, the number of detected defects per N lines of code, the depth of inheritance, the integrity of the code, the (collective code ownership), and so on.

7. Incorporating refactoring into assessment (absolute or relative) during each planning (release planning, PI planning, iteration planning, replenishment, if possible).

8. Collecting and sharing acquired knowledge and practices of technical debt management at least once in IP (Innovation and Planning). This is usually one in five iterations of innovation and planning in SAFe.

If there are certain contractual obligations, or the project is delivered according to stricter methodologies, I would suggest the following actions:

• Prescribing refactoring in RFP (request for proposal) in response to the customer's request in SoW (statement of work).

• Describing in detail the non-functional requirements in the requirements documents.

• Standardizing project/product quality management procedures.

• Determining the permissible error of technical debt in quality metrics.

• Putting refactoring in the calculation of the cost of quality.

• Maintaining the relevance of technical documentation.

I will share some practical examples of working with technical debt on past projects. On one of them, we worked in cooperation with several vendors, and the whole backend was written in coffeescript. For about six months, we offered "decaffeination": we actively persuaded the CTO and the architect on the part of the customer to start moving towards the conversion of code to JavaScript, with the prior support of other teams.

We tried to argue differently, held many meetings, drew diagrams, and worked with objections and rejections from key stakeholders. In the end, we actually convinced the technical management of customers and even a few interested businesspeople in favor of conversion, which would significantly reduces the cost of product development and maintenance. And we got the green light! Decaffeination of the whole backlog took about four months of our team's work and motivated its members during this time. Moreover, when the teams and product owners of the customer took a month-long vacation after a successful release, decaffeination was the only task of our team in two sprints.

On another project, we immediately agreed that we allocated every eighth day of the sprint to code refactoring, while the testers tested build regression scenarios in terms of user acceptance testing (UAT). Two more days were left to correct the defects and prepare for the sprint review. We agreed on this at the level of CTO/CDO (Chief Delivery Officer). And this approach was reflected in the tacit agreements of the teams.

How does technical debt affect project development, and what are the consequences of ignoring it?

The accumulation of technical debt and its timely non-repayment steadily lead to a deterioration in the product, service quality, or overall result.

In the pursuit of a good cause — a quick return on investment (ROI) — the business launches a time bomb, by forgetting about the technical part of the project, and turning a blind eye to the reasoning of engineers. I mean the cost of support, which will increase closer to completion and surge sharply when the product is transferred to operations and support. The product will become so unreasonably expensive for further development that it will be cheaper to rewrite it from scratch.

With further evolution, for example, the launch of one or more related projects, the investment spent on restarting will be meaningless. The constant deterioration of one of the key constraints of the project — and this, in fact, quality — will automatically affect other constraints: schedule, risks, resources, costs, and content. This will offset the results obtained in each area.

When debt becomes a destructive element of the system, it decomposes it from within, leading to inevitable entropy. Problems accumulate like a snowball, and untimely corrections would hardly help.

All this can be avoided by establishing a timely communication and technical debt management plan, especially in the context of flexible product delivery practices — the good old Agile. And indeed, almost all frameworks, such as Scrum, SAFe, RAD, XP, FDD, DSDM, partly Kanban and Lean and their hybrids —  ScrumBan, Scrum + XP, etc. — are essentially aimed at:

• daily interaction between technical teams and businesspeople;

• simplification of communication;

• the ability to reconcile complex issues transparently, quickly, and comprehensively.

Despite the fact that frequent changes in requirements are directly proportional to the emergence of technical debt, most Agile frameworks are balanced to respond quickly and minimize both risks and technical barriers.

Well-established communication between teams and stakeholders is based on the principles of transparency and trust: honest assessments, clear DoD, timely articulated risks, public recognition of mistakes on both sides. All this creates the necessary social environment for open dialogue, which, in turn, ensures mutual understanding and alignment of the backlog in line with current priorities.

Conclusions

The following are key technical debt management recommendations:

• Define a technical debt management strategy in advance.

• Agree on this strategy with key project participants.

• Address the issue of technical debt openly at general meetings (iteration reviews, retrospective, iteration planning, PI planning), or through a proxy (scrum master, product owner), etc.

• Prioritize debt tasks along with features and risks in the process of "combing" the product backlog.

• Incorporate refactoring into the assessment of performance, stories, and updates.

• Talk about technical debt practices during the retrospective and take action items in each iteration.

• Constantly monitor quality metric trends to identify and reduce the impact of debt on the product in a timely manner.

• Make your DoD transparent to all project participants (a Wiki page or physical printout on the team board) and update it regularly.

I sincerely hope that this article has added new tools to your arsenal of work with technical debt that you can use in the future. Or at least, now you have food for thought.

In this article Andriy Troyanov, Delivery Manager at Innovecs, shares his experience with technical debt, approaches to its rapid repayment, and minimization on previous and current projects.