(This piece is a work in progress, a placeholder for developing thoughts on technical debt and measuring the performance of software development teams. I will update it as my opinions change.)

An oft-cited benefit of paying down technical debt is that it will allow you to move faster. This is typically characterized in terms of increased velocity.

But if you are measuring velocity by factoring in “accidental complexity”, then when you remove the source of that accidental complexity, your velocity will not actually appear to improve. This point is well put by Moneyball for software teams: an imperfect heuristic for quantifying dev performance. This runs contrary to what we’d generally expect from a measure of developer velocity, and also contrary to the business case that is usually made for paying down technical debt: to simplify, reduced debt = increased velocity.

The aforementioned article presents a reasonable way out of the impasse of a) needing to factor tech debt into your estimates, b) wanting tech debt reductions to be visible as increased velocity i.e. more “story points” per cycle. (Whether it is necessary or desirable to adopt this approach is debatable, and personally I’d still bias towards avoiding measurement of velocity at all, and having developers do PERT estimates only as necessary. This seems to be less common outside small, high trust teams.)

But even if you are measuring velocity this way, it is entirely possible that you could pay down all the tech debt and still not have your velocity  (as a measurement of essential complexity) increase:

  • new sources of accidental complexity can emerge that are not at all under your control, whether from vendors, dependencies that are suddenly EOLd or have hard to patch security flaws, and so on
  • eliminating a given set of sources of accidental complexity may enable your team to take on other types of projects that are subject to additional kinds of accidental complexity. These new tasks or projects may have higher business value, but that won’t be reflected on your velocity (or probably reflected anywhere because We can’t measure developer productivity). This is reminiscent of the way in which improving tail latency can actually cause median latency to increase: previously, requests from slow clients would time out or never be initiated in the first place, but after improving performance for those clients, their requests will now succeed, albeit at latencies greater than your current median.
  • even if your velocity does go up, will the signal be strong enough to be distinguishable from statistical noise (maybe one of your senior developers finally got their kid into an after school program and now they are back to “full performance” level)? do you have tools sophisticated enough to discern this?

Returning to the moneyball comparison from above: sport performance is generally easily and unambiguously measurable (did the ball go in the net?), and what it means to do well can be reduced to “win competitions”. Software development is inherently harder to measure, and has many axes along which something can be called “a success”. Moreover, whether a software project turns out to have been successful may not be discernible for months or years down the road, for reasons business or market reasons that have nothing to do with the development project “in itself”. For these reasons, it is at best a waste of time, if not actively harmful, to foist such methods into software management.

That said, my thoughts on this topic are always challenged by the ideas of Hubbard (e.g. How to Measure Anything) whose main argument boils down to, if it matters you can notice it, and if you can notice it, you can (at least sort of) measure it.