A Developer Blog

What are you measuring?

Sunday, 24th September 2006

There is a common saying amongst management types - If you can't measure it, you can't manage it. I think this is complete and utter hogwash, because many of the most important things can't be objectively measured. This doesn't stop them from trying, though.

One thing that ends up happening is a kind of quest, seeking to find the right Key Performance Indicators, or KPIs, to use as measures.

Very often, though, the presence of the KPIs acts to distort - and corrupt - the very business that they are measuring and trying to improve.

Here are some examples that I've seen in my own career ...

KPI	Goal	Resulting Behaviour	Actual Outcome
Bugs fixed per day	Get Bugs fixed faster, so we can ship earlier.	Every trivial bug, no matter how minor, was formally logged (with all the effort that this entailed).	Lots of bugs were logged and fixed, but actual progress towards shipping slowed.
Average age of open issues	Make sure that issues aren't left around forever	All the old, trivial issues that nobody cared about were closed, while newer urgent issues that actually impacted on real customers were ignored.	Customer satisfaction fell, because all the effort was being expended on issues that nobody was affected by.
Lines of new code per day	Encourage developers to be more productive	People would use copy & paste when a similar situation arose, instead of refactoring a common routine to reuse. Everyone's efforts were on writing new functionality, so the existing codebase wasn't routinely tested, and any bugs found were left for later.	The quality of the codebase fell dramatically, and maintenance became a real nightmare.
Average length of phone call	Provide prompt, rapid, service.	Instead of working with the customer to solve the problem, walking through various possibilities to troubleshoot the issue, a quick suggestion would be made and the customer asked to phone back if that suggestion didn't work.	Call volumes went through the roof as customers were forced to call back five or seven or a dozen times to get a solution to a simple problem.

In each case, the source of the problem was a poorly chosen KPI which encouraged the people involved into behaviour that was detrimental to the business but advantageous to the KPI.

Gaming of a KPI usually becomes apparent, and there are often ways around the problems to measure the right things for the right reasons.

The danger come when gaming of the KPI isn't apparent - or when management becomes convinced that the measures of the KPI are more important than the behaviour they are promoting.

Eric Gunnerson has a good discussion on the effect of poor KPI selection and the effects this can have.

External Links

Eric Gunnerson's C# Compendium

http://blogs.msdn.com/ericgu/default.aspx

Bugs, blogs, and surrogates

http://blogs.msdn.com/ericgu/archive/2006/09/21/765495.aspx