Bad Metrics Make for Bad Decisions

Photo by Luke Chesser on Unsplash: https://unsplash.com/photos/JKUTrJ4vK00

Many of the metrics that dominate news are misleading. For example, take the recently-fabled Twitter 5%. Twitter claims that at most 5% of its daily users are bots. Leaving aside the fact that figure has a 4.3% confidence interval, Twitter’s management not only didn’t answer the right question but also Elon Musk and the rest of those covering the story didn’t call them on it.

The relevant question is how much daily traffic is attributable to bots. If 5% of daily users are bots, then the bot-driven daily traffic is likely to be multiples of that figure.

This isn’t an isolated example.

The CDC focused our attention on near-term COVID deaths when the major concern should have been promoting policies that enhanced the long-run well-being of the American people. Figuring that out is beyond the CDC’s mandate and ability, which immediately tells you they weren’t the right organization to have been setting the pace for government action.

The saying goes what gets measured gets managed. Fair enough, but equally or more important is its natural corollary. What doesn’t get measured gets mismanaged.

This isn’t an academic question in the consumer analytics space. If a company doesn’t know the relative value of customer acquisition and customer retention or the impact of service quality on future purchases then on what basis is it allocating its time, money, and effort?

Managers looking to maximize the long-run value of their enterprises need to seek out the answers to these critical questions.