Sports uses metrics much better and more creatively to measure success than news organizations do. Sports metrics (sports fans are more likely to call them stats) also illustrate how misleading numbers can be.
You know who owns the Super Bowl record for most passes completed in a Super Bowl? Peyton Manning, who set that record this month in perhaps the worst loss of his career.
Manning, the Denver Broncos’ quarterback, completed nearly twice as many passes as Seattle Seahawks’ quarterback Russell Wilson in the Super Bowl for 74 more yards than Wilson. But no one watching that game thought that Manning played a better game.
Wilson completed a higher percentage of his passes (72 percent to 69 percent) for more yards per completion (11 to 8). Most important, Wilson threw for two touchdowns and no interceptions and Manning threw for one touchdown and two interceptions (one of them for a touchdown). In quarterback rating, Wilson blew Manning away, 123.1 to 73.5.
So what were the best metrics to measure the better Super Bowl quarterback this year? Passing yards and receptions can be really important metrics, but they aren’t the best metrics and they are most useful in the context of other metrics, including the final score. The losing quarterback often passes for more yards than the winner, passing a lot late in the game to catch up while the other team runs the ball to kill the clock. But Manning’s record 5,477 yards this year was one of the best measures of his great regular season, when his team won 13 games and lost only three.
Still, when it came to this year’s Super Bowl, a football coach (or fan) overusing passing completions or yards to measure performance would be evaluating quarterbacks wrong. In the same way, a newsroom overusing or misusing a key media metric, such as page views, Facebook fans or Twitter followers, will be evaluating its performance wrong.
I blogged yesterday about how important metrics are to measuring the changes that Digital First Media is pursuing in Project Unbolt.
I have likened Klout to the quarterback rating of social media: A rating based on multiple statistics using a formula that few people actually understand and producing numbers make sense only in comparison, but it seems to have some validity despite its flaws.
By itself, a 90 quarterback rating or a 60 Klout rating doesn’t tell you much. Neither is measuring 90 or 60 of anything. They’re just numbers on a scale that few understand (the highest passer rating possible on the NFL scale is 158.3 where did they come up with that?). But the numbers might be useful in measuring progress. If your Klout score has moved from 50 to 60, that probably means your social media influence is growing, just as if your quarterback rating was 80 last year, this year’s 90 rating probably means you’re having a better year.
Or they might be helpful in comparing performance. A quarterback with a 100 rating is almost always going to be better than a quarterback with an 80 rating, just as a journalist with a 60 Klout score almost certainly has more social-media influence than one with a 40. But if the ratings are 100 and 95 for quarterbacks or 60 and 55 with Klout, you might want to look at some other metrics, rather than assuming that the higher measurement that close really reflects better performance.
Manning’s 115.1 rating this year was outstanding, but the quarterback rating says Nick Foles had a better year (119.2). Foles had a great year, sure, but Manning threw for more than twice as many touchdowns. Manning’s highest quarterback rating ever was 121.1 in 2004. But he threw for more yards and touchdowns this year with the same number of interceptions. Until the Super Bowl, this might have been his best year, but not his best quarterback rating.
So if you use Klout or page views or Twitter followers or engagement minutes or any metric to measure success in changing your newsroom, step back now and then and ask what isn’t the measurement showing. Does the measurement match up with what you’re seeing? If not, seek some more measurements. You don’t want to assume that your view is accurate (we can all deceive ourselves, especially in times of change). But you want to learn what your metrics measure and what they don’t.
Sometimes you can overanalyze and lose sight of a goal. In my baseball blog, I wrote recently about why I prefer wins as a measurement of a starting pitcher’s performance over WAR, a stat attempting to measure the hypothetical wins a player would produce above those delivered by a “replacement” who might fill that person’s spot from the bench or the minor leagues.
Baseball stat geeks tend to like WAR better than wins in measuring a starting pitcher’s performance. But I noted that winning each game is the starting pitcher’s job and goal, and gave an example where WAR was seriously inaccurate when measuring a pitcher’s actual wins against an actual replacement’s wins. WAR fans fired back by citing Felix Hernandez, who pitched exceptionally well in 2010 and won the Cy Young Award despite a mediocre 13-12 won-loss record.
My point here is not to revisit that baseball debate but to note that you should be aware of the limitations of whatever metrics you use. If metrics conflict, try to look at the broader context.
Use metrics to help you understand, not as a replacement for understanding.
I’ll have another post about metrics, discussing how you look beyond the surface to see what the numbers are telling you.