It happens twice a year. I’ll write about not taking data at face value. I’m often triggered by somebody saying with downright religious fervor, “The data never lie,” “If you can’t measure it, it doesn’t exist," or “I’m completely driven by the data.” Good grief; I hope not.
Data are essential tools of analytics and decision making, of course. Effective leaders cannot function properly without meaningful numbers that tell true stories about trends and performance. Folks who tremble at the sight of data and, in turn, refuse to assess quantitative evidence to make judgments are just as misguided as those who blindly worship at the altar of numerics. That’s why the “lies, damn lies, and statistics” derogation is equally troubling. As is often the case, wisdom is to be found between the two extremes.
Still, data are not infallible. As long as we mere mortals are involved in their collection, categorization, analysis, and interpretation, data mislead and lie every day. Too often, data can be manipulated and presented to reflect biases if not fully conscious agendas of some kind. More often, however, data such as those found in predictive models - that are of necessity laced with assumptions, estimations (informed guesses, really), choices about variables included and excluded, use of relevant and irrelevant past-performance measures, and unconscious biases - can be more innocently and unintentionally misleading. Don’t get me wrong. I still want the best possible models, but end-users need to leaven use of these models with their own independent thinking and common sense.
This critique about data does not apply to mathematics and the physical sciences where, absent human error, the numbers are generally accurate and do reflect reality. Rather these observations relate to the use of data in business, polling, journalism, sports, and the social sciences where it can be more, well, pliable if not manipulable.
In her 2020 book, “Counting: How We Use Numbers to Decide What Matters,” Deborah Stone writes that, “Numbers enjoy an aura of objectivity and precision unwarranted by their origins.” She rightly adds that, “We can’t do without numbers, so the challenge is how we can do a lot better with them.” Her concerns about counting are many, offering that “numbers don’t speak for themselves, but for their creators” and “if you torture the numbers long enough, they’ll confess to anything.” We witness this everyday with politicians, pundits, ideologues, and in popular media.
This caution about counting includes all the ways in which some performance metrics, in Stone’s words, can “distort people’s goals and make them aim to achieve good numbers instead of good results.” Does this ring a bell, teach-to-the-test supporters in education? She cites elaborate data-driven Key Performance Metrics that Uber uses to assess driver performance that do not consider ride quality, people skills, and route difficulty. I’m sure Uber’s evaluative measures are statistically accurate, but there’s no way in hell they tell the truth. Of course, Uber drivers will teach, or better, drive to the test, which apparently does not or did not include customer service.
Ultimately, Stone calls for humility when weighing data. She advises, “We need humility when we count so we don’t mistake numbers for life. Numbers grow from human imagination. We shouldn’t put forth numbers or rely on them without examining the human judgment they stand on.” You can count on that.
