Statistics are used to summarize and clarify the nature of complex society. Debates over social problems generally require statistical answers. Therefore, statistics can be used as a weapon in political struggles over social problems and social policy. Statistics is shaped by our language, culture, and society therefore we should not see these statistics as “hard facts”.
The ways people produce statistics might be flawed. There are four basic sources of bad statistics: bad guesses, deceptive definitions, flawed measurements, and biased samples. Guessing is a common way for activists because most of the time, there are not any good statistics available for a new social problem. Therefore, activists have to make guesses about the size of the social problem. Since they want to draw attention to this new social problem and since they believe that this new problem is big and important, they tend to exaggerate and overestimate the problem’s size. These guesses are repeated routinely among officials, experts, activists, and other reporters and are treated as straightforward and fact. People start to assume that these numbers must be correct because they appear everywhere. Also, over time, people may change the original guess to embellish statistics. Furthermore, people develop an emotional stake to promote these numbers and attack those who question them. Thus, large guesses are reported everywhere even if there are new findings about this problem.
Bad statistics may be a product of poor definitions. If examples substitute for definitions, our understanding of the problem may be distorted. Also, activists tend to make general, broad, and inclusive definitions that support larger estimates of a problem’s size at the cost of “false positives” (mistakenly identifying cases as part of the problem). If the problem is defined too broad, then there will be bigger statistics. Therefore, whenever we encounter statistics, we should ask “How is the problem defined?” and “Is the definition reasonable?”
Flawed measurement also results in bad statistics. Sometimes, survey questions are worded carefully in order to encourage people to respond in the desired way. Also, people who conduct surveys can decide how to interpret the results. Moreover, activists tend to devise measurements that will minimize “false negatives” (incorrectly identifying cases as not being part of the problem). Therefore, like definitions, measurements also involve choices and these choices shape the statistics. The other source for bad statistics is biased samples. If sample does not accurately represent the population and if samples are not random, then the statistics which are produced from these samples will be flawed.
Although some statistics are born bad through guessing, dubious definitions, questionable measures, and poor sampling, there are also other statistics which mutate and become bad over time. That is to say, good statistics can turn into bad statistics (mutant statistics) through mangling, misusing, or misunderstanding. Detecting mutant statistics is not easy at all because it requires tracing the history of a number. Innumeracy -“difficulties grasping the meanings of numbers and calculations”- is the fundamental source for mutation. Mutant statistics might be derived from the fact that both advocates and their audiences are innumerate. At the same time, mutant statistics might have their roots in manipulation: “conscious attempts to turn statistical information to particular uses”.
There are four common ways of creating mutant numbers: generalization, transformation, confusion, and compound errors. The first way of creating mutant statistics is making inappropriate generalizations from a statistic. If there are elementary forms of error while collecting evidence such as flawed definitions, poor measurements, and bad samples and if generalization is made from this evidence, then mutant statistics emerge. The second way of creating mutant statistics is transforming a number’s meaning. Transformation may be deliberate efforts to mislead people so as to support their claims. At the same time, transformation may be inadvertent. In such cases, people try to repeat the number that means one thing but accidentally interpret it to mean something completely different. The third way of creating mutant statistics is garbling complex statistics. Sometimes, complex statistics such as odds ratios can produce confusion and eventually correct but difficult to understand statistic may turn into easy to understand but completely wrong number. Lastly, the fourth way of creating mutant statistics is creating chains of bad statistics. Sometimes, people may distort the bad statistics further in the process of calculating more statistics. That is to say, there can be a chain of errors if one questionable number is used to create second statistic, this second statistic is used to create a third statistic and so on.
On the other hand, inappropriate comparisons also create bad statistics. There are some common errors in comparing two or more time periods, places, groups, and social problems. The comparisons over time might create bad statistics due to poor projections, changing the way to measure the social problem and also unchanging measures. Additionally, the statistics gathered from different places are based on different definitions and different measures, thus they are not really comparable. Moreover, the comparisons among groups might lead us to make wrong choices due to different group sizes or unmentioned variables (such as social class). Furthermore, the flaws such as estimating the dollar costs of various social problems or focusing on some narrowly defined population where the problem is relatively concentrated might cause bad statistics while comparing social problems.
Furthermore, stat wars cause confusion, distress press and intimidate people. It becomes so hard for people to evaluate competing statistics. Usually, there are few people who make the effort to examine original sources in order to reconcile a stat war. People generally assume that the people who offer contradictory statistics are the ones telling lies. Therefore, many bad statistics endure because no one attempts to question them.
In conclusion, bad statistics are potentially important because they can lead people to make wrong policy choices. Therefore, it is very important to watch out for bad statistics. There are three general approaches to thinking about statistics: naive, cynical and critical. “Naive” presume that statistics are generally accurate therefore they accept statistics without questioning. In contrast, “Cynical” do not trust numbers because they believe that statistics are flawed and these flaws are probably intentional. On the other hand, “Critical” approach statistics thoughtfully, evaluate numbers, and distinguish between good and bad statistics. Therefore, the best solution is adopting critical approach and becoming better judges of the numbers we encounter. We should think critically about statistics, we should approach them thoughtfully and we should keep in mind the signs of bad statistics. We should not accept statistics without asking questions because every statistic reflects its creator’s choices. We should not forget that every statistic has flaws and therefore, we should decide whether these flaws are severe enough to damage the statistic’s usefulness.
(This article is prepared by using the content of Joel Best’s “Dammed Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists” book.)
Yazarın diğer yazıları için tıklayınız.