Conflicting research results – now what?
Many people are confused by news reports about research studies that conflict with previous studies or long standing ideas. These conflicting research results bring on many questions. Who is right? Who should I believe? What should I do? What does it mean when experts disagree?
It is common for researchers and experts to disagree, especially on research frontiers where new ideas challenge our current understanding. Most researchers strive to increase our understanding and sometimes that upsets the established point of view. In 1997, Nobel laureate John Polanyi wrote, "every issue of a scientific journal, every scientific conference, and every informal meeting between scientists is devoted to testing the current orthodoxy in order to see whether it can be improved."
Researchers who propose new ideas and report new data must describe and defend their methods and conclusions. Other researchers ask questions, challenge methods and conclusions, and present alternative ideas and data. Over time, as information accumulates and our understanding increases, a consensus usually emerges built on the foundations of solid reproducible research.
The ability of other researchers to replicate new findings is key to accepting new findings. In the late 80s, researchers claimed to have produced energy using cold fusion. When other researchers could not replicate the work, cold fusion was discredited. If work cannot be replicated by others, the findings will not be accepted. On the other hand, scientific rebels that were initially ignored or scorned eventually become the new orthodoxy after others confirm their discoveries and conclusions.
Today we continue to hear conflicting research studies. For example, a report last year by Frank Mitloehner, University of California Associate Professor and Air Quality Specialist, and two colleagues argued that a 2006 United Nations Food and Agriculture Organization report was not accurate when it reported, "the livestock sector was responsible for 18 percent of the greenhouse gas emissions measured in carbon dioxide equivalents. This is a higher share than transport." Mitloehner argued that the UN report did not use consistent methods when analyzing the livestock and transportation sectors. In one news report, Mitoehner described the comparison as a "classical apples-and-oranges analogy that confused the issue." Subsequent news reports quote one UN report author as accepting the criticism that the comparison with transport data was inaccurate. This admission by one of the co-authors of the UN report is not the end of this controversy over greenhouse gas emissions from animal agriculture and other sectors. The UN is expected to publish a revised report.
So what are we to make of this and other research controversies? It is important to remember a few key things about research:
- Research is a process that involves some trial and error, mistakes being ferreted out and corrected, new questions being asked and answered, and hypothesis being tested and replicated. It is common for researchers to take two steps forward, one step backward, and zigzag as their original hypothesis are proven wrong or they find surprising results. Researchers need time and resources to replicate the work of others and accumulate the ideas, data, models, answers, and experience to hone in on a consensus understanding.
- Many research projects are very controlled and applicable to specific conditions. Results from these very specific studies cannot be easily applied broadly. Without more studies under other conditions, the initial results, while valid, are only useful in those original specific conditions. It is dangerous to extrapolate initial results.
- Some apparent conflicting results and conclusions can be due to small differences in experimental conditions or assumptions. These details may explain differences in results and conclusions that summaries and news reports miss. It is important to look for details about how the study was done and the assumptions made. Refereed research reports need to be clear and complete so that others can replicate the work and small differences in the experimental conditions or assumptions can be discerned.
- Researchers, even very experienced ones, can become biased or make mistakes. Most work very hard to avoid bias and find mistakes before the results are presented or published. Subsequent work that confirms initial findings strengthens the research community's confidence in the original work.
When you read about new research, especially surprising research, take the initial results and conclusions with a proverbial grain of salt. Be both skeptical and open to accepting the new information. You can either wait until the initial results have been replicated and confirmed by others or you can begin to use the result to see if they work for you. Remember that a single research study does not prove a new idea or challenge the current understanding. It takes an accumulation of work and clear descriptions explaining the physical, chemical, and biological processes that leads to our best understanding.
Published in Dairy Star July 2, 2010