Posted on 15 September 2021 by Guest Author

This is a re-post from the Thinking is Power website maintained by Melanie Trecek-King where she regularly writes about many aspects of critical thinking in an effort to provide accessible and engaging critical thinking information to the general public. Note: This article is the second of a two-part series on “doing your own research.” To read the first article click here.

It seems nearly everyone is “doing their own research” these days. And to some extent it’s understandable: we want to make good decisions and there’s a seemingly endless amount of information available at our fingertips.

Unfortunately, access to information simply isn’t enough. Although it’s difficult to admit, we aren’t as knowledgeable or as unbiased as we’d like to think we are. We often resort to “doing our own research” when we want (or don’t want) something to be true…and so we set out to find “evidence” to make our case. Due to an unfortunate mixture of motivated reasoning and confirmation bias, we end up wildly misled yet even more confident we’re right.

Needless to say, this isn’t how real research works. What you’re actually doing is looking for the results of someone else’s research. The real question is, how do you decide which source to trust?

In a previous article I tried to point out the dangers of doing your own research, which was essentially a plea for intellectual humility and trusting experts. But if you want to “do your own research,” you basically have two options: Find and trust the expert consensus or become an expert yourself and do your own (real) research.

What is a consensus? And why is it trustworthy?

Scientific research these days is highly specialized, with each subfield having its own nomenclature, methodologies, statistical analyses, etc., required to represent and communicate its nuance and complexity. Experts have the knowledge, skills, and experience necessary to evaluate the quality of evidence provided by any particular study, and importantly, to put it into context with the larger body of literature. Scientists researching heart disease are not trained to understand the literature on black holes, for instance, or even eye diseases. (Note: This goes for you, too. The scientific literature is where specialized scientists talk to each other. It’s not for non-experts.)

Contrary to how science is often taught in school, there is no single scientific method. Instead, science is a community of experts using diverse methods to gather evidence and scrutinize claims. The social aspect of science is a major reason it’s so reliable, as there’s a systematic way to correct for the biases, errors, and even fraud of individual scientists.

At the heart of the process of science is peer-review, in which research must pass the critical scrutiny of other experts before it’s published in a scientific journal. However, a single study is never the final answer: conclusions must be replicated and fit in with the larger body of evidence before scientists will accept them.

The peer-reviewed literature is where experts share their research with other experts and can be very difficult for non-experts to read. Conversely, the popular media often sensationalizes findings when they’re “translated” for the general public.
Sources: Steinberg, et al (2017); Peter Hess, Inverse

When independent and diverse lines of evidence converge onto a conclusion, the conclusion is considered strong, and experts generally accept it. The result is an expert consensus, or the collective position of experts based on an evaluation of the body of evidence. It’s not the final word, but the starting point upon which the vast majority of experts agree. Scientists can then build upon the foundation of knowledge that was gained through the process of science to learn more about what they don’t know.

The consensus is likely incomplete and it’s open to challenge, but it’s very unlikely to be completely overturned. If there is a problem with the consensus, you’re not going to find it with your “research”…it’s going to be an expert who does. And because the incentive structure in science rewards scientists who discover new things or prove established knowledge wrong, overturning the consensus would be a career-winning strategy, the kind of which Nobel prizes are made. The likelihood of thousands of scientists giving up fame and fortune for the sake of maintaining a conspiracy, as suggested by many denialists and pseudoscience promoters, is next to none.

Most people generally trust experts and use the consensus as a short-cut for decision-making. For example, if five electricians told me the wiring in my house was in danger of starting a fire, I would almost certainly get it fixed! Industry denial campaigns, such as those by tobacco and the fossil fuel companies, know full well that the public trusts the expert consensus, which is why their strategy includes telling the public there isn’t one, using bulk fake experts to counter a real consensus, or even suggesting consensus isn’t a part of the process of science.

One reason for the confusion around the importance of consensus in science is that the word has different meanings. For many of us, it’s a popular opinion or general agreement. But expert consensus isn’t the result of group-think, and it isn’t democratic. It’s the result of highly specialized experts independently evaluating the body of evidence and arriving at a similar conclusion. No other system of acquiring knowledge is as reliable or trustworthy, and it’s due largely to the community of experts checking each other’s work.

Finally, it’s not an appeal to authority to accept the consensus of experts. It’s the prudent thing to do! What is fallacious is appealing to those who aren’t experts, are experts in another area, or who represent a minority opinion to support a claim. If you value expert opinion, the consensus position should matter more than a cherry picked “expert.”

In short, the expert consensus is the most reliable form of knowledge for non-experts. Learn to trust them…they know more than you.

How to find the consensus

Sometimes a consensus is measured by gauging expert opinion while other times it’s by evaluating evidence. It can take significant time and research for experts to reach a consensus, and some issues have more agreement than others. That said, some topics that are controversial from the public’s standpoint (e.g. evolution, climate change, safety of vaccines) are about as settled as science can get.

Finding the consensus, if there is one, can be challenging, but it’s still orders of magnitude easier than doing all the research yourself.

–Research syntheses combine the results from studies to assess the weight of the evidence. Systematic reviews synthesize the literature and condense what’s known on a given question while meta-analyses are systematic reviews that use statistical methods to summarize the results.

Remember that individual studies aren’t the final word. If individual studies are like pieces of a puzzle, these papers help to put the puzzle together. Essentially, systematic reviews and meta-analyses help us avoid the trap of potentially being misled by a single study.

To find research syntheses, search for your keywords and the words “systematic review” or “meta-analysis.” Be sure to check the quality of the journal, and keep in mind that syntheses are only as good as the research that goes into them.

–Synthesis reports are syntheses of syntheses. (Yes, it’s very meta.) Synthesis reports aren’t always available, but if they are, they provide excellent evidence of a consensus.

The best examples of synthesis reports come from climate change research. For example, Cook et al (2016) synthesized the consensus estimates from six independent studies and found a robust scientific consensus that humans are causing the climate to change.

Seven independent studies into the consensus of human-caused global warming all show a strong consensus.
(Cook et al. 2016). Illustration: John Cook.

Another example from climate change research is the Intergovernmental Panel on Climate Change (IPCC). Every few years since 1988, the IPCC selects top scientists from around the world to evaluate and synthesize thousands of climate-related studies to provide the most reliable and comprehensive assessments about the causes, impacts, and future risks of a changing climate. (Their most recent assessment states, “It is unequivocal that human influence has warmed the atmosphere, ocean and land.”)

To find a synthesis report, search for your keywords and the phrase “synthesis report” or “research synthesis.”

–Position statements are official points of view on scientific issues by reputable scientific organizations and/or professional societies. These reports are generally based on reviews of published literature and are typically the most common expression of a scientific consensus.

To find position statements, search for your keywords along with a relevant scientific body. If you’re unfamiliar with the scientific organizations regarding a particular issue this may take a bit of searching. (However, it’s also a good indicator that searching for the expert consensus is a better choice than trying to read the literature.) Be wary of front or astroturf organizations that create unreliable societies to promote pseudoscience or science denial.

Examples of authoritative governmental scientific organizations, non-profit organizations, or professional societies include:

• Center for Disease Control and Prevention (CDC)
• National Institutes of Health (NIH)
• National Oceanic and Atmospheric Administration (NOAA)
• National Science Foundation (NSF)
• American Association for the Advancement of Science (AAAS)
• American Medical Association (AMA)
• National Academy of Sciences (NAS), especially the Based on Science section

 A very reliable indicator of consensus is if the majority of prestigious organizations have arrived at a similar conclusion.

–Clinical practice guidelines are recommendations produced by various healthcare organizations and medical professional bodies to help clinicians diagnose and treat patients using the best available evidence. These statements are based on systematic reviews and generally the consensus positions of experts in their relevant medical fields.

[Finding clinical practice guidelines.]

–Polls and surveys measure scientists’ opinions on a topic. While these provide useful information, especially if the results are in line with other measures of the consensus, they aren’t as reliable as literature reviews.

As always, it’s important to use credible polling organizations. In addition, pay attention to who was surveyed. Was it all scientists? Top scientists, such as those in the AAAS? Were they experts in their fields?

The higher the expertise in climate science the stronger the consensus on human-caused climate change. Illustration: John Cook.

A few more tips for your search

We’re more likely to fall for misinformation when it confirms what we already believe. If you’re looking for evidence that you’re right, you will absolutely be able to find it online! So instead of asking Google a leading (or loaded) question, use neutral search terms. Or better yet, break out of confirmation bias by trying to find evidence that you’re wrong.

It’s essential to check the reliability of sources. While peer-reviewed journals are the most reliable source of scientific information, not all are created equal. One way to check the quality of a journal is to search for its Journal Impact Factor (JIF), which calculates its influence in the scientific community based on the number of times its articles are cited.

Unfortunately, the more dangerous forms of deceit online involve setting up fake science journals or organizations to promote pseudoscience or denial, and it can be easy to fall for “studies” that appear to be scientific but are of significantly lower quality. In particular, be wary of predatory journals that publish false or misleading information and lack quality publication practices and pseudoscience journals that pretend to be scientific but are created for the sole purpose of publishing fake science.

If you’re using sources of information that aren’t peer-reviewed, it’s even more important to check its reliability. Don’t waste time on an unknown site’s “About” section, instead look laterally to see what other reliable sources have to say about that.

[Learn more: Don’t be fooled…fact check!]

The take-home message

While many of us like to think of ourselves as independent thinkers, most of our beliefs come down to the sources we choose to trust. Unfortunately some have decided that “thinking for themselves” means automatically not trusting information from the government, the media, or even academia, leaving them vulnerable to misinformation that reinforces their existing beliefs or biases.

We trust experts all the time, to grow our food, to fly our planes, to wire our homes, etc. It’s only when expert opinion conflicts with our core beliefs that we resort to denial…and to “doing our own research” to find the “real truth.” Instead of cherry picking studies or experts to try and support your cherished beliefs, be honest with yourself about what you believe and why…and why you trust the sources you do.

But you can’t out-research experts. You have to be an expert to do that. If your “research” leads you to sources that tell you the experts are all mistaken or lying, you’re doing it wrong. You haven’t found trustworthy sources the experts have all missed. You’ve been misled by propaganda that appealed to your emotions or biases.

The bottom line is: If you want to believe in things that are “true,” to the best of human capabilities to know “the truth,” learn to trust experts, because they have  the knowledge and skills needed to evaluate evidence.

However, if you’re really convinced nearly all of the world’s experts are wrong, become an expert and do your own (real) research.

To learn more

• Forbes: What does ‘scientific consensus’ mean?
• Credible Hulk: Scientific Consensus isn’t ‘part’ of the scientific method: It’s a consequence of it
• Medium: Scientific consensus: What it is and why this is a good time to start caring
• Time: Science isn’t always perfect – but we should still trust it
• Kyle Hill: How to read a scientific paper

Special thanks to Lynnie Bruce, Jonathan Stea, and John Cook for their feedback