10 Types of Study Bias

Back in 1903, a few years after the discovery of X-rays by German researchers, a French scientist named René Blondlot announced that he'd discovered yet another previously unknown form of radiation — N-rays. They could only be observed using peripheral vision, and seen as a corona when electricity was discharged from crystals. Eventually, Blondlot's research was refuted by an American scientist, Robert Wood, who visited the Frenchman's lab and found that Blondot still observed N-rays, even after Wood secretly removed the crystal during one of the experiments.

But after that, something strange happened. For years, other French scientists continued to publish papers describing their observations of N-rays, as if they actually existed. Perhaps out of nationalistic pride, French scientists wanted to see N-rays, and so they did [sources: Lee, Simon].

Those N-ray findings were an extreme example of one of the simplest most widely recognized reasons that studies can go wrong — confirmation bias. That's when a researcher takes the hypothesis that he or she starts out with ("marijuana is beneficial/detrimental") and shapes the study methodology or the analysis of the data in a way that confirms the original premise, whether or not it's actually justified [source: Sarniak]. Lay people are prey to confirmation bias as well. If they support (or despise) a sitting president of the U.S., for instance, they tend to look for information that confirms their view and disregard anything that refutes it.

Researchers who've done meta-analyses of scientific research have found that early, small-scale studies — ones that end up being frequently cited in other work — often overstate their results [source: Fanelli, et al.].

That can happen because of sampling bias, in which researchers conducting small studies base their findings upon a group that isn't necessarily representative of the larger population. Universities often use students for their studies but the findings for this group don't necessarily project to the wider population.

It's a problem that's seen in both medical studies and social science research. For example, if a political science researcher who's studying attitudes about gun control does surveys in an area where most people are Second Amendment supporters, that will skew the results in a way that doesn't necessarily reflect the views of the larger U.S. population.

But sampling bias can occur in bigger studies as well. One famous example of sampling bias occurred during the 1936 U.S. presidential campaign, when Literary Digest conducted a mail survey of 2.4 million people and predicted — incorrectly — that Republican Alf Landon would handily beat incumbent Democrat Franklin Roosevelt. The problem was that the magazine used phone directories, drivers' registrations and country club memberships to find people to poll — a method that tended to reach relatively affluent voters (cars and phones were luxury items back then), rather than the poorer ones among whom Roosevelt was popular. The erroneous results hastened the end of the publication [source: Oxford Math Center].

How do scientists determine whether a new drug will cure or help a particular disease? Usually with a study involving two groups of people. For instance, if the scientists are studying the effectiveness of a new antihistamine on allergy sufferers, they would give the trial medication to one group of patients and a placebo (sugar pill) to the other group, called the control group. Neither group is supposed to know whether they have been given the medication and the study participants are randomly assigned to each group.

This is referred to as a randomized double-blind placebo control study and is considered the gold standard of clinical trials. "Double-blind" refers to the fact that neither the scientists nor the participants know which allergy patients are in which group until after the experiment is over.

There are several reasons for doing this but one is to avoid selection bias. Let's say you want to study whether people who work at night are more likely to develop headaches. So, you recruit a group of people who work at night, and another group who work during the day, and then compare them. Your results show that the people who work at night are more likely to have aching temples.

But that doesn't necessarily mean that night work is the cause, because it could be that people who work at night tend to be poorer, have more unhealthy diets or more stress. Such factors might bias your results, unless you could make sure that the two groups are similar in every other way except for their schedules [sources: Institute for Work and Health, CIRT].

Channeling bias occurs when a patient's prognosis or degree of illness influences which group he or she is put into in a study. It's a particular problem in nonrandomized medical trials, ones in which doctors select which patients are going to receive the drug or surgical procedure that's going to be evaluated.

It's not hard to figure out why it happens, because physicians, after all, generally want to help the people that they treat, and are trained to weigh the risks versus the rewards for a treatment.

Let's look at a hypothetical example of a study intended to evaluate the effectiveness of a certain surgical procedure on the hand. Surgeons might be more inclined to pick younger, healthier patients to get the operation, because they have lower risks of complications afterward, and more of a need to have full hand function.

In turn, they might be less likely to perform it on older patients who face higher post-operative risks and don't need to have the same degree of hand function because they're no longer working. If researchers aren't careful, the group that gets the surgery in the study will consist of younger patients, and the group that doesn't will be mostly older ones. That could produce a very different result than if the two groups were otherwise identical [source: Pannucci and Wilkins].

The order that questions are asked in a survey or study can influence the answers that are given. That's because the human brain has a tendency to organize information into patterns. The earlier questions — in particular, the ones that come just before a particular query — may provide information that subjects use as context in formulating their subsequent answers, or affect their thoughts, feelings and attitudes. That effect is called priming [sources: Pew, Sarniak].

Pew Research gave this example from a December 2008 poll: "When people were asked 'All in all, are you satisfied or dissatisfied with the way things are going in this country today?' immediately after having been asked 'Do you approve or disapprove of the way George W. Bush is handling his job as president?'; 88 percent said they were dissatisfied, compared with only 78 percent without the context of the prior question."

Another example of the question-order bias effect comes from the General Social Survey, a major long-term study of American attitudes. In 1984, GSS participants were asked to identify the three most important qualities for a child to have, and given a card with a list of qualities. When "honest" was high on the list, it was picked by 66 percent of respondents. But when it came near the end, only 48 percent of people picked it as one of their top three. A similar pattern was seen with other qualities [source: Henning].

Not only do researchers need to be careful about whom they pick to be in groups in studies, but they also have to worry about how they solicit, record and interpret the data that they get from these subjects. Interviewer bias, as this problem is called, is more of an issue in medical studies when the interviewer knows the research subject's health status before questioning him or her.

A 2010 medical journal article on how to identify and avoid bias cites the hypothetical example of a study that's attempting to identify the risk factors for Buerger's disease, a rare disorder in which arteries and veins in the arms and legs become swollen and inflamed. If the interviewer already knows that a research subject has the disease, he or she is likely to probe more intensely for known risk factors, like smoking. So, the interviewer may ask people in the risk group, "Are you sure you've never smoked? Never? Not even once?"— while not subjecting patients in the control group to these kinds of questions [source: Pannucci and Wilkins].

An interviewer also can cause errant results in a study by giving subjects non-verbal cues when asking questions, such as with gestures or facial expressions, or tone of voice [source: Delgado, et al.].

In studies where people are questioned about something that occurred in the past, their recollections may be affected by current realities. Recall bias, as this phenomenon is known, can be a major problem when researchers are investigating what factors could have led to a health condition, and interviews are the prime source of information. For example, since there's a widespread — though unsubstantiated — belief that autism is somehow caused by the measles-mumps-rubella (MMR) vaccine, parents of children on the autism spectrum are more likely to recall that their child was immunized prior to showing signs of autism, and draw a connection between the two events [source: Pannucci and Wilkins].

Similarly, mothers of children with birth defects may be more likely to remember drugs that they took during pregnancy than mothers of fully abled children. One study also found that pilots who knew they had been exposed to the herbicide Agent Orange had a greater tendency to remember skin rashes that they experienced in the year after exposure [source: Boston College].

This is another bias that can occur with social science surveys. People want to be agreeable so they are more likely to answer in the affirmative to a "yes/no" or "agree/disagree" question — particularly if they are less-educated or have less information. One way to get around this bias is to ask participants to choose between two statements (the forced choice format) rather than have them agree or disagree to one statement. The two statements would give two different views of a subject.

And in addition to being agreeable, survey respondents also want to be seen as likeable. "Research has shown that respondents understate alcohol and drug use, tax evasion and racial bias; they also may overstate church attendance, charitable contributions and the likelihood that they will vote in an election," notes Pew Research. Therefore, the questions have to be framed in a way that gives participants an "out" for admitting to less-than-desirable behavior. So, a question on voting could be phrased as: "In the 2012 presidential election between Barack Obama and Mitt Romney, did things come up that kept you from voting, or did you happen to vote?"

One common type of bias stems from an uncomfortable reality in the scientific culture. Researchers have a continual need to publish articles in journals, in order to sustain their reputations and rise in academia. That publish-or-perish mentality might exert an influence upon the outcomes of hypotheses, because as one critic notes, academia tends to bias toward statistically significant, "positive" results [source: van Hilten].

Indeed, meta-analyses show that journals are much more likely to publish studies that report a statistically significant positive result than ones that don't. Publication bias is stronger in some fields than others; one 2010 study found that papers in the social sciences are 2.3 times more likely to show positive results than papers in the physical sciences [source: Fanelli].

As Ian Roberts, a professor of epidemiology and public health at the London School of Hygiene and Tropical Medicine, noted in a 2015 essay, clinical trials showing that a treatment works are much more likely to be published than those showing that it doesn't have any benefit or even harmful.

In some ways, this is the flip side of publication bias. Negative results from a study get shoved in a metaphorical file drawer instead of being published. Critics see it as a particular problem when it comes to studies of new medications, which these days often are sponsored by the companies that developed them [source: Pannucci and Wilkins].

File-drawer bias can be significant. A study published in the New England Journal of Medicine in 2008 compared the results of published studies on antidepressants to data from a U.S. Food and Drug Administration registry of research that included unpublished information. It found that 94 percent of the published studies reported drugs having positive effects. But when the unpublished studies were included, the number with positive results dropped to 51 percent [source: Turner, et al.].

In an effort to get more information into the public domain, Congress in 2007 passed a law requiring researchers to report results of many human studies of experimental treatments to ClinicalTrials.gov. In 2016, the U.S. Food and Drug Administration strengthened the rules, requiring more thorough reporting of clinical trials, including drugs and devices that were studied but never brought to market [source: Piller].

But some critics worry that the laws won't have much teeth since there is no increase in enforcement staffing.