How Scientific Peer Review Works

Peer review, also known as refereeing, is the cornerstone of science. It is a process whereby a scientist's research is assessed for quality before it is funded or published. The "peer" in peer review means that the scientist in question will submit their work to other experts in the field. In other words, if the scientist is a biologist studying the migration habits of a particular bird, they will submit that research to colleagues who have sufficient knowledge of birds and bird migration to give a thorough and proper evaluation. It's the job of the reviewers to comment on the quality, significance and originality of the research. Reviewers aren't the ultimate arbiters about whether research should be funded or published, but their comments inform the decision makers.

Who the decision makers are depends on the type of peer review taking place. One type happens at the beginning of a scientist's research project, when the person is submitting a research proposal to be considered for a grant. In this case, the decision maker is the funding body that will award the grant based, in large part, on the review given by the referees. The other type of peer review happens at the end of a scientist's research project, when they prepares to have the results of their investigation published in a scientific journal. All scholarly journals, just like consumer magazines, have editors who reign as the ultimate decision makers, but they rely on comments from the reviewers to decide which papers they will publish.

The rest of this article will focus on the peer review process used by scholarly journals, which are quite different than mainstream publications, such as Popular Science, Discover and Scientific American. They feature articles written by research scientists for other scientists to read. By publishing in a journal, a scientist helps to spread scientific knowledge and stimulate further research and discovery. Specifically, a scientist can:

The last bullet is a key aspect of what scholars refer to as "publish or perish." It's a particularly descriptive way to characterize the pressure scholars feel to publish work in leading academic journals. Frequent publication improves a scientist's visibility, which in turn raises the reputation of the sponsoring institution, which in turn attracts more funding dollars. If a scientist fails to publish regularly, they risk losing this financial lifeline. The scientist might be asked to stop their research and, in the worst-case scenario, lose their position. That's the "perish" part — and why academic journals are so important to researchers all over the world.

Not all scientific journals use a peer-review system, but the most prestigious do. You've no doubt heard of the New England Journal of Medicine, the Journal of the American Medical Association (JAMA), Science and Nature. All of these are peer-reviewed journals representing the pinnacle of scientific publication. Their reputations owe much to the peer review processes they employ to ensure the quality of their content. Although these premier journals cast a long shadow, there are many other well-respected scientific, technical and medical publications. There are also many that fall far below the high standards set by a Nature or Science. In total, there are approximately 21,000 peer-reviewed journals publishing more than 1 million research papers a year [source: Sense About Science].

Next, we'll look at the typical peer review process used by these journals.

The basic steps in the peer-review process have been around for a while. In fact, a medical journal published in the 1700s alerted contributors that all submissions would be "distributed according to the subject matter to those members who are most versed in these matters" [source: Ware]. This time-honored tradition continues today, although it's not as simple as it sounds. Getting research published in a peer-reviewed journal can be time-consuming and difficult.

It all starts with a scientist and their research. When the research is completed, the scientist writes a paper describing the experimental procedure and the results. The scientist then submits it to a journal that publishes papers in their field. For example, if the scientist's studying an aspect of breast cancer formation, they might submit their paper to CA: A Cancer Journal for Clinicians, a widely circulated oncology journal. Starting with a prestigious journal in a topic area is common practice. If a paper isn't accepted there, the scientist moves on to their second choice, third choice and so on.

The path to acceptance begins with the journal editors. They first review the submission to make sure it fits both the journal's subject-matter focus and its editorial platform. For example, some journals prefer to publish only groundbreaking research and may overlook even good papers that don't, in the opinion of the editors, drive the field forward. Only a small percentage of papers survive this initial evaluation. Those that do enter the formal peer review system.

Generally, the process of peer review involves an exchange between a journal editor and a team of reviewers, also known as referees. After the referees receive a paper from the editor, they read it closely and provide individual critiques, usually within two to four weeks. In their critiques, they:

These activities are common to all types of peer review. What varies is whose identities are known and whose are concealed. In the most traditional approach to peer review, known as single-blind review, reviewers know the author's identity, but not vice versa. Blinding the identity of reviewers enables them to comment freely and not worry about disgruntled authors seeking retribution for negative reviews. Another approach is double-blind review, in which the identities of the author and referees are both hidden, making it easier for reviewers to focus on the paper itself without being swayed by any preconceived ideas about the author or their institution. Finally, many journals have adopted open peer review. In this model, the author's and reviewers' identities are known to each other, a situation that forces reviewers, who can't hide behind a veil of anonymity, to provide more thoughtful critiques.

Regardless of the approach, peer review has several benefits. Let's look at those next.

Think of peer review as a quality-control system. When a team of reviewers gives a green light to a particular paper, they are saying the science described in the paper is valid and trustworthy. This is similar to what quality-control inspectors do at a manufacturing plant. They check products by sight, sound, feel, smell or even taste to locate imperfections that might cause harm or dissatisfaction in the end-user audience. Inspectors adhere to strict quality standards, discarding any product that doesn't meet the standard. Peer review does the same thing by setting a scientific standard.

Validating scientific results benefits everyone, from the scientists doing the work to consumers like you who eventually hear about the research on the evening news. Let's look briefly at the value peer review brings to various stakeholders:

Still, there are many scientists who question the value of peer review. According to this group, the negative aspects of peer review far outweigh its benefits. Next, we'll present some of the arguments against peer review.

Peer review has earned its enemies over the years. You might think these are scientists to whom peer review has not been kind, but that's not the case. A growing number of researchers are shining the bright light of science — controlled experimentation and careful observation — into the dark corners of a process that has been around for hundreds of years. What they're finding may surprise you.

Consider a study conducted by BMJ (British Medical Journal), one of the most respected peer-reviewed journals in medicine. BMJ Editor Fiona Godlee and two colleagues took a paper about to be published in their journal and introduced eight deliberate errors. Then they sent the paper to 420 reviewers. The median number of errors detected by the 221 respondents was two. Nobody found more than five, and 16 percent didn't find any errors at all. This seems to suggest that peer review doesn't really increase the quality of published research, or does so to only a small degree. Another BMJ study showed that a single, seasoned editor could judge the quality and significance of research just as effectively as a team of external referees.

Then there's the issue of detecting fraud. As the Hwang stem cell case clearly demonstrates, peer review isn't a fraud-detection system. Referees are much more likely to find and flag plagiarism than falsified data. That's because reviewers don't generally have access to the actual data on which a paper is based. If a scientist knowingly and deliberately sets out to falsify data, a team of reviewers may not be able to detect it. However, such data will not be able to stand up to the intense scrutiny of the larger scientific community. In fact, the ability of scientists to duplicate the results of published research is another hallmark of science and another quality-control mechanism that extends beyond peer review.

Quality is not the only issue. Some critics argue that peer review slows down advances in scientific and medical knowledge. It can take a year for an article to move through the peer-review system and become published. Some journals have introduced a fast-track approach to streamline the submission process, but it's usually reserved for very high-quality work. Papers below this standard may languish for months. That's a long time to wait, especially if the research promises to provide valuable information about a disease or other issue that affects public health and safety. And yet moving methodically and with great care is not a bad thing when you're dealing with the safety of human beings.

Finally, a few critics have suggested that peer review leads to the suppression of some scientists' results. There are two ways this suppression might play out. First, a reviewer — an established scientist in their field — might reject research that challenges their particular point of view or contradicts their own findings. Such a reviewer might be accused of maintaining the "scientific establishment" at the cost of innovative ideas. The other form of suppression involves the work of Third World researchers. Studies have shown that a vast majority of mainstream journal articles come from scientists in developed countries, with the majority coming from U.S.-based scientists. Very few Third World researchers see their work published in mainstream journals, such as Nature and Science. Even when a developing nation succeeds in publishing in a peer-reviewed journal, the journal might not be listed in the Science Citation Index, a commercial database of scholarly publications widely used by researchers.

Because of limitations such as these, many scientists are campaigning to change the peer-review process. The next section highlights some of those changes and how peer review is evolving.

Like many other systems and processes, peer review has been forced to adapt to changes brought on by the computer and various online technologies. The last 25 years of peer review can be roughly organized into three eras based on the dominant technology: the PC era, the Internet era and the Web 2.0 era. Let's look at each of these to understand how peer review has evolved and continues to evolve.

The PC era is marked by the introduction of peer-review software to streamline the process. In the late 1980s and early 1990s, this software resided locally, on the desktop computers of editors. Two popular applications were Peer Review Plus and the Editorial Management System, or EMS. At the heart of these systems were relational databases to manage information about reviewers, editorial advisory board members and manuscript information. These databases replaced cumbersome and time-consuming manual filing systems. Early software also incorporated word processing capabilities, making it easier to make and track changes. Most journals using these systems reported that they increased efficiency by facilitating the selection of appropriate reviewers and accelerating the processing of manuscripts.

In the late 1990s, desktop systems gave way to Web-based systems and ushered in the internet era. Web-based peer review systems offered several advantages. First, all stakeholders in the process — editor, reviewer and scientist — were linked electronically, eliminating the need for costly faxes and overnight shipments. They also enabled editorial office personnel to manage all aspects of the review process, including data entry, data retrieval, correspondence, reporting, workflow control and manuscript file management. Even better, editors and their assistants could access all of this from any location.

The internet also encouraged another type of peer review known as post-publication review. Post-publication review is a variation of open review in which all readers, not just referees selected by the journal editor, are able to review and comment on a paper. In some cases, readers can even rate the paper on a numerical scale following publication. This is similar to the customer review feature on Amazon, which enables readers to provide a score and post comments on a book they have recently finished. Of course, rating the latest Stephen King novel doesn't carry the same implications as rating a paper on the safety and efficacy of a drug, which is why many scientists don't like post-publication review. They say it encourages gut reactions in favor of more thoughtful, well-considered reviews. Still, many see it as a useful supplement to formal peer review.

Some scientists are also finding favor with the applications and tools of Web 2.0, the term used to describe the next generation of Web-based tools that enhance creativity, communication and collaboration. More researchers are turning to blogs and social networking sites to share their ideas and connect with other scientists. In some cases, they are actually posting raw experimental results and draft papers for others to see and comment on. Advocates of these open-access practices argue that real-time collaboration encourages scientific progress. Opponents worry about the integrity of the scientific process. How, they wonder, can a researcher know whether a comment is coming from a trusted source? And how does a researcher know their ideas won't be stolen or exploited?

There will certainly be more questions as the Web 2.0 era of peer review gives way to whatever might follow in the coming years. But one thing is certain: Peer review, in one form or another, will remain a cornerstone of the scientific process — not because it's the best system, but because it's the best system we have.