South Korean scientists Woo Suk Hwang (L) and Shin Yong Moon announcing that they cloned a human embryo in February 2004. Dr. Hwang's work, later found to be fraudulent, shook the public's faith in science.

Ron Wurzer/­Getty Images

Introduction to How Scientific Peer Review Works

­Most people didn't know much about scientific peer review five years ago. Then, in December 2005, South Korean scientist Dr. Hwang Woo Suk stunned the world by admitting that his stem cell research -- research that was supposed to revolutionize health care by helping to cure diseases ranging from diabetes to Parkinson's -- used fabricated data. Although the revelation brought disgrace to Hwang and poured fuel on the stem cell controversy, it had a more damaging effect on the public's perception of science itself. Suddenly, there were reports questioning how Science, the prestigious U.S. journal that published Hwang's findings, could have been so easily duped. Other reports condemned the process of science itself as antiquated and flawed.

­At the heart of that process is scientific peer review, a quality-control system that requires all new scientific discoveries, ideas and implications to be scrutinized and critiqued by expert scientists before they become widely accepted. Peer review has been around for nearly 300 years, so it is not new. It just seems that way sometimes because of the attention it has received in the wake of the stem cell scandal. Unfortunately, increased awareness does not always translate into increased understanding. Many myths and misconceptions about peer review still exist, and many average citizens don't see how a system of checks and balances is important either to science or to their day-to-day decision making.

To make matters worse, the conventions of peer review, just like the conventions of many traditional systems and processes, are being challenged by Web 2.0 technologies. A new generation of scientists is turning to the Web as a vehicle to post raw experimental results, fledgling theories and draft papers. Proponents of "open access" practices argue that science is improved in a more democratic and collaborative atmosphere. Critics warn that such promiscuity with research data undermines the very integrity of the scientific endeavor.

These are some of the issues we will explore in this article. We'll cover the basics -- what is scientific peer review, how does it work and what is its historical context -- before moving on to an analysis of what it can and can't do. Finally, we'll examine some of the current trends in peer review to understand how the system is evolving and may continue to evolve.

But first, let's expand a bit on the basic definition of peer review.

Philip Campbell, (R) the editor-in-chief of Nature greets Colin Norman, the news editor at Science in 2007. If Campbell gives your work the nod, you're golden.

Miguel Riopa/AFP/­Getty Images

Peer Review: The Basics

­Peer review, also known as refereeing, is the cornerstone of science. It is a process whereby a scientist's research is assessed for quality before it is funded or published. The "peer" in peer review means that the scientist in question will submit his work to other experts in the field. In other words, if the scientist is a biologist studying the migration habits of a particular bird, he will submit that research to colleagues who have sufficient knowledge of birds and bird migration to give a thorough and proper evaluation. It's the job of the reviewers to comment on the quality, significance and originality of the research. Reviewers aren't the ultimate arbiters about whether research should be funded or published, but their comments inform the decision makers.

Who the decision makers are depends on the type of peer review taking place. One type happens at the beginning of a scientist's research project, when he is submitting a research proposal to be considered for a grant. In this case, the decision maker is the funding body that will award the grant based, in large part, on the review given by the referees. The other type of peer review happens at the end of a scientist's research project, when he prepares to have the results of his investigation published in a scientific journal. All scholarly journals, just like consumer magazines, have editors who reign as the ultimate decision makers, but they rely on comments from the reviewers to decide which papers they will publish.

The rest of this article will focus on the peer review process used by scholarly journals, which are quite different than mainstream publications, such as Popular Science, Discover and Scientific American. They feature articles written by research scientists for other scientists to read. By publishing in a journal, a scientist helps to spread scientific knowledge and stimulate further research and discovery. Specifically, a scientist can:

  • Announce formally the results of his work
  • Associate his name with an important discovery, thereby creating a permanent record
  • Promote his research interests and attract additional funders

The last bullet is a key aspect of what scholars refer to as "publish or perish." It's a particularly descriptive way to characterize the pressure scholars feel to publish work in leading academic journals. Frequent publication improves a scientist's visibility, which in turn raises the reputation of the sponsoring institution, which in turn attracts more funding dollars. If a scientist fails to publish regularly, he risks losing this financial lifeline. He might be asked to stop his research and, in the worst-case scenario, lose his position. That's the "perish" part -- and why academic journals are so important to researchers all over the world.

Not all scientific journals use a peer-review system, but the most prestigious do. You've no doubt heard of the New England Journal of Medicine, the Journal of the American Medical Association (JAMA), Science and Nature. All of these are peer-reviewed journals representing the pinnacle of scientific publication. Their reputations owe much to the peer review processes they employ to ensure the quality of their content. Although these premier journals cast a long shadow, there are many other well-respected scientific, technical and medical publications. There are also many that fall far below the high standards set by a Nature or Science. In total, there are approximately 21,000 peer-reviewed journals publishing more than 1 million research papers a year [source: Sense About Science].

Next, we'll look at the typical peer review process used by these journals.

After the research is finished, it's time to start analyzing the data and seeing if your experiment turned up anything worthy of publishing.

John A. Rizzo/­Getty Images

Steps in the Peer Review Process

­The basic steps in the peer-review process have been around for a while. In fact, a medical journal published in the 1700s alerted contributors that all submissions would be "distributed according to the subject matter to those members who are most versed in these matters" [source: Ware]. This time-honored tradition continues today, although it's not as simple as it sounds. Getting research published in a peer-reviewed journal can be time-consuming and difficult.

It all starts with a scientist and his research. When the research is completed, the scientist writes a paper describing the experimental procedure and the results. He then submits it to a journal that publishes papers in his field. For example, if he's studying an aspect of breast cancer formation, he might submit his paper to CA: A Cancer Journal for Clinicians, a widely circulated oncology journal. Starting with a prestigious journal in a topic area is common practice. If a paper isn't accepted there, the scientist moves on to his second choice, third choice and so on.

­The path to acceptance begins with the journal editors. They first review the submission to make sure it fits both the journal's subject-matter focus and its editorial platform. For example, some journals prefer to publish only groundbreaking research and may overlook even good papers that don't, in the opinion of the editors, drive the field forward. Only a small percentage of papers survive this initial evaluation. Those that do enter the formal peer review system.

Generally, the process of peer review involves an exchange between a journal editor and a team of reviewers, also known as referees. After the referees receive a paper from the editor, they read it closely and provide individual critiques, usually within two to four weeks. In their critiques, they:

  • Comment on the validity of the science, identifying scientific errors and evaluating the design and methodology used
  • Judge the significance by evaluating the importance of the findings
  • Determine the originality of the work based on how much it advances the field. Reviewers also identify missing or inaccurate references.
  • Recommend that the paper be published or rejected. Editors don't have to heed this recommendation, but most do.

These activities are common to all types of peer review. What varies is whose identities are known and whose are concealed. In the most traditional approach to peer review, known as single-blind review, reviewers know the author's identity, but not vice versa. Blinding the identity of reviewers enables them to comment freely and not worry about disgruntled authors seeking retribution for negative reviews. Another approach is double-blind review, in which the identities of the author and referees are both hidden, making it easier for reviewers to focus on the paper itself without being swayed by any preconceived ideas about the author or his institution. Finally, many journals have adopted open peer review. In this model, the author's and reviewers' identities are known to each other, a situation that forces reviewers, who can't hide behind a veil of anonymity, to provide more thoughtful critiques.

Regardless of the approach, peer review has several benefits. Let's look at those next.

Taming a Claim

How can you tell if a scientific claim is based on validated research? It's not easy, but one thing you can do is look for a full reference to the peer-reviewed paper. Such references follow a very specific style and always give the name of the journal. A typical reference is shown below:

Aisen PS, Schafer KA, Grundman M, Pfeiffer E, Sano M, Davis KL, Farlow MR, Jin S, Thomas RG, Thal LJ, for the Alzheimer's Disease Cooperative Study. Effects of rofecoxib or naproxen vs. placebo on Alzheimer disease progression. JAMA 2003; 289: 2819-2826.

The Value of Peer Review

­­­

­Think of peer review as a quality-control system. When a team of reviewers gives a green light to a particular paper, they are saying the science described in the paper is valid and trustworthy. This is similar to what quality-control inspectors do at a manufacturing plant. They check products by sight, sound, feel, smell or even taste to locate imperfections that might cause harm or dissatisfaction in the end-user audience. Inspectors adhere to strict quality standards, discarding any product that doesn't meet the standard. Peer review does the same thing by setting a scientific standard.

­Validating scientific results benefits everyone, from the scientists doing the work to consumers like you who eventually hear about the research on the evening news. Let's look briefly at the value peer review brings to various stakeholders:

  • For authors, peer review provides a patina of respectability on their work. A scientist who publishes in his field's most prestigious journal gets to bask in the glow of the publication's reputation. He may get called for more interviews and may have future research viewed more favorably by funding bodies.
  • For journal editors, peer review informs their decision-making process. An editor can publish a paper with much greater confidence if he knows that paper has been thoroughly vetted by a team of qualified referees. The editor's management of the peer-review process is directly related to the reputation of the journal. If he consistently selects papers of the highest quality, he will enhance the reputation of his journal. If, on the other hand, he allows the occasional substandard paper to be published, he can erode the journal's credibility.
  • For other scientists, peer review acts as a mechanism to help prioritize what they read. Considering there are 21,000 scholarly peer-reviewed journals available, this is a significant benefit for the average overworked scientist [source: Sense About Science]. By focusing only on the top four or five journals in his field, a scientist can assume he's reading the most important papers of the highest quality. It's sort of like using the New York Times bestseller list to determine which novel you're going to read next.
  • For nonscientists, peer review acts like a quality standard that helps make sense of scientific claims. Those claims -- about everything from health care remedies to vacuum cleaners -- fill news stories, TV ads and Web sites. Ethical and conscientious writers and producers will indicate whether research cited in an article or ad has been published and provide the name of the journal. By making sure scientific claims are based on research published in a respected, peer-reviewed journal, consumers can feel a measure of protection against hucksters trying to use "science" to sell their products.

­Still, there are many scientists who question the value of peer review. According to this group, the negative aspects of peer review far outweigh its benefits. Next, we'll present some of the arguments against peer review.

Will she find any errors in papers that she reviews?

DAJ/­Getty Images

Limitations of Peer Review

­Peer review has earned its enemies over the years. You might think these are scientists to whom peer review has not been kind, but that's not the case. A growing number of researchers are shining the bright light of science -- controlled experimentation and careful observation -- into the dark corners of a process that has been around for hundreds of years. What they're finding may surprise you.

Consider a study conducted by BMJ (British Medical Journal), one of the most respected peer-reviewed journals in medicine. BMJ Editor Fiona Godlee and two colleagues took a paper about to be published in their journal and introduced eight deliberate errors. Then they sent the paper to 420 reviewers. The median number of errors detected by the 221 respondents was two. Nobody found more than five, and 16 percent didn't find any errors at all. This seems to suggest that peer review doesn't really increase the quality of published research, or does so to only a small degree. Another BMJ study showed that a single, seasoned editor could judge the quality and significance of research just as effectively as a team of external referees.

Then there's the issue of detecting fraud. As the Hwang stem cell case clearly demonstrates, peer review isn't a fraud-detection system. Referees are much more likely to find and flag plagiarism than falsified data. That's because reviewers don't generally have access to the actual data on which a paper is based. If a scientist knowingly and deliberately sets out to falsify data, a team of reviewers may not be able to detect it. However, such data will not be able to stand up to the intense scrutiny of the larger scientific community. In fact, the ability of scientists to duplicate the results of published research is another hallmark of science and another quality-control mechanism that extends beyond peer review.

Quality is not the only issue. Some critics argue that peer review slows down advances in scientific and medical knowledge. It can take a year for an article to move through the peer-review system and become published. Some journals have introduced a fast-track approach to streamline the submission process, but it's usually reserved for very high-quality work. Papers below this standard may languish for months. That's a long time to wait, especially if the research promises to provide valuable information about a disease or other issue that affects public health and safety. And yet moving methodically and with great care is not a bad thing when you're dealing with the safety of human beings.

Finally, a few critics have suggested that peer review leads to the suppression of some scientists' results. There are two ways this suppression might play out. First, a reviewer -- an established scientist in his field -- might reject research that challenges his particular point of view or contradicts his own findings. Such a reviewer might be accused of maintaining the "scientific establishment" at the cost of innovative ideas. The other form of suppression involves the work of Third World researchers. Studies have shown that a vast majority of mainstream journal articles come from scientists in developed countries, with the majority coming from U.S.-based scientists. Very few Third World researchers see their work published in mainstream journals, such as Nature and Science. Even when a developing nation succeeds in publishing in a peer-reviewed journal, the journal might not be listed in the Science Citation Index, a commercial database of scholarly publications widely used by researchers.

­Because of limitations such as these, many scientists are campaigning to change the peer-review process. The next section highlights some of those changes and how peer review is evolving.

Will the future of sc­ientific peer review look anything like this?

Image courtesy Amazon

The Changing Face of Peer Review

­Like many other systems and ­pro­cesses, peer review has been forced to adapt to changes brought on by the computer and various online technologies. The last 25 years of peer review can be roughly organized into three eras based on the dominant technology: the PC era, the Internet era and the Web 2.0 era. Let's look at each of these to understand how peer review has evolved and continues to evolve.

The PC era is marked by the introduction of peer-review software to streamline the process. In the late 1980s and early 1990s, this software resided locally, on the desktop computers of editors. Two popular applications were Peer Review Plus and the Editorial Management System, or EMS. At the heart of these systems were relational databases to manage information about reviewers, editorial advisory board members and manuscript information. These databases replaced cumbersome and time-consuming manual filing systems. Early software also incorporated word processing capabilities, making it easier to make and track changes. Most journals using these systems reported that they increased efficiency by facilitating the selection of appropriate reviewers and accelerating the processing of manuscripts.

In the late 1990s, desktop systems gave way to Web-based systems and ushered in the Internet era. Web-based peer review systems offered several advantages. First, all stakeholders in the process -- editor, reviewer and scientist -- were linked electronically, eliminating the need for costly faxes and overnight shipments. They also enabled editorial office personnel to manage all aspects of the review process, including data entry, data retrieval, correspondence, reporting, workflow control and manuscript file management. Even better, editors and their assistants could access all of this from any location.

The Internet also encouraged another type of peer review known as post-publication review. Post-publication review is a variation of open review in which all readers, not just referees selected by the journal editor, are able to review and comment on a paper. In some cases, readers can even rate the paper on a numerical scale following publication. This is similar to the customer review feature on Amazon, which enables readers to provide a score and post comments on a book they have recently finished. Of course, rating the latest Stephen King novel doesn't carry the same implications as rating a paper on the safety and efficacy of a drug, which is why many scientists don't like post-publication review. They say it encourages gut reactions in favor of more thoughtful, well-considered reviews. Still, many see it as a useful supplement to formal peer review.

Some scientists are also finding favor with the applications and tools of Web 2.0, the term used to describe the next generation of Web-based tools that enhance creativity, communication and collaboration. More researchers are turning to blogs and social networking sites to share their ideas and connect with other scientists. In some cases, they are actually posting raw experimental results and draft papers for others to see and comment on. Advocates of these open-access practices argue that real-time collaboration encourages scientific progress. Opponents worry about the integrity of the scientific process. How, they wonder, can a researcher know whether a comment is coming from a trusted source? And how does a researcher know his ideas won't be stolen or exploited?

­There will certainly be more questions as the Web 2.0 era of peer review gives way to whatever might follow in the coming years. But one thing is certain: Peer review, in one form or another, will remain a cornerstone of the scientific process -- not because it's the best system, but because it's the best system we have.

­­

Lots More Information

Sources

  • Brown, Tracey. "Peer Review and the Acceptance of New Scientific Ideas." Sense About Science. May 2004. (Dec. 2, 2008) http://www.senseaboutscience.org.uk/index.php/site/project/33
  • Dauben, Joseph W. "Science." World Book Multimedia Encyclopedia. 2004.
  • Gibbs, W. Wayt. "Lost Science in the Third World." Scientific American. August 1995.
  • National Climatic Data Center. A Paleo Perspective on Global Warming. "Peer Review: Why does it matter?" April 17, 2006. (Dec. 2, 2008) http://www.ncdc.noaa.gov/paleo/globalwarming/peerreview.html
  • "Nature's peer review trial." Nature. December 2006. (Dec. 2, 2008) http://www.nature.com/nature/peerreview/debate/nature05535.html
  • Sense About Science. "I Don't Know What to Believe: Making Sense of Science Stories." 2005. (Dec. 2, 2008) http://www.senseaboutscience.org.uk/index.php/site/project/29/
  • Smith, Richard. "Peer review: reform or revolution?" BMJ. Sept. 27, 1997. (Dec. 2, 2008) http://www.bmj.com/cgi/content/full/315/7111/759
  • Wager, Elizabeth. "Ethics: What is it for?" Nature. 2006. (Dec. 2, 2008) http://www.nature.com/nature/peerreview/debate/nature04990.html
  • Waldrop, M. Mitchell. "Science 2.0" Scientific American. May 2008.
  • Ware, Mark. "Peer review: benefits, perceptions and alternatives." Publishing Research Consortium. 2008. (Dec. 2, 2008) www.publishingresearch.net/documents/PRCsummary4Warefinal.pdf
  • White, Caroline. "Little evidence for effectiveness of scientific peer review." BMJ. Volume 326, February 2003. (Dec. 2, 2008) http://www.bmj.com/cgi/content/full/326/7383/241/a

­