Highly cited studies on biomarkers often report larger effect size estimates than subsequent studies

Frequently cited studies involving associations of biomarkers report effect sizes that are often larger when compared to summary estimates from meta-analyses evaluating the same associations, according to a study in the June 1 issue of JAMA.

"Many new biomarkers are continuously proposed as potential determinants of disease risk, prognosis, or response to treatment. The plethora of statistically significant associations increases expectations for improvements in risk appraisal. However, many markers get evaluated only in 1 or a few studies. Among those evaluated more extensively, few reach clinical practice. This translational attrition requires better study. Are the effect sizes proposed in the literature accurate or overestimated?" the authors write.

John P. A. Ioannidis, M.D., D.Sc., of the Stanford University School of Medicine, Stanford, Calif., and University of Ioannina School of Medicine, Ioannina, Greece, and Orestis A. Panagiotou, M.D., of the University of Ioannina School of Medicine, conducted a review of the medical literature to examine biomarkers that had been evaluated in at least 1 highly cited study and for which at least 1 meta-analysis had been performed for that same association to compare the effect size of these associations. Eligible articles were those that had received more than 400 citations in the ISI Web of Science and that had been published in any of 24 highly cited biomedical journals. MEDLINE was searched for subsequent meta-analyses on the same associations (same biomarker and same outcome). The researchers identified 35 highly cited studies published between 1991 and 2006 for inclusion in the study. The highly cited biomarkers included genetic risk factors, blood proteins, other blood biomarkers, and infectious agent biomarkers. Cancer-related and cardiovascular-related outcomes predominated.

The researchers found that for 30 of the 35 associations (86 percent), the highly cited studies had a stronger effect estimate than the largest study; for 3 the largest and highly cited study coincided; and only twice was the effect estimate stronger in the largest than in the highly cited study. The relative risk estimate was in the opposite direction in the highly cited than in the largest study in 5 associations, and the increase was more than 2-fold greater in another 20 associations (more than 4-fold in 13 associations).

For 29 of the 35 highly cited studies (83 percent), the corresponding meta-analysis found a smaller effect. The relative risk estimate was in the opposite direction in the highly cited than in the meta-analysis in 4 associations and the increase was more than 2-fold greater in another 14 associations (more than 4-fold in 7 associations), the authors write. Only 15 associations were nominally statistically significant based on the largest studies. Thirty-two of the 35 associations showed nominally statistically significant increased risk based on the meta-analyses.

The researchers write that several reasons could explain false-positive and inflated results among the examined highly cited investigations. "Many of these studies were relatively small and among the first to report on the association of interest. Discoveries made in small studies are prone to overestimate or underestimate the actual association. Interest in publishing major discoveries leads to selective reporting from chasing significance."

"… our study documents that results in highly cited biomarker studies often significantly overestimate the findings seen from meta-analyses. Evidence from multiple studies, in particular large investigations, is necessary to appreciate the discriminating ability of these emerging risk factors. Rapid clinical adoption in the absence of such evidence may lead to wasted resources."

(JAMA. 2011;305[21]2200-2210. Available pre-embargo to the media at www.jamamedia.org)

Editor's Note: Please see the article for additional information, including other authors, author contributions and affiliations, financial disclosures, funding and support, etc.

Editorial: The Thin Line Between Hope and Hype in Biomarker Research

In an accompanying editorial, Patrick M. M. Bossuyt, Ph.D., of the University of Amsterdam, the Netherlands, writes that it would be premature to doubt all scientific efforts at marker discovery and unwise to discount all future biomarker evaluation studies.

"However, the analysis presented by Ioannidis and Panagiotou should convince clinicians and researchers to be careful to match personal hope with professional skepticism, to apply critical appraisal of study design and close scrutiny of findings where indicated, and to be aware of the findings of well-conducted systematic reviews and meta-analyses when evaluating the evidence on biomarkers."

(JAMA. 2011;305[21]2229-2230. Available pre-embargo to the media at www.jamamedia.org)

Editor's Note: Please see the article for additional information, financial disclosures, funding and support, etc.

Source: JAMA and Archives Journals