Skip to main content

Main menu

  • Home
  • Content
    • Current Issue
    • Online First
    • Multimedia
    • Collections
    • Past Issues
    • Articles by Subject
    • Articles by Type
    • Supplements
    • The Issue in Brief (Plain Language Summaries)
    • Call for Papers
  • Info for
    • Authors
    • Reviewers
    • Media
    • Job Seekers
  • About
    • Annals of Family Medicine
    • Editorial Staff & Boards
    • Sponsoring Organizations
    • Copyrights & Permissions
    • Announcements
  • Engage
    • Engage
    • e-Letters (Comments)
    • Subscribe
    • RSS
    • Email Alerts
    • Journal Club
  • Contact
    • Feedback
    • Contact Us
  • Careers

User menu

  • My alerts

Search

  • Advanced search
Annals of Family Medicine
  • My alerts
Annals of Family Medicine

Advanced Search

  • Home
  • Content
    • Current Issue
    • Online First
    • Multimedia
    • Collections
    • Past Issues
    • Articles by Subject
    • Articles by Type
    • Supplements
    • The Issue in Brief (Plain Language Summaries)
    • Call for Papers
  • Info for
    • Authors
    • Reviewers
    • Media
    • Job Seekers
  • About
    • Annals of Family Medicine
    • Editorial Staff & Boards
    • Sponsoring Organizations
    • Copyrights & Permissions
    • Announcements
  • Engage
    • Engage
    • e-Letters (Comments)
    • Subscribe
    • RSS
    • Email Alerts
    • Journal Club
  • Contact
    • Feedback
    • Contact Us
  • Careers
  • Follow annalsfm on Twitter
  • Visit annalsfm on Facebook
Research ArticleMethodology

Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study

Robert Ewart, Harald Lausen and Norman Millian
The Annals of Family Medicine November 2009, 7 (6) 542-546; DOI: https://doi.org/10.1370/afm.1017
Robert Ewart
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Harald Lausen
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Norman Millian
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF
Loading

Abstract

PURPOSE We wanted to investigate the frequency of undisclosed changes in the outcomes of randomized controlled trials (RCTs) between trial registration and publication.

METHODS Using a retrospective, nonrandom, cross-sectional study design, we investigated RCTs published in consecutive issues of 5 major medical journals during a 6-month period and their associated trials registry entries. Articles were excluded if they did not have an available trial registry entry, did not have analyzable outcomes, or were secondary publications. The primary outcome was the proportion of publications in which the primary outcome of the trial was, without disclosure, changed between that recorded in the trial registry and that reported in the final publication. The secondary outcome was the proportion of publications in which the secondary outcome was changed without disclosure.

RESULTS We reviewed 158 reports of RCTs and included 110 in the analysis. In 34 (31%), a primary outcome had been changed, and in 77 (70%), a secondary outcome had been changed.

CONCLUSIONS There are substantial and important undisclosed changes made to the outcomes of published RCTs between trial registration and publication. This finding has important implications for the interpretation of trial results. Disclosure and discussion of changes would improve transparency in the performance and reporting of trials.

  • Endpoints
  • outcomes
  • trials registration

You want more scepticism at the very foundation of your work. Scepticism, the tonic of minds, the tonic of life, the agent of truth, the way of art and salvation. Joseph Conrad, in a letter to John Galsworthy, 1901

INTRODUCTION

The primary outcome (or endpoint) is an essential element of a randomized controlled trial (RCT), and specification of outcomes is central to the design and reporting of these trials. Changing the outcomes of a clinical trial without explanation calls its validity into question. To allow authors to change their outcomes without reason or disclosure threatens the underpinnings of the scientific enterprise and threatens the ability of practicing physicians to apply published research to patient care. Acceptable reasons to change outcomes include results of other trials, intertrial comparisons, theoretical advances in the relevant field, and technical advances (for example, in measurement or treatment). These changes present few problems if they are reported and interpreted appropriately. Unfortunately, outcomes may be changed in ways that violate the fundamental principles of trial design and interpretation1 and, therefore, raise doubts about the validity of the conclusions. These changes, if made, should be carefully reasoned, limited in number, and clearly labeled as hypothesis generating rather than hypothesis supporting.

During the past several years efforts have been made to encourage “full transparency with respect to performance and reporting of clinical trials,”2 because the “credibility of research depends on a critical assessment by others of the strengths and weaknesses in study design, conduct, and analysis.”3 As one part of this larger endeavor, most major medical journals and many funding agencies now require that trial details be registered. Registration includes the “definitions of the primary and secondary outcome measures.”2 The purpose of our current research is to estimate the extent to which outcomes are changed, without disclosure in the final report, between trial registration and publication.

How prevalent is this problem of changing outcomes? Chan and colleagues4,5 and Hahn and colleagues6 independently have looked at changes in outcomes that occur between trial protocols and publication (or nonpublication). This approach is likely to give an accurate estimate of outcome changes, because researchers are required to submit amendments to trial protocols to their institutional review boards. Protocols, however, are not usually available (although Hawkey7 has argued that they should be), making it nearly impossible for a typical practicing physician to critique trials using this information. Trial registries are an almost ideal solution. They are easy to find on the Internet, and the information is well-organized and concise. This system provides easier access and better transparency for the typical journal reader. With this background in mind, we set out to examine how often outcomes are changed between trial registration and final publication.

METHODS

Our primary outcome was the proportion of trials in which the primary outcome of the trial under consideration had been changed without disclosure from registration to publication. Our secondary outcome was the proportion of trials in which the secondary outcome of the trial under consideration had been changed without disclosure from registration to final publication.

To determine the frequency of changes in outcomes, we assembled consecutive issues of 5 major medical journals. These 5 journals were chosen because they have high impact factors and are widely considered of high quality. We identified all reports of randomized controlled trials for a 6-month period. We searched each report for a reference to a trials registry. We then excluded trials for the following reasons: there was no publicly accessible trial registration recorded in the report; the registration database did not record a primary outcome or the outcome recorded was too vague to make any judgments; and the trial publication was not the main report of the trial results. If there were multiple reports from the same trial, we used what appeared to be the main one. If multiple reports considered different outcomes of the same study, we either combined them and considered them together as one study or considered them as separate studies depending on the circumstances. We kept a database of all excluded trials.

For each included report we extracted the primary and secondary outcomes as listed in the registration database and in the final publication (or, if necessary, in the associated design and methods publication). We did not search the archives of the trials registries to see whether the outcomes had been changed since registration. Rather, we took them as is on the day we searched. For quality assurance, each of the 3 authors reviewed all the articles independently. The few disagreements were resolved easily by consensus after reviewing the articles a second time. The authors were not masked to the identity of the research reports.

We considered outcomes to have been changed only if there were substantive differences, rather than minor differences in wording that did not appear to change the intent of the trial. We counted outcomes as unchanged if the authors acknowledged the change and made any statement indicating that the changes were made before any analyses were done. Secondary outcomes were counted as unchanged if the authors said they would be published separately (although we made no attempt to confirm event).

As this sample of articles was not generated randomly, the assumptions underlying the calculation of confidence intervals do not hold; therefore, the confidence intervals presented here (calculated by the modified Wald method and at the 95% level8) should be interpreted cautiously.

RESULTS

We found a total of 158 RCTs in the 5 journals (Table 1⇓). Forty-eight were excluded, and 110 were included in the analysis. Trials were registered in 5 databases. Table 2⇓ gives further details of the included and excluded trials, and Table 3⇓ gives further details on the registry databases.

View this table:
  • View inline
  • View popup
Table 1.

Study Flow (N = 158)

View this table:
  • View inline
  • View popup
Table 2.

Journals Publishing Trials Reports Reviewed in the Study

View this table:
  • View inline
  • View popup
Table 3.

Databases of Reported Trials

Thirty-four reports (31%; 95% confidence interval [CI], 23%–40%) had a change in one or more primary outcomes. Seventy-seven reports (70%; 95% CI, 61%–78%) had a change in one or more secondary outcomes.

For the primary outcomes, most changes were by deletion. For the secondary outcomes, the changes were more evenly split between addition of outcomes and deletion. Table 4⇓ expands on these results. Although not part of our research question, we noted that there were almost no differences in outcomes when comparing trials funded by pharmaceutical companies with those that had noncommercial sponsorship. In Table 5⇓ are 3 examples of changes made between registration and publication. These were representative of the different types of changes we encountered.

View this table:
  • View inline
  • View popup
Table 4.

Summary of Outcome Changes of Included Studies (n = 110)

View this table:
  • View inline
  • View popup
Table 5.

Examples of Changes Made Between Trial Registration and Publication of Findings

DISCUSSION

Several biases could have effected our results. It is possible that the outcomes in the trial registry were incorrect. For example, difficulty in database formats, errors in data entry, misperception of registry importance, or deliberate falsification could all invalidate the entries. Trials registries are not useful, however, if they do not provide an accurate representation as to what was intended.

It is also possible that the authors of the articles had made changes in their registry entries after data analysis but before publication, and our search would not have uncovered this change. During our data-checking process, we found 3 instances in which authors had changed their registry entries after trial publication. In all 3 cases, the original registry entry had not listed any outcomes,15–17 but the subsequent changes listed outcomes identical to the final publications.18–20 If these sorts of changes were common, we would have underestimated the frequency of outcome changes.

Further, our selection of articles was not random. We selected articles from what are often considered better journals. This focus could have resulted in a lower percentage of changed outcomes (as these articles are presumably reviewed more closely before publication) or a higher percentage (as the financial stakes may be higher with a correspondingly higher incentive to produce good results).

Finally, our exclusion criteria were subjective, and other researchers reviewing our database might make slightly different decisions. Although almost all of the decisions on changes in outcomes were straightforward, other reviewers of these same data might arrive at slightly different results. We doubt, however, that the conclusions would change.

Outcomes can be changed either by addition, deletion, or by some combination of the 2. The issue with addition of outcomes is that it asks the question after the answer is known. Deletion, while more subtle, may also turn a statistically insignificant result into a significant one. When a study has multiple outcomes, one may be significant by chance alone. In the past, authors have been tempted to mine data for statistically significant outcomes after a study was completed. It is well-understood that such activity is unacceptable.21,22 Instead, authors may be tempted to list multiple outcomes in advance and then eliminate those that were not significant—in essence, data mining in reverse. Any use of multiple outcomes requires P value adjustments to account for chance findings of statistical significance. Reducing the number of outcomes can, therefore, make an outcome appear significant where it would not have been before the reduction. Although there is considerable disagreement in the statistical literature about the best approach to doing these adjustments, there is no argument about the necessity.23

We suggest several potential remedies for the issues discussed here. (1) The International Committee of Medical Journal Editors24 requirement that authors register the World Health Organization Minimal Registration Data Set25 should be enforced by all databases (or all journals). For example, the American registry does not require the registration of any outcomes,26 an omission that is easily remedied. (2) Trial registration databases should have the archive of each trial linked directly to the trial registration, which would make it easier to follow changes. (3) Our investigation was about transparency, not misrepresentation. We believe most changes were made in good faith and without an understanding of the issues involved. To aid authors in avoiding these problems, the next revision of the CONSORT statement27 should require reporting of more details about trial registration and changes that have been made therein.28 (4) Journal editors should expect that trials be registered at the outset and certainly before any interim statistical analyses are performed. (5) Reviewers should routinely check the trial protocol, the registry, and any quoted design and methods publications to ensure that the authors have not altered their outcomes without appropriate comment and justification.

Although our results must be regarded as only approximate, they nevertheless indicate a substantial and important lack of transparency in the reporting of clinical trials.

Acknowledgments

The authors wish to acknowledge the contributions of Ms Joan Stolz, who reviewed drafts of this article and made helpful editorial comments. The authors also wish to acknowledge the constructive comments of anonymous reviewers of numerous earlier drafts of this article.

Footnotes

  • Conflicts of interest: none reported

  • Received for publication March 26, 2008.
  • Revision received March 5, 2009.
  • Accepted for publication March 6, 2009.
  • © 2009 Annals of Family Medicine, Inc.

REFERENCES

  1. ↵
    Evans S. When and how can endpoints be changed after initiation of a randomized clinical trial? PLoS Clin Trials. 2007;2(4):e18.
    OpenUrlCrossRefPubMed
  2. ↵
    De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. http://www.icmje.org/clin_trial.pdf Accessed Feb 3, 2007.
  3. ↵
    von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573–577.
    OpenUrlCrossRefPubMed
  4. ↵
    Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–2465.
    OpenUrlCrossRefPubMed
  5. ↵
    Chan A-W, Krleza-Jeri K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–740.
    OpenUrlAbstract/FREE Full Text
  6. ↵
    Hahn S, Williamson PR, Hutton JL. Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002;8(3):353–359.
    OpenUrlCrossRefPubMed
  7. ↵
    Hawkey CJ. Journals should see original protocols for clinical trials. [Letter]. BMJ. 2001;323(7324):1309.
    OpenUrlFREE Full Text
  8. ↵
    Measuring usability: quantitative usability, statistics & six sigma by Jeff Sauro. Web page. http://www.measuringusability.com/wald.htm. Accessed May 26, 2008.
  9. Pfizer. Effect and safety of detrol LA in men with overactive bladder symptoms with or without bladder outlet obstruction. In: Clinical-Trials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct/show/NCT00147654. Accessed May 25, 2008.
  10. Kaplan SA, Roehrborn CG, Rovner ES, Carlsson M, Bavendam T, Guan Z. Tolterodine and tamsulosin for treatment of men with lower urinary tract symptoms and overactive bladder: a randomized controlled trial. JAMA. 2006;296(19):2319–2328.
    OpenUrlCrossRefPubMed
  11. European Organization for Research and Treatment of Cancer. Radiation therapy, surgery, and chemotherapy in treating patients with rectal cancer that can be surgically removed. In: ClinicalTrials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct/show/NCT00002523. Accessed May 25, 2008.
  12. Bosset JF, Collette L, Calais G, et al.; EORTC Radiotherapy Group Trial 22921. Chemotherapy with preoperative radiotherapy in rectal cancer. N Engl J Med. 2006;355(11):1114–1123.
    OpenUrlCrossRefPubMed
  13. University of Minnesota. National Institute of Dental and Craniofacial Research (NIDCR). Obstetrics and periodontal therapy (OPT) study. In: ClinicalTrials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct2/show/NCT00066131 Accessed May 28, 2008.
  14. Michalowicz BS, Hodges JS, DiAngelis AJ, et al; OPT Study. Treatment of periodontal disease and the risk of preterm birth. N Engl J Med. 2006;355(18):1885–1894.
    OpenUrlCrossRefPubMed
  15. ↵
    Pronyk PM, Hargreaves JR, Kim JC, et al. Effect of a structural intervention for the prevention of intimate-partner violence and HIV in rural South Africa: a cluster randomised trial. Lancet. 2006;368(9551):1973–1983.
    OpenUrlCrossRefPubMed
  16. Bussel JB, Kuter DJ, George JN, et al. AMG 531, a thrombopoiesis-stimulating protein, for chronic ITP. N Engl J Med. 2006;355(16): 1672–1681.
    OpenUrlCrossRefPubMed
  17. ↵
    National Institute of Allergy and Infectious Diseases (NIAID). CQSP in Malawi: chloroquine and sulfadoxine-pyrimethamine efficacy for the treatment of malaria in Malawi. In: ClinicalTrials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct2/show/NCT00125489. Accessed May 26, 2008.
  18. ↵
    Sanofi-Aventis. Safety and efficacy of enoxaparin in percutaneous coronary intervention (PCI) patients, an international randomized evaluation (STEEPLE). In: ClinicalTrials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct2/show/NCT00077844. Accessed May 26, 2008.
  19. National Institute of Allergy and Infectious Diseases (NIAID). Islet cell transplants for diabetes. In: ClinicalTrials.gov [Internet]. Bethesda, MD: National Library of Medicine. http://www.clinicaltrials.gov/ct2/show/NCT00014911. Accessed May 26, 2008.
  20. ↵
    Laufer MK, Thesing PC, Eddington ND, et al. Return of chloroquine antimalarial efficacy in Malawi. N Engl J Med. 2006;355 (19):1959–1966.
    OpenUrlCrossRefPubMed
  21. ↵
    Montalescot G, White HD, Gallo R, et al; STEEPLE Investigators. Enoxaparin versus unfractionated heparin in elective percutaneous coronary intervention. N Engl J Med. 2006;355(10):1006–1017.
    OpenUrlCrossRefPubMed
  22. ↵
    Shapiro AM, Ricordi C, Hering BJ, et al. International trial of the Edmonton protocol for islet transplantation. N Engl J Med. 2006;355(13):1318–1330.
    OpenUrlCrossRefPubMed
  23. ↵
    Tannock IF. False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons. J Natl Cancer Inst. 1996;88(3–4):206–207.
    OpenUrlFREE Full Text
  24. ↵
    Mills JL. Data torturing. N Engl J Med. 1993;329(16):1196–1199.
    OpenUrlCrossRefPubMed
  25. ↵
    Ludbrook J. Multiple comparison procedures updated. Clin Exp Pharmacol Physiol. 1998;25(12):1032–1037.
    OpenUrlCrossRefPubMed
  26. ↵
    International Committee of Medical Journal Editors. Statement of purpose. In: Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication. http://www.icmje.org/index.html#state. Accessed Jun 26, 2008.
  27. ↵
    International Committee of Medical Journal Editors. Table 1: minimal registration data set. In: Is This Clinical Trial Fully Registered?: A Statement from the International Committee of Medical Journal Editors. http://www.icmje.org/clin_trialup.htm#table1. Accessed May 26, 2008.
  28. ↵
    ClinicalTrials.gov Registration requirements. http://prsinfo.clinicaltrials.gov/. Accessed May 26, 2008.
  29. CONSORT Statement web site. http://www.consort-statement.org/.
  30. Goldbeck-Wood S. Commentary: changes between protocol and manuscript should be declared at submission. BMJ. 2001;322: 1460–1461.
    OpenUrl
PreviousNext
Back to top

In this issue

The Annals of Family Medicine: 7 (6)
The Annals of Family Medicine: 7 (6)
Vol. 7, Issue 6
1 Nov 2009
  • Table of Contents
  • Index by author
  • In Brief
  • Annual Indexes
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on Annals of Family Medicine.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study
(Your Name) has sent you a message from Annals of Family Medicine
(Your Name) thought you would like to see the Annals of Family Medicine web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
11 + 6 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study
Robert Ewart, Harald Lausen, Norman Millian
The Annals of Family Medicine Nov 2009, 7 (6) 542-546; DOI: 10.1370/afm.1017

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Get Permissions
Share
Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study
Robert Ewart, Harald Lausen, Norman Millian
The Annals of Family Medicine Nov 2009, 7 (6) 542-546; DOI: 10.1370/afm.1017
Reddit logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • Acknowledgments
    • Footnotes
    • REFERENCES
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF

Related Articles

  • PubMed
  • Google Scholar

Cited By...

  • Peer reviewed evaluation of registered end-points of randomised trials (the PRE-REPORT study): protocol for a stepped-wedge, cluster-randomised trial
  • Systematic review protocol assessing the processes for linking clinical trial registries and their published results
  • Many Randomized Trials of Physical Therapy Interventions Are Not Adequately Registered: A Survey of 200 Published Trials
  • Despite Law, Fewer Than One In Eight Completed Studies Of Drugs And Biologics Are Reported On Time On ClinicalTrials.gov
  • Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
  • Discrepancies between research advertisements and disclosure of study locations in trial registrations for USA-sponsored research in Russia
  • In This Issue: Multifaceted Research
  • Google Scholar

More in this TOC Section

  • Designing and Implementing an Electronic Health Record–Embedded Card Study in Primary Care: Methods and Considerations
  • Considerations Before Selecting a Stepped-Wedge Cluster Randomized Trial Design for a Practice Improvement Study
  • Case Study With a Participatory Approach: Rethinking Pragmatics of Stakeholder Engagement for Implementation Research
Show more METHODOLOGY

Similar Articles

Subjects

  • Methods:
    • Quantitative methods
  • Other research types:
    • Health policy

Content

  • Current Issue
  • Past Issues
  • Past Issues in Brief
  • Multimedia
  • Articles by Type
  • Articles by Subject
  • Multimedia
  • Supplements
  • Online First
  • Calls for Papers

Info for

  • Authors
  • Reviewers
  • Media
  • Job Seekers

Engage

  • E-mail Alerts
  • e-Letters (Comments)
  • RSS
  • Journal Club
  • Submit a Manuscript
  • Subscribe
  • Family Medicine Careers

About

  • About Us
  • Editorial Board & Staff
  • Sponsoring Organizations
  • Copyrights & Permissions
  • Contact Us
  • eLetter/Comments Policy

© 2023 Annals of Family Medicine