Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study

INTRODUCTION

The primary outcome (or endpoint) is an essential element of a randomized controlled trial (RCT), and specification of outcomes is central to the design and reporting of these trials. Changing the outcomes of a clinical trial without explanation calls its validity into question. To allow authors to change their outcomes without reason or disclosure threatens the underpinnings of the scientific enterprise and threatens the ability of practicing physicians to apply published research to patient care. Acceptable reasons to change outcomes include results of other trials, intertrial comparisons, theoretical advances in the relevant field, and technical advances (for example, in measurement or treatment). These changes present few problems if they are reported and interpreted appropriately. Unfortunately, outcomes may be changed in ways that violate the fundamental principles of trial design and interpretation¹ and, therefore, raise doubts about the validity of the conclusions. These changes, if made, should be carefully reasoned, limited in number, and clearly labeled as hypothesis generating rather than hypothesis supporting.

During the past several years efforts have been made to encourage “full transparency with respect to performance and reporting of clinical trials,”² because the “credibility of research depends on a critical assessment by others of the strengths and weaknesses in study design, conduct, and analysis.”³ As one part of this larger endeavor, most major medical journals and many funding agencies now require that trial details be registered. Registration includes the “definitions of the primary and secondary outcome measures.”² The purpose of our current research is to estimate the extent to which outcomes are changed, without disclosure in the final report, between trial registration and publication.

How prevalent is this problem of changing outcomes? Chan and colleagues^4,5 and Hahn and colleagues⁶ independently have looked at changes in outcomes that occur between trial protocols and publication (or nonpublication). This approach is likely to give an accurate estimate of outcome changes, because researchers are required to submit amendments to trial protocols to their institutional review boards. Protocols, however, are not usually available (although Hawkey⁷ has argued that they should be), making it nearly impossible for a typical practicing physician to critique trials using this information. Trial registries are an almost ideal solution. They are easy to find on the Internet, and the information is well-organized and concise. This system provides easier access and better transparency for the typical journal reader. With this background in mind, we set out to examine how often outcomes are changed between trial registration and final publication.

METHODS

Our primary outcome was the proportion of trials in which the primary outcome of the trial under consideration had been changed without disclosure from registration to publication. Our secondary outcome was the proportion of trials in which the secondary outcome of the trial under consideration had been changed without disclosure from registration to final publication.

To determine the frequency of changes in outcomes, we assembled consecutive issues of 5 major medical journals. These 5 journals were chosen because they have high impact factors and are widely considered of high quality. We identified all reports of randomized controlled trials for a 6-month period. We searched each report for a reference to a trials registry. We then excluded trials for the following reasons: there was no publicly accessible trial registration recorded in the report; the registration database did not record a primary outcome or the outcome recorded was too vague to make any judgments; and the trial publication was not the main report of the trial results. If there were multiple reports from the same trial, we used what appeared to be the main one. If multiple reports considered different outcomes of the same study, we either combined them and considered them together as one study or considered them as separate studies depending on the circumstances. We kept a database of all excluded trials.

For each included report we extracted the primary and secondary outcomes as listed in the registration database and in the final publication (or, if necessary, in the associated design and methods publication). We did not search the archives of the trials registries to see whether the outcomes had been changed since registration. Rather, we took them as is on the day we searched. For quality assurance, each of the 3 authors reviewed all the articles independently. The few disagreements were resolved easily by consensus after reviewing the articles a second time. The authors were not masked to the identity of the research reports.

We considered outcomes to have been changed only if there were substantive differences, rather than minor differences in wording that did not appear to change the intent of the trial. We counted outcomes as unchanged if the authors acknowledged the change and made any statement indicating that the changes were made before any analyses were done. Secondary outcomes were counted as unchanged if the authors said they would be published separately (although we made no attempt to confirm event).

As this sample of articles was not generated randomly, the assumptions underlying the calculation of confidence intervals do not hold; therefore, the confidence intervals presented here (calculated by the modified Wald method and at the 95% level⁸) should be interpreted cautiously.

RESULTS

We found a total of 158 RCTs in the 5 journals (Table 1⇓). Forty-eight were excluded, and 110 were included in the analysis. Trials were registered in 5 databases. Table 2⇓ gives further details of the included and excluded trials, and Table 3⇓ gives further details on the registry databases.

Thirty-four reports (31%; 95% confidence interval [CI], 23%–40%) had a change in one or more primary outcomes. Seventy-seven reports (70%; 95% CI, 61%–78%) had a change in one or more secondary outcomes.

For the primary outcomes, most changes were by deletion. For the secondary outcomes, the changes were more evenly split between addition of outcomes and deletion. Table 4⇓ expands on these results. Although not part of our research question, we noted that there were almost no differences in outcomes when comparing trials funded by pharmaceutical companies with those that had noncommercial sponsorship. In Table 5⇓ are 3 examples of changes made between registration and publication. These were representative of the different types of changes we encountered.

DISCUSSION

Several biases could have effected our results. It is possible that the outcomes in the trial registry were incorrect. For example, difficulty in database formats, errors in data entry, misperception of registry importance, or deliberate falsification could all invalidate the entries. Trials registries are not useful, however, if they do not provide an accurate representation as to what was intended.

It is also possible that the authors of the articles had made changes in their registry entries after data analysis but before publication, and our search would not have uncovered this change. During our data-checking process, we found 3 instances in which authors had changed their registry entries after trial publication. In all 3 cases, the original registry entry had not listed any outcomes,^15–17 but the subsequent changes listed outcomes identical to the final publications.^18–20 If these sorts of changes were common, we would have underestimated the frequency of outcome changes.

Further, our selection of articles was not random. We selected articles from what are often considered better journals. This focus could have resulted in a lower percentage of changed outcomes (as these articles are presumably reviewed more closely before publication) or a higher percentage (as the financial stakes may be higher with a correspondingly higher incentive to produce good results).

Finally, our exclusion criteria were subjective, and other researchers reviewing our database might make slightly different decisions. Although almost all of the decisions on changes in outcomes were straightforward, other reviewers of these same data might arrive at slightly different results. We doubt, however, that the conclusions would change.

Outcomes can be changed either by addition, deletion, or by some combination of the 2. The issue with addition of outcomes is that it asks the question after the answer is known. Deletion, while more subtle, may also turn a statistically insignificant result into a significant one. When a study has multiple outcomes, one may be significant by chance alone. In the past, authors have been tempted to mine data for statistically significant outcomes after a study was completed. It is well-understood that such activity is unacceptable.^21,22 Instead, authors may be tempted to list multiple outcomes in advance and then eliminate those that were not significant—in essence, data mining in reverse. Any use of multiple outcomes requires P value adjustments to account for chance findings of statistical significance. Reducing the number of outcomes can, therefore, make an outcome appear significant where it would not have been before the reduction. Although there is considerable disagreement in the statistical literature about the best approach to doing these adjustments, there is no argument about the necessity.²³

We suggest several potential remedies for the issues discussed here. (1) The International Committee of Medical Journal Editors²⁴ requirement that authors register the World Health Organization Minimal Registration Data Set²⁵ should be enforced by all databases (or all journals). For example, the American registry does not require the registration of any outcomes,²⁶ an omission that is easily remedied. (2) Trial registration databases should have the archive of each trial linked directly to the trial registration, which would make it easier to follow changes. (3) Our investigation was about transparency, not misrepresentation. We believe most changes were made in good faith and without an understanding of the issues involved. To aid authors in avoiding these problems, the next revision of the CONSORT statement²⁷ should require reporting of more details about trial registration and changes that have been made therein.²⁸ (4) Journal editors should expect that trials be registered at the outset and certainly before any interim statistical analyses are performed. (5) Reviewers should routinely check the trial protocol, the registry, and any quoted design and methods publications to ensure that the authors have not altered their outcomes without appropriate comment and justification.

Although our results must be regarded as only approximate, they nevertheless indicate a substantial and important lack of transparency in the reporting of clinical trials.