To evaluate a process for identifying adverse events through review of medical records, multiple reviews of 360 medical records from two teaching hospital were performed. The data from these multiple reviews provided information about the validity and reliability of our two-phase review process. In particular, it was found that the initial phase, involving review of the medical records by medical-record-room administrators using explicit criteria, was valid, with a sensitivity of 84% and negative predictive value of 92%. Results also showed that the second phase, involving judgments by physicians guided by an adverse event analysis form, was reliable (Spearman Brown Rm = 0.78, m = 2) and demonstrated construct validity when compared with a review by a set of senior physicians employing a different method of review (Kappa = 0.57). In addition, it was found that these cases classified as difficult to judge from causation were judged less reliable (Rm = 0.48, for difficult case, Rm = 0.65, for other cases, m = 1). These results indicate that a two-step review process of medical records can produce judgments about adverse events that are both reliable and valid.