Clinical methodPragmatic strategies that enhance the reliability of data abstracted from medical records☆
Introduction
Retrospective research designs often rely on the abstraction of patients' medical records to obtain research data. This valid approach to data collection is, however, based on several assumptions about the validity of the data. These include (a) the data needed for the research will be present in the record; (b) the data in the record will be in a form that can be abstracted, or manipulated, for research purposes (e.g., grams converted to ounces); (c) the data in the record will accurately represent what was, in fact, the case; (d) data addressing any single item that is recorded in more than one place in the medical record will be consistently recorded by one or more individuals who enter that data; and (e) medical record entries will be interpretable in a manner common to all those who access the record (Allison et al., 2000). A source of data variance (validity error) is introduced into the data abstraction process in every instance when one of these assumptions proves to be invalid.
The reliability of these data is an equally compelling concern. The potential for interrater variance is an additional concern when data are to be abstracted by more than one individual. This challenge increases when information is to be obtained from a large number of providers, in multiple locations, each of whom may use a different medical record form and format, and also when data are collected over a long period.
This article describes the process and procedure used to promote interrater reliability (IRR) in the abstraction of data for a California statewide study of the quality of breast and cervical cancer screening services. San Diego State University's Cancer Clinical Quality Assurance Project funded by and in collaboration with the Cancer Detection Section of the California Department of Health Services conducted the abstraction and developed the processes and procedures described here (http://www.qap.sdsu.edu). Several proactive strategies that serve the purpose of leading to standard interpretations of clinical data are discussed. Lessons learned from this project can assist nurse researchers to develop high-quality information retrieval methods, when multiple observers (or abstractors) are used during the data collection process.
Section snippets
Review of the literature
The validity of information contained in the medical record has been tested in a number of recent investigations. Clegg et al. (2001) noted that the decentralization of the health-care system, with the result that therapies are being implemented in a wider variety of settings, has made the process of collection of outcomes data using chart review and abstraction both more difficult and more expensive. They compared the results of using two methods of self-report (a mailed questionnaire vs. an
Instruments
This project developed two computerized database applications for the abstraction of breast and cervical cancer screening and diagnostic information from patient medical records. The researchers recognized and acknowledged concerns about the validity of certain types of data contained in the medical record that are documented in the literature cited above. Certain design elements were strategically incorporated into the computerized medical record abstraction tool in the interest of promoting
Results
The overall concordance (Po) computed during the combined data abstraction phases ranged from 96% to 100% for six of nine major demographic outcome variables and from 90% to 94% for two additional items. The overall degree of agreement computed during this same timeframe for the documentation of procedures and their results ranged from 89% to 99% for seven variables. Assuming an expected proportion of agreement between two individual reviewers to be 50%, the upper and lower bounds of kappa over
Discussion
Two of the abstracted data elements were associated with lower computed interrater agreements. This finding was likely associated with the more subjective nature of these data sources. The patient self-report of breast symptoms was the first of these subjective sources. The second source of subjective data was the lexicon of terminology used by clinical providers when describing the findings of a clinical examination. Whether the provider translated that clinical finding into a predictive
Summary and conclusion
The IRR assessment process designed for this project provided the following benefits: (a) facilitated consistency in the data collection process; (b) allowed for continuous quality improvement by providing feedback and suggestions for improvement throughout the abstraction process; (c) provided a forum for resolving questions encountered in field abstraction; and (d) established benchmarks to evaluate data consistency and the IRR process itself. The processes and procedures established for the
References (14)
- et al.
The art and science of chart review
Joint Commission Journal on Quality Improvement
(2000) - et al.
How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record
American Journal of Medicine
(2000) - et al.
Comparison of self-reported initial treatment with medical records: Results from the prostate cancer outcomes study
American Journal of Epidemiology
(2001) - et al.
Reliability of health information on the net: An examination of experts' ratings
Journal of Medical Internet Research
(2002) - et al.
Accounting for overlap? An application of Mezzich's kappa statistic to test interrater reliability of interview data on parental accident and emergency attendance
Journal of Advanced Nursing
(2001) - et al.
Methodological considerations in the design of an obstetric database abstracted from medical records
Methods of Information in Medicine
(1997) - et al.
Estimating hospital deaths due to errors: Preventability is in the eye of the reviewer
Journal of the Medical Association
(2001)
Cited by (26)
Researching the Appropriateness of Care in the Complementary and Integrative Health Professions Part 5: Using Patient Records: Selection, Protection, and Abstraction
2019, Journal of Manipulative and Physiological TherapeuticsCitation Excerpt :When the data source is the patient record, the complexity of data can be challenging for abstractors to interpret in a standardized manner. Therefore, a comprehensive abstraction guide is essential, including priorities of source for data elements, standardization of terminology, definitions for unstructured data elements, and a process for guideline revision during data collection.9,10 The utility of the abstraction tool used is maximized by logically organizing the content to be user friendly and approximate the organization of the medical record.
Reliability of a Canadian database for primary care nursing services' clinical and administrative data
2018, International Journal of Medical InformaticsLooking through the retrospectoscope: Reducing bias in emergency medicine chart review studies
2014, Annals of Emergency MedicineCitation Excerpt :Thus, the sensitivity and specificity of the medical record is low because there may be errors and idiosyncrasies in the reading, interpreting, coding, and transcribing of the data. Solution: The variables to be collected from the chart, as well as how these variables are defined, should be determined a priori and documented in a coding guide for abstractors.6,7,11,12 When the methods are reported, the coding rules for each abstracted element should be provided.
Research strategies that result in optimal data collection from the patient medical record
2012, Applied Nursing ResearchMedical record review to recover missing data in a Portuguese birth cohort: Agreement with self-reported data collected by questionnaire and inter-rater variability
2011, Gaceta SanitariaCitation Excerpt :The interobserver variability was low and did not threaten data precision. Standardized training of abstractors and rigorous quality assurance were proposed as critical criteria to improve the quality and accuracy of clinical record review16–18,34,35. When research involves data collection by distinct observers, the extent to which different observers perceive and record the same information should be evaluated.
Factors Associated With Non-Normal Birth Outcomes for Low-Risk Women in an Inner-City Hospital
2010, Journal of Midwifery and Women's Health
- ☆
Supported in part by funds received from the State of California, Department of Health Services, Cancer Detection Section. All analysis, interpretations, and conclusions presented in this article are those of the authors and not the State of California. There are no known biases in the data presented that would affect the results.