Methods to Achieve High Interrater Reliability in Data Collection From Primary Care Medical Records

Clare Liddy; Miriam Wiens; William Hogg

doi:10.1370/afm.1195

Article Figures & Data

Figures

Tables
Additional Files

Download figure
Open in new tab
Download powerpoint
Figure 1.
Timeline of interrater reliability checks in phase I.

Tables

Figures
Additional Files

View popup

Table 1.

κ Statistics and Percent Agreement Values for Selected Data Abstraction Items (N = 132 Charts)

		Percent Agreement
Item Abstracted	κ (95% CI)	Observed	Expected^a
ASA= acetylsalicylic acid (aspirin); CVD = cardiovascular disease; eGFR = estimated glomerular filtration rate; HbA_1c=hemoglobin A_1c; TIA = transient ischemic attack.
^a Percent agreement expected (p_e) is a measure of the agreement that is expected to occur by chance between the raters.
^b Measurement period was the 12 months before first date of abstraction.
^c β-blocker, angiotensin-converting enzyme inhibitor, angiotensin receptor blocker.
^d β-blocker, angiotensin-converting enzyme inhibitor, angiotensin receptor blocker, diuretic, calcium channel blocker.
Diagnoses
Cardiovascular disease	0.92 (0.85–0.99)	96.2	0.53
Chronic kidney disease	0.72 (0.58–0.87)	90.9	0.67
Diabetes	0.89 (0.82–0.97)	94.7	0.50
Dyslipidemia	0.90 (0.81–1.00)	97.0	0.69
Hypertension	0.80 (0.67–0.92)	93.2	0.66
Peripheral vascular disease	0.79 (0.62–0.97)	96.2	0.82
Stroke/TIA	0.89 (0.77–1.00)	97.7	0.79
Quality of care indicators^b
1 blood pressure recorded	1.00 (1.00–1.00)	100.0	0.87
1 HbA_1c test ordered	0.87 (0.78–0.96)	93.9	0.54
ASA recommended	0.83 (0.74–0.93)	91.7	0.33
CVD medications^c recommended	0.85 (0.76–0.94)	92.4	0.50
eGFR test ordered	0.74 (0.62–0.87)	90.2	0.62
Fasting blood glucose test ordered	0.70 (0.56–0.84)	88.6	0.63
Glycemic control medications recommended	0.83 (0.75–0.92)	90.9	0.45
Hypertension medications^d recommended	0.79 (0.68–0.90)	90.2	0.54
Lipid profile test ordered	0.71 (0.58–0.83)	87.1	0.56
Lipid-lowering medications recommended (statin or other)	0.79 (0.68–0.90)	90.9	0.57
Smoking status recorded	0.75 (0.63–0.88)	90.2	0.60
Smoking cessation counseling	0.88 (0.77–0.98)	96.2	0.69
Smoking cessation medication	0.88 (0.77–0.98)	96.2	0.70
Smoking cessation program referral	0.87 (0.77–0.98)	96.2	0.70
Waist circumference measured	0.00 (0.00–0.00)	95.5	0.95
Weight management program referral	0.86 (0.74–0.97)	95.5	0.68

Supplemental Appendix

�IDOCC�Improved Delivery of Cardiovascular Care through Outreach Facilitation. Data Collection Hand Book for Chart Abstractors (CAs)

Files in this Data Supplement:

Supplemental data: Appendix - PDF file, 33 pages, 1.5 MB
The Article in Brief

Methods to Achieve High Interrater Reliability in Data Collection From Primary Care Medical Records

Clare Liddy , and colleagues

Background Patient chart audits are often the only way to collect required data for research. There is little guidance, however, about methods to assess inter-rater reliability, the degree of agreement when a measurement is repeated under identical conditions by different raters. This study describes a 4-part data collection quality monitoring procedure and the process to measure data inter-rater reliability.

What This Study Found The 4-part data quality monitoring procedure included standardized protocols and forms, extensive training, continuous monitoring of inter-rater reliability, and a quality improvement feedback mechanism. There was excellent agreement between chart abstractors in this study, and no charts needed to be reabstracted, thus supporting the effectiveness of this training and data collection approach.

Implications

These findings offer a guide and benchmark for other medical chart review studies in primary care.