Validating the 8 CPCSSN Case Definitions for Chronic Disease Surveillance in a Primary Care Database of Electronic Health Records

Study Sample

CPCSSN consists of 10 primary care research networks (PCRNs) across Canada with 475 participating primary care sentinel clinicians (family physicians and nurse practitioners) contributing quarterly data on more than 600,000 patients. All 10 PCRNs have received approval from their host institution research ethics boards, as well as Health Canada ethics approval for collecting this information. Our study included patient data extracted on June 30, 2012, from 6 of the 10 PCRNs. Four of the 10 networks were excluded for a variety of reasons, including being the pilot test site for this study and data collection problems. Each of the participating networks reviewed 400 patient charts, except in British Columbia, which had a smaller number of participating sentinel clinicians and accordingly reviewed 200 charts. Charts were randomly selected using an age-stratified method, with 90% of the charts drawn from those patients older than 60 years. In addition, 25 charts each for epilepsy and parkinsonism were chosen nonrandomly because of the lower prevalence rates of these diseases. The total sample of 2,200 charts from all the sites ensured a margin of error of less than 10% for all outcomes of interest, with the exception of epilepsy and parkinsonism.

Case Definitions

The CPCSSN case definitions were specifically developed for use in primary care contexts. The definitions use a combination of International Classification of Diseases, Ninth Revision (ICD-9) codes (used by primary care physicians for service billing purposes in Canada) and numeric and textual data (including spelling variants, etc) drawn from a number of sections within the EHR, including the problem and encounter diagnoses, billing, laboratory test results, and prescribed medications (Supplemental Appendix). The case definitions were constructed with guidance from published evidence and both general and specialist physicians, and required several revisions before validation and implementation using computerized case finding algorithms. Table 1 provides a description of the case definitions. Each is unique to the respective chronic condition and includes varying EHR data elements. The detailed case definitions are available from CPCSSN at http://www.cpcssn.ca/research-resources/case-definitions.

Statistical Analysis

Sensitivity, specificity, and positive and negative predictive values (PPVs and NPVs) for each case definition were calculated. The data were summarized using 2 × 2 tables comparing the CPCSSN case definition diagnosis (either a case or noncase) with the chart review diagnosis (either a case or noncase) for each condition. Generalized estimating equations quantified the effect of clustering at the physician, site, or network level. The estimated intracluster correlation was then used to more accurately estimate the appropriate 95% confidence intervals for sensitivity, specificity, PPV, and NPV. For all metrics, the lower limit of the 95% confidence interval was considered the lower limit of the plausible range and was compared with standard cutoffs. Because acceptable limits for individual metrics need to be suited to the question of interest, we considered all measures above 70% acceptable, with any falling into the 70% to 80% range meriting additional investigation. All data were analyzed using the SAS 9.3 statistical platform (SAS Institute).

Overall Study Sample

In total, 1,920 patient charts were reviewed from regional PCRNs from the provinces of British Columbia, Alberta (2 PCRNs), Ontario, Nova Scotia, and Newfoundland. Collectively, these PCRNs included 126 sentinel clinicians from 33 practice sites, using 4 different EHR systems (Nightingale On Demand [Nightingale Informatix Corporation], PS Suite EMR [TELUS Health], Wolf EHR [TELUS Health], and Med Access [TELUS Health]). A description of the patients selected for the study is displayed in Table 2. This sample was age stratified with oversampling in older age-groups; we expected higher prevalence rates for the 8 chronic conditions.

There was a shortfall of 280 charts from the planned sample of 2,200. One hundred forty-nine charts were excluded because of EHR access challenges. Forty-four charts were excluded because of an insufficient number of patients with parkinsonism to satisfy the additional 25 per network. Eighty-seven charts were excluded because the EHR record was incomplete (n = 51), the patient had left the practice (n = 12), or was deceased (n = 24) (Figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Sample inclusion and exclusion flow chart.

APBRN = Atlantic Practice Based Research Network; BCPCReN = British Columbia Primary Research Care Network; MaRNet-FP = Maritme Family Practice Research Network; NAPCPReN = Northern Alberta Primary Care Research Network; SAPCReN = Southern Alberta Primary Care Research Network; UTOPIAN = University of Toronto Practice Based Research Network.

^a 350 Patients chosen at random, 25 patients case positive for epilepsy, 25 patients case positive for parkinsonism.

^b 175 Patients chosen at random, 13 patients case positive for epilepsy, 12 patients case positive for parkinsonism.

Validation

Table 3 summarizes the validation metrics of sensitivity, specificity, PPV, and NPV for each of the 8 case definitions. Overall, sensitivity ranged from 77.8% (osteoarthritis) to 98.8% (parkinsonism). Specificity was high for all 8 conditions, with the lowest being observed for hypertension (93.5%). COPD had the lowest PPV (72.1%) and hypertension the highest (92.9%). Hypertension had the lowest NPV (86.0%) with values greater than 90% for all other conditions and many above 99%.

Comparison With Other Studies

The validation metrics for the CPCSSN case definitions are comparable to, or better than, similar international databases. CPCSSN’s sensitivity rate for diabetes (95.6%) compares favorably with the United Kingdom’s Clinical Practice Research Datalink (formerly General Practice Research Database) (90.6%),⁷ as well as with a sample of EHR data from 17 physicians in 1 Canadian province (83.1%)⁸ and a general practice in Yorkshire, United Kingdom (98.3%).⁹ With respect to hypertension, the CPCSSN algorithm was both reasonably accurate (PPV of 92.9%) and comprehensive (sensitivity of 84.9%). Our findings are comparable to those observed in an EHR-based surveillance system in Sweden,¹⁰ while performing better than algorithms based exclusively on Canadian administrative data.¹¹ The case definition for depression has validity properties that are comparable to EHR-based algorithms in the United States¹² and better than algorithms based exclusively on billing and pharmacy data.¹³ A diagnostic algorithm for COPD used in a UK Clinical Practice Research Datalink study was less sensitive and specific,¹⁴ and algorithms derived from administrative data had lower sensitivity and PPV than those observed in our study.^15–17 Little is known about diagnostic algorithms for osteoarthritis, though validation results from a predictive algorithm used in a computerized, diagnostic database in the United States had outcomes similar to those of this study.¹⁸

The CPCSSN case definition for dementia performed better than those constructed from Quebec billing data (sensitivity of 12.9%–39.7%)¹⁹ and Canadian administrative hospital discharge data (sensitivity of 32.3%–66.9%, specificity of 100.0%).²⁰ The epilepsy results were similar to definitions using administrative databases for various Canadian provinces (PPV of 75.5%–98.9%, NPV of 94.0%–97.4%),²¹ and for definitions combining diagnoses and medications to identify epilepsy cases in a US managed care organization data system (PPV of 79.2%–84.1%).²² Lastly, cases of parkinsonism validated in an American Veteran’s Health Administration database showed strong PPV (81%) when using ICD-9 codes alone but had poor sensitivity (18.7%); sensitivity improved when ICD-9 codes and medications were combined (42.5%), but PPV suffered (53.3%).²³

Strengths and Weaknesses

Previous validation studies using EHR data have been highly variable in their methodology; diagnostic algorithms and reference standard sources have varied across studies, and as a result, establishing acceptable and comparable values for the conventional epidemiological measures of validity (sensitivity, specificity, PPV, NPV) was a complicated task.²⁴ For our study, randomly selecting charts enabled estimation of sensitivity and specificity, as well as positive and negative predictive value. Other studies have selected patients who were case positive, permitting only an assessment of the positive predictive value of the definition. ²⁵ Further, our study is unique in that the validation of case definitions took place across multiple regional networks involving multiple EHRs. Last, this study has a relatively large total sample size compared with similar studies involving primary care chart abstraction.²⁵ There were some limitations to our study. Using chart abstraction as the reference standard implies that the study is limited to information in the EHR. Missing diagnoses and incomplete documentation will limit the accuracy of the algorithms. Further, each case definition is currently limited to lifetime prevalence and for depression does not distinguish between chronic and episodic depression. The relatively low prevalence of epilepsy and parkinsonism in the general practice population necessitated oversampling with patients who were flagged as having the condition according to the algorithm. If the entire sample was selected this way, it would be inappropriate to speak about the sensitivity and specificity of these 2 case definitions. Given that the chart abstractors were completely blinded to the CPCSSN algorithmic diagnosis and that all charts were reviewed, however, we believe the calculated sensitivity and specificity are reasonable estimates. Another limitation is that interrater reliability was not measured; nevertheless, the standardized training manual and rigorous oversight during the review process promoted consistency. Although there were nonrandom exclusions from the study sample, it is unlikely that this would change the conclusions of the study. The largest exclusion was related to 1 network being unable to finish data collection because of technical problems. Other exclusions, such as insufficient number of parkinsonism cases and charts being incomplete or for deceased patients, are likely to have negligible impact on the overall study results. Including different EHRs is both a strength and a limitation, as doing so replicates reality; however, the study was not powered to detect differences between EHRs.

In conclusion, the CPCSSN case definitions show excellent sensitivity and specificity for hypertension, diabetes, epilepsy, dementia, and parkinsonism, and the validity in general for all 8 conditions was very good. CPCSSN has set precedence for systematically validating the case definitions used within our primary care database by way of an explicit, consistent, and robust methodology. The use of validated EHR-based clinical data from community-based primary care settings is essential to understand, inform, and evaluate disease epidemiology, as well as to improve primary care clinical practice, organizational development, and health system policy and planning.