Abstract
PURPOSE Cognitive diagnostic work-up in primary care is not always physically feasible, owing to chronic disabilities and/or travel restrictions. The identification of dementia might be facilitated with diagnostic instruments that are time efficient and easy to perform, as well as useful in the remote setting. We assessed whether the Telephone Interview for Cognitive Status (TICS) might be a simple and accurate alternative for remote diagnostic cognitive screening in primary care.
METHODS We administered the TICS (range, 0-41) for 810 of 1,473 older people aged 84.5 (SD, 2.4) years. We scrutinized electronic health records for participants with TICS scores ≤30 and for a random sample of participants with TICS scores >30 for a dementia diagnosis using all data from the Prevention of Dementia by Intensive Vascular Care (preDIVA) trial for 8-12 years of follow-up. We used multiple imputation to correct for verification bias.
RESULTS Of the 810 participants, 155 (19.1%) had a TICS score ≤30, and 655 (80.9%) had a TICS score >30. Electronic health records yielded 8.4% (13/154) dementia diagnoses for participants with TICS ≤30 vs none with TICS >30. Multiple imputation for TICS >30 yielded a median of 7/655 (1.1%; interquartile range, 5-8) estimated dementia cases. After multiple imputation, the optimal cutoff score was ≤29, with mean sensitivity 65.4%, specificity 87.8%, positive predictive value 11.9%, negative predictive value 99.0%, and area under the curve 77.4% (95% CI, 56.3%-90.0%).
CONCLUSIONS In the present older population, the TICS performed well as a diagnostic screening instrument for excluding dementia and might be particularly useful when face-to-face diagnostic screening is not feasible in family practice or research settings. The potential reach to large numbers of people at low cost could contribute to more efficient medical management in primary care.
INTRODUCTION
In recent decades, increasing emphasis has been placed on early intervention in the care and treatment of persons with dementia to help provide more efficient medical management and advanced care planning.1,2 The family physician (FP) plays a pivotal role in detecting dementia. Despite growing attention for elder care in the Western world, dementia remains underdiagnosed in primary care.3 Recognizing dementia in primary care can be challenging, particularly with regard to lack of time and limited availability of simple diagnostic screening instruments.4–6 The identification of dementia in primary care might be facilitated by diagnostic instruments that are more time efficient and easier to perform, particularly in the setting of remote assessments.
Several cognitive tests, including the broadly used Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment test, are used for detecting cognitive disorders in the primary care setting.7–10 Disadvantages of these tests include limited suitability for illiterate or visually impaired patients, copyright protections, and requirement of a face-to-face assessment.11,12 The latter is sometimes challenging in large-scale research and in routine practice because in-person assessments are not always feasible, owing to long traveling distances, physical disabilities, or safety considerations such as in the current state of a global pandemic.13
Telephone-administered diagnostic screening tests might comprise easy and accurate alternatives in the diagnostic workup of dementia. The Telephone Interview for Cognitive Status (TICS) is the most broadly used and researched telephone-administered diagnostic screening instrument for detecting dementia.14,15 Diagnostic studies have shown TICS test characteristics to outperform other telephone-administered screening instruments such as the TELE and the Short Portable Mental Status Questionnaire.16–18 Previous studies have shown high test-retest and inter-rater reliability of the TICS, with high negative predictive values that are particularly useful for excluding dementia.19–23 Those studies mostly recruited participants with clinical signs of incipient dementia from memory clinics or retirement homes and reported test accuracies comparable to the MMSE.24–27 We investigated the validity of the TICS for identifying dementia in older persons recruited from family practices in a large population sample in the Netherlands.
METHODS
Study Design and Participants
Participants were from the Prevention of Dementia by Intensive Vascular Care (preDIVA) trial, which investigated the effect of multicomponent lifestyle intervention on incident dementia in community-dwelling older people from the Netherlands in primary care setting, during the period 2006-2015.28 Dementia at the end of the trial was meticulously assessed using in-person cognitive screening visits and extensive evaluation of electronic health records (EHRs). Four years after the end of the trial, all participants who were alive and without dementia at the end of the trial were invited to participate in the present follow-up study.
Assessment of the TICS
For those who agreed to participate, the validated Dutch version of the TICS was administered by a trained research nurse.24,28 The TICS comprises 11 items (range, 0-41 points) and assesses several cognitive domains including orientation, attention, anterograde episodic memory, language, praxis, and mathematical skills.15 The test was standardized for use in adults aged 60 to 98 years and usually took <10 minutes to complete.
Identification of Dementia
After TICS administration, we verified the dementia status for all participants with a score ≤30 using FP medical records. For feasibility reasons, medical records were only evaluated for a subsample of participants with a TICS score >30 from 2 randomly selected health centers, representing approximately one-quarter of all participants. The conservative threshold of ≤30 was selected on the basis of a validation study of the Dutch TICS translation, which found an optimal cutoff score of ≤26 for detecting dementia.24 Owing to an absence of false-negative TICS findings in the random subsample in the main analysis, we assessed that it would be unlikely for dementia diagnoses to be made within this score range for the remainder of the group participants.
In the Netherlands, FPs collect and maintain records of all medical reports pertaining to their registered patients (>98% of the Dutch population is registered). Medical records were scrutinized for potential dementia diagnoses as in the preDIVA trial.28 All data pertaining to dementia diagnoses, including mentions of cognitive symptoms or dementia by the FP, reports from hospital referrals, and outpatient clinic visits, were retrieved from the medical records. These data were subsequently presented for evaluation to the individual members of an adjudication committee comprising an FP and 2 neurologists, all specialists in dementia, blinded to TICS scores. Discrepancies regarding judgment on dementia diagnosis were resolved via discussion until a consensus was reached.28 As in the preDIVA trial, participants for whom a dementia diagnosis remained uncertain were classified as having no dementia.
Statistical Analysis
We compared population characteristics using Wilcoxon signed rank tests for nonnormally distributed variables and Student t tests for normally distributed variables. Educational level (categorized as <7 years, 7-12 years, and >12 years), presence of comorbidities (heart disease, stroke, type 2 diabetes, symptoms of apathy and depression [Geriatric Depression Scale]), and cognitive performance were compared between participants with TICS scores ≤30 and >30.
Test characteristics of the TICS data were investigated by calculating the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for a range of cutoff scores in the total population with a dementia status. For a sensitivity analysis, we assessed the potential influence of verification bias, which might occur in diagnostic studies when only a subset of subjects receives the reference standard confirmation of the diagnosis, depending on test results. For this purpose, we used multiple imputation (MI) of the dementia status of individuals with a TICS >30 who were not part of the random verification sample and whose dementia status was therefore unknown.29 Using the multiple imputation by Chained Equations package in R (the R Foundation), we performed a series of 100 imputations with 50 iterations based on preDIVA baseline age, sex, TICS score, and age at the time of the TICS measurement. These variables were selected as providing the greatest accuracy in repeated random subsampling cross-validation of the original data compared with several other models, which also included apolipoprotein E genotype, Mini-Mental State Examination score, Visual Association Test score, and subjective memory complaints (yes/no).30,31 After imputation, estimates of the 100 imputed data sets were combined using Rubin’s rules.32 Similar to the original data, test characteristics of the imputed data were investigated by calculating the sensitivity, specificity, PPV, and NPV for a range of cutoff scores. Optimal TICS cutoff scores were determined on the basis of the cutoff for which sensitivity and specificity were most balanced. We performed data collection, cleaning, and analyses using R version 3.6.2. Data are presented as mean (SD) or median (interquartile range) unless otherwise specified.
RESULTS
Of 1,473 individuals invited to participate in the study, 663 (45%) declined, and 810 (55%) agreed to participate (Figure 1). Participants were younger (mean age, 84.5 [2.4] years) than nonparticipants (mean age, 86.0 [2.4] years; P = .02). Participants also had more education (18.6% vs 10% with >12 years, respectively; P < .01), higher median MMSE scores at final pre-DIVA assessment (29.0 [28.0-30.0] vs 29.0 [27.3-30.0]; P < .01), and a lower incidence of stroke (5.7% vs 10.9%; P < .01) compared to nonparticipants.
Of the 810 participants, 155 (19.1%) had a TICS score ≤30 (mean, 28.0 [2.4]), and 655 (80.9%) had a TICS score >30 (mean, 34.5 [2.2]) (Figure 1). Dementia status was verified for 99.4% (154/155) of participants with a TICS score ≤30 and for 25.6% (168/655) of a random subsample of participants with a TICS score >30. The mean time between TICS assessment and verification using medical records was 5.9 months for participants with a TICS score ≤30 and 10.1 months for participants with a TICS score >30.
Table 1 summarizes the population characteristics of the participants from the verified data set. Participants with a TICS score ≤30 were older than those with a TICS score >30 at the time of TICS assessment (84.9 [2.4] vs 84.3 [2.5] years; P = .01), and fewer had >12 years of education (12.3% vs 20.2%; P < .001). Participants with a TICS score ≤30 also had lower median MMSE scores at the final pre-DIVA assessment (28 [27-29] vs 29 [28-30]; P < .001) and a greater incidence of stroke (9.0% vs 4.9%; P = .03) compared to participants with a TICS score >30. Of participants with a TICS score >30, none had received a diagnosis of dementia compared to 8.4% (13/154) of participants with a TICS score ≤30. Those with a dementia diagnosis had a median TICS score of 27 (23-28). Participants with a TICS score >30 and an unverified medical status were similar to participants with verified medical status in terms of age at TICS assessment, educational level, median MMSE score at final preDIVA assessment, comorbidities, and mean TICS score.
The test characteristics of the TICS, computed with the verified data and the imputed data, are summarized in Table 2. The optimal TICS cutoff value for the verified data was at a TICS score of ≤28, with a sensitivity of 76.9%, a specificity of 81.2%, a PPV of 14.7%, and an NPV of 98.8%. Figure 2 shows the receiver operating characteristic curve of the verified data, with an area under the curve of 90.3%.
Imputation Sensitivity Analysis
The sensitivity analysis using MI showed the imputed data to include a total of 20/810 (2.5%) modeled dementia cases, of whom 13/155 (8.4%) had a TICS score ≤30, and a median of 7 (5.0-8.0) cases had a TICS score >30 (1.1% [7/655]). The optimal TICS cutoff value of the imputed data was at ≤29, with a mean sensitivity of 65.4%, specificity of 87.8%, PPV of 11.9%, and NPV of 99.0% (Table 2). Sensitivity was lower for the imputed data for all cutoff values, whereas specificity was structurally higher compared with the verified data.
DISCUSSION
We investigated the validity of the TICS in an older population recruited from primary care. At an optimal cutoff value of 29, the TICS had a relatively low sensitivity, high specificity, and an almost optimal NPV, possibly owing to the relatively low prevalence of dementia (3.2%) in the study population. Verification bias appeared to cause overestimation of the TICS sensitivity; this overestimation was mitigated after multiple imputation. The estimated specificity of the TICS was not greatly affected by the verification bias and remained relatively stable after multiple imputation. Given the high NPV in this primary care population with low prevalence of dementia, the TICS appears to be a particularly useful diagnostic instrument for excluding dementia and could therefore be helpful as part of routine comprehensive geriatric assessment in primary care. In the case of a positive test result, the FP could be prompted to pursue further (face-to-face) investigation.
The TICS test characteristics in the present study are generally comparable to those in prior studies using the TICS, with similar or somewhat greater specificity and NPV compared to other telephone-administered screening measures.21–23 Depending on the study population and cutoff thresholds used, we report similar specificity (range, 80%-86%) but slightly lower sensitivity (range, 78%-98%) compared to prior studies investigating the diagnostic accuracy of the TICS.18–25 The vast majority of studies investigated the TICS in the context of memory clinics, including in mixed cohorts of healthy control subjects, persons with mild cognitive impairment, and patients with dementia,21–25 which might explain the low dementia prevalence in the present study (3.2%) compared to prior studies (9%-41%).21–25 The low incidence of dementia in our study might have been caused by respondents having relatively high cognitive performance, which was confirmed by the higher dementia prevalence rate among nonrespondents (8.9% [59/663]) on the basis of retrieval via FPs’ EHRs.
Two studies that did investigate the TICS in a population-based setting were quite different from one another in terms of study populations. One included community-dwelling older people taking part in a dementia screening program in one of the boroughs of New York City, and the other included individuals characterized by low levels of education and/or illiteracy in rural Greece.26,27 Both studies had greater numbers of dementia cases compared to our study (9% and 10.5%, respectively, vs 3.2%) and greater sensitivity and specificity of at least 80% at optimal cutoff scores of 28 and 25, respectively. Similar optimal cutoff values, ranging from 24 to 26, were reported in other studies, rendering our cutoff score of 29 relatively high.23,25,33 This might be caused, in part, by the fact that detected dementia cases in the present study population were categorized to the mild disease spectrum.
Strengths and Limitations
Strengths of the present study include the relatively large sample size of 810 participants compared to prior studies (range, 51-746),20–25,33 the older community-dwelling population for whom the TICS might offer a useful remote alternative, the blinded adjudication of outcomes including a 1-year follow-up after the diagnosis of dementia, the completeness of follow-up on all-cause dementia (98%), and complete verification of all cases with a relatively high TICS cutoff score (≤30). In addition, given that language comprehension is an important element of diagnostic screening in dementia, we used the Dutch validated version of the TICS.
The present study also has limitations. The first involves factors associated with the telephone administration of diagnostic screening such as the limited suitability for those with hearing problems, inadvertent use of external cues while answering questions (ie, using a calendar to answer what day it is), and receiving help from a partner and/or family member. In addition, because of the cognitive demands of using a telephone, the TICS might not be useful for detecting persons with moderate or severe dementia.
Second, selection bias might have occurred, given that almost one-half of the surviving preDIVA cohort declined participation in the TICS study. Therefore, the present results might not be generalizable to the older population as a whole. This was further substantiated by the fact that our study participants were relatively healthy, with generally lower baseline history of cardiac disease, stroke, and diabetes compared to nonparticipants. Our study population also had a higher mean TICS score (33.3 [3.4]) compared to prior studies (range, 21.0-29.5)20–25,33 and might therefore resemble a relatively cognitively unimpaired older population. Additional validation with a more representative older population is warranted, although it would not be expected to substantially change the high NPV, given the overall low dementia prevalence in primary care. Even with a doubled prevalence of dementia cases, the NPV would still be quite high, despite the relatively low sensitivity.
Third, we did not include participants with an uncertain dementia diagnosis in our analysis. This concerned 1.2% (10/810) of participants and was therefore unlikely to affect the overall results. Fourth, when screening for memory impairments in those with complaints in early phases of the disease, a diagnosis of dementia is often not yet formally made by FPs. Even if suspicion exists, this might lead to underestimation of the prevalence of dementia in EHRs.34 In contrast, this leads to high specificity and therefore high internal validity of diagnostic labels by FPs.35
Fifth, we screened a random sample of medical files for participants with a TICS score >30 to study the diagnostic accuracy of the TICS for dementia, which could have led to verification bias. We minimized the risk of verification bias by performing multiple imputation according to Rubin’s rules.32 Because the TICS status was verified in a subsample of participants from 2 randomly selected health centers for feasibility reasons, a slight risk of selection bias might have occurred. Differences in study results between the MI sensitivity analysis and the observed original data could be explained by training of the MI model on a data set with relatively few dementia cases. This might have caused greater uncertainty in the imputed model, owing to chance playing a larger role in the prediction of dementia outcome, causing a relatively large number of imputed cases compared to what would be expected with the original data.
The relatively steep changes in sensitivity and specificity from 25 to 30 points in the present study suggest that a slightly finer scale for this cognitive function range might be better suited to establish an optimal cutoff point, combining the greatest sensitivity and specificity. Future studies might explore these potential improvements, although they would necessitate much larger samples with many more dementia cases to be able to make strong inferences. This might also allow the TICS to better distinguish mild cognitive impairment from normal cognition, for which it is currently not well suited.26
Last, we did not notify FPs of TICS scores. Notification might have accelerated or advanced diagnostic workup toward a diagnosis of dementia in some cases with low values and evaluated as without dementia, leading to underestimation of both sensitivity and prevalence. However, there was a relatively long period to mean verification (average, 8.1 months), making it unlikely that outcomes of ongoing diagnostic trajectories were missed.
CONCLUSION
In an older population, we found the TICS to be a useful diagnostic screening instrument for excluding dementia and that it might be particularly useful in family practice or research settings when face-to-face screening is not feasible. The potential reach to large numbers of people at low cost might contribute to more efficient population management in primary care for older people at increased risk of cognitive decline and dementia.
Footnotes
Conflicts of interest: authors report none.
Author contributions: H.A.: evaluation of data, drafting of manuscript; E.J.: data collection, drafting of manuscript; M.H-B.: data management, concept and design, critical revision of manuscript; J.W.v.D.: data management, critical revision of manuscript; L.L.v.W.: data management, critical revision of manuscript; E.v.B.: drafting of manuscript, critical revision of manuscript; W.A.v.G.: concept and design, critical revision of manuscript; E.R.: concept and design, critical revision of manuscript; E.P.M.v.C.: concept and design, critical revision of manuscript
- Received for publication April 26, 2021.
- Revision received July 30, 2021.
- Accepted for publication August 17, 2021.
- © 2022 Annals of Family Medicine, Inc.