Development, Scoring, and Test-Retest Reliability of a New Tool to Assess Primary Care Practice Performance

Context: Multiple tools to assess primary care quality exist but none have been rigorously developed and validated.

Objective: To develop and establish reliability of a tool to assess primary care structures (e.g., care team makeup) and processes (e.g., care coordination) that are associated with better quality and patient outcomes.

Study Design and Analysis: We generated a list of 314 candidate items for the Tool for Advancing Practice Performance (TAPP) based on data from a scoping review, Delphi study, and qualitative interviews with high-performing primary care practices. We refined this list using criteria for removing/retaining items and expert review for redundancy and clarity. We developed the tool and revised it based on feedback from cognitive interviews and pilot testing with primary care providers and staff (e.g., lead clinician, practice manager). The final tool was administered to 401 primary care practices from December 2021 to March 2022. We considered three scoring methods: two different unidimensional factor analysis models and a simple total summed score. A subset of practices completed the tool a second time; these data were used to assess test-retest reliability by calculating an intraclass correlation and Cohen’s Weighted Kappa.

Setting or Dataset: United States Population Studied: Primary care practices varying in size, ownership type, and patient population served Intervention/Instrument: TAPP Outcome Measures: TAPP scores Results: After expert review, 8 cognitive interviews, and 15 pilot tests, the tool consisted of 113 items organized into 8 domains related to primary care practice (e.g., reduce clinical risk factors, expand access to care). The final tool was completed by 259 practices across 26 states (64.6% response rate). There was moderate to high internal consistency within most domains. The scores from the factor analysis models and the simple total summed score were highly correlated (r>0.97). For test-retest reliability, we found moderate to high reliability for the 8 domains and high reliability for the simple total summed score across the 51 practices that completed a second survey.

Conclusions: The TAPP is a novel, low-burden tool consisting of 113 items to assess primary care structures and processes. Since the simple total summed score was highly correlated with the more complex factor scores, it may be a suitable replacement and thus, we recommend a total score to those who wish to use this tool.