The Challenges of Measuring, Improving, and Reporting Quality in Primary Care

Process Complexity

Industrial QI approaches have improved health care delivery in mechanical linear domains, such as elective spine surgery,¹⁶ ventilator-associated pneumonia bundles,^17,18 and central line bundles,^19,20 where processes do not change appreciably for different patients. These processes have relatively few variables —5 for ventilator bundles and 8 for central lines—and it is reasonable to assume that measurable process steps are causally connected to patient-oriented outcomes. Such is not the case for primary care, which is more complex than other specialties (eg, cardiology, psychiatry, and obstetrics/gynecology), because so many more inputs and outputs are managed during each visit.^21,22

For example, a study of primary care physicians’ workflow calculated a mean of 37 tasks performed per visit in no discernable pattern within visit or physician, concluding “the number and sequence of tasks is varied and unpredictable.”²³ This complexity is problematic for certain QI techniques, which have a practical limit on the number of variables that can be reliably measured and controlled. Typically, if more than 20 variables are included in a process control algorithm, the impact of measurement inaccuracies, for instance, results in cumulative noise that makes distinguishing individual variable effects very difficult (Thomas Edgar, PhD, Chair, Department of Chemical Engineering, University of Texas at Austin, personal communication).

Process Standardization

Traditional QI assumes that the highest quality is achieved when a linear process occurs the same way each time.⁴ Primary care, however, depends on meeting a wide variety of patient needs. More than one-third of health problems initially encountered at a primary care center do not lend themselves to a diagnosis, and about one-half are unlikely to result in a definitive diagnosis that would trigger a standard care pathway.²⁴ Care of patients in the primary care setting must account for each patient’s comorbidities, disease severity, medication tolerance, beliefs, desires, and socioeconomic factors. Given the paucity of evidence, primary care physicians must often rely on creativity, problem solving, and adaptability to develop a custom care plan.

Process Controls

Input variables in processes, such as hospital central line bundles, are largely controlled by clinicians (washing hands, using chlorhexidine antiseptics, etc). In contrast, complex adaptive systems have no single point of control.⁵ Input variables for primary care processes and health outcomes are often beyond the control of the physician and patient,²⁵ such as socioeconomic factors²⁶ and poverty.²⁷ Some successes in hot spots of improving care for disadvantaged populations²⁸ may be generalizable, but the improved outcomes of these programs were more likely achieved by creating care plans adapted to the unique circumstances of high-risk patients. Additionally, for these approaches to spread more rapidly and consistently to other vulnerable populations, physician payment models must change.²⁹

Outcome Goal Clarity: Multimorbidity

A body of literature concludes that specialists are more likely than generalists to follow disease-specific guidelines, but this literature assumes one correct diagnostic or treatment option based on the organ of interest.³⁰ For example, if a patient with a history of a heart attack develops erectile dysfunction after starting a β-blocker, what is the correct measure of quality? The cardiac disease-specific answer would be to continue the β-blocker. The primary care physician addresses the human factors; a patient-centered discussion of risks and benefits might cause the patient to accept a small increased risk of sudden death in return for a viable sex life and therefore choose to discontinue the β-blocker. When primary care physicians deviated from single-disease guidelines, reasons were deemed medically appropriate in most cases.^31,32

Outcome Goal Clarity: Unique Patient Priorities

Traditional QI assumes that patients want and will do what experts say is best. In reality, however, human beings have a wide range of priorities that may not be aligned with those of their caregivers. Dr Iona Heath, former president of the Royal College of General Practitioners, commented on a reform proposal for the National Health Service by a British think tank in 2011, “… patients are presented as units of health need, indistinguishable but for their medical histories. They are assumed to be uniformly rational and committed to making decisions designed to promote their health ….”³³ She concluded, “I find no evidence of any understanding [by the think tank authors] that patients can also be abusive, manipulative, and self-destructive.” Although most patients are well meaning and cooperative, their personal goals, attitudes, and beliefs often confound guideline writers and physicians. Primary care physicians provide the most patient-centered care when they attempt to understand the meaning behind varying patient priorities and adjust diagnostic and treatment plans accordingly.

Goal Timing

In traditional QI metrics, there is little variance of the timing of intervention goals. For example, the Center for Medicare and Medicaid Services (CMS) core quality indicator set includes a hospital metric that aspirin will be prescribed to every patient with an acute myocardial infarction at hospital discharge.³⁴ In primary care, the best time frame to attempt to achieve an idealized goal (if ever) is more complex. A well-supported patient with new-onset diabetes might achieve and tolerate a hemoglobin A_1c of <7.0% within a matter of months. For another patient who recently experienced the loss of a spouse with a resulting change in diet and activity, a hemoglobin of A_1c <10% might represent a more reasonable goal until the social stressors improve.

Poor Risk-Adjustment Tools

Risk adjustment of summative quality scorecards is crucial to insure that primary care physicians and practices are evaluated fairly, because the characteristics of the associated patient population have a large effect on the reported results of standard QI metrics. Studies of these quality scorecards conclude that caring for complex patients in a safety-net setting strongly predicts failure to meet common quality goals in hospitals^37,38 and primary care settings.^39–41 The National Quality Forum has recognized that socioeconomic factors are important contributors to patient outcomes, that current measurements do not account for these factors, and that adequate risk adjustments for quality outcomes do not currently exist.⁴² QI leaders have discussed attempting risk adjustment for patient panels,⁴³ but rigorous methods are lacking, and existing models give different results.⁴⁴ As a result, current summative quality scorecards disadvantage physicians who care for the most vulnerable patient populations. Physicians who are more attentive to the social and cultural context, and who adjust care plans accordingly, are more likely to deliver positive patient outcomes,⁴⁵ though measuring such contextual care takes considerable effort.⁴⁶

Goal Target Number

There is a clash of expectations between the reality of primary care and traditional notions of industrial QI, such as Six Sigma targets, which imply adherence to process measures 99.99966% of the time.⁴⁷ Uptake of recommended tests or treatments is never 100% in the outpatient setting, even in populations with little socioeconomic deprivation. In vulnerable populations, patient preferences and daily practicalities often disrupt recommended ambulatory care. For example, in primary care, even though many patients say they are willing to undergo colon cancer screening when asked by their physicians,⁴⁸ uptake of recommended screening is low, often measured at less than 50% of eligible patients.^49,50 Even in primary care centers of excellence, chronic disease targets are met and sustained less than 50% of the time despite extra resources, such as health coaches.⁵¹ The impossibility of achieving 100% uptake makes it much more difficult to draw a summative conclusion about which primary care practices are providing high-quality care when contrasted against elective surgeries where nearly 100% compliance with preoperative antibiotic guidelines could reasonably be achieved.

Scorecard Comprehensiveness

The comprehensiveness of services offered by primary care physicians creates a tension between (1) the desire of funders of health care systems—such as Medicare and insurance companies—to be reassured with numerical data that participating primary care practices provide high-quality care, and (2) the practical limits, practice burdens, and dubious validity of data collection and reporting. On the one hand, a few well-chosen measures will represent only a fraction of the work delivered at primary care centers. CMS has just begun to evaluate patient-oriented outcome measures, and only for hip and knee replacement surgery.⁵² For a practice that mostly performs such surgeries, this metric may allow policy makers to determine whether the practice is performing well. A 2-disease measure would be inappropriate and inadequate for primary care physicians.²²

On the other hand, hundreds of measures become cumbersome and expensive to collect and report. Even worse is that these myriad measures will still be unlikely to capture the richness of the interactions between the primary care practice professionals and patients. Direct observational research of family physicians has shown that even when diagnoses are clustered into groups (eg, low back pain syndromes and headaches), 25 of these clusters account for only 60% of the diagnoses.⁵³ Family physicians have been observed to manage nearly 500 different diagnoses in 3,344 recorded patient encounters in a national study.²² Such a complex interaction of variables makes it unrealistic to represent a complex adaptive system with a finite number of dependent and independent variables.⁸ The cost to a practice is also burdensome. Just to maintain a PCMH certification has been estimated to cost $120,000 per physician per year.⁵⁴ Thus, given the current state of inadequate measurement in primary care, summative scorecards, such as a PCMH certification, which purport to distinguish “good” and “bad” primary care practices, are woefully inadequate at best and misleading at worst. Scorecards constructed using EHR data represent an ideal case as imagined by system designers that often include incomplete or inaccurate information and differ from the reality experienced by patients and physicians.⁵⁵

Shared-Decision Reporting

Future quantitative quality scorecards should borrow a concept from the British National Health Service QOF pay-for-performance system currently called exception reporting.⁶⁵ British general practitioners are given a list of reasons that allow them to remove individual patients from quality reports. Examples include patients new to their panel, patients who decline recommended tests or treatments, and patients with comorbidities that render a disease-specific quality metric inappropriate. Exception reporting may not be the best label; we concur with others who suggested alternative descriptions, such as patient choice or shared decisions.⁶⁶

Target Ranges Without Absolute Goals

Quality scorecards should never include metrics where 0% or 100% are the targets. Intermountain Health expects variance of 5% to 15% in some of its quality measures and scrutinizes physicians for complying with a protocol too much.⁶⁷ Targets should also be adjusted for socioeconomic status, though existing identifiable covariates have not proven to be very useful.

Measure When Physicians Do Not Order Tests or Treatments

Physicians should be measured and acknowledged when they do not subject their patients to interventions of dubious value. In traditional QI, process measures almost always dictate what should be done, not what should be avoided. The true value of primary care often occurs when physicians choose to not order a test, refer a patient to another physician, or start a new treatment.⁶⁸ The Choosing Wisely campaign provides a thoughtful framework to develop new metrics to reduce unnecessary tests and treatments.⁶⁹

Measure Other Aspects of Primary Care Capacity Associated With Better Outcomes

Other aspects of primary care are associated with better outcomes or lower costs and should be measured. Examples include the comprehensiveness of services provided by family physicians that is associated with lower Medicare costs and hospitalizations⁷⁰; increased physician-patient continuity associated with lower costs and fewer complications of common chronic conditions,⁷¹ fewer hospitalizations,⁷² and lower overall mortality⁷²; smaller primary care practice size associated with reduced hospitalizations from preventable conditions⁷⁴; and the rate of generic prescription writing associated with lower costs.⁷⁵ Other measures associated with lower total costs for larger primary care systems could include increased time for office visits for complex patients, 24/7 access to local clinic professionals, and careful selection of referral specialists.⁷⁶

A better measurement of a practice may be that it regularly undertakes self-reflection and measures its patient care, with regulators worrying less about exactly what each practice is attempting to measure and improve. Doing so would encourage the positive features of a complex adaptive system: self-organization, emergence, and coevolution with the practice’s environment.⁵⁵

De-emphasize Measures of Patient Satisfaction

Improved patient satisfaction scores have been linked to inappropriate services, such as prescribing unnecessary antibiotics.⁷⁷ A full review of the advantages and disadvantages of patient satisfaction scores is beyond the scope of this article, other than to note that patients’ goals and treatment preferences are not the same as patient satisfaction, and there is evidence that greater patient satisfaction scores are associated with worse outcomes, including higher mortality rates.^78–80 The enormous gaps in knowledge about the relevance of patient satisfaction scores to quality demand caution and further research in this area.

Measure Outcomes More Important to Patients

A lofty goal would be to measure more patient-centric outcomes at a population level, such as premature death rates (adjusted for age, comorbidities, and socioeconomic factors), disability (absence from work or school, limitation of activities, etc), ease of access, and some measures of patient experience of care (eg, respect, feeling heard, sensitivity, and treatment burden). Such measures may be more difficult to develop and implement, and they may often be statistically insignificant, even for a relatively large primary care practice, but they are likely to be more meaningful for achieving societal goals.^81,82

Peer-Led Reviews of Patterns of Care

Perhaps a better approach to reassure payers, regulators, and patients of high-quality primary care would be to replace most quantitative measures with adequately funded peer-reviewed assessments of patterns of care, as proposed recently in Scotland.⁶³ Peer-reviewed assessments would also be more consistent with evaluation principles of a complex adaptive system, where qualitative assessments of relationships between agents provides a more meaningful understanding of the quality of care provided than externally mandated numeric targets.^35,56

Final Thoughts

Family physicians’ comfort with complexity, ambiguity, and uncertainty—and their ongoing relationships with their patients—allow them to negotiate flexible diagnostic and treatment plans with their patients based on patient-specific risk factors and probabilities of serious disease.^68,84 At its heart, traditional QI assumes there is a definitive and measurable right answer in a given situation. In contrast, primary care physicians often deliver high-value care by doing the best they can with the patient care cards they are dealt., knowing that perfection will never be achieved. Adaptability rather than standardization should be the cornerstone of complex primary care and chronic disease care.⁸⁵ The national trends of rigid metrics and simplistic noncomprehensive scorecards must be reversed for primary care to do an even better job of delivering better patient care at a lower cost. At a minimum, measures that better respect the complexity and value of primary care would help promote the sustainable primary care workforce that is desperately needed.