Skip to main content

Main menu

  • Home
  • Current Issue
  • Content
    • Current Issue
    • Early Access
    • Multimedia
    • Podcast
    • Collections
    • Past Issues
    • Articles by Subject
    • Articles by Type
    • Supplements
    • Plain Language Summaries
    • Calls for Papers
  • Info for
    • Authors
    • Reviewers
    • Job Seekers
    • Media
  • About
    • Annals of Family Medicine
    • Editorial Staff & Boards
    • Sponsoring Organizations
    • Copyrights & Permissions
    • Announcements
  • Engage
    • Engage
    • e-Letters (Comments)
    • Subscribe
    • Podcast
    • E-mail Alerts
    • Journal Club
    • RSS
    • Annals Forum (Archive)
  • Contact
    • Contact Us
  • Careers

User menu

  • My alerts

Search

  • Advanced search
Annals of Family Medicine
  • My alerts
Annals of Family Medicine

Advanced Search

  • Home
  • Current Issue
  • Content
    • Current Issue
    • Early Access
    • Multimedia
    • Podcast
    • Collections
    • Past Issues
    • Articles by Subject
    • Articles by Type
    • Supplements
    • Plain Language Summaries
    • Calls for Papers
  • Info for
    • Authors
    • Reviewers
    • Job Seekers
    • Media
  • About
    • Annals of Family Medicine
    • Editorial Staff & Boards
    • Sponsoring Organizations
    • Copyrights & Permissions
    • Announcements
  • Engage
    • Engage
    • e-Letters (Comments)
    • Subscribe
    • Podcast
    • E-mail Alerts
    • Journal Club
    • RSS
    • Annals Forum (Archive)
  • Contact
    • Contact Us
  • Careers
  • Follow annalsfm on Twitter
  • Visit annalsfm on Facebook
Research ArticleOriginal Research

Clinical Quality Measure Exchange is Not Easy

Robert L. Phillips, Lars Peterson, Ted E. Palen, Scott A. Fields, Michael L. Parchman and John Johannides
The Annals of Family Medicine May 2021, 19 (3) 207-211; DOI: https://doi.org/10.1370/afm.2649
Robert L. Phillips
1American Board of Family Medicine, Lexington, Kentucky
MD, MSPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: bphillips@theabfm.org
Lars Peterson
1American Board of Family Medicine, Lexington, Kentucky
MD, PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ted E. Palen
2Colorado Permanente Medical Group, Institute for Health Research, Aurora, Colorado
PhD, MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Scott A. Fields
3OCHIN, Portland, Oregon
MD, MHA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael L. Parchman
4Kaiser Permanente of Washington Health Research Institute, Seattle, Washington
MD, MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
John Johannides
1American Board of Family Medicine, Lexington, Kentucky
PMP
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF
Loading

Abstract

PURPOSE The Trial of Aggregate Data Exchange for Maintenance of Certification and Raising Quality was a randomized controlled trial which first had to test whether quality reporting could be a by-product of clinical care. We report on the initial descriptive study of the capacity for and quality of exchange of whole-panel, standardized quality measures from health systems.

METHODS Family physicians were recruited from 4 health systems with mature quality measurement programs and agreed to submit standardized, physician-level quality measures for consenting physicians. Identified measure or transfer errors were captured and evaluated for root-cause problems.

RESULTS The health systems varied considerably by patient demographics and payer mix. From the 4 systems, 256 family physicians elected to participate. Of 19 measures negotiated for use, 5 were used by all systems. There were more than 15 types of identified errors including breaks in data delivery, changes in measures, and nonsensical measure results. Only 1 system had no identified errors.

CONCLUSIONS The secure transfer of standardized, physician-level quality measures from 4 health systems with mature measure processes proved difficult. There were many errors that required human intervention and manual repair, precluding full automation. This study reconfirms an important problem, namely, that despite widespread health information technology adoption and federal meaningful use policies, we remain far from goals to make clinical quality reporting a reliable by-product of care.

Key words
  • health information technology
  • quality measures
  • family physicians
  • quality indicators
  • health care
  • certification

INTRODUCTION

Family physicians provide nearly 20% of all clinical outpatient visits, nearly 200 million visits in the United States annually.1 Frontline clinicians continue to report failures of certified electronic health records (EHRs) to meet federal certification requirements and to meet electronic reporting needs, the latter of which is estimated to cost them $15 billion per year.2-11 Reducing reporting burden and enhancing the portability and utility of clinical data for quality improvement is an American Board of Family Medicine (ABFM) goal that is aligned with federal policy.12,13 In 2013, the ABFM began development of a Qualified Clinical Data Registry to assist practices whose EHRs were not serving their quality reporting needs and to reduce reporting burden. That registry, called PRIME, launched in 2016 and now actively extracts data from EHRs in more than 800 practices in 47 states (https://www.primeregistry.org).

Beyond reducing reporting burden, the ABFM aims to help practices easily access their own data for quality assessment and improvement and to reduce continuous certification burden. The ABFM recognized that many primary care physicians would not be able to connect directly to PRIME and that another passive data path was needed by which an EHR vendor, institution, or clinic system could send aggregated quality measures to the PRIME registry.12,14,15

The objective of this study was to assess whether quality measure reporting could be made a by-product of clinical care and quality improvement. We report here on the functional testing of quality measure exchange, that is to say, a test of the reliable exchange of whole-panel, quality measures from clinical networks. This component of the study was integral to a broader effort called the TRial of Aggregate Data Exchange for Maintenance of certification and Raising Quality (TRADEMaRQ).14,16,17 The study partners were 4 health systems with mature, established clinician quality-measurement and tracking systems. All 4 systems were willing to test ways to automate direct transmission of clinical measures.

METHODS

While TRADEMaRQ was a randomized, controlled trial, we report on the descriptive study component of the measure exchange testing experience.

Recruitment

Each of our clinical partners recruited family physicians in their networks or participating networks via face-to-face and e-mail solicitations. The ABFM provided study information in written form or via webinar, after which those physicians wanting to participate provided consent. Eligible participants were made aware that participation was voluntary and that the ABFM would not know their identity.

Measures and Data Sharing

Quality measures were calculated per each organization’s protocol and uploaded to the ABFM’s Secure File Transfer Protocol (SFTP) server. Measure exchange was designed to occur every 2 weeks.

Southeast Texas Medical Associates (SETMA) had an extensive history of reporting on quality metrics and publishes them, by clinician, on their website. The only additional work that was required was to format the results per the specifications of the study and to limit the results sent to family physicians participating in the study. They reported that quality control and verification of new files took approximately 4 to 6 hours and finalized files were placed into an automated job on the report server for reporting at 2-week intervals. Before each file set was submitted, they were given a brief inspection for correct formatting to make sure the data did not appear abnormal.

Kaiser Permanente Washington (KPWA) chose to report their clinical measures from their Clinical Improvement & Prevention department’s Healthcare Effectiveness Data and Information Set (HEDIS) measures, which are used both for monthly reporting and Medicare Star Rating calculations; however, KPWA outsourced these calculations to an National Committee for Quality Assurance-certified vendor who consumed KPWA data to produce annual HEDIS measures. KPWA also receives a “measure engine” from the vendor which allows KPWA to run HEDIS calculations themselves on a rolling basis.

Kaiser Permanente Colorado (KPCO) explained that although development of their reporting process was the most labor-intensive effort (350 hours), data processing and maintenance efforts (350 hours combined) were also substantial, resulting in an average 9 hours spent per data file transfer. KPCO developed a program specifically for this study to produce physician-level measures and it was run biweekly by a data specialist who also had to review and approve outputs.

OCHIN (a nonprofit health care innovation center focused on the needs of community health centers, small practices, and critical access hospitals) did not provide documentation about their process or related problems.

The study was approved by the American Academy of Family Physicians’ Institutional Review Board after all parties entered into an Office for Human Research Protections-approved federal-wide authority consortium agreement.

RESULTS

Enrollment and Participation

Across the 4 networks, there were 2,570 eligible family physicians, and 256 participated (Table 1). Participation ranged from 4.4% of OCHIN physicians to 100% of eligible SETMA physicians.

View this table:
  • View inline
  • View popup
Table 1.

TRADEMaRQ-Enrolled Physicians

Data and Transfers

The 4 partners agreed to 19 common clinical quality measures with our site co-principal investigators; across the 19 measures, however, initially only 4 were used by all health systems (Table 2) and only 5 were used by all systems by the end of the study. More than 15 examples of transmission or receipt errors were identified (Table 3). KPCO documented 17 different ways that their process was interrupted, failed, or had to be revised. Some types of errors, such as breaks in data delivery, changes in measure values, and nonsensical measure results were readily identifiable. Only SETMA had no identified errors—the only system to routinely list physician-level measures on their website for public viewing. KPWA noted 2 complications: (1) HEDIS measures change over time, both the measures required and the measure definitions; and, (2) an annual data validation process that typically delays data availability until the second quarter. Once the data were available from the vendor, KPWA wrote programs to automatically upload a physician-level measure file. KPWA staff noted, “we have internal dashboards with more complete and timely data than what is reported in HEDIS, and this drives our providers absolutely crazy because the two do not agree, and we as a medical group are being judged based on HEDIS measures, not on what we know is data that better reflects our true practice of medicine and delivery of care. But no one is willing to stop producing our internal dashboards because those are used to actually implement changes that improve quality of care, not the HEDIS measures.” This revelation helped explain complaints we were receiving from KPWA physicians who noted that their quality measures in their certification portfolio did not reflect their internal dashboard measures.

View this table:
  • View inline
  • View popup
Table 2.

TRADEMaRQ Measures across Participating Health Systems

View this table:
  • View inline
  • View popup
Table 3.

Common Measure Delivery or Receipt Errors and Hazards Experienced in TRADEMaRQ

DISCUSSION

This paper reports on a test of exchange of whole-panel, standardized quality measures from clinical networks and health systems with mature internal quality measure processes, all of whom reported to HEDIS and Centers for Medicare & Medicaid Services (CMS). Of 19 measures that the 4 agreed were common and standard, only 5 measures were used by all 4. Dozens of identified errors or hazards in measure calculation, data management, data delivery, and physician inclusion occurred. These problems often required human intervention and manual repair, precluding fully automated file transfers. In the end, there was a steady stream of reported data and most of the enrolled physicians were able to interact with them on the secure ABFM website. We conclude that it was really a test of system-level breakdown in measure definition, process, capture, and transmission rather than about the particulars of individual clinician quality; it is a symptom of the bigger system problem.

The “passive” data reporting path that TRADEMaRQ sought to test was related to the intentions of the Health Information Technology for Economic and Clinical Health Act.18 Specifically, it promoted the Office of the National Coordinator for Health Information Technology (ONC) Direct Project which aimed to create a simple, secure, scalable, standards-based way to transmit health information from a sender to a trusted recipient over the Internet.19 Direct Project priority 2 specifically mentions reporting of quality measures to a variety of potential recipients. The TRADE-MaRQ aims also aligned with the ONC Nationwide Health Information Exchange intentions, namely to support secure health information exchange for measuring and improving quality tied to incentives for improvement and required reporting.20,21 This study reconfirms an important problem, namely, that despite continued growth in health information technology adoption and calls for interoperability, most clinical practices struggle to produce standard quality measures from their clinical data and to meet a growing number of reporting needs.22 Several large-scale, federal demonstration and research efforts have run headlong into this problem and are struggling to get measure data out of participating practices.11 Although many clinicians are now reporting to the federal Quality Payment Program and hardship exceptions are made for small practices, they are at increasing risk of being left behind because NCQA’s patient-centered medical home recognition program is moving to annual electronic measure reporting, and many public health registries and even some payers are also requiring electronic measure submission.5 Quality reporting is unlikely to disappear but the reliability of what is reported is unclear, meaningfulness uncertain, and the cost high.6,7

A major limitation of our study is that the problems identified may not generalize to other health systems.

CONCLUSION

TRADEMaRQ and other recent studies suggest that we remain far from clinical data liquidity goals of federal agencies. The ABFM’s goals to help physicians make reporting a by-product of care and ongoing quality improvement, and reduce physicians’ reporting burdens, remain aspirations. While small and independent practices struggle with turning their EHR data into reportable measures, large health systems also struggle with managing measure definitions, internal vs external quality processes, and costly processes for managing reporting.

Acknowledgments

The authors gratefully acknowledge the assistance of Stanley Xu, PhD, Head of Biostatistics and a Senior Investigator at the Kaiser Permanente Institute for Health Research.

Footnotes

  • Conflicts of interest: authors report none.

  • To read or post commentaries in response to this article, go to https://www.AnnFamMed.org/content/19/3/207/tab-e-letters.

  • Funding support: This project was supported by grant number R21HS022583 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

  • Supplemental materials: Available at https://www.AnnFamMed.org/content/19/3/207/suppl/DC1/

  • Received for publication October 6, 2019.
  • Revision received August 13, 2020.
  • Accepted for publication August 19, 2020.
  • © 2021 Annals of Family Medicine, Inc.

References

  1. 1.↵
    1. Centers for Disease Control and Prevention
    . NAMCS Summary Tables. Published 2012. https://www.cdc.gov/nchs/data/ahcd/namcs_summary/2012_namcs_web_tables.pdf
  2. 2.↵
    1. Centers for Medicare & Medicaid Services
    . Medicare program; Merit-Based Incentive Payment System (MIPS) and Alternative Payment Model (APM) incentive under the physician fee schedule, and criteria for physician-focused payment models. Fed Regist. 2016; 81: 77008–77831.
    OpenUrl
  3. 3.
    1. Centers for Medicare & Medicaid Services
    . Transforming clinical practice initiative. Published 2016. Accessed 2016. https://innovation.cms.gov/initiatives/Transforming-Clinical-Practices/
  4. 4.
    1. Agency for Healthcare Research and Quality
    . About EvidenceNOW: background and stories from the field. Published 2016. Accessed Feb 8, 2017. https://www.ahrq.gov/evidencenow/about/index.html
  5. 5.↵
    1. NCQA
    . Recognition redesign annual reporting. Published 2017. Accessed Jan 14, 2018. https://www.ncqa.org/programs/recognition/practices/patient-centered-medical-home-pcmh/pcmh-redesign/annual-reporting
  6. 6.↵
    1. Murray K,
    2. Hilligoss B,
    3. Hefner J, et al
    . The quality reporting reality at a large Academic Medical Center: reporting 1600 unique measures to 49 different sources. Int J Acad Med. 2017; 3(1): 10–15.
    OpenUrl
  7. 7.↵
    1. Casalino LP,
    2. Gans D,
    3. Weber R, et al.
    US physician practices spend more than $15.4 billion annually to report quality measures. Health Aff (Millwood). 2016; 35(3): 401–406.
    OpenUrlAbstract/FREE Full Text
  8. 8.
    1. Xierali IM,
    2. Hsiao C-J,
    3. Puffer JC, et al.
    The rise of electronic health record adoption among family physicians. Ann Fam Med. 2013; 11(1): 14–19.
    OpenUrlAbstract/FREE Full Text
  9. 9.
    1. Hsiao C-J,
    2. Decker SL,
    3. Hing E,
    4. Sisk JE.
    Most physicians were eligible for federal incentives in 2011, but few had EHR systems that met meaningful-use criteria. Health Aff (Millwood). 2012; 31(5): 1100–1107.
    OpenUrlAbstract/FREE Full Text
  10. 10.
    1. Peterson LE,
    2. Blackburn B,
    3. Ivins D,
    4. Mitchell J,
    5. Matson C,
    6. Phillips RL Jr..
    Do family physicians electronic health records support meaningful use? Health Care (Don Mills). 2015; 3(1): 38–42.
    OpenUrl
  11. 11.↵
    1. Cohen DJ,
    2. Dorr DA,
    3. Knierim K, et al
    . Primary care practices’ abilities and challenges in using electronic health record data for quality improvement. Health Aff (Millwood). 2018; 37(4): 635–643.
    OpenUrlCrossRef
  12. 12.↵
    1. Phillips R,
    2. Kennedy J.
    3. Jaén C,
    4. Stelter K,
    5. Puffer J.
    Transforming physician certification to support physician self-motivation and capacity to improve quality and safety. Journal of Enterprise Transformation. 2016; 6(3-4): 162–169.
    OpenUrl
  13. 13.↵
    1. Washington V,
    2. DeSalvo K,
    3. Mostashari F,
    4. Blumenthal D.
    The HITECH era and the path forward. N Engl J Med. 2017; 377(10): 904–906.
    OpenUrl
  14. 14.↵
    1. Phillips R.
    ABFM to simplify maintenance of certification (MOC) for family physicians and make it more meaningful: a family medicine registry. J Am Board Fam Med. 2015; 28(3): 431–433.
    OpenUrlFREE Full Text
  15. 15.↵
    1. Cook DA,
    2. Holmboe ES,
    3. Sorensen KJ,
    4. Berger RA,
    5. Wilkinson JM.
    Getting maintenance of certification to work: a grounded theory study of physicians’ perceptions. JAMA Intern Med. 2015; 175(1): 35–42.
    OpenUrl
  16. 16.↵
    1. DeMaio S.
    Simplifying MOC for famly physicians: the TRADE-MaRQ study. NEJM Knowledge+. Published Mar 26, 2015. https://knowledgeplus.nejm.org/blog/simplifying-moc-for-family-physicians-the-trademarq-study/
  17. 17.↵
    1. Tachibana C.
    How can we streamline board certification for family physicians? Kaiser Permanente Washington. Published Feb 5, 2015. Accessed Apr 12, 2108. https://www.kpwashingtonresearch.org/news-and-events/recent-news/news-2015/how-can-we-streamline-board-certification-family-physicians
  18. 18.↵
    1. Anderson KM,
    2. Marsh CA,
    3. Flemming AC,
    4. Isenstein H,
    5. Reynolds J.
    Quality measurement enabled by health IT: overview, possibilities, and challenges. Published Jul 2012. Accessed Feb 9, 2013. https://healthit.ahrq.gov/portal/server.pt/community/ahrq-funded_projects/654/health_it-enabled_quality_measurement/30886
  19. 19.↵
    DIRECT Project Overview. Published Oct 11, 2010. Accessed Feb 9, 2013. https://www.oregon.gov/oha/HPA/OHIT-HITOC/Documents/NHINDirectProjectOverviewWeb.pdf
  20. 20.↵
    1. Nationale Health Collaborative
    . EXCHANGE ROADMAP: the landscape and a path forward. Published 2012. Accessed Feb 9, 2013. https://www.nationalehealth.org/hie-roadmap
  21. 21.↵
    1. Chassin MR,
    2. Loeb JM,
    3. Schmaltz SP,
    4. Wachter RM.
    Accountability measures — using measurement to promote quality improvement. N Engl J Med. 2010; 363(7): 683–688.
    OpenUrlCrossRefPubMed
  22. 22.↵
    1. Krist AH,
    2. Beasley JW,
    3. Crosson JC, et al
    . Electronic health record functionality needed to better support primary care. J Am Med Inform Assoc. 2014; 21(5): 764–771.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top

In this issue

The Annals of Family Medicine: 19 (3)
The Annals of Family Medicine: 19 (3)
Vol. 19, Issue 3
1 May 2021
  • Table of Contents
  • Index by author
  • Back Matter (PDF)
  • Front Matter (PDF)
  • The Issue In Brief
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on Annals of Family Medicine.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Clinical Quality Measure Exchange is Not Easy
(Your Name) has sent you a message from Annals of Family Medicine
(Your Name) thought you would like to see the Annals of Family Medicine web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
6 + 4 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Clinical Quality Measure Exchange is Not Easy
Robert L. Phillips, Lars Peterson, Ted E. Palen, Scott A. Fields, Michael L. Parchman, John Johannides
The Annals of Family Medicine May 2021, 19 (3) 207-211; DOI: 10.1370/afm.2649

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Get Permissions
Share
Clinical Quality Measure Exchange is Not Easy
Robert L. Phillips, Lars Peterson, Ted E. Palen, Scott A. Fields, Michael L. Parchman, John Johannides
The Annals of Family Medicine May 2021, 19 (3) 207-211; DOI: 10.1370/afm.2649
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • CONCLUSION
    • Acknowledgments
    • Footnotes
    • References
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF

Related Articles

  • PubMed
  • Google Scholar

Cited By...

  • Physicians Choice of Board Certification Activity Is Unaffected by Baseline Quality of Care: The TRADEMaRQ Study
  • Google Scholar

More in this TOC Section

  • Adverse Outcomes Associated With Inhaled Corticosteroid Use in Individuals With Chronic Obstructive Pulmonary Disease
  • Family-Based Interventions to Promote Weight Management in Adults: Results From a Cluster Randomized Controlled Trial in India
  • Teamwork Among Primary Care Staff to Achieve Regular Follow-Up of Chronic Patients
Show more Original Research

Similar Articles

Subjects

  • Methods:
    • Cross-disciplinary
  • Other research types:
    • Health services
  • Other topics:
    • Health informatics
    • Organizational / practice change

Keywords

  • health information technology
  • quality measures
  • family physicians
  • quality indicators
  • health care
  • certification

Content

  • Current Issue
  • Past Issues
  • Early Access
  • Plain-Language Summaries
  • Multimedia
  • Podcast
  • Articles by Type
  • Articles by Subject
  • Supplements
  • Calls for Papers

Info for

  • Authors
  • Reviewers
  • Job Seekers
  • Media

Engage

  • E-mail Alerts
  • e-Letters (Comments)
  • RSS
  • Journal Club
  • Submit a Manuscript
  • Subscribe
  • Family Medicine Careers

About

  • About Us
  • Editorial Board & Staff
  • Sponsoring Organizations
  • Copyrights & Permissions
  • Contact Us
  • eLetter/Comments Policy

© 2025 Annals of Family Medicine