Leveraging Free-Form Comments to Assess and Improve Patient Satisfaction

INTRODUCTION

Health care organizations frequently use patient-reported satisfaction data to measure performance and identify opportunities to improve the quality of care.^1-3 Accurately assessing patient satisfaction has long vexed health care clincians.² Often clinicians attempt to measure satisfaction using free-form comments from patients.^4,5 Software tools, which streamline the assessment of comments, have been developed; however, these tools typically require users with advanced data analytics skill sets.⁶

A team of 3 operations management professors and 2 family medicine physicians collaborated to develop a system for clinicians to easily analyze large sets of patients comments and identify drivers of patient satisfaction. We employed text-analysis techniques to find recurring themes within patient narratives and to assess their relationship with satisfaction. Recurring themes were then used to build a spreadsheet-based comment analysis tool.

METHODS

Developing a Comment Analysis Tool

We leveraged the findings of the theme identification process to create the clinician Review Comment Analysis Tool using Microsoft Excel (Microsoft Corp) which allows users to analyze up to 5,000 comments. To use the tool, users simply paste in patient comments. The tool then automatically measures the prevalence of each theme’s key words. This is used to create a summary (Figure 1), that applies the odds ratios from the regression, to predict the likelihood that a patient will leave a positive review, as well as the relative prevalence for each theme.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Family Medicine Residency of Idaho comment analysis tool results.

RESULTS

The tool was tested using 721 patient comments collected during 2019 by the Family Medicine Residency of Idaho (FMRI). As shown in Figure 1, FMRI patients were more likely to leave a highly satisfied review than a less-satisfied review (by a ratio of 1.5 to 1). The finding that Theme 1 (Scheduling) was the highest-loading negative theme validated ongoing efforts to improve these processes. Feedback from the FMRI staff indicated they appreciated the ease with which the tool allowed them to quickly analyze comments and provide insights into areas of success and potential improvement.

DISCUSSION

The findings of this study can be used in several ways. First, by understanding which themes have the greatest influence on patient satisfaction, clinicians can monitor their interactions with patients to ensure that satisfaction is emphasized within their practices. Second, given the proliferation of online reviews, clinicians can read through comments to see which themes frequently appear. For larger sets of comment data, the spreadsheet tool, which will be provided free-of-charge to interested clinicians, gives users (without specialized skills) actionable measures of patient satisfaction.

A limitation of this study is that patient comments typically reflect how their expectations were met, which does not directly equate to the quality of care received.¹ Nevertheless, patient experiences and outcomes are not unrelated, as satisfaction may lead patients to be more involved in their care.² An understanding of patient satisfaction may also highlight systemic issues and identify underlying drivers of outcome-related underperformance.⁵ Thus, measuring how comments reflect the themes provides insights ranging from specific, addressable actions of clinicians to breakdowns of organizational processes. The tool we developed, however, represents only 1 input to a comprehensive quality program, as patient satisfaction should not be the sole focus of improvement efforts.³

The tool developed can also be leveraged by future researchers. Researchers can reduce vast sets of comment text into numerical data, suited for quantitative analyses, using the thematic scoring method. The tool also facilitates longitudinal analyses of patient feedback; for example, a follow-up to this effort is an investigation of how FMRI’s patient satisfaction was impacted by COVID-19. Additionally, the use of this tool for future research is suited for today’s family medicine environment, as the broad dimensions of patient experiences encompassed by the themes share commonality with aspects of patient-centered care.⁸