Abstract
Context eConsult is an online service through which primary care practitioners (PCP) submit patients’ cases to specialists and receive a recommendation. eConsult reduces patients’ unnecessary in-person visits and improves timely access to specialists’ advice.
Objective We explored the extent of reusability of past specialists’ responses as answers to current and future PCP questions.
Study Design and Analysis We used natural language processing and machine learning (ML) experimentations to cluster the most similar questions. We then automatically built an answer bank through summarization of the associated past specialists’ answers to the topmost similar PCP questions. This machine generated answer was then presented to enable PCPs to proactively refine their question prior to submitting it to the specialist, if still needed.
Setting or Dataset We randomly selected a sample of 3000 PCP questions and the associated answers from eConsult, submitted during 2020.
Population Studied The population is an unbiased random sample of patient cases for whom PCP asked a question from the specialist.
Intervention/Instrument The online software system is the means of intervention-instrumentation to provide care more efficiently and effectively rather than requiring in-person visits for all.
Outcome Measures We used ROUGE to measure the accuracy of the correspondence among the terms in the generated response and those in the actual specialist response to evaluate the results.
Results We aggregated past specialists’ answers associated with the similar questions to automatically generate relevant answers and evaluated their quality against actuals. We summarized the associated answers to the topmost 5 similar questions and compared them with the actual answers. The top ROUGE measures were 19% and 12% for Neurology-Migraine and Pediatrics respectively.
Conclusion The combination of clustering and summarization of the past conversations can play an important role in monitoring, maintenance and re-usability of medical knowledge. There is still an identifiable gap between the machine-generated medical answers and the actual answers using eConsult’s historical data as the diversity and variability of the new questions are still higher than what the summary of aggregation of answers to those in the past can sufficiently provide and be specific enough to fully qualify to replace human answers.
- © 2023 Annals of Family Medicine, Inc.