Using Machine Learning to Predict Primary Care and Advance Workforce Research

Peter Wingrove; Winston Liaw; Jeremy Weiss; Stephen Petterson; John Maier; Andrew Bazemore

doi:10.1370/afm.2550

Article Figures & Data

Tables

Additional Files

View popup

Table 1

Prescriptions and Procedures and Comparison of Train and Test Data Sets, by Specialty

Specialty	Drugs Prescribed, Mean No.	Procedure Codes, Mean No.	Train, n (%)	Test, n (%)
Allergy/immunology	12.9	8.7	1,611 (0.6)	1,625 (0.6)
Anesthesiology	16.4	7.4	16,087 (5.7)	16,110 (5.7)
Cardiology	38.1	21.3	11,465 (4.1)	11,170 (4.0)
Dermatology	12.8	17.3	5,609 (2.0)	5,498 (1.9)
Emergency medicine	8.6	5.5	18,689 (6.6)	18,663 (6.6)
Endocrinology	31.7	9.7	2,376 (0.8)	2,497 (0.9)
Gastroenterology	16.2	13.2	5,999 (2.1)	5,960 (2.1)
Hematology-oncology	19.2	18.5	5,638 (2.0)	5,572 (2.0)
Infectious disease	21.8	7.0	2,337 (0.8)	2,328 (0.8)
Nephrology	36.6	13.3	3,735 (1.3)	3,691 (1.3)
Neurology	27.4	9.4	6,431 (2.3)	6,217 (2.2)
Neurosurgery	5.1	8.8	1,976 (0.7)	2,008 (0.7)
Obstetrics and gynecology	5.4	4.6	11,361 (4.0)	11,505 (4.1)
Ophthalmology	13.7	13.2	8,837 (3.1)	8,755 (3.1)
Orthopedic surgery	5.9	13.5	10,980 (3.9)	11,095 (3.9)
Otolaryngology	9.9	10.5	4,322 (1.5)	4,262 (1.5)
Pathology	5.6	10.2	4,682 (1.7)	4,831 (1.7)
Physical medicine and rehabilitation	14.5	10.6	3,610 (1.3)	3,438 (1.2)
Plastic surgery	3.1	5.7	1,864 (0.7)	1,795 (0.6)
Primary care	61.4	11.7	100,682 (35.6)	101,498 (35.9)
Psychiatry	20.9	4.0	15,075 (5.3)	14,974 (5.3)
Pulmonology	22.8	12.7	5,282 (1.9)	5,395 (1.9)
Radiation oncology	3.5	14.5	1,926 (0.7)	1,903 (0.7)
Radiology	4.3	35.7	11,840 (4.2)	11,816 (4.2)
Rheumatology	33.4	15.1	1,975 (0.7)	2,030 (0.7)
Surgery	5.1	9.2	13,536 (4.8)	13,278 (4.7)
Urology	17.3	20.1	4,568 (1.6)	4,579 (1.6)
Total			282,493 (100)	282,493 (100)

Note: Prescribing data are from the 2014-2016 Centers for Medicare and Medicaid Services (CMS) Medicare Fee-For-Service Provider Utilization and Payment Data: Part D Prescriber Public Use Files.19 Procedure data are from 2014-2016 CMS Medicare Fee-For-Service Provider Utilization and Payment Data: Physician and Other Supplier Public Use Files.²⁰

View popup

Table 2

F1 Scores for Random Forests, by Specialty and Type of Training Data

Specialty	F1 Score, Mean (95% CI)
Specialty	Combined	Prescription Only	Procedure Only
Allergy/immunology	0.912	0.860^a	0.903^a
Allergy/immunology	(0.906-0.917)	(0.855-0.865)	(0.901-0.905)
Anesthesiology	0.951	0.624^a	0.963^a
Anesthesiology	(0.950-0.952)	(0.615-0.632)	(0.962-0.964)
Cardiology	0.938	0.917^a	0.929^a
Cardiology	(0.935-0.940)	(0.913-0.919)	(0.927-0.932)
Dermatology	0.966	0.940^a	0.964^b
Dermatology	(0.964-0.968)	(0.937-0.943)	(0.963-0.965)
Emergency medicine	0.897	0.716^a	0.914^a
Emergency medicine	(0.894-0.899)	(0.708-0.723)	(0.911-0.916)
Endocrinology	0.865	0.869^a	0.623^a
Endocrinology	(0.858-0.871)	(0.863-0.875)	(0.615-0.630)
Gastroenterology	0.923	0.901^a	0.900^a
Gastroenterology	(0.922-0.924)	(0.897-0.905)	(0.897-0.903)
Hematology-oncology	0.874	0.872	0.677^a
Hematology-oncology	(0.872-0.876)	(0.869-0.876)	(0.673-0.680)
Infectious disease	0.745	0.758^a	0.474^a
Infectious disease	(0.740-0.750)	(0.754-0.763)	(0.462-0.486)
Nephrology	0.885	0.866^a	0.882^a
Nephrology	(0.882-0.887)	(0.864-0.868)	(0.879-0.884)
Neurology	0.885	0.895^a	0.732^a
Neurology	(0.883-0.887)	(0.892-0.897)	(0.725-0.739)
Neurosurgery	0.650	0.377^a	0.631^a
Neurosurgery	(0.645-0.656)	(0.364-0.389)	(0.626-0.635)
Obstetrics and gynecology	0.920	0.928^a	0.868^a
Obstetrics and gynecology	(0.917-0.923)	(0.927-0.929)	(0.865-0.870)
Ophthalmology	0.982	0.975^a	0.987^a
Ophthalmology	(0.982-0.982)	(0.974-0.976)	(0.987-0.987)
Orthopedic surgery	0.884	0.760^a	0.901^a
Orthopedic surgery	(0.880-0.888)	(0.756-0.765)	(0.900-0.903)
Otolaryngology	0.932	0.874^a	0.951^a
Otolaryngology	(0.927-0.937)	(0.868-0.880)	(0.949-0.952)
Pathology	0.987	0.005^a	0.990^a
Pathology	(0.986-0.987)	(0-0.012)	(0.990-0.990)
Physical medicine and rehabilitation	0.586	0.380^a	0.492^a
Physical medicine and rehabilitation	(0.583-0.589)	(0.374-0.387)	(0.483-0.500)
Plastic surgery	0.533	0.314^a	0.383^a
Plastic surgery	(0.527-0.539)	(0.287-0.341)	(0.378-0.388)
Primary care	0.920	0.911^a	0.878^a
Primary care	(0.917-0.922)	(0.910-0.912)	(0.876-0.880)
Psychiatry	0.930	0.938^a	0.740^a
Psychiatry	(0.929-0.932)	(0.936-0.940)	(0.734-0.745)
Pulmonology	0.836	0.843^a	0.818^a
Pulmonology	(0.834-0.837)	(0.839-0.848)	(0.814-0.822)
Radiation oncology	0.939	0.691^a	0.976^a
Radiation oncology	(0.933-0.945)	(0.680-0.702)	(0.975-0.977)
Radiology	0.979	0.275^a	0.984^a
Radiology	(0.977-0.980)	(0.272-0.279)	(0.983-0.985)
Rheumatology	0.916	0.916	0.726^a
Rheumatology	(0.913-0.919)	(0.913-0.918)	(0.719-0.733)
Surgery	0.774	0.624^a	0.735^a
Surgery	(0.767-0.781)	(0.614-0.634)	(0.731-0.740)
Urology	0.962	0.950^a	0.962
Urology	(0.958-0.965)	(0.947-0.953)	(0.961-0.963)
Macro F1	0.876	0.745^a	0.821^a
Macro F1	(0.874-0.878)	(0.741-0.748)	(0.820-0.823)

Note: Prescribing data are from 2014-2016 Centers for Medicare and Medicaid Services (CMS) Medicare Fee-For-Service Provider Utilization and Payment Data: Part D Prescriber Public Use Files.19 Procedure data from 2014-2016 CMS Medicare Fee-For-Service Provider Utilization and Payment Data: Physician and Other Supplier Public Use Files.²⁰
↵a P <.01 (paired t test vs combined).
↵b P <.05 (paired t test vs combined).

View popup

Table 3

Predicted vs Actual Counts of Physicians, by Specialty, for Combined Random Forest Models

Specialty	Predicted	Actual	Predicted/Actual, %
Allergy/immunology	1,590	1,625	97.8
Anesthesiology	16,100	16,110	99.9
Cardiology	10,790	11,170	96.6
Dermatology	5,569	5,498	101.3
Emergency medicine	17,886	18,663	95.8
Endocrinology	2,330	2,497	93.3
Gastroenterology	6,064	5,960	101.7
Hematology-oncology	5,468	5,572	98.1
Infectious disease	1,704	2,328	73.2
Nephrology	3,702	3,691	100.3
Neurology	5,747	6,217	92.4
Neurosurgery	1,342	2,008	66.8
Obstetrics and gynecology	11,425	11,505	99.3
Ophthalmology	8,758	8,755	100.0
Orthopedic surgery	11,008	11,095	99.2
Otolaryngology	4,021	4,262	94.3
Pathology	4,790	4,831	99.2
Physical medicine and rehabilitation	2,154	3,438	62.7
Plastic surgery	1,419	1,795	79.1
Primary care	105,225	101,498	103.7
Psychiatry	14,912	14,974	99.6
Pulmonology	5,525	5,395	102.4
Radiation oncology	1,881	1,903	98.8
Radiology	11,547	11,816	97.7
Rheumatology	2,086	2,030	102.8
Surgery	14,949	13,278	112.6
Urology	4,498	4,579	98.2

Note: The predicted counts are based on the combined models and averaged all 9 sets of predictions. The actual counts are the number of physicians by specialty in the Test data set. Values are rounded to the nearest integer.

View popup

Table 4

Model Agreement and Specialty Match Using 2016 Data

Specialty	Count	Models Predicting the Same Specialty, %	Specialty Match, %^a	Specialty Mismatch, %^b
Allergy/immunology	1,625	97.1	89.6	7.5
Anesthesiology	16,110	97.9	94.3	3.6
Cardiology	11,170	96.9	90.4	6.5
Dermatology	5,498	98.8	96.7	2.1
Emergency medicine	18,663	98.3	87.0	11.3
Endocrinology	2,497	95.8	83.3	12.5
Gastroenterology	5,960	97.2	92.4	4.8
Hematology-oncology	5,572	94.9	84.9	10.0
Infectious disease	2,328	91.1	61.2	29.9
Nephrology	3,691	96.7	86.9	9.8
Neurology	6,217	94.5	83.1	11.4
Neurosurgery	2,008	80.6	48.3	32.3
Obstetrics and gynecology	11,505	96.7	90.6	6.1
Ophthalmology	8,755	99.1	97.9	1.2
Orthopedic surgery	11,095	94.6	86.1	8.5
Otolaryngology	4,262	96.8	89.5	7.3
Pathology	4,831	99.3	97.8	1.5
Physical medicine and rehabilitation	3,438	83.2	41.6	41.6
Plastic surgery	1,795	80.7	42.2	38.5
Primary care	101,498	98.3	92.6	5.7
Psychiatry	14,974	97.9	92.1	5.8
Pulmonology	5,395	96.1	83.2	12.9
Radiation Oncology	1,903	95.9	91.0	4.9
Radiology	11,816	99.1	96.4	2.7
Rheumatology	2,030	97.6	91.7	5.9
Surgery	13,278	91.7	77.7	14.0
Urology	4,579	97.3	94.5	2.8
Overall	282,493	97.0c	89.4^c	7.6^c

For this analysis, we applied the 2014, 2015, and 2016 combined random forests to 2016 Test data, for a total of 3 predictions based on prescribing and procedure data for a single year. Model agreement is defined as all 3 models predicting the same specialty.
↵a All 3 models predicted the self-reported specialty.
↵b All 3 models predicted a specialty that differed from the self-reported category.
↵c Mean across all specialties weighted by number in each specialty.