Annals of Family Medicine
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

Annals of Family Medicine 5:361-367 (2007)
© 2007 Annals of Family Medicine, Inc.
doi: 10.1370/afm.703
This Article
Right arrow Abstract
Right arrow Full Text (PDF)
Right arrow In Brief
Right arrow TRACK Discussion: Submit a Comment
Right arrow TRACK Discussion: View Comments
Right arrow Alert me when this article is cited
Right arrow Alert me when TRACK Comments are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Forrest, C. B.
Right arrow Articles by Starfield, B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Forrest, C. B.
Right arrow Articles by Starfield, B.

Specialty Referral Completion Among Primary Care Patients: Results From the ASPN Referral Study

Christopher B. Forrest, MD, PhD1, Efrat Shadmi, PhD, RN2, Paul A. Nutting, MD, MSPH3 and Barbara Starfield, MD, MPH4

1 Department of Pediatrics, Children’s Hospital of Philadelphia, University of Pennsyl-vania School of Medicine, Philadelphia, Pa
2 Department of Nursing, Faculty of Social Welfare and Health Studies, University of Haifa, Haifa, Israel
3 Center for Research Strategies and the Department of Family Medicine, University of Colorado, Denver, Colo
4 Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Md

CORRESPONDING AUTHOR: Christopher B. Forrest, MD, PhD, Abramson Research Center 1335, Children’s Hospital of Philadelphia, 3400 Civic Center Blvd, Philadelphia, PA 19104, forrestc{at}email.chop.edu


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
PURPOSE This study describes referral completion from the perspectives of patients and primary care physicians and identifies predictors of adherence to the referral recommendation.

METHODS We observed a cohort of 776 referred patients from the offices of 133 physicians in 81 practices and 30 states. Referring physicians and patients completed self-administered questionnaires at the time of the referral decision and 3 months later.

RESULTS Physicians reported that 79.2% of patients referred had a specialist visit, and 83.0% of patients indicated they completed the referral. The most common reasons for not completing the referral were "lack of time" and patient belief that the "health problem had resolved." The {kappa} statistic for patient-physician agreement on referral completion was 0.34, indicating only fair concordance. Patients in Medicaid plans were less likely than others to complete the referral, and more likely to experience a health plan denial. A longer duration of the patient relationship with the primary care physician and physician/staff scheduling of the specialty appointment were both positive predictors of referral completion.

CONCLUSIONS About 8 in 10 patients referred from primary care complete a specialty referral within 3 months. Findings from this study suggest that referral completion rates may be increased by assisting patients with scheduling their specialty appointments and promoting continuity of care.

Key Words: Referral and consultation • appointment adherence • patient acceptance of health care • primary health care • practice-based research • delivery of health care • continuity of patient care


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Specialty referrals are one of several management options available to primary care physicians. The effectiveness of specialty referrals as a management tool depends not only on the quality of physician decision making—when and for whom to obtain specialty care—but also on the likelihood that patients referred choose to complete the referral by attending one or more specialist visits.

Although much is published on the determinants of primary care physician referral decision making,1–3 much less is known about referral completion. Based on chart audits and physician survey, prior studies have found that 63% to 82% of patients referred from primary care settings ultimately attend a consultation with the type of specialist to whom they were referred.4–6 Beyond these descriptive estimates, there is little information on the associations among patient, referring physician, and health care delivery system characteristics with referral completion.

A necessary step toward improving the quality and outcomes of the specialty referral process is developing a better understanding of the determinants of referral completion. To fill this knowledge gap, we report findings from the ASPN Referral Study, a prospective cohort study that examined the content and results of physicians’ referral decisions.3,7–9 The study was specifically designed to (1) estimate and contrast rates of referral completion from the perspectives of patients and primary care physicians; (2) identify patient, physician, and delivery system predictors of referral completion; and (3) understand from the perspective of patients why some chose not to or could not complete the referral.


   METHODS
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
RESULTS
 DISCUSSION
 REFERENCES
 
Physician Recruitment
Physician recruitment was directed to family physician and general internal medicine members of the Ambulatory Sentinel Practice Network (ASPN), physicians affiliated with Medical Group Management Association, other practice-based research networks (Minnesota Academy of Family Physicians Research Network, the Wisconsin Research Network, and the Dartmouth COOP), and the larger community of physicians. The study was publicized by means of direct mailings to physicians, articles in research network newsletters, notices in journals, and presentations at conferences. Telephone contact was made with physicians expressing interest. Physicians were included in the study if they practiced in the United States, were not in training, and participated in both the baseline and follow-up surveys.

Data Collection
The Johns Hopkins School of Public Health’s Committee on Human Research and the Colorado Multi-Institutional Review Board approved study protocols, physician disclosure letters, and patient informed consent forms. Data were collected from August 1997 to April 1999. Before recruiting patients, physicians completed a questionnaire with demographic and practice characteristic items. Each practice selected a coordinator who communicated with the research team, learned study protocols, trained office staff, monitored data quality, and completed a questionnaire about the organizational and financial components of their practices.

During 15 consecutive workdays, physicians and their staff invited all patients who were referred to a specialist to participate in the study. We defined a referral as a physician’s decision to send the patient to see a specialist practitioner (physicians and non-physicians with specialized skills were included) for a face-to-face encounter. Referrals to laboratory and imaging facilities, emergency departments, hospitals for inpatient admission, and curbside consultations in which the referring physician obtains oral advice from a specialist without a patient visit were excluded from the study.

For each referral, physicians and patients (or caregivers for children and youth) completed a questionnaire, and office staff recorded its occurrence in a referral log. The physician baseline questionnaire had items on the type of specialist referred to; the physician’s reason for referral and expectation of the consultant, the expected duration of the specialist’s involvement, urgency of the referral, and duration of the relationship with the patient; and whether the physician or staff made an appointment with the specialist, and whether the physician transferred relevant patient information to the specialist. Only sociodemographic items from the patient baseline questionnaire were used in this analysis.

Approximately 3 months after each referral was made, follow-up surveys were conducted with referred patients and their referring physicians. Practices mailed questionnaires to patients who participated in the baseline survey and who were among the first 10 referrals made by the physician. To enhance response rate, a $2 bill was sent with the patient questionnaire; patients who completed the questionnaire were entered into a $500 lottery.

Because we kept a record of the universe of referred patients, we had a list of all eligible patients. For the 6% of patients with more than one referral, we randomly selected 1 referral for inclusion in the follow-up survey. There were 1,022 patients eligible for the follow-up survey. Of these, 796 patients returned a completed questionnaire (response rate 78%). The questionnaire asked whether patients saw the specific specialist type (eg, cardiologist, general surgeon, etc) to whom they were referred. We customized each patient questionnaire with the name of the referring physician, the date when the referral was made, and the type of specialist referred to.

Referring physicians used medical records to complete follow-up questionnaires and determine referral completion status. Physicians completed questionnaires for 776 of the 796 patients (97% response rate) who participated in the patient follow-up survey. These 776 patients constituted the final patient sample.

Data Analysis
The physician and patient follow-up surveys provided independent assessments of whether the referral was completed. To estimate interobserver concordance, we calculated the percentage of agreement and {kappa} statistic, which corrects for chance agreement.10 Using patient reports as the criterion, we also estimated the sensitivity, specificity, and predictive values of physician reports of referral completion. These statistics provide information on the degree to which physicians are aware of patient-reported referral completion.

Cross-tabulations with Pearson {chi}2 analysis were done to identify characteristics of patients, physicians, practices, and health plans that may be associated with patient-reported referral completion. An {alpha} level of .20 was used to report statistically significant associations with referral completion. This critical value level was selected, rather than the more conventional .05 level, to ensure that we included all variables that may be important covariates in the multivariable logistic regression predictive model.11

The referral completion predictors that were significant in univariate analysis were entered into a stepwise logistic regression analysis with forward selection. The analysis goal was to identify the most parsimonious set of predictors that explains variability in the chances of referral completion. The unit of analysis was the referred patient. We used the generalized estimating equation12 to account for the clustering of visits within physicians. Data analysis was conducted using STATA statistical software, version 8.13 Groups of variables were entered into the regression model as follows: referral decision, physician-patient relationship, physician, practice, and then patient characteristics. The likelihood ratio test11 was used to determine whether the variable was a significant predictor of referral completion. A critical value of 0.05 was selected as the criterion for retaining variables in the final regression model.


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
DISCUSSION
 REFERENCES
 
The study sample was composed of 776 patients referred by 133 physicians from 81 unique practices located in 30 states. The mean age of patients was 48.2 years (SD, 19.6), and 59.9% were female (Table 1Go). Only 3.7% of patients were uninsured, and 11.0% had Medicaid financing. Most referrals (64.0%) were expected to be short-term (less than 3 months). The most common reason for a referral was to obtain a therapeutic procedure (eg, a surgery) or diagnostic test (eg, an endoscopy). Almost one-half of the patient sample had a 5-year or longer relationship with their referring physicians. Referring physicians scheduled the specialty appointment for 78.8% of referrals made and sent to the specialist relevant patient information for 85.3%.


View this table:
[in this window]
[in a new window]

  		Table 1. Characteristics of Study Sample
 
Physicians reported that 79.2% (615 of 776) of patients referred had a specialist visit, and 83.0% (644 of 776) of patients reported that they completed the referral (Table 2Go). According to 132 referred patients who did not see a specialist, the most common, non–mutually exclusive reasons for choosing not to complete the referral were (1) the patient’s belief that the health problem had resolved (47.5%), (2) lack of time (37.3%), and (3) the patient disagreed with the physician on the need for referral (26.5%). Patients who requested the referral were more likely to complete the referral than those who did not (92.4% vs 81.9%; P=.02).


View this table:
[in this window]
[in a new window]

  		Table 2. Patient-Physician Agreement on Referral Completion
 
For 2.3% of all referrals made, patients said that their health care insurance plan did not approve the referral. Patients with Medicaid financing were more likely than their insured counterparts to have their referral denied by their health plan (7.1% vs 1.2%; P<.01).

Patients and physicians agreed on referral completion for 79.3% (615 of 776). The {kappa} statistic for patient-physician agreement was 0.34, representing only fair agreement. Using patients’ reports as the reference standard, the sensitivity of physician reports of referral completion was 85.5%, the specificity was 51.5%, the positive predictive value was 89.6%, and the negative predictive value was 42.2%.

All patient, referral decision, health plan, physician, and practice variables, which are listed in Table 1Go, were tested for statistical association with patient-reported referral completion; only those with a P value of less than .20 are shown in Table 3Go. Variables not associated with referral completion were (1) patient characteristics—age, sex, education, and income; (2) health plan—gatekeeping arrangements and capitation payment; (3) referral decision—type of specialist referred to, primary expectation of specialist, urgency, referring physician gave name of specialist to patient, and referring physician transferred information to specialist; and (4) physician and practice—sex, census division, and practice size.


View this table:
[in this window]
[in a new window]

  		Table 3. Associations Between Patient-Reported Referral Completion and Patient, Referral Decision, Physician, Practice, and Health Plan Characteristics: Univariate Analyses
 
A stepwise logistic regression with forward selection was done to select the most parsimonious set of predictors of patient-reported referral completion (Table 4Go). Patients in Medicaid plans were less likely than all others to complete the referral. On the other hand, a longer duration of relationship with the primary care physician and scheduling of the specialty appointment by the referring physician’s office staff were both positive predictors of referral completion.


View this table:
[in this window]
[in a new window]

  		Table 4. Predictors of Referral Completion: Logistic Regression Analysis
 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
REFERENCES
 
According to both patients and referring physicians, about 8 in 10 referrals are completed within 3 months. We attempted to minimize the effects of recall bias by using a follow-up period of 3 months; customizing each patient questionnaire with the specific name of the referring physician, the date when the referral was made, and the specific type of specialist referred to; and having referring physicians use patients’ medical records to complete questionnaires.

Physicians in this study volunteered to participate, and many were active in primary care practice-based research networks. Their referral practices may systematically differ from all US physicians because of this self-selection process. Our physician sample has been compared with a nationally representative sample of primary care physicians who participated in the National Ambulatory Medical Care Survey, and results suggested similar rates of referral overall and by age, sex, and disease status between the 2 groups.9 Even though all physician participants were interested in research on specialty referrals, the potential direction of any bias on referral completion estimates is unclear.

We believe the effects of nonresponse bias are likely to be minimal, although 22% of patients did not return the follow-up survey. Physician-estimated referral completion for this subgroup was 76%, which is close to the 80% physician-based estimate for the sample who participated in the follow-up survey. Responders and nonresponders to the patient follow-up questionnaire did not differ significantly in terms of payer status or likelihood that physicians or their staff scheduled the specialty appointment. The nonresponders were more likely than responders to report a relationship of less than 5 years with their primary care physician (39% vs 50%, respectively).

Doctor-patient agreement on which patients actually attended a specialty appointment was only fair. Physicians’ perceptions of whether the referral was completed were sensitive, but not specific. In other words, the largest source of disagreement resulted from physicians believing that patients had completed the referral, while the patients reported that they had not. We found that physicians were no better than a chance flip of a coin in reporting on referral completion when patients had, in fact, not seen the specialist. This low level of agreement could result from primary care physicians not following up on all referred patients or incomplete transfer of information from specialists back to primary care practices, leading to a lack of medical record documentation. It should also be noted that we had no source other than patient and physician reports for verifying whether the referral was actually completed, which raises the question of which source is most valid. We elected to use the patient as the criterion in this study, because referral completion is an event that they experience directly, whereas physicians are in effect proxy observers.

Referral completion awareness could be improved with electronic records that automatically update the patient’s primary care records once a specialist appointment has occurred. Such an electronic system could also be used to notify physicians when patients cancel a specialty appointment, perhaps with the reason for the cancellation appended to the notice. Lack of physician knowledge of referral completion could lead to overuse quality problems, resulting from inadvertent duplication of tests, procedures, or prescribing of medications. False-positive physician reports, on the other hand, could result in patients not receiving needed services.

Scheduling the appointment with the specialist at the time the referral was made had a strong positive effect on referral completion. Other researchers have found a similar association between assistance in scheduling the specialist’s appointment and referral completion for youth referred from school-based health centers,5 patients referred from an emergency department to outpatient psychiatry,14 and patients referred for mammograms.15 Staff or physician scheduling of the specialty referral appointment is low-cost, practical, and, as this study shows, an important determinant of referral completion.

Research from other countries has found that longer waiting times increase the chances of nonattendance for a new referral visit.16 One reason why specialty appointment waiting times may decrease referral completion is that patients’ health problems can resolve during the wait, which was the case for about one-half of all patients who did not complete the referral in our study. Another study on nonattendance in a pediatric dermatology clinic found similar rates of health problem resolution.17 It is unclear, however, whether physicians would similarly report that the problem prompting referral had in fact resolved.

Medicaid insurance was associated with reduced chances of completing the referral. The proportion of patients with Medicaid financing (11%) in this study was low, which suggests that practices with higher proportions would experience reduced rates of referral completion. Other research has found that Medicaid beneficiaries are less likely than those with commercial insurance to access specialty care for chronic care management.18 One explanation for why referral completion rates were so low for Medicaid patients is that their health plans were more likely than their insured counterparts to deny the referral. Another explanation that we did not examine is that Medicaid patients may have more difficulty scheduling a specialist appointment because of low clinician acceptance of Medicaid payment. Nonetheless, Medicaid financing should alert physician organizations to patients who may need special efforts to facilitate referral completion.

A longer relationship between the patient and the referring physician was another important positive predictor of referral completion. Better continuity between doctors and patients has been associated with reduced rates of primary care no-shows.19 One reason for this association may be that a longer relationship offers greater opportunities for doctors and patients to align their goals regarding health care treatment and outcomes.20 A shared understanding of common goals may increase patients’ trust that their physicians are acting on their behalf, thereby increasing patient’s motivation to follow specialty referral recommendations made by their doctor. Patient motivation to see the specialist, according to our results, is a key reason for attending the appointment: 9 in 10 patients requesting the referral completed it.

Referral completion involves a negotiation process between doctors and patients. This dialogue is influenced by continuity of the doctor-patient relationship. Not all patients choose to participate in their physicians’ management decisions: 1 in 4 patients who did not complete the referral disagreed with their doctor on the need for a specialist, and about 1 in 2 stated the problem prompting the referral had resolved. Findings from this study suggest that referral completion rates may be increased by assisting patients with scheduling their specialty appointments and promoting continuity of care.


   ACKNOWLEDGMENTS
 
Laurie Vorel and Jim Werner provided technical assistance with data collection and project implementation. We are grateful to the physicians and office coordinators who donated their time and provided their energy to make this study possible. Their involvement was invaluable.


   FOOTNOTES
 
Conflicts of interest: none reported

Funding support: This study was funded by grant No. R01 HS09377 from the Agency for Healthcare Research and Quality.

Received for publication December 18, 2005. Revision received February 17, 2007. Accepted for publication February 22, 2007.


   REFERENCES
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Wilkin D, Smith A. Explaining variation in general practitioner referrals to hospital. Fam Pract. 1987;4(3):160–169.[Abstract/Free Full Text]
  2. Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice profiling. When good apples look bad. JAMA. 1994;272(11):871–874.[Abstract]
  3. Forrest CB, Nutting PA, von Schrader S, Rohde C, Starfield B. Primary care physician specialty referral decision making: patient, physician, and health care system determinants. Med Decis Making. 2006;26(1):76–85.[Abstract/Free Full Text]
  4. Byrd JC, Moskowitz MA. Outpatient consultation: interaction between the general internist and the specialist. J Gen Intern Med. 1987;2(2):93–98.[Medline]
  5. Hacker KA, Weintraub TA, Fried LE, Ashba J. Role of school-based health centers in referral completion. J Adolesc Health. 1997;21(5):328–334.[CrossRef][Medline]
  6. Bourguet C, Gilchrist V, McCord G. The consultation and referral process. A report from NEON. Northeastern Ohio Network Research Group. J Fam Pract. 1998;46(1):47–53.[Medline]
  7. Forrest CB, Nutting P, Werner JJ, Starfield B, von Schrader S, Rohde C. Managed health plan effects on the specialty referral process: results from the Ambulatory Sentinel Practice Network referral study. Med Care. 2003;41(2):242–253.[CrossRef][Medline]
  8. Starfield B, Lemke KW, Bernhardt T, Foldes SS, Forrest CB, Weiner JP. Comorbidity: implications for the importance of primary care in ‘case’ management. Ann Fam Med. 2003;1(1):8–14.[Abstract/Free Full Text]
  9. Forrest CB, Nutting PA, Starfield B, von Schrader S. Family physicians’ referral decisions: results from the ASPN referral study. J Fam Pract. 2002;51(3):215–222.[Medline]
  10. Fleiss J. Statistical Methods for Rates and Proportions. New York, NY: John Wiley & Sons, Inc; 1981.
  11. Hosmer D, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons, Inc; 1989.
  12. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986;42(1):121–130.[CrossRef][Medline]
  13. Stata Statistical Software. Release 8.0. College Station, Tex: Stata-Corp LP; 2003.
  14. Craig TJ, Huffine CL, Brooks M. Completion of referral to psychiatric services by inner city residents. Arch Gen Psychiatry. 1974;31(3):353–357.[Medline]
  15. Wolosin RJ. Effect of appointment scheduling and reminder postcards on adherence to mammography recommendations. J Fam Pract. 1990;30(5):542–547.[Medline]
  16. Leung GM, Castan-Cameo S, McGhee SM, Wong IO, Johnston JM. Waiting time, doctor shopping, and nonattendance at specialist outpatient clinics: case-control study of 6495 individuals in Hong Kong. Med Care. 2003;41(11):1293–1300.[CrossRef][Medline]
  17. Hon KL, Leung TF, Wong Y, Ma KC, Fok TF. Reasons for new referral non-attendance at a pediatric dermatology center: a telephone survey. J Dermatolog Treat. 2005;16(2):113–116.[CrossRef][Medline]
  18. Cabana M, Bruckman D, Rushton JL, Bratton SL, Green L. Receipt of asthma subspecialty care by children in a managed care organization. Ambul Pediatr. 2002;2(6):456–461.[CrossRef][Medline]
  19. Alpert JJ, Robertson LS, Kosa J, Heagarty MC, Haggerty RJ. Delivery of health care for children: report of an experiment. Pediatrics. 1976;57(6):917–930.[Abstract/Free Full Text]
  20. Donaldson MS. Continuity of care: a reconceptualization. Med Care Res Rev. 2001;58(3):255–290.[Abstract/Free Full Text]



This article has been cited by other articles:


Home page
 DOC NewsHome page
S. Watson
Getting to the Roots of Referral Dropout
DOC News, November 1, 2007; 4(11): 3 - 13.
[Full Text]

Home page
 Ann Fam MedHome page
K. C. Stange
In This Issue: Clinical Diagnosis and Management
Ann. Fam. Med, July 1, 2007; 5(4): 290 - 291.
[Full Text] [PDF]

TRACK Comments:

Read all TRACK Comments

Specialty Referral Completion Among Primary Care Patients: Mission impossible?
Kam lun Ellis Hon, FAAP
Annals of Family Medicine, 1 Aug 2007 [Full text]
The Role of Patient-Centered Care in Specialty Referral Completion
Dennis Z Kuo, MD, MHS
Annals of Family Medicine, 2 Aug 2007 [Full text]
Responses to Hon and Kuo
Christopher B Forrest
Annals of Family Medicine, 21 Aug 2007 [Full text]
Untangling the web of referral coordination
Christopher J. Stille
Annals of Family Medicine, 27 Aug 2007 [Full text]
This Article
Right arrow Abstract
Right arrow Full Text (PDF)
Right arrow In Brief
Right arrow TRACK Discussion: Submit a Comment
Right arrow TRACK Discussion: View Comments
Right arrow Alert me when this article is cited
Right arrow Alert me when TRACK Comments are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Forrest, C. B.
Right arrow Articles by Starfield, B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Forrest, C. B.
Right arrow Articles by Starfield, B.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS