Time Spent in Face-to-Face Patient Care and Work Outside the Examination Room
=============================================================================

* Andrew Gottschalk
* Susan A. Flocke

## Abstract

**PURPOSE** Contrary to physicians’ concerns that face-to-face patient time is decreasing, data from the National Ambulatory Medical Care Survey (NAMCS) indicate that between 1988 and 1998, durations of primary care outpatient visits have increased. This study documented how physicians spend time during the workday, including time outside the examination room, and compared observed face-to-face patient care time with that reported in NAMCS.

**METHODS** Using time-motion study techniques, for each of 11 physicians, 2 patient care days were randomly selected and documented by direct observation. Physician time spent on face-to-face patient care and 54 activities outside the examination room were documented. Data represent 12,180 minutes of work and 611 outpatient visits.

**RESULTS** The average workday duration was 8.6 hours, and face-to-face patient care accounted for 55% of the day. Work outside the examination room relevant to a patient currently being seen averaged 14% of the day. Work related to a patient not physically present accounted for one fifth (23%) of the workday. The combination of face-to-face time and time spent on visit-specific work outside the examination room assessed by direct observation was significantly less than the 2003 NAMCS estimate of visit duration assessed by physician report (13.3 vs 18.7 minutes, *P* <.001).

**CONCLUSIONS** Nearly one half of a primary care physician’s workday is spent on activities outside the examination room, predominately focused on follow-up and documentation of care for patients not physically present. National estimates of visit duration overestimate the combination of face-to-face time and time spent on visit-specific work outside the examination room by 41%.

*   Direct observation
*   primary care
*   time factors
*   time and motion study
*   office visits

## INTRODUCTION

Primary care physicians have expressed discontent and concern that face-to-face time with patients is diminishing and that their administrative burdens are increasing.1–3 Concerns are fueled by data suggesting that shorter visits are associated with lower patient satisfaction4,5 and possibly poorer quality of care.6 Recent findings from multiple data sources indicate, however, that the duration of the visit in a primary care setting is increasing,7 the number of patients being seen during an average week is decreasing, and the number of hours spent working during the week has remained the same.8 Specifically, longitudinal data using the National Ambulatory Medical Care Survey (NAMCS) for the decade 1988–1998 indicate that physician-reported face-to-face interaction time has increased 2.0 minutes to an average of 16.3 minutes per encounter.7 NAMCS data from 2003 indicate that among general and family physicians, the average visit duration is 18.7 minutes.9

What could account for the discrepancy between physician perceptions and national data? Data for the NAMCS are based on physician reports at the completion of each sampled visit. Gilchrist et al10 showed that compared with *direct observation* of patient care, physicians’ reports using the NAMCS data card overestimate visit duration by an average of 3.5 minutes. The accuracy of physician report of visit duration using these methods thus may be questioned. One possible explanation for physicians’ overestimate of face-to-face time is that visit complexity and multiple competing demands for limited time may affect physician perception of the actual visit duration.1 A second possibility is that activities outside the examination room, such as reviewing the medical record and other administrative tasks related to the patient visit, are included in the physician estimate of the face-to-face visit duration.

This study documented primary care physicians’ use of time outside the examination room during a typical workday. Our purpose was to test the hypothesis that a combination of face-to-face time and work outside the examination room related to care of the current patient more closely represents the NAMCS estimates of visit duration. Our findings will help to interpret the visit duration literature and other studies that use the NAMCS data as a source of face-to-face visit duration. By examining how physicians use time outside the examination room, we can also critically consider how office systems might better support clinicians.

## METHODS

### Setting and Sample

We studied family physician time use using a cross-sectional observational study design and time-motion techniques. Eleven physicians from 8 practices participated. Each was a full-time family physician who agreed to serve as a preceptor for a first-year medical student for a 6-week period during which this study was conducted. Practices were located in northeast Ohio, Los Angeles, Calif, and Nashville, Tenn. Two practices were solo and the rest were single-specialty group practices. The median number of physicians in the group practices was 6. Participating physicians were comparable to the national membership of the American Academy of Family Physicians (AAFP) in regard to the percentage who were male (82% vs 68%); the number of years since residency training (12.7 vs 11.6 years); and the distribution of insurance type of the practice population—Medicare (22.4% vs 22.1%), Medicaid (7.8% vs 12.6%), capitated or managed care (40.5% vs 35%), traditional indemnity or fee for service (22.4% vs 21.5%), and none (6.6% vs 8.9%).11 The average number of visits per day was also similar to the estimated number from the American Medical Association Socioeconomic Monitoring System 1998 survey of family physicians (29.1 vs 25.8).8 None of the participating physicians used electronic medical records.

### Data Collection

Data were collected by 5 trained medical student observers; 4 collected data on 2 physicians and 1 collected data on 3 physicians. Training included (1) participation in the development of the protocol and definition and refinement of the time use categories, and (2) collection of 2 or more hours of pilot data in a hospital-based family practice clinic. The pilot work was conducted to practice using the form in real time and to identify problems with categorizing activities or other feasibility problems with the protocol itself. Problems identified during the pilot were discussed by the group to reach agreement on the final definitions of each category and the coding protocol.

Two patient care days per physician were randomly selected for observation from within a 6-week window during June and July of 2003. Half-days of patient care and days scheduled specifically for clinical teaching (ie, precepting residents or medical students) were excluded. Physicians were asked to indicate when and where their workday began, and time and motion data were collected from that point forward until the physician indicated that the workday was complete.

The observers followed the physician everywhere, excluding the rest room, to document how time was spent throughout the day. Time use was categorized into 55 distinct activities. The 2 main categories were face-to-face patient care and activities outside the examination room. Subcategories of time outside the examination room included (1) visit-specific work, (2) work related to a patient not currently being seen, and (3) other work outside the examination room, which included administration, academic activities, personal time, and patient care in other settings.

Measurement of face-to-face patient care time began when the physician entered the examination room and stopped when the physician exited. All work conducted within the examination room with the patient present was categorized as face-to-face patient care. Visit-specific work was defined as work outside the examination room specific to patients who were physically present and being cared for by the physician at the practice at that moment. For example, reviewing a patient’s chart immediately before entering the room or completing the encounter form after exiting the room would be coded in this subcategory. The second subcategory was work specific to patients not currently being seen (ie, patients who were not present). This subcategory included work related to patients seen on a prior day, seen earlier in the day, or to be seen on an upcoming day. Specific activities under each of the subcategories are displayed in Table 1⇓.

View this table:
[Table 1.](http://www.annfammed.org/content/3/6/488/T1)

Table 1.  
**Physician Time Use During 21 Randomly Selected Patient Care Days** 

Digital watches were used to record the start and stop time of each time use category to the nearest minute. If an activity took less than 1 minute, the start time and the number of seconds in 10-second intervals were recorded. If a physician was engaged in multiple acts at once, the time spent in the most patient care–related category was coded. During data collection, the observer asked the physician to clarify any activities that were ambiguous.

Five individual activities (completing encounter form, communicating with insurance companies, arranging for tests or consultations, looking up allowed referrals, and completing disability or school forms) were combined to assess the amount of time the physician spent on “paperwork”-related activities. These activities represent documentation tasks typically required by particular insurance plans.12 The total amounts of time spent on paperwork activities were summed for both patients currently being seen and patients not currently being seen.

At the end of each observation day, physicians were asked to rate how typical the day was. This measure was assessed with a single item and a 5-point Likert scale, with 1 indicating typical and 5 indicating not at all typical. The average response was 1.2. To assess the potential for a Hawthorne effect, we asked physicians to rate the degree to which the presence of the observer changed their workday. This measure was also assessed with a single item and a 5-point Likert scale, with 1 being not at all changed and 5 being changed a lot. The average response was 1.2.

This protocol was approved by the Institutional Review Board of University Hospitals of Cleveland.

### Statistical Analysis

The workday was the unit of analysis. For each day, the total time spent in each of the 55 time use activities and the total duration of the workday were calculated. The mean, standard deviation, and mean percentage of total time spent in each activity were computed. Totals for each main category were also computed. A 1-sample *t* test was used to compare the observed mean face-to-face time with estimated visit duration time from the 2003 NAMCS survey.9 A 1-sample *t* test was also used to compare the combination of observed face-to-face time and time outside of the examination room related to care of the current patient with the 2003 NAMCS survey estimate. Tests were evaluated using a significance level of *P* <.05.

## RESULTS

One data collection day was excluded because it did not represent a complete day of patient care. A total of 21 days of observation included 611 patient visits and 12,180 minutes of physician work that was allocated into 1 of 55 distinct activities (Table 1⇑). The mean number of patients seen per day was 29.1 (SD, 8.8). The average duration of the patient care day was 9.7 hours (SD, 1.7). One hour (SD, 0.5) was specific to personal activities, such as lunch, socializing with staff, and personal telephone calls. Time spent on personal activities was excluded from the total time, leaving just the work categories and an average time devoted to work per day of 8.6 hours. Descriptive data for each activity are given in Table 1⇑.

Face-to-face patient care accounted for 54.9% of the day. Visit-specific work outside the examination room accounted for 14.5% of the total workday. The most common activities outside the examination room included reviewing the medical record and writing notes, completing the encounter form, arranging for tests or consultations, and writing prescriptions.

Work related to patients not currently being seen averaged 22.9% of the workday. Common activities included reviewing or writing medical record notes, making telephone calls to or receiving them from patients, reviewing laboratory results, and fielding patient-related questions from staff. Time used attending to administrative and academic activities, and patient care in other settings (eg, the home or the hospital) accounted for only 7.5% of the total workday. Time spent in the categories combined to represent the paperwork category did not account for a large portion of the workday. Paperwork combined for both the patient currently being seen and for patients not being seen averaged only 34 minutes or 6.5% of the total workday.

Next we compared our direct observation of time spent in face-to-face patient care with national data collected as part of the NAMCS in 2003. The average face-to-face patient care time reported for general and family medicine physicians in that survey was 18.7 minutes.9 The average face-to-face patient care time measured by direct observation in our study was significantly less (10.7 vs 18.7 minutes, *P* <.001). When time spent on visit-specific work outside the examination room was combined with face-to-face time, the average time per patient visit was closer to the NAMCS estimate but was still significantly less (13.3 vs 18.7 minutes, *P* <.001).

## DISCUSSION

This study is the first we are aware of that documents how primary care physicians spend time during the workday outside the examination room. It may not be a surprise to community practicing physicians that only 55% of time is spent in face-to-face patient care. What may be surprising is that paperwork activities account for only an average of 34 minutes or 6.5% of the total workday. We also found that a large portion of physician time is spent on work related to patients who are not currently in the office. This work accounted for almost one fifth of the workday, on average, and predominately involved writing or dictating notes, patient care–related telephone calls, and interpreting laboratory results. Documenting, disseminating information to patients, and aiding with decision making are vital to providing good-quality care to patients and maintaining continuous healing relationships,13 but office systems could help streamline the physician’s role in these tasks.14 Implementing such systems as electronic prescribing, electronic medical records, and telephone call triage protocols can increase the efficiency of information management and decision support.15–19 In addition, use of automated templates for patient letters and use of a billing specialist to assist with forms and formularies can reduce physician time spent on information management and dissemination tasks.

Greater time spent on administrative tasks has been shown to be associated with low physician job satisfaction independent of compensation, financial incentives, and care management restrictions,20 while face-to-face patient time has been shown to affect both patient and physician satisfaction.1,4,20,21 Because physician compensation is generally visit based, the work outside the examination room is largely uncompensated care and may contribute to physicians’ dissatisfaction. Use of office systems such as electronic medical records and electronic prescribing is not a panacea,1,22 but reducing the time physicians spend on paperwork activities can have a positive effect beyond satisfaction. For example, the average 34 minutes per day spent on paperwork may seem small, but it translates into about 2 patient visits a day for physicians. Seeing 2 additional patients per day may have an important impact on increasing practice revenue, as well as on increasing access to care for patients.

The average face-to-face time for general and family physicians reported from the 2003 NAMCS data is 75% greater than the face-to-face time observed in this study. The NAMCS visit-specific card that the physician completes at the end of each visit clearly defines the visit duration as time spent in face-to-face care. Although others have reported a significant overestimation of the visit duration using the NAMCS card compared with direct observation,10 these investigators could only speculate that other activities outside the examination room that are related to the visit, such as writing notes and completing forms, may seep into physicians’ estimates. In this study, we addressed this hypothesis by combining the observed face-to-face time with visit-specific time spent outside the examination room. The combination of these time use categories diminishes but does not completely close the gap in direct observation compared with physician reported face-to-face time. National estimates of visit duration overestimate the combination of face-to-face time and time spent on visit-specific work outside the examination room by 41%. This observation leads us to conclude that both visit-specific work outside the examination room and work related to patients not currently being seen affect physician “telescoping”23,24 (overestimating) of the visit duration. Investigators using the NAMCS data for other analyses involving visit duration should be cautioned about this overestimation. Identification of visit and physician factors that affect the telescoping of reported visit duration would be useful for future research studies wherein visit duration is a primary variable.

Our study has limitations that deserve mention. The labor-intensive protocol limited the number of physicians who could be included in the study and, thus, generalization of these findings to other types of physicians or different settings should be done with caution. Estimates of time spent in nursing homes, on hospital rounds, and on home visits may not be adequately represented with the time and motion methods used in this study. A better estimate may be the average time spent per month in each of these settings, because many primary care physicians schedule these activities on a monthly basis. The 11 family physicians included in the study represented a variety of geographic locations and were similar to the average members of the AAFP in terms of number of years in practice and distribution of patient insurance type. The average number of patients seen per day by study physicians was also similar to the estimated number seen by family physicians surveyed in 1998 by the American Medical Association Socioeconomic Monitoring System. The 2 observation days per physician were randomly selected from a 6-week period, and physicians were not informed of the specific hypotheses to minimize the potential of a Hawthorne effect.25 We did not ask the physicians to estimate how much face-to-face time they spent with each patient. This decision was driven by the primary purpose of the study, which was to observe and document complete and unaltered primary care physician patient care days. Asking the study physicians to complete an NAMCS-like card after each of the 622 patient visits would have altered work flow outside the examination room. We did assess how typical the day was from the physician’s perspective, and all data collection days were rated very typical or typical.

In conclusion, these data shed light on the seeming discrepancy between primary care physicians’ experiences that face-to-face visit time is getting shorter and the national data indicating that visit duration has increased over the past decade. It is possible that face-to-face time with patients has diminished over the past decade, while patient-related demands outside the examination room have increased. Longitudinal data that accurately document face-to-face time as well as time use outside the examination room are required, however, to address this hypothesis.

## Acknowledgments

We wish to thank the physicians who amiably allowed the observers to shadow and document time use for 2 full days. Douglas Cook, Amy Huang, Maria Huang, Scott Martin, and Vanessa Richlin contributed to the project development and data collection. Jason Chao, MD, MS, contributed to the protocol development. Ginger Pomiecko assisted with data management.

## Footnotes

*   *Conflicts of interest: none reported* 

*   Received for publication December 9, 2004.
*   Revision received March 18, 2005.
*   Accepted for publication March 25, 2005.


*   © 2005 Annals of Family Medicine, Inc.

## REFERENCES

1.  Mechanic D. Physician discontent: challenges and opportunities. JAMA. 2003;290:941–946.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.290.7.941&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12928472&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000184828800032&link_type=ISI) 

2.  Kassirer JP. Doctor discontent. N Engl J Med. 1998;339:1543–1545.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJM199811193392109&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9819454&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000077051800009&link_type=ISI) 

3.  Burdi MD, Baker LC. Physicians’ perceptions of autonomy and satisfaction in California. Health Aff* (Millwood)*. 1999;18:134–145.
    
    

4.  Lin CT, Albertson GA, Schilling LM, et al. Is patients’ perception of time spent with the physician a determinant of ambulatory patient satisfaction? Arch Intern Med. 2001;161:1437–1442.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/archinte.161.11.1437&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=11386893&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000169136400010&link_type=ISI) 

5.  Gross DA, Zyzanski SJ, Borawski EA, Cebul RD, Stange KC. Patient satisfaction with time spent with their physician. J Fam Pract. 1998;47:133–137.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9722801&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000075393900009&link_type=ISI) 

6.  Wilson A, Childs S. The relationship between consultation length, process and outcomes in general practice: a systematic review. Br J Gen Pract. 2002;52:1012–1020.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNTIvNDg1LzEwMTIiO3M6NDoiYXRvbSI7czoyMjoiL2FubmFsc2ZtLzMvNi80ODguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

7.  Mechanic D, McAlpine DD, Rosenthal M. Are patients’ office visits with physicians getting shorter? N Engl J Med. 2001;344:198–204.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1056/NEJM200101183440307&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=11172143&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000166402800007&link_type=ISI) 

8.  Weeks WB, Wallace AE. Time and money: a retrospective evaluation of the inputs, outputs, efficiency, and incomes of physicians. Arch Intern Med. 2003;163:944–948.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/archinte.163.8.944&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12719204&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000182477400011&link_type=ISI) 

9.  Hing E, Cherry DK, Woodwell DA. *National Ambulatory Medical Care Survey: *2003* Summary.* Hyattsville, Md: National Center for Health Statistics. Available at: [http://www.cdc.gov/nchs/data/ad/ad365.pdf](http://www.cdc.gov/nchs/data/ad/ad365.pdf).
    
    

10. Gilchrist VJ, Stange KC, Flocke SA, McCord G, Bourguet CC. A comparison of the National Ambulatory Medical Care Survey (NAMCS) measurement approach with direct observation of outpatient visits. Med Care. 2004;42:276–280.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/01.mlr.0000114916.95639.af&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15076827&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000189293000010&link_type=ISI) 

11. American Academy of Family Physicians. Facts about family practice. Web site. Available at: [http://www.aafp.org/x530.xml](http://www.aafp.org/x530.xml). Accessed November 1, 2004.
    
    

12. Flocke SA, Orzano AJ, Selinger HA, et al. Does managed care restrictiveness affect the perceived quality of primary care? A report from ASPN. Ambulatory Sentinel Practice Network. J Fam Pract. 1999;48:762–768.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12224673&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 

13. Committee on Quality of Health Care in America, Institute of Medicine. *Crossing the Quality Chasm: A New Health System for the 21st Century.* Washington, DC: National Academy Press; 2001.
    
    

14. Spann SJ. Report on financing the new model of family medicine. Ann Fam Med. 2004;2(Suppl 3):S1–S21.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTI6IjIvc3VwcGxfMy9zMSI7czo0OiJhdG9tIjtzOjIyOiIvYW5uYWxzZm0vMy82LzQ4OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

15. Committee on Quality of Health Care in America, Institute of Medicine. Using information technology. In: *Crossing the Quality Chasm: A New Health System for the 21st Century.* Washington, DC: National Academy Press; 2001:164–180.
    
    

16. Bates DW, Ebell M, Gotlieb E, Zapp J, Mullins HC. A proposal for electronic medical records in U.S. primary care. J Am Med Inform Assoc. 2003;10:1–10.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=14640303&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 

17. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med. 2003;114:397–403.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0002-9343(03)00057-3&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12714130&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000182436400009&link_type=ISI) 

18. Fox GN, Weidmann E, Diamond DE, Korbey AA. Hand-held electronic prescribing. J Fam Pract. 2001;50:449–454.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=11350711&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000168628000011&link_type=ISI) 

19. Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA. 1998;279:1024–1029.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.279.13.1024&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9533503&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000072669700032&link_type=ISI) 

20. Grembowski D, Ulrich CM, Paschane D, et al. Managed care and primary physician satisfaction. J Am Board Fam Pract. 2003;16:383–393.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMTYvNS8zODMiO3M6NDoiYXRvbSI7czoyMjoiL2FubmFsc2ZtLzMvNi80ODguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21. Probst JC, Greenhouse DL, Selassie AW. Patient and physician satisfaction with an outpatient care visit. J Fam Pract. 1997;45:418–425.
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=9374968&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=A1997YG26600007&link_type=ISI) 

22. Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? JAMA. 2003;290:259–264.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1001/jama.290.2.259&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12851283&link_type=MED&atom=%2Fannalsfm%2F3%2F6%2F488.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000183989800033&link_type=ISI) 

23. Sudman S, Bradburn N. *Response Effects in Surveys: A Review and Synthesis.* Chicago, Ill: Aldine Publishing Company; 1974.
    
    

24. Sudman S, Bradburn N. *Asking Questions: A Practical Guide to Questionnaire Design.* San Francisco, Calif: Jossey-Bass Inc; 1982.
    
    

25. Goodwin MA. *Using Direct Observation in Primary Care Research and the Hawthorne Effect. Defining the Nature and Impact of Nurse Observers on Patients and Family Physicians Interactions* [dissertation]. Department of Epidemiology and Biostatistics, Cleveland, Ohio: Case Western Reserve University, 2003.