HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Supplemental Tables 1-3 In Brief TRACK Discussion: Submit a Comment Alert me when this article is cited Alert me when TRACK Comments are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Galliher, J. M. Articles by Pace, W. D. Search for Related Content PubMed PubMed Citation Articles by Galliher, J. M. Articles by Pace, W. D. Related Collections Quantitative methods PBRN research Professional practice

Representativeness of PBRN Physician Practice Patterns and Related Beliefs: The Case of the AAFP National Research Network

¹ AAFP National Research Network, Leawood, Kansas
² Department of Sociology, University of Missouri–Kansas City, Kansas City, Missouri
³ Department of Family Medicine, University of Colorado at Denver Health Sciences Center, Denver, Colorado
⁴ Department of Basic Medical Science, University of Missouri–Kansas City School of Medicine, Kansas City, Missouri

CORRESPONDING AUTHOR: James M. Galliher, PhD, National Research Network, American Academy of Family Physicians, 11400 Tomahawk Creek Pkwy, Leawood, KS 66211, jgallihe{at}aafp.org

	ABSTRACT

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
PURPOSE We wanted to compare survey responses from members of a national practice-based research network (PBRN) with those of a larger sample of family physicians to assess the generalizability of findings from the PBRN to the larger physician population.

METHODS The American Academy of Family Physicians National Research Network (AAFP NRN) conducted 3 separate national surveys among random samples of AAFP active members and physician members of the AAFP NRN. The surveys assessed self-reported clinical behaviors and beliefs related to hepatitis C, hyperlipidemia, and pharyngitis. Bivariate comparisons were conducted to detect statistical differences between the AAFP and AAFP NRN respondents on both demographic and clinically relevant survey items. Multivariate analyses of outcomes were found to be statistically significant at the bivariate level.

RESULTS Response rates to the surveys ranged from 53% to 59% for AAFP members and 60% to 72% for AAFP NRN members. The most consistent differences (P <.05) in demographic comparisons were for percentage of time spent in patient care, practice location, practice type, and census region. Bivariate comparisons found the groups differed on 8 (12%) of 66 clinically relevant survey items, with the Bonferroni correction for multiple comparisons reducing these items to 4 (6%). These comparisons were followed by multivariate analyses of outcomes that were found statistically significant at bivariate level.

CONCLUSIONS The AAFP NRN and AAFP membership differed on several demographic characteristics, but network members were overall more representative than not of the AAFP active membership in their self-reported clinical behaviors and related beliefs.

Key Words: Representativeness • generalizability of study results • practice-based research networks • PBRNs • research methods • statistical analysis

	INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
The most systematic investigations concerning primary care have been conducted in practice-based research networks (PBRNs). PBRNs appeared in the United States in the late 1970s.¹ The Ambulatory Sentinel Practice Network (ASPN) was the first well-recognized national PBRN, established in 1978.¹ During the past decade the number of federally registered primary care PBRNs has grown rapidly to more than 110.² Based on prior accomplishments of PBRNs, Green has argued that "the feasibility of these networks has been established." ³ Recognizing this viability, the Agency for Healthcare Research and Quality (AHRQ) funded 4 rounds of infrastructure grants between 2000 and 2003 to existing and developing networks.²

Nerenz has stated, "research in primary care...generally presumes that knowledge gained in a study is generalizable" and can be made widely available for improving health care.⁴ Yet, the generalizability of PBRN findings is possible only to the extent that study clinicians and their practice patterns or their patients are representative of the universe of primary care clinicians.⁴ Yet, representativeness of PBRN clinicians cannot be assumed. Nutting et al assert that "the clinicians in these networks are volunteers.... This voluntary nature of the organization creates the potential for selection and observer bias in the studies conducted in networks." ⁵ Wetzel et al have also reported significant differences in demographic characteristics of family physicians willing to participate in a quality improvement study compared with nonresponders. They question whether network-based studies can be generalized.⁶

Stange has stated that PBRN physicians are likely to be systematically different from the average clinician.⁷ This view is repeated by Croughan⁸: "physicians who participate in research and/or practice-based research networks are different from physicians who do not participate." As Croughan is quick to acknowledge, however, the difficulty is in "delineating and describing those differences." Nutting et al argue: "[T]here remains strong reason to suspect that physicians who devote substantial portions of their time to research are not completely typical of the larger universe of family [and primary care] physicians." ⁵ Further, Stange asserts that studies of PBRN physician behavior are likely to be biased toward higher standards of care than studies of a representative sample of non-PBRN physicians.⁷ Thus, PBRN studies that evaluate care practices, including clinician responses to patient problems, recognition of disease, and natural history studies where an intervention is part of the analysis, all depend on the generalizability of PBRN clinicians.

Research Question and Hypothesis
To gain a better understanding of the comparability of PBRN physicians’ clinical decision making with non-network physicians, direct comparisons are needed in studies that enroll both groups of physicians. This work is based on 3 studies conducted by the American Academy of Family Physicians National Research Network (AAFP NRN) in which survey data were collected from both independent random samples of AAFP active members and AAFP NRN physician members. Using these 3 data sets, we address the question: To what extent are AAFP NRN physicians representative of the larger population of family physicians with respect to their self-reported clinical practices, behaviors, knowledge, and beliefs about selected clinical issues? Based on the work of Croughan,⁸ Nutting et al,⁵ and Stange,⁷ we hypothesize that self-reported practice patterns, knowledge, and beliefs about clinical care issues between these 2 groups will differ.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
The Surveys
The AAFP NRN conducted 3 national surveys used for this analysis: hepatitis C survey (2003),⁹ hyperlipidemia survey (2003–2004),¹⁰ and pharyngitis survey (2004).¹¹ For each survey, a random sample of participants was selected from AAFP active members identified within the AAFP Member Master Database. This database contains demographic information on all AAFP members. A second set of participants for each survey consisted of all physician members of the AAFP NRN (Table 1). Approximately 90% of AAFP NRN members belong to the AAFP, thus the AAFP NRN represents a subset of the AAFP membership. AAFP NRN members who also were members of the AAFP were excluded from selection into the AAFP sample—they could not be in both surveyed groups. The specific methods and results from these surveys have been published.^9–11

Accounting for Missing Data
We selected 14 demographic characteristics from the AAFP Member Master Database for comparing AAFP survey respondents with the AAFP population; these data are shown in Supplemental Table 1 (available at http://annfammed.org/cgi/content/full/7/6/547/DC1). Missing data per demographic factor for each survey ranged from 0% to 4%, with only 4 items (9.5%) across the 3 population databases having greater than 1% with data missing.

Although there are alternative techniques for imputing missing data and ongoing debates^12–17 concerning their use, we chose a straightforward approach to the imputation of missing data on both the categorical and continuous variables. We first created a composite factor based on 7 demographic items for which no data were missing. These items included census region (4 categories) and 6 dichotomous components of practice variables (physician’s practice includes coronary care unit, emergency room, intensive care unit, obstetrics, pediatrics, and surgery), where each variable’s response categories were yes and no. Combining these 7 variables resulted in a composite with 28 categories, with each category representing a census region (1 to 4) and the number of practice components endorsed (0 to 6).

Each of the 5 categorical demographic items (Supplemental Table 1, at http://www.annfammed.org/cgi/content/full/7/6/547/DC1, provides information about the physicians’ sex, practice arrangement, practice base, practice location, and practice major owner), with missing data included as a category, was then cross-classified separately with the composite variable. If a given resulting cell had missing cases, those specific cases were randomly assigned to a given category on the demographic variable in proportion to that category when missing cases were excluded. Thus, if sex had 10 missing cases for 1 of 28 cells and the proportion of men for that cell was 70%, then 7 of the missing cases were assigned randomly as men with the other 3 cases assigned as women. For the continuous variables of age and years in practice, individual means were calculated for each of the 28 cells on the composite factor. If there were missing cases on 1 of the continuous variables per category, those cases were assigned the mean for that category.

AAFP Respondents Compared With the AAFP Population
Before making comparisons between AAFP and AAFP NRN respondents, we assessed the comparability of the AAFP respondents to the larger AAFP population from which the random sample was selected for each survey across 15 demographic items. The results (Supplemental Table 2, available at: http://annfammed.org/cgi/content/full/7/6/547/DC1) show that of the 45 demographic comparisons across the 3 surveys, the AAFP respondents differed from the larger population at the P <.10 level on 8 (18%) of these factors (3 for hepatitis survey, 3 for hyperlipidemia survey, and 2 for pharyngitis survey). Based on the results from the statistical comparisons (² goodness-of-fit test for categorical variables,¹⁸ and the 1-sample Z test for continuous variables),¹⁹ the AAFP sample respondents’ survey data were weighted to better represent the distributions from the AAFP population (weights available upon request). We elected not to weight the AAFP NRN response to the total AAFP NRN membership because of their much higher overall survey response rates, and because some members of the AAFP NRN were not members of the AAFP with the requisite data available from the AAFP Member Master Database.

Selection of Survey Items for Group Comparisons
The 3 survey instruments included several demographic items and 139 clinical questions (survey instruments available upon request). The topic areas of these clinical items are shown in Supplemental Table 3, available online at http://annfammed.org/cgi/content/full/7/6/547/DC1. Among the clinical questions, 76 (55%) that were considered by a clinician author (W.D.P.) as conceptually most closely related to self-reported clinical practices, behaviors, knowledge, and beliefs were initially selected for primary analysis. Ten were discarded because they lacked ample variation in responses from both AAFP respondents and AAFP NRN respondents, leaving 66 for comparison. The remaining 63 clinical items not directly related to the research question (such as availability of rapid testing for streptococcus infection) were excluded.

We limited our analysis of the 66 comparisons to (1) focus on clinical decision making and (2) remove items that would clearly not show differences between the 2 groups because they lacked variability in responses.

Statistical Analysis
Statistical analysis was conducted using SAS 9.1 (Cary, North Carolina, SAS Institute Inc, 2003). For bivariate comparisons, we used ² tests for categorical demographic and outcome variables and t tests for continuous variables. Differences are reported both at the P <.05 level and after correcting for multiple comparisons. For the multivariate analysis, we used logistic and linear regression procedures for categorical and continuous outcome variables, respectively.

Survey Approval
Each survey was approved as exempt by the Social Science Institutional Review Board of the University of Missouri–Kansas City. The research comparing AAFP members with AAFP NRN physicians across the 3 surveys was approved as exempt by the AAFP Institutional Review Board.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Response Rates and Demographic Comparisons
Response rates varied across the 3 surveys (Table 1). The AAFP mailings for the hepatitis C, hyperlipidemia, and pharyngitis surveys resulted in response rates of 53% (634), 58% (676), and 59% (583). The AAFP NRN response rates were 72% (174), 60% (155), and 71% (186). For both the hepatitis C and pharyngitis surveys, the AAFP NRN’s response rates were statistically higher (P <.001) relative to the response rates of the AAFP members. This difference was expected because the AAFP NRN members had previously elected to contribute to research projects. The response rates were not statistically higher for AAFP NRN members for the hyperlipidemia survey (60% vs 58%), as there were no follow-up mailings to nonrespondents.

Comparisons of the AAFP with AAFP NRN respondents on selected demographic items from each survey are displayed in Table 2. The demographic comparisons showing the largest and most consistent differences—although not all were statistically significant at P <.05 level across the 3 surveys—were for percentage of time spent in direct patient care, practice location, practice type, and census region distribution. AAFP NRN physicians reported less time spent in direct patient care (P <.001), were more likely to represent urban areas and less likely to represent rural areas (P=.016 to .054), were more likely to work in a residency or university practice (P <.001), and were more likely to be from the Northeast and less likely to be from the Midwest (P=.006 to .099). AAFP NRN physicians also were older (P=.046 to .186), and had spent more years in practice (P=.015 to .067).

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
To our knowledge, this study is the first to directly compare self-reported clinical behaviors and beliefs of PBRN network members—in this case members of the AAFP NRN—with sampled respondents representing the larger population of family physicians from which the great majority of network members come (in this case members of the AAFP). Moreover, our research compared PBRN members with their nonnetwork counterparts across 3 separate surveys representing diverse foci of clinical care—hepatitis C, hyperlipidemia, and pharyngitis. The results of these comparisons are consistent: there are few substantive differences observed between the 2 groups in their self-reported practice patterns, knowledge, and beliefs related to these 3 clinical areas. When controlling for the increased probability of a type 1 error across the 66 substantive comparisons and 3 surveys (ie, Bonferroni correction), the number of statistically significant bivariate comparisons was reduced from 8 (12%) (P <.05) to 3 (5%) (P =.001). The multivariate analysis controlling for known demographic variables also reduced the number of statistically significant comparisons from 8 to 3. These findings indicate that the AAFP NRN physician members represent a good approximation of the AAFP membership as a whole in their self-reported practice patterns and clinical beliefs. Controlling for known demographic differences further reduces self-reported clinical differences.

The question of representativeness and generalizability of PBRN study results to the larger population of primary care is of central importance for clinical and policy reasons. Nonetheless, this question has not been thoroughly addressed during the 3 decades of PBRN activity in the United States. According to Stange⁷: "The main reason that the representativeness...is not frequently assessed is a practical one. If it were easy to study nonrespondents and others not included in a study sample, they would have been included in the sample to begin with. Obtaining access to nonrespondents or a measure that truly represents the population from which the study sample came is usually difficult and frequently impossible." This study presents direct comparisons between PBRN physician members (AAFP NRN) and the larger population of active AAFP members.

There were consistent differences between AAFP members and AAFP NRN members in selected demographic characteristics. The demographic differences thus highlight the need to look deeper into the clinical arena to determine whether these demographic differences extend to clinical care. Published studies vary on the representativeness of PBRN network clinicians, patient populations, or problems seen. Green et al showed that patient visits to physicians participating in the Ambulatory Sentinel Practice Network (ASPN) were generally representative of the larger population of primary care patient visits,²⁰ and Nutting et al found that services provided by ASPN practices were similar to those offered by non-PBRN practices.⁵ Binns et al compared the provision of selected services between PBRN members and data collected as part of the National Ambulatory Care Survey (NAMCS) and showed a number of differences between the 2 groups.²¹ The instruments were not the same, however, and the PBRN members included a much higher percentage of pediatricians than does NAMCS. Thus, the differences shown in the Binns et al project are difficult to interpret as PBRN vs non-PBRN physician differences or data collection and sampling frame differences.

The results of the current research, coupled with earlier work reported by Green et al²⁰ and Nutting et al,⁵ support the overall conclusion that the AAFP NRN is representative of the larger population of family physicians represented by the AAFP in terms of clinical beliefs and self-reported clinical practices. Thus, the results of research conducted in the AAFP NRN, after controlling for known demographic variables, can be generalized to the larger arena of family medicine, including its clinicians, patients, and patterns-of-care delivery.

These 3 sets of study results suggest that the 2 groups are not likely to differ systematically in other clinical areas (eg, depression care), although these results do not preclude such differences. Furthermore, whereas this body of work does not immediately translate to regional networks, it at least questions the contention by Croughan, Nutting, and others that PBRN members, in general, have clinical behaviors and beliefs that are different from those of nonmembers who practice in similar settings.

There are limitations to this research. First, although there were responses to many items from 3 separate surveys on which to base the statistical comparisons between the AAFP membership and AAFP NRN physicians, the similarities between their responses may simply be fortuitous. Other clinical areas (diabetes, depression) might show a greater number of differences in practice patterns, knowledge, and beliefs between the 2 groups. Regional PBRNs, particularly those with selected membership requirements, such as only in Federally Qualified Health Centers or within a single medical organization, might differ more widely from the universe of primary care physicians.

Second, the 66 specific items selected for comparing the 2 physician groups might well have been biased in favor of not finding many statistical differences. When we conducted the analysis using all 129 (66 + 63) clinical items with sufficient variability across the 3 surveys, however, the results did not substantially change—19% (9 of 77 for hepatitis C survey, 7 of 21 for hyperlipidemia survey, and 8 of 31 for pharyngitis survey) were found statistically significant (P <.05); after using the Bonferroni correction, 6 (5%) of these differences remained (P =.001).

Third, respondents may not have understood all survey questions posed to them and consequently responded to idiosyncratic interpretations of certain questions. Because these surveys were conducted by mail, respondents did not have an opportunity to ask for clarification. Comparable responses thus do not guarantee similarity of interpretation or meaning, though there is no reason to suspect that AAFP NRN members would systematically interpret questions differently than AAFP members. Fourth, whereas the response rates for AAFP-sampled members were relatively high for physician surveys (average response 57%), the results may not reflect the practice patterns and beliefs of nonrespondents. To account for this discrepancy, AAFP respondents’ results were weighted to better reflect the full AAFP membership, though this weighting had little effect on the full membership survey results. Finally, this study focused on self-reported care patterns, which may vary greatly from actual care patterns²² and may not vary uniformly between AAFP NRN members and the AAFP membership. We did not obtain information on the actual practice patterns of surveyed physicians.

The Institute of Medicine’s report on primary care published in the mid-1990s highlighted practice-based research networks for their potential to study and understand primary care.²³ The results of this research and that reported by Green et al²⁰ and Nutting et al⁵ lend credibility to this statement. Our analysis of these 3 studies has shown that physician member volunteers of a national PBRN are more similar than not to their nonnetwork counterparts in their self-reported clinical practice patterns, knowledge and beliefs, and patients.

	FOOTNOTES

 
Conflicts of interest: none reported

Funding support: This study was funded in part by a grant to the AAFP NRN by the AAFP and AAFP Foundation’s Joint Grant Awards Program.

Received for publication February 18, 2008. Revision received February 17, 2009. Accepted for publication February 24, 2009.

	REFERENCES

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 

1. Nutting PA. A case study of the Ambulatory Sentinel Practice Network. Practice-based research networks in the 21st century: the pearls of research. In: Proceedings from the Conference Convened by the AAFP Task Force to Enhance Family Practice Research. Leesburg, VA. Leawood, KS: American Academy of Family Physicians.
2. Agency for Healthcare Research and Quality (AHRQ). Fact Sheet: AHRQ Support for Primary Care Practice-Based Research Networks (PBRNs). 2006. http://www.ahrq.gov/research/pbrn/pbrnfact.htm.
3. Green LA. The history of PBRNs: the establishment of practice-based primary care research networks in the United States. In: Proceedings from the conference convened by the AAFP Task Force to Enhance Family Practice Research. Leawood, KS: American Academy of Family Physicians; 1999.
4. Nerenz DR. Primary care research from a health system perspective. J Fam Pract. 1996;42(2):186–191.[Medline]
5. Nutting PA, Baier M, Werner JJ, Cutter G, Reed FM, Orzano AJ. Practice patterns of family physicians in practice-based research networks: a report from ASPN. Ambulatory Sentinel Practice Network. J Am Board Fam Pract. 1999;12(4):278–284.[Medline]
6. Wetzel D, Himmel W, Heidenreich R, et al. Participation in a quality of care study and consequences for generalizability of general practice research. Fam Pract. 2005;22(4):458–464.[Abstract/Free Full Text]
7. Stange KC. Practice-based research networks. Their current level of validity, generalizability, and potential for wider application. Arch Fam Med. 1993;2(9):921–923.[Abstract/Free Full Text]
8. Croughan Mary. Generalizability in PBRN studies: are the results of practice-based research generalizable? Methods for practice-based research networks: challenges and opportunities. In: Proceedings from the Practice-Based Research Networks Methods Conference. Convened by the AAFP Task Force, Plan to Enhance Family Practice Research. San Antonio, Texas. Leawood, KS: American Academy of Family Physicians; 2003.
9. Clark EC, Yawn BP, Galliher JM, Temte JL, Hickner J. Hepatitis C identification and management by family physicians. Fam Med. 2005;37(9):644–649.[Medline]
10. Eaton CB, Galliher JM, McBride PE, Bonham AJ, Kappus JA, Hickner J. Family physician’s knowledge, beliefs, and self-reported practice patterns regarding hyperlipidemia: a National Research Network (NRN) survey. J Am Board Fam Med. 2006;19(1):46–53.[Abstract/Free Full Text]
11. Park SY, Gerber MA, Tanz RR, et al. Clinicians’ management of children and adolescents with acute pharyngitis. Pediatrics. 2006; 117(6):1871–1878.[Abstract/Free Full Text]
12. Allison PD. Multiple imputation for missing data: a cautionary tale. Sociol Methods Res. 2002;28(3):301–309.[CrossRef]
13. Sinharay S, Stern HS, Russell D. The use of multiple imputation for the analysis of missing data. Psychol Methods. 2001;6(4):317–329.[Medline]
14. Allison PD. Missing Data. Thousand Oaks, CA: Sage; 2002.
15. Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods. 2002;7(2):147–177.[CrossRef][Medline]
16. Cohen J, Cohen P, West SG, Aiken LS. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. 3rd edition. Mahwah, NJ: Lawrence Eribaum; 2003.
17. Scheuren F. Multiple imputation: how it began and continues. Am Stat. 2005;59(4):315–319.[CrossRef]
18. Hays WL. Statistics. 5th ed. New York, NY: Harcourt Brace College Publishers; 1994.
19. Blalock HM Jr. Social Statistics. 2nd ed. New York, NY: McGraw-Hill; 1972.
20. Green LA, Miller RS, Reed FM, Iverson DC, Barley GE. How representative of typical practice are practice-based research networks? A report from the Ambulatory Sentinel Practice Network Inc (ASPN). Arch Fam Med. 1993;2(9):939–949.[Abstract/Free Full Text]
21. Binns HJ, Lanier D, Pace WD, et al. Primary Care Network Survey (PRINS) Participants. Describing primary care encounters: the Primary Care Network Survey and the National Ambulatory Medical Care Survey. Ann Fam Med. 2007;5(1):39–47.[Abstract/Free Full Text]
22. Montano D, Phillips W. Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit. Am J Public Health.1995;85(6):795–800.[Abstract/Free Full Text]
23. Donaldson MS, Yordy KD, Lohr KN, Vanselow NA, eds. Primary Care: America’s Health in a New Era. Washington, DC: National Academy Press; 1996.

This article has been cited by other articles:

				L. S. Acheson and J. J. Frey III In This Issue: Multifaceted Research Ann. Fam. Med, November 1, 2009; 7(6): 482 - 483. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Supplemental Tables 1-3 In Brief TRACK Discussion: Submit a Comment Alert me when this article is cited Alert me when TRACK Comments are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Galliher, J. M. Articles by Pace, W. D. Search for Related Content PubMed PubMed Citation Articles by Galliher, J. M. Articles by Pace, W. D. Related Collections Quantitative methods PBRN research Professional practice