Combining Web-Based and Mail Surveys Improves Response Rates: A PBRN Study From PRIME Net

INTRODUCTION

Research using surveys for data collection is common in practice-based research networks (PBRNs). The continuing growth of electronic communications, potential for cost savings, and more rapid results has created interest in Internet-based survey designs. Regardless of the method or technology used in performing a survey, it is clear that an adequate response rate is still critical to the scientific validity of survey findings.¹ Several investigators have shown that the mode of the survey measurably affects response rates in various biomedical venues.² Our objective was to ascertain the value of a paper follow-up to an electronically based survey conducted on clinicians across 3 PBRNs.

METHODS

We performed a secondary, retrospective analysis on data obtained from a completed survey on clinician attitudes toward the screening and treatment of hepatitis C. The survey was conducted in the Primary Care Multi-Ethnic Research Network (PRIME Net).³ At the time of this study, PRIME Net was a collaboration of 3 PBRNs: the Research Involving Outpatient Settings Network (RIOS Net),⁴ the Colorado Research Network (CaReNet),⁵ and the South Eastern Regional Clinicians Network (SERCN).⁶

The survey instrument⁷ consisted of 28 questions exploring clinicians’ approaches to hepatitis C in their practices and available resources for care of patients infected with the hepatitis C virus.

Invitations to participate in the survey were sent to all 805 active clinician members of the 3 PRIME Net PBRNs for whom a valid e-mail and postal address was available. Before the survey began, clinicians received 1 or more advance e-mail messages from their home PBRN describing the purpose of the survey and requesting participation.

At the start of the survey, each clinician received a personalized e-mail solicitation from his or her home PBRN with a link to the Web-based version of the survey. The customized link had the target’s unique identifying code embedded to provide automatic log-in and response identification. We repeated the e-mail solicitation to nonresponders at intervals of 1 to 2 weeks for a total of 5 e-mail solicitations. The Web page for the survey also contained a link to a PDF version of the questionnaire that could be printed out and faxed or mailed back.

We mailed the first round of the paper survey questionnaire to nonresponders when the fourth e-mail solicitation was sent. The postal survey packet contained a personalized cover letter with a Web address that allowed the respondent the option to complete the questionnaire through the Web. A stamped, addressed envelope was provided for return mail. We sent 2 complete paper mailings and 1 postcard reminder to nonresponders only.

Continuing medical education credit, links to additional free hepatitis C continuing medical education opportunities, and periodic drawings for $75 gift certificates were offered as incentives. Procedures for both the electronic and paper formats assured respondent anonymity while allowing for targeted follow-up solicitations to nonrespondents. After the survey began, ad hoc general e-mail announcements were sent through individual network list-serves to all clinicians regardless of whether they had responded. (The purpose of these messages was to remind everyone to complete the questionnaire they had already received.)

RESULTS

A total of 301 electronic and 97 paper responses (97 postal mail and 3 faxes) yielded a valid overall PBRN response rate of 61% (Table 1⇓). Despite a total of 5 rounds of e-mail solicitations, almost one-quarter of the total responses received were in paper form (Table 1⇓).

Response times were several days faster for Web-based electronic questionnaires than for postal-based paper questionnaires (Figure 1⇓). Each e-mail solicitation had an immediate (ie, within a few days) yet diminishing response. The paper-based solicitations appeared to have a more delayed (ie, 1 week or so) response, as similarly reported in the literature (Figure 1⇓).^8–10

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Practice-based research network survey responses timeline.

The estimated total costs to create and operate the Web site (technology support) and to mail the paper questionnaires (postage, envelopes, and stationery) were $4,260 and $6,230, respectively, for a cost of $14.20 and $62.30 per usable response for the electronic and paper questionnaires, respectively.

DISCUSSION

Despite multiple solicitations for the electronic mode, 24% of total responses were in paper form. This finding suggests that paper is still an important survey mode in PBRN research. We suspect that had the paper option been provided earlier in the process, the proportion of paper responses might have been even higher. Our experience in the PBRN realm is perhaps not surprising in light of the increasing number of randomized trials that also report an added value of paper in optimizing survey response rates.^2,9,11–16 Only 2 respondents chose to print the PDF version of the survey and fax it back (as reported similarly).¹⁰

Perhaps the most touted feature of Web-based surveys is their potential for cost savings compared with paper.^6,17 If the costs of developing and hosting the Web-based survey are taken into account, the differences in cost per usable survey response may not be as large as anticipated. Interestingly, there is an economy of scale dependent on sequencing. Electronic surveys have higher initial fixed costs but limited marginal costs. Postal surveys have lower fixed costs but higher marginal costs.⁸ Reversing the sequence in which electronic- and paper-based questionnaires were offered would substantially increase the total cost.

The mixed-mode design of this survey was an attempt to reduce overall cost while maintaining adequate response rate. The higher cost of paper in this setting may be justifiable because it is possible we received paper responses from clinicians who might have responded only to paper regardless of the number of electronic solicitations. On the other hand, possibly some paper responders did so because they might have believed the paper questionnaire was their last opportunity to respond. With either case, those performing surveys of PBRN clinicians may do well to consider using both electronic and paper methods to enhance the total response rate. In our study, the response rate to the electronic version of the survey alone was insufficient to support the internal validity of the survey results.

The conclusions that can be drawn from this analysis are somewhat limited because of the lack of a control group. Nevertheless, this study provides important data about surveys conducted in the context of clinicians in a PBRN.^11–17 Results reported here suggest that in PBRN research surveys, using a mixed-mode research method that includes paper may yield a greater response rate than electronic alone.