Abstract
PURPOSE Best-worst scaling (BWS) is a survey method for assessing individuals’ priorities. It identifies the extremes—best and worst items, most and least important factors, biggest and smallest influences—among sets. In this article, we demonstrate an application of BWS in a primary care setting to illustrate its use in identifying patient priorities for services.
METHODS We conducted a BWS survey in 2014 in Boston, Massachusetts, to assess the relative importance of 10 previously identified attributes of Papanicolaou (Pap) testing services among women experiencing homelessness. Women were asked to evaluate 11 sets of 5 attributes of Pap services, and identify which attribute among each set would have the biggest and smallest influence on promoting uptake. We show how frequency analysis can be used to analyze results.
RESULTS In all, 165 women participated, a response rate of 72%. We identified the most and least salient influences on encouraging Pap screening based on their frequency of report among our sample, with possible standardized scores ranging from+1.0 (biggest influence) to −1.0 (smallest influence). Most important was the availability of support for issues beyond health (+0.39), while least important was the availability of accommodations for personal hygiene (−0.27).
CONCLUSIONS BWS quantifies patient priorities in a manner that is transparent and accessible. It is easily comprehendible by patients and relatively easy to administer. Our application illustrates its use in a vulnerable population, showing that factors beyond those typically provided in health care settings are highly important to women in seeking Pap screening. This approach can be applied to other health care services where prioritization is helpful to guide decisions.
- stated preferences
- conjoint analysis
- methods
- best-worst scaling
- homeless
- cervical cancer
- vulnerable populations
INTRODUCTION
Best-worst scaling (BWS) is a survey method for assessing individuals’ priorities: what they view as best and worst among a set of items.1 BWS has been used to illuminate patients’ and clinicians’ priorities, such as which disease symptoms are most important to patients,2 what factors most influence patients in choosing a doctor3 or using a medication,4 and what is important for clinicians when deciding on treatments.5 BWS belongs to the conjoint analysis family of methods, which collectively serve to identify preferences and trade-offs that contribute to individuals’ choices with respect to “goods.”6 These methods have been noted for their cognitive and administrative simplicity,7 as well as their focus on the patient perspective.8,9 In this article, we present the BWS method in the context of cervical cancer screening priorities for homeless women.
Homeless women die from cervical cancer at a rate nearly 6 times that of the general population.10 The rate of guideline-adherent screening among homeless women has been reported to be approximately 50%, compared with 82.8% of the insured US population and 61.9% of the uninsured population.11–16 The literature identifies financial, transportation, and scheduling issues, and sources of care as critical obstacles for underserved women.17–19 BWS can distinguish among obstacles to identify those most salient for choice—improving on a list of important factors by quantifying the relative importance of each.
METHODS
Participants
We recruited adult women experiencing homelessness in Boston, Massachusetts, from 5 sheltered settings: a homeless medical respite center; a health care for the homeless outpatient medical clinic; an emergency (ie, overnight) shelter; a transitional housing program; and a residential substance use treatment program. Inclusion criteria were being aged 18 years or older, verbally proficient in English or Spanish, and awake and alert, as determined by the recruiter. We excluded women reporting a hysterectomy. We conducted recruitment at each site until the pool of eligible women was exhausted or the target quota for each site was achieved. Power calculations for this survey design are not yet available (see de Bekker-Grob et al20 for most recent guidance); we sought a minimum sample size of 150 women to provide reasonable precision in our estimates balanced with feasibility of recruitment.21
Measures
BWS identifies the relative importance of the attributes of a good.22 We used the BWS object case method, in which the importance of each contributing attribute of a decision is evaluated relative to that of all others.23 We asked women to evaluate previously identified attributes of a hypothetical Papanicolaou (Pap) smear screening intervention24 in terms of their relative influence on homeless women’s decision to be tested: women indicated which they thought would have the biggest and smallest influence on screening decisions. The 11 attributes encompassed 4 categories: provider (sex of patient’s choice, provider kindness, and provider is familiar); setting (accepting setting and no-cost testing); procedure (convenience of testing time/scheduling, explanation/questions answered); and personal fears/concerns (personal hygiene accommodations, testing is not contingent on substance use, counseling about results is provided, and a general category described as “support is provided around all issues the woman is facing”).24
Interviewers showed women a sequence of 11 sets of 5 attributes each on a tablet computer. For each set, the woman indicated first which 1 had the biggest influence and then which 1 had the smallest influence, for a total of 2 responses per set (11 biggest influences and 11 smallest influences, or 22 data points per respondent). BWS is considered superior to rating scales because it avoids common biases,25 and superior to rankings because it is cognitively simpler to identify items that are the extremes among a list than those that are in a middle position (eg, the best and worst items of a list are relatively easy to identify, whereas the middle items of a list are more difficult to distinguish among).7 Each set of 5 attributes was presented singly; a sample question is given in Figure 1. The 11 sets were determined by an experimental design that ensured that each attribute was presented 5 times in total and viewed with each other twice (a so-called balanced incomplete block design).26,27 The order of the 5 attributes within each set was randomly assigned before the questionnaire was administered; the order of the 11 sets was randomly assigned for each respondent.
Sample screen shot of a question from the survey.
Pap = Papanicolaou.
Note: Respondents were shown 11 questions similar to this, each with a different set of 5 attributes (ie, “objects,” predefined per the experimental design, so that each was seen 5 times and in combination with each other attribute twice during the course of the survey). They were instructed to choose which attribute would have the biggest influence on women’s decision to get tested, and which would have the smallest. A button appeared highlighted with color when it was “clicked.” After choosing biggest and smallest, respondents proceeded to the next question until completion of the 11 total sets.
Demographic and health data were collected at the end of the questionnaire. The questionnaire was available in English or Spanish, for self-completion or with interviewer assistance. (It is available from the authors on request.) Participants were given a $10 gift card as remuneration. Verbal informed consent was obtained before the start of the survey.
Statistical Analysis
We analyzed BWS response data in 2 ways: as frequency counts for the number of times each attribute was chosen as biggest and smallest across the series of 11 sets presented to each respondent, and as a standardized “score” for each attribute. The score was calculated as the difference between the frequency of being chosen as biggest vs smallest divided by the availability of each attribute (meaning the number of times it appeared across the design, or 5 × number of respondents)23,26—that is, (frequency of biggest – frequency of smallest)/(5 × 165). We generated confidence intervals for the standardized scores by bootstrapping respondents 1,000 times (eg, for the overall group, 165 women were sampled with replacement 1,000 times to generate different survey samples).28 The standardized score indicates the relative strength of influence, or salience, of an attribute across all women in the sample. Standardized scores are on a scale from −1.0 to +1.0, with a score of 0 indicating no salience, and scores toward ±1.0 indicating increasing salience. In our design, scores toward +1.0 indicate salience as a biggest influence on testing, scores toward −1.0 as a smallest influence, and a score of 0 indicates no salience to the decision to undergo testing. Scores have been shown to provide similar information as coefficients from a conditional logistic regression choice model, but with greater ease of calculation and interpretation.23,29,30 Finally, we conducted stratified analyses by factors hypothesized a priori to affect priorities: time since last Pap smear (within 12 months vs not), and self-reported addiction status (addicted vs not). Analyses were conducted using Stata 11 (StataCorp LP).
Although our data set was complete—that is, all women completed all choice screens—this technique is robust to missing data because it assumes no choice when an item is left missing. Any item that is not selected from a set is considered neither best nor worst, so it contributes nothing to the numerator of the standardized score, yet contributes to the denominator by virtue of being available for choice. A profusion of missing data from a single respondent might suggest miscomprehension of the task or a decision not to participate, and could trigger exclusion from the set.
The study was approved by the Harvard T. H. Chan School of Public Health Institutional Review Board.
RESULTS
Participants
A total of 165 women completed the survey between February and June 2014 (72% response rate); reasons for declining included lack of interest, scheduling conflicts, and competing priorities at that moment. Demographic, health history, and questionnaire completion data are shown in Table 1. Of note, 90% of women reported having had a Pap test within the past 3 years.
Sample Characteristics and Questionnaire Completion (N = 165)
Priorities
Women most frequently cited the biggest influence on the decision to pursue testing as “support is available for all issues the woman is facing” (370 women), followed by no-cost testing (277 women) (Table 2). Personal hygiene accommodations and the provider being familiar to the woman were most frequently cited as having the smallest influence on the decision to pursue testing (309 and 262 women, respectively).
Subjective Priority of Attributes Influencing Acceptance of Pap Smears Among Homeless Women: Frequency Counts and Standardized Score
The relative priority assigned to each attribute is represented by the standardized scores depicted in Figure 2. For example, the score for “support is available for all issues the woman is facing” is computed as (370 − 51)/(5 × 165) = 0.39. The length of the bars represents the relative salience of each attribute on the screening decision, standardized to a scale ranging from −1.0 to +1.0; the 95% confidence intervals indicate the precision of each estimated score (overlapping intervals indicate that the difference is not significant). Stratified analyses showed that personal hygiene accommodations were significantly less important for women who reported addiction, but no other attributes differed significantly across strata (results not shown).
Standardized scores for the 11 hypothetical screening attributes.
Note: Standardized “biggest – smallest” scores and 95% confidence intervals for the 11 hypothetical screening attributes influencing homeless women’s decision to be tested. A total of 165 women participated, each of whom chose biggest and smallest attributes from 11 sets of 5 attributes each (3,630 total choices).
Interviewers used their judgment to assess participant comprehension and cooperation, and we conducted analyses with and without the 5 women who did not meet the criteria for comprehension and cooperation. We found no difference in results and therefore included all women in our analytic data set.
DISCUSSION
BWS offers a simple and transparent, patient-centered method to assess the relative importance of factors that guide decisions. The method is feasible for both patients and investigators, and extends insights to priorities that can directly inform the allocation of resources by dictating what should be done first among a list of possibilities.
Homeless women were willing and able to complete our survey, as indicated by our response rate and minimal missing data. Our results showed distinctions with the literature: characteristics of providers and settings were relatively low in priority for our women compared with those reported in the literature,17,19,31–33 as were provider sex and perceived barriers to care due to substance use.17,24,34 Our illustration of BWS demonstrates a sorting on importance of the multitude of factors identified as barriers or facilitators in decisions—critical for policy decisions and informative for intervention design.
The limitations of BWS impose some constraints on results. First, this method provides a prioritization among those attributes included in the task, so results are limited to and relative among the attributes included in the BWS questions—excluding factors omitted from the inquiry. Second, good practice dictates that BWS attributes be derived empirically from qualitative research and the descriptors be pretested to measure comprehension and understanding.35 Nevertheless, as qualitative descriptors, attributes are subject to respondent interpretation: they may be interpreted differently than intended, or differently across respondents. Our most important attribute, “support is available on all issues the woman is facing,” was derived from a compilation of concerns expressed by women in our focus groups (such as housing, emotional support, transportation, child care), and intended to capture things not typically provided in a health care setting.24 The broad scope of this attribute may be responsible for its relative importance to women—it may have been differently interpreted across women and multiple factors included in the single attribute. BWS surveys can be designed to offer finer distinctions among attributes to address this limitation. The object case that we used measures priorities across attributes defined categorically, for example, support vs cost vs hygiene accommodations.23 The “profile” case extends the design to allow for levels within each attribute for those that are ordinal or continuous: a little or a lot of support; low, moderate, or high cost.23 By adding more distinctions within each attribute, the number of possible combinations of choices increases and, hence, the sample size required to collect data on all combinations increases. We could have further distinguished among the components of “support for all issues” to illuminate what within that category was important to women, but our intent was to distinguish among those factors internal and external to the health care system. The BWS design can be selected to assess the specific trade-offs of interest to the investigator, and pretesting can inform interpretation of the attribute descriptive labels.
A third limitation is that population samples are rarely available for survey research, so selection bias is often a concern. Ninety percent of the women in our sample reported having had a Pap smear within the past 3 years (the current recommended screening interval36), far above the 51.4% screening rate for the entire Boston Health Care for the Homeless Program.37 We recruited women exclusively from sheltered locations and excluded women who were not awake and alert, and thereby possibly selected women who are screened at a higher rate. Moreover, our qualitative data revealed some confusion among women in distinguishing between pelvic examinations and Pap smears, which may have led to their overreporting of Pap smears in our survey.24 One approach to address selection bias in samples is to frame questions in terms of a general population to which respondents belong, or to ask respondents to consider policy options—both of which can help avoid personal reports. We framed our question by asking our respondents to consider the priorities of “women like them” to encourage population insights and possibly inform choices beyond the screening history of our particular women. Nevertheless, the external validity of our results is limited.
Finally, BWS assumes that preferences are sufficiently similar across individuals in the sample that the mean is representative for the group. This assumption may not be reasonable for all samples, and methods have been proposed to explore subgroup differences (eg, data segmentation and latent class analysis38,39).
In conclusion, BWS quantifies patient priorities in a manner that is transparent and accessible. It is easily comprehendible by survey participants and relatively simple to administer. Our application illustrates its use in a vulnerable population, showing that factors beyond those typically provided in health care settings are highly important to women in seeking Pap tests. This approach can be applied to other health care areas where prioritization is helpful to guide decisions.
Acknowledgments
The authors gratefully acknowledge the assistance of Adrianna Saada, MPH, and Emely Santiago in data collection. We also thank the organizations who generously allowed us to collect data from their patients and clients. Finally, we are most appreciative of the women who participated in the study, without whom this research would have been impossible.
Footnotes
Conflicts of interest: authors report none.
Funding support: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award number R21CA164712.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funding agreement guaranteed the authors’ independence in the research.
Previous presentations: Preliminary results were presented at the 36th Annual Meeting of the Society for Medical Decision Making; October 18–22, 2014; Miami, Florida.
- Received for publication May 12, 2015.
- Revision received February 8, 2016.
- Accepted for publication February 17, 2016.
- © 2016 Annals of Family Medicine, Inc.
