Relationship Quality and Patient-Assessed Quality of Care in VA Primary Care Clinics: Development and Validation of the Work Relationships Scale

Study Design

We drew upon Lanham et al’s 7 characteristics to develop a set of 19 items assessing the perceived quality of work relationships⁷; in some cases items were developed in collaboration with other researchers working across multiple studies and adapted for use in the VA.^13,14 All items were tested for face validity in a pilot project of 5 primary care clinics and refined based on insights gained. The resulting WRS was administered as part of a mixed methods survey of learning and relationships conducted among 17 VA primary care clinics in South and Central Texas. Questionnaires were distributed to all clinicians and staff of each clinic.

Information on patient experiences within these clinics was drawn from the Survey of Healthcare Experiences of Patients (SHEP), an ongoing initiative of the VA’s Office of Quality and Performance.¹⁵ The SHEP collects regular input from a stratified random sample of VA patients who have received inpatient or outpatient care in the preceding month. We selected 4 SHEP variables from fiscal years 2010 to 2011 for the purpose of assessing overall clinic function and patient satisfaction during the study period. “Getting care quickly” assesses whether patients felt care was provided “as soon as you thought you needed” for urgent and regular care concerns in the previous 12 months; reported percentages reflect responses of either usually or always. “Overall rating of personal doctor/nurse” and “overall rating of health care” are scored from 0 to 10, where a rating of 0 indicates worst possible and 10 indicates best possible; reporting reflects the percentage of responses that endorse a rating of 9 or 10. “Clinician wait time” describes how long the patient waited before being seen; reporting reflects the percentage of responses indicating wait times of 20 minutes or less. Data for all variables were aggregated at the clinic level.

Study team members also observed clinic members interacting within each of the clinics for 3- to 5-day periods between January 2009 and September 2011. Key informants representing all clinic roles were selected based on these observations and asked to participate in semistructured individual interviews guided by open-ended questions regarding relationships within the clinic (Supplemental Appendix, available at www.annfammed.org/content/11/6/543/suppl/DC1).

All study procedures were approved by the appropriate institutional review boards.

Data Analysis

Descriptive statistics and classical item analysis statistics were computed using procedures in SAS 9.2 (SAS Institute). Rasch item analysis and principal components analyses were conducted. A partial-credit Rasch model was used to evaluate the original 19 items. The final subset of items to include in the scale were selected based on item-response convergence with respect to the underlying latent trait, the in-fit and out-fit statistics, and established ordinality of the responses. Unidimensionality was established by the Martin-Löf test and the results from a principal components analysis on the person-item residuals. SAS PROC Mixed was used to evaluate clinic differences in WRS scores. K-nearest neighbor analysis was used to impute missing data where 2 or fewer responses to the WRS were missing. Class statement was included to account for clustering of patient and respondent data within clinics. Spearman rank-order correlations between SHEP variables and mean WRS scores were used to assess the relationship between WRS relationship scores and patient perceptions of care quality.

As an additional means of assessing construct validity, we used mean clinic WRS scores to identify the 3 highest- and 3 lowest-scoring clinics and examined whether interview data revealed variation in clinic work relationships that was congruent with variability in scale findings.¹⁶ All interviews were transcribed and uploaded into the qualitative software program NVivo 8 (QSR International). We then conducted content analysis of all interview transcripts using a codebook developed by the research team and based upon Lanham et al’s 7 relationship characteristics. All members of the research team participated in coding, with at least 2 coders reviewing each transcript, and all transcripts were compared for agreement. For this analysis, we focused on clinic members’ descriptions of staff and clinician interactions and the presence or absence of Lanham et al’s relationship characteristics, noting how particular characteristics (eg, trust) clustered across responses from members of high- and low-scoring clinics. Case studies were prepared by the first author and then reviewed and refined by the rest of the research team.

WRS Development: Item Evaluation

Using WRS responses to questionnaires collected as part of the clinic member survey, we fit a partial-credit Rasch model to item responses. Initial analysis of the degree of fit between scale items and the scale’s latent trait showed that 15 of the 19 original items had an in-fit and out-fit mean square ranging between 0.5–1.5, indicating they had good fit to the model and were productive for measurement (Supplemental Table, available at www.annfammed.org/content/11/6/543/suppl/DC1). The 4 items that fell outside this range were excluded. The original scale was coded on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree); however, inspection of the person-item map (Supplemental Figure 1, available at www.annfammed.org/content/11/6/543/suppl/DC1) showed disordered thresholds between the middle 2 categories (3 = slightly disagree and 4 = slightly agree) on a number of items. We therefore collapsed these categories to create a 5-point scale with a middle neutral response option. The likelihood ratio statistic for the Martin-Löf test supported the unidimensionality of the WRS. Principal component analysis on the person-item residuals showed little clear residual structure in the item responses. Item scores were summed to produce a total score ranging from 15 to 90. Higher scores suggest higher quality relationships. The 15-item WRS (Table 1) was found to have excellent internal reliability (Cronbach’s α = .95), and inter-item correlations ranging from 0.29 to 0.80. Figure 1 displays the person-item map for the 15-item WRS.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Person-item map for 15-item Work Relationships Scale.

Clinic Variation in WRS Scores

Clinic WRS scores are displayed in Table 2. Significant differences in WRS scores emerged between clinics (F value = 7.14, P <.0001). There was considerable variation in the number of patients enrolled in each clinic (mean = 7,630, range = 2,634–21,621) and the number of staff in each clinic (mean = 43.8, range = 17–92), but clinic WRS scores were not associated with either patient enrollment (r² = −0.39, P = .12) or number of staff (r² = −0.31, P =.22).

WRS Scores and Patient-Assessed Quality

We also assessed patient satisfaction with clinic-level quality of care, using the SHEP variables “getting care quickly” (mean = 70.4, SD = 7.8), “overall rating of personal doctor/nurse” (mean = 50.1, SD = 7.8), “overall rating of health care” (mean = 63.2, SD = 8.4), and “clinician wait time” (mean = 69.8, SD = 11.8). Two variables were significantly associated with mean WRS score: “overall rating of personal doctor/nurse” (r²= 0.53, P <.05) and “overall rating of health care” (r²= 0.50, P <.05; Table 3). Relationships between mean WRS score and the 2 remaining variables—”getting care quickly” (r²= 0.19, P= .50) and “clinician wait time” (r²= 0.08, P= .78)—were not significant.

Relationships in High- and Low-Scoring Clinics

Two hundred forty-seven clinic members representing all 17 clinics also participated in semistructured interviews. For this analysis, we examined all interviews from the 6 clinics that scored highest and lowest on the WRS. Five primary relationship characteristics emerged as consistently present in clinics with higher WRS scores: (1) rich or face-to-face communication, particularly during problem solving and conflict resolution; (2) heedful interrelating; (3) trust; (4) respectful interaction; and (5) mindfulness, particularly across clinic roles. In contrast, low-scoring clinics showed a marked lack of these characteristics. Table 4 provides quotes illustrating the presence or absence of these characteristics in high- and low-scoring clinics. Of the 2 remaining relationship characteristics, we found that diversity overlapped in this context with mindfulness and concluded that diversity did not improve our ability to characterize clinic-level relationship quality. We also found that clinic members across all clinics described using both social- and task-related communication. As comparable use of both types of communication was described across high- and low-scoring clinics, we determined that these communication styles may not function well as markers of relationship quality.

Considerable differences emerged in patterns of communication and relating between low- and high-scoring clinics. Low-scoring clinics were more likely to make use of e-mail and other lean communication methods to transmit patient and clinic-related information. Clinicians and staff in these clinics often described good working relationships with those they work with most closely, such as those within their discipline (eg, other nurses) or those on their team (as in a physician-nurse pairing), while expressing reservations about relationships with individuals in more distant clinic roles or with upper management, which at times appeared to result in conflict and inefficiency. Clinicians and staff were less likely to feel comfortable raising potential concerns and sometimes felt as though workloads were not fairly distributed. Clinic members in low-scoring clinics often described feeling as though their concerns, when raised, went unheard and unaddressed.

High-scoring clinics, by contrast, tended to augment lean communication strategies with face-to-face communication to convey information, resolve conflict, and engage in problem solving. Those in high-scoring clinics described more heedful interrelating, with a strong sense of teamwork and shared purpose, as well as greater mutual helpfulness and task coordination among diverse clinic members. Trust and respectful interaction were more frequently observed in these clinics, where staff members described feeling comfortable asking questions and expressing opinions and were notably appreciative of the skills and hard work of their colleagues. Members of high-scoring clinics also described an environment in which concerns were heard and acknowledged, reflecting greater mindfulness.