Inter-Rater Reliability of Coding of Clinician Communication within a Randomized Trial to Reduce Low-Value Spinal Imaging

Context: Back pain is one of the most common reasons for primary care visits and is followed by low- value spinal imaging in a high proportion of visits. A watchful waiting (WW) approach involves following a conservative treatment plan while negotiating a waiting period prior to pursuing imaging. This study is part of a randomized trial of a standardized patient (SP)-delivered intervention to improve clinician communication and increase pursuit of WW during visits with patients with back pain.

Objective: To determine the inter-rater reliability of trained coders in assessment of intra-visit clinician communication behaviors during study visits.

Study Design and Analysis: Study visits with SPs portraying patients with acute back pain were recorded and transcribed. Guided by a coding manual, two coders independently rated targeted communication based on a modification of the Four Habits model.

Setting or Dataset: Primary care and urgent care clinics in Sacramento, CA.

Population Studied: The sample includes 52 primary care clinicians for which coding reliability was assessed on the initial 27 visits.

Intervention/Instrument: Two research assistants were trained and supervised by a medical sociologist using a codebook. SP visit transcripts were coded independently by each blinded research assistant, with disagreement resolved by consensus.

Outcome Measures: Inter-observer agreement for coding of targeted behaviors was assessed using Cohen’s weighted kappa.

Results: Weighted kappa item ratings in the following Four Habits were: 1. Invest in the beginning: 0.62 (95% CI: 0.36-0.88); 2. Elicit the patient’s perspective: 0.71 (0.48-0.94); 3. Demonstrate empathy: 0.60 (0.38-0.82); 4. Invest in the end: 0.68 (0.42-0.94).

Conclusions: Using a modification of the Four Habits model, trained coders achieved moderate to substantial inter-rater reliability when coding for physician communication behaviors related to a watchful waiting approach to reduce low-value imaging for acute back pain.