Research ArticleOriginal Research

A Nationwide Flash-Mob Study for Suspected Acute Coronary Syndrome

Angel M.R. Schols, Robert T. A. Willemsen, Tobias N. Bonten, Martijn H. Rutten, Patricia M. Stassen, Bas L. J. H. Kietselaer, Geert-Jan Dinant and Jochen W.L. Cals
The Annals of Family Medicine July 2019, 17 (4) 296-303; DOI: https://doi.org/10.1370/afm.2401

Abstract

PURPOSE Our primary objective was to evaluate the Marburg Heart Score (MHS), a clinical decision rule, or to develop an adapted clinical decision rule for family physicians (FPs) to safely rule out acute coronary syndrome (ACS) in patients referred to secondary care for suspected ACS. The secondary objective was to evaluate the feasibility of using the flash-mob method, an innovative study design, for large-scale research in family medicine.

METHODS In this 2-week, nationwide, prospective, observational, flash-mob study, FPs collected data on possible ACS predictors and assessed ACS probability (on a scale of 1-10) in patients referred to secondary care for suspected ACS.

RESULTS We collected data for 258 patients in 2 weeks by mobilizing approximately 1 in 5 FPs throughout the country via ambassadors. A final diagnosis was obtained for 243 patients (94.2%), of whom 45 (18.5%) received a diagnosis of ACS. Sex, sex-adjusted age, and ischemic changes on electrocardiography were significantly associated with ACS. The sensitivity of the MHS (cut-off ≤2) was 75.0%, specificity was 44.0%, positive predictive value was 24.3%, and negative predictive value was 88.0%. For the FP assessment (cut-off ≤5), these test characteristics were 86.7%, 41.4%, 25.2%, and 93.2%, respectively.

CONCLUSIONS For patients referred to emergency care, ACS could not be safely ruled out using the MHS or FP clinical assessment. The flash-mob study design may be a feasible alternative research method to investigate relatively simple, clinically relevant research questions in family medicine on a large scale and over a relatively short time frame.

Key words

INTRODUCTION

The Marburg Heart Score (MHS), a clinical decision rule based on 5 signs and symptoms (Table 1), has shown promising results in assisting family physicians (FPs) to identify patients with a low probability of acute coronary syndrome (ACS) as the underlying cause of chest pain in the primary care population.1–3 In contrast to the History, Electrocardiogram, Age, Risk factors, and Troponin (HEART) score validated in emergency departments, additional diagnostic tests, such as electrocardiography (ECG) or cardiac troponin, are not included in the MHS.4–6 Recently, a large meta-analysis of 3,099 primary care patients with chest pain identified 2 additional predictors of ACS, that is, the FP’s suspicion of a serious diagnosis and a pain that feels like pressure.7 However, the validity of the MHS and other possible predictors—including the FP’s clinical assessment, which is infrequently assessed and compared with decision rules—in safely ruling out ACS in otherwise referred primary care patients is unclear.3,8,9

View this table:
Table 1

Components of the Marburg Heart Score1–3

Assessing the accuracy of clinical decision rules in family practice requires large prospective studies, which are time consuming and costly. Recently, an innovative research method, the flash-mob method, has been used in hospital-based studies, allowing for the investigation of 1 simple research question on a large scale and over a short time frame.10,11 Flashmob research is based on the concept of flash mobs, “a large public gathering at which people perform an unusual or seemingly random act and then disperse, typically organized by means of the Internet or social media.”12 Previous research has shown the numeric strength of multiple hospitals and professional and social networks in flash-mob research to obtain sufficient data over a short time period.10,11 The geographically widespread organization of family medicine and the relatively few numbers of relevant patients per FP are complicating factors in large-scale conventional research, which might be overcome in part by this method. To the best of our knowledge, this flash-mob method has not been used in a nationwide study in family medicine.

The primary aim of this prospective study was to evaluate the MHS or to develop an adapted clinical decision rule for FPs to safely rule out ACS in patients referred to secondary care for suspected ACS. Our secondary aim was to evaluate the feasibility of using the flash-mob method for large-scale, relatively inexpensive, and rapid research in family medicine.

METHODS

Study Design

We performed a 2-week, nationwide, prospective, observational, flash-mob study among FPs. The study protocol was approved by the Medical Ethical Committee of Zuyderland Medical Centre (The Netherlands; 17-N-119) and prospectively registered at www.trialregister.nl (NTR6789).

The flash-mob steering committee invited FPs and organizations from their professional and social networks to act as ambassadors for the study (Figure 1). These ambassadors were asked to actively spread the word and distribute study materials among FPs nationwide. All ambassadors as well as the steering committee members used traditional professional networks as well as their social media channels to advertise the study. In addition, information about the study was distributed via personal communication, scientific journals, and our website (www.huisartshartweek.nl).13,14 Furthermore, study materials were sent to specialized FPs, 8 departments of family medicine, >100 FP out-of-hours services, and ambulance services and were included in a Dutch FP scientific journal, Huisarts en Wetenschap, distributed to approximately 14,000 professionals.

Figure 1
Figure 1

Implementation of the flash-mob method in the present study.

a Specialized FPs receive a special 2-year training in heart and vascular disease or acute medicine.

Setting and Participants

After the Dutch College of General Practitioners annual conference where the study was promoted, all patients referred by FPs for suspected ACS from November 20, 2017 to December 3, 2017 were eligible for inclusion. During that time, all FPs in the Netherlands could register patients they referred during practice hours and out-of-hours shifts using a short 1-page case report form (CRF) either on paper or online (taking <2 min). In the Netherlands, FPs perform a gate-keeping function, and hospital care can only be accessed after referral by an FP. More than 95% of all episodes of care are completely covered in primary care.15

We defined ACS as acute myocardial infarction or unstable angina pectoris. The study was originally planned for 1 week, which we extended to 2 weeks to increase the sample size. We excluded patients who were unable to provide informed consent, were not seen in person by an FP (telephone referral), or were eventually not referred to hospital. All patients were informed about the study by the participating FP and had to provide oral informed consent.

Case Report Form

The CRF consisted of 10 consultation items including MHS items, whether the attending FP immediately suspected a serious diagnosis, whether the patient reported that the pain felt like pressure, the duration of the symptoms, signs of ischemia on ECG if performed, and the attending FP’s assessment of ACS probability on a scale of 1 to 10. In addition, we registered the patient’s name, sex, date of birth, name of the patient’s own FP, whether the attending physician was an FP (if so, years of work experience) or FP trainee, and the time the patient was seen (daytime/out-of-hours and date). FPs were not asked to calculate the MHS.

Reference Standard

The final diagnosis was obtained by contacting all FPs of included patients via post/mail followed by at least 2 telephone/fax reminders at least 6 weeks after the initial consultation. We asked FPs to select the most probable final diagnosis from a short list—ACS, stable angina, other cardiac diagnosis, or noncardiac diagnosis—and to base their answer on the (discharge or other) letter from the cardiologist.

Statistical Analysis

We performed a univariate analysis to detect possible ACS predictors. The odds of having ACS were compared between patients with and without each possible predictor by calculating odds ratios with 95% CIs and Pearson χ2 tests for each possible (dichotomous) predictor. The variable sex-adjusted age dichotomized age in line with the original MHS variable using a cut-off of age ≥65 years for women and age ≥55 years for men.

The MHS and the FP assessment of ACS probability were dichotomized into positive (high risk) and negative (low risk) results, with a cut-off value of ≤2 for MHS (based on previous MHS research) and ≤5 for FP assessment (based on practical usability).1–3 For the MHS, we performed additional analyses using a cut-off value of ≤1 (also based on previous MHS research) because this cut-off value showed greater sensitivity and negative predictive value (NPV), therefore decreasing the chance of missing patients with ACS in the referred population.1 For each strategy, the sensitivity, specificity, positive predictive value (PPV), and NPV were calculated with corresponding 95% CIs according to the efficient-score method (corrected for continuity).16,17 See the Supplemental Appendix (http://www.AnnFamMed.org/content/17/4/296/suppl/DC1/) for the sensitivity and specificity of all other possible cut-off values for the FP assessment.

The diagnostic ability of the MHS and FP assessments to discriminate between patients with and without ACS was assessed by plotting sensitivity against 1-specificity on a receiver operating characteristic curve for both strategies and calculating the area under the curve (AUC).

We performed sensitivity analyses for cases with missing MHS items and imputed a 0 score for missing MHS items for patients with chest pain, assuming these missing items would have led to a negative score for that specific item. Statistical analyses were performed using IBM SPSS Statistics Data Editor version 23 (International Business Machines Corp) and the Vassarstats website: http://vassarstats.net/clin1.html.

RESULTS

Patients Registered via Flash Mob

A total of 241 FPs (including FP trainees) from all provinces in the country registered 258 patients, 113 (43.8%) online and 145 (56.2%) on paper. Of these patients, 203 (78.7%) were included by an FP (median work experience 8 years [interquartile range, 15.3 years]) and 55 (21.3%) by an FP trainee. A total of 182 (70.5%) patients were registered during office hours. Figure 2 summarizes patient inclusion and follow-up. A final diagnosis was obtained for 243 (94.2%) patients, and an MHS could be determined for 186 (72.1%) patients.

Figure 2
Figure 2

Patient inclusion and follow-up.

Final Diagnosis and Univariate Analyses of Possible ACS Predictors (n = 243)

Of the 243 patients with a final diagnosis, 45 (18.5%) received a diagnosis of ACS, of whom 34 (75.6%) had a myocardial infarction, 10 (22.2%) unstable angina, and 1 (2.2%) ACS confirmed but not otherwise speci fied. Stable angina was diagnosed for 11 (4.5%) patients, another cardiac diagnosis for 40 (16.5%) patients, and a noncardiac diagnosis for 153 (63.0%) patients.

Table 2 summarizes patient characteristics and investigated predictors. Three possible predictors—sex, sex-adjusted age, and ischemic changes on ECG—were significantly associated with ACS in the univariate analysis.

View this table:
Table 2

Patient Characteristics, Investigated Predictors, and Univariate Analyses of Possible ACS Predictors

FP Assessment of ACS Probability (n = 243)

Table 3 shows the test characteristics of the FP assessment. According to the FP assessment, 88 of 243 (36.2%) patients were at low risk of ACS (≤5) and 155 (63.8%) at high risk (>5). A total of 6 (2.5%) patients would have been falsely classified as not having ACS when using the FP assessment. The AUC for the FP assessment was 0.72 (95% CI, 0.63-0.81).

View this table:
Table 3

Test Characteristics of the Marburg Heart Score and the FP Assessment

Marburg Heart Score (n = 186)

Table 3 shows the test characteristics of the MHS using cut-off values of ≤2 and ≤1. Of the 186 patients for whom an MHS could be calculated, 75 (40.3%) were at low risk of ACS (0-2) and 111 (59.7%) at intermediate-high risk (3-5). The ACS incidence in the MHS group was 19.4%. Nine (4.8%) patients would have been falsely classified as not having ACS when using the MHS with a cut-off value of ≤2. When using the MHS with a cut-off value of ≤1, 2 (1.1%) patients would have been missed, and 24 (12.9%) patients would have been classified as true negative. The AUC for the MHS was 0.64 (95% CI, 0.54-0.74), and that for FP assessment in the group for which an MHS could be determined was 0.71 (95% CI, 0.61-0.80) (Figure 3).

Figure 3
Figure 3

ROC curves for FP assessment of ACS probability and the Marburg Heart Score in patients referred for suspected ACS (n = 186).

ACS = acute coronary syndrome; AUC = area under the curve; FP = family physician; ROC = receiver operating characteristic.

Sensitivity Analyses MHS (n = 215)

The sensitivity analysis showed a slightly greater sensitivity (76.9%), specificity (44.9%), and NPV (89.8%) of the MHS for ACS using a cut-off of ≤2. The PPV was 23.6%. The sensitivity analysis of the MHS using a cut-off of ≤1 showed a sensitivity of 94.9%, specificity of 17.0%, PPV of 20.2%, and NPV of 93.8% for ACS.

Alternative Scenario Analysis

In a strategy wherein both the MHS and FP assessments were negative, no patient would have been missed, whereas 35 of 186 (18.8%) patients would have been classified as true negative (Table 3).

DISCUSSION

This was the first nationwide flash-mob study in family medicine, which serves as a proof of concept for large-scale clinical diagnostic research among FPs within a short time frame. We showed that flash-mob research in family medicine is feasible by collecting data on 258 patients with suspected ACS in only 2 weeks. We were able to involve and motivate many organizations and FPs by mobilizing nearly 1 in 5 FPs (approximately 1,800 FPs) throughout the country via ambassadors (see Figure 1 and the Supplemental Appendix, http://www.AnnFamMed.org/content/17/4/296/suppl/DC1/, for more information). Recruiting this number of FPs in the context of traditional research would have been extremely challenging and both resource and time consuming. Prior studies on signs and symptoms and clinical decision rules in similar populations (including nonreferred patients, allowing for a larger group eligible for recruitment) showed inclusion rates of less than 30 to 40 patients per week.1–3,18–20 In addition, traditional studies often have a limited number of participating physicians, which may lead to clustering of data collected by individual physicians. The large number of different FPs and the widespread inclusion of patients scattered throughout the country strengthens the external validity of the present study. Moreover, FPs did not receive additional instructions or training; thus, the study conditions reflected aspects of daily practice.

Strengths and Limitations

Some selection bias might have occurred given the fact that very ill patients might not have been included in this study. For these severe cases, however, the use of a clinical decision rule would not be advised. We asked FPs to only register referred patients to allow for the determination of a final diagnosis based on assessment in secondary care, in contrast to prior studies that also included nonreferred patients using a reference standard based on clinical follow-up and thereby risking diagnostic misclassification.1–3,7 Cardiologists did not perform a uniform assessment, yet all patients received a standard diagnostic work-up according to current clinical practice. We did not provide an exact definition of myocardial infarction or unstable angina but asked FPs to base their answer on the final diagnosis in the cardiologist’s final report and to allow for a follow-up of at least 6 weeks. Cardiologists were blinded with regard to the MHS results, and FPs were not asked to calculate MHS values. A substantial proportion of FPs, however, did fill out the CRF as well as the final diagnosis form. The FP’s assessment was registered at the end of the CRF and might therefore have been slightly biased by the preceding items. The significant association between ischemic changes on ECG and ACS was influenced by incorporation bias because ECG abnormalities are part of the reference standard. Although MHS values could only be determined for 186 patients owing to missing items, sensitivity analyses showed only minor differences in test characteristics after imputation. Even though one-half of patients without an MHS had no chest pain, we allowed them to be included in an attempt to develop an adapted clinical decision rule for all referred patients with suspected ACS. We did not specifically investigate clinically relevant conditions other than ACS. There is an ongoing debate as to whether unstable angina is a justified diagnosis in the setting of high-sensitivity troponin assays, yet FPs do not have troponin assays available at the point of care, and unstable angina is still mentioned as part of ACS in the Dutch FP guideline.21,22

Comparison With Existing Literature

In contrast to prior studies of the MHS, we found an insufficient diagnostic accuracy of the MHS to safely rule out ACS at a cut-off value of ≤2 (NPV 88.0% compared to 97.7% to 98.1% in prior studies).1–3 This difference might be explained by spectrum bias because the present study was performed with a referred population with a greater incidence of ACS (19.4%) compared to prior MHS studies (3.7% and 2.5%).1–3,23 In addition, in those prior studies, a lower-risk population was represented because patients with stable coronary artery disease were included along with patients with unstable presentations of chest pain. This explanation is in line with our present finding that the MHS could accurately rule out ACS in the subgroup that was estimated to be at low risk according to the FP and is comparable with the finding of a prior study in a primary care population with an ACS incidence of 22%.8 That study also found that a clinical decision rule could be used to safely rule out ACS in patients considered to be at low risk according to the FP’s assessment. The overall AUC we found for FP assessment was also in line with that study, which showed an AUC for FP risk estimation of 0.75 (95% CI, 0.68-0.82) compared to 0.72 (95% CI, 0.63-0.81) in the present study.8

Implications for Clinical Practice and Future Research

The MHS and FP assessments individually showed insufficient diagnostic accuracy to safely rule out ACS in referred patients. When combined, however, they safely reduced the number of referrals by 19% by applying the MHS only for referred patients considered to be at low risk of having an ACS by the FP’s assessment. Yet, such a strategy meets practical limitations. FPs would have to apply this strategy after they made the decision to refer, meaning FPs should incidentally correct their decision to refer’. Therefore, the suggested strategy should be validated in a sufficiently large cohort including both referred and nonreferred patients with suspected ACS.

Although we consider flash-mob research in family medicine feasible and that it may be considered for use as a new research method, it should be noted that the flash-mob method is not suitable for all diagnostic research in family medicine. It should only be used if (1) the research question can be answered with a small data set per patient, (2) the research question is relatively simple and is considered to be relevant and urgent according to FPs, (3) the patient selection and data collection are sufficiently robust and self-explanatory such that very few instructions are necessary, and (4) the flash-mob study can be performed in a large region to include patients among a large number of FPs.

Acknowledgments

The authors would like to thank all participating FPs, FP out-of-hours services, ambulance services, and specifically all ambassadors of the flash mob study: all 126 specialized FPs (specialized in heart and vascular disease or acute medicine), FPs at all 8 departments of family medicine, and the 10 professional organizations (Dutch Heart Foundation [Hartstichting], employment agency for FP trainees [Stichting Beroepsopleiding Huisartsen (SBOH)], Dutch ambulance organization [Ambulancezorg Nederland], The Netherlands Society of Cardiology [Nederlandse Vereniging voor Cardiologie (NVvC)], the educational programs for specialized FPs in heart and vascular disease or acute medicine [NHG-opleidingen HartVaatHAG en kaderhuisartsen spoedzorg], the national organization of FP out-of-hours services [InEen], national organization of aspirant FPs [LOVAH], the patient association on heart and vascular disease [Harteraad], and the Netherlands Organisation for Health Research and Development [ZonMw]).

Footnotes

  • Conflicts of interest: authors report none.

  • To read or post commentaries in response to this article, see it online at http://www.AnnFamMed.org/content/17/4/296.

  • Funding support: This work was supported by a Veni-grant, assigned to JWLC (91614078), of the Netherlands Organisation for Health Research and Development (ZonMw).

  • Ethical approval: We obtained ethical approval from the Medical Ethics Committee of Zuyderland Medical Centre in The Netherlands (17-N-119).

  • Supplementary materials: Available at http://www.AnnFamMed.org/content/17/4/296/suppl/DC1/.

  • Received for publication June 21, 2018.
  • Revision received January 25, 2019.
  • Accepted for publication February 5, 2019.

References

  1. ↵
    1. Bösner S,
    2. Haasenritter J,
    3. Becker A,
    4. et al
    . Ruling out coronary artery disease in primary care: development and vali dation of a simple prediction rule. CMAJ. 2010; 182(12): 1295–1300.
    OpenUrlAbstract/FREE Full Text
    1. Haasenritter J,
    2. Bösner S,
    3. Vaucher P,
    4. et al
    . Ruling out coronary heart disease in primary care: external validation of a clinical prediction rule. Br J Gen Pract. 2012; 62(599): e415–e421.
    OpenUrlAbstract/FREE Full Text
  3. ↵
    1. Haasenritter J,
    2. Donner-Banzhoff N,
    3. Bösner S
    . Chest pain for coronary heart disease in general practice: clinical judgement and a clinical decision rule. Br J Gen Pract. 2015; 65(640): e748–e753.
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Backus BE,
    2. Six AJ,
    3. Kelder JC,
    4. et al
    . Chest pain in the emergency room: a multicenter validation of the HEART Score. Crit Pathw Cardiol. 2010; 9(3): 164–169.
    OpenUrlCrossRefPubMed
    1. Backus BE,
    2. Six AJ,
    3. Kelder JC,
    4. et al
    . A prospective validation of the HEART score for chest pain patients at the emergency department. Int J Cardiol. 2013; 168(3): 2153–2158.
    OpenUrlCrossRefPubMed
  6. ↵
    1. Six AJ,
    2. Cullen L,
    3. Backus BE,
    4. et al
    . The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study. Crit Pathw Cardiol. 2013; 12(3): 121–126.
    OpenUrlCrossRefPubMed
  7. ↵
    1. Aerts M,
    2. Minalu G,
    3. Bösner S,
    4. et al.,
    5. International Working Group on Chest Pain in Primary Care (INTERCHEST)
    . Pooled individual patient data from five countries were used to derive a clinical prediction rule for coronary artery disease in primary care. J Clin Epidemiol. 2017; 81: 120–128.
    OpenUrlCrossRef
  8. ↵
    1. Bruins Slot MH,
    2. Rutten FH,
    3. van der Heijden GJ,
    4. Geersing GJ,
    5. Glatz JF,
    6. Hoes AW
    . Diagnosing acute coronary syndrome in primary care: comparison of the physicians’ risk estimation and a clinical decision rule. Fam Pract. 2011; 28(3): 323–328.
    OpenUrlCrossRefPubMed
  9. ↵
    1. Schriger DL,
    2. Elder JW,
    3. Cooper RJ
    . Structured clinical decision aids are seldom compared with subjective physician judgment, and are seldom superior. Ann Emerg Med. 2017; 70: 338–344,e3.
    OpenUrl
  10. ↵
    1. Semler MW,
    2. Stover DG,
    3. Copland AP,
    4. et al
    . Flash mob research: a single-day, multicenter, resident-directed study of respiratory rate. Chest. 2013; 143(6): 1740–1744.
    OpenUrl
  11. ↵
    1. Alsma J,
    2. van Saase JLCM,
    3. Nanayakkara PWB,
    4. et al.,
    5. FAMOUS Study Group
    . The power of flash mob research: conducting a nationwide observational clinical study on capillary refill time in a single day. Chest. 2017; 151(5): 1106–1113.
    OpenUrl
  12. ↵
    Flash mob. Oxford Dictionaries website. https://en.oxforddictionaries.com/definition/flash_mob. Accessed May 3, 2018.
  13. ↵
    1. Schols AMR,
    2. Cals JWL
    . [Flash mob research on decision rule for acute coronary syndrome]. Ned Tijdschr Geneeskd. 2017; 161: D2280.
    OpenUrl
  14. ↵
    1. Schols A,
    2. Cals J
    . [New: flash mob research Family Practitioner HEART Week]. Huisarts Wet. 2017: 60(11): 603.
    OpenUrl
  15. ↵
    1. van Weel C,
    2. Schers H,
    3. Timmermans A
    . Health care in the Netherlands. J Am Board Fam Med. 2012; 25(Suppl 1): S12–S17.
    OpenUrlAbstract/FREE Full Text
  16. ↵
    1. Newcombe RG
    . Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 1998; 17(8): 857–872.
    OpenUrlCrossRefPubMed
  17. ↵
    1. Wilson EB
    . Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927; 22(158): 209–212.
    OpenUrlCrossRef
  18. ↵
    1. Buntinx F,
    2. Truyen J,
    3. Embrechts P,
    4. Moreel G,
    5. Peeters R
    . Evaluating patients with chest pain using classification and regression trees. Fam Pract. 1992; 9(2): 149–153.
    OpenUrlCrossRefPubMed
    1. Nilsson S,
    2. Scheike M,
    3. Engblom D,
    4. et al
    . Chest pain and ischaemic heart disease in primary care. Br J Gen Pract. 2003; 53(490): 378–382.
    OpenUrlAbstract/FREE Full Text
  20. ↵
    1. Bösner S,
    2. Becker A,
    3. Abu Hani M,
    4. et al
    . Accuracy of symptoms and signs for coronary heart disease assessed in primary care. Br J Gen Pract. 2010; 60(575): e246–e257.
    OpenUrlAbstract/FREE Full Text
  21. ↵
    1. Braunwald E,
    2. Morrow DA
    . Unstable angina: is it time for a requiem? Circulation. 2013; 127(24): 2452–2457.
    OpenUrlFREE Full Text
  22. ↵
    1. Rutten F,
    2. Bakx J,
    3. Bruins Slot M,
    4. et al
    . [GP guideline Acute coronary syndrome]. Huisarts Wet. 2012; 55: 564–570.
    OpenUrl
  23. ↵
    1. Willis BH
    . Spectrum bias—why clinicians need to be cautious when applying diagnostic test studies. Fam Pract. 2008; 25(5): 390–396.
    OpenUrlCrossRefPubMed
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Get Permissions

Jump to section

Subjects

Keywords