A Core Outcome Set for Multimorbidity Research (COSmm)
======================================================

* Susan M. Smith
* Emma Wallace
* Chris Salisbury
* Maxime Sasseville
* Elizabeth Bayliss
* Martin Fortin

## Abstract

**PURPOSE** We aimed to develop a consensus-based set of core outcomes specifically for studies in multimorbidity.

**METHODS** We undertook a consensus study following the COS-STAR (Core Outcome Set-STAndards for Reporting) guidelines for the design and reporting of core outcome sets. A Delphi panel of experts completed a web-based survey with 2 rounds. Panelists were presented with a range of outcomes that had been identified in previous workshops and a related systematic review. They indicated their level of agreement on whether each outcome should be included in the core set using a 5-point Likert scale, and outcomes reaching a prespecified consensus level were included.

**RESULTS** Of 30 individuals invited to be panelists, 26 from 13 countries agreed. All 26 completed both rounds of the survey. The Delphi panel reached consensus on 17 outcomes for inclusion in a core outcome set for multimorbidity (COSmm). The highest-ranked outcomes were health-related quality of life, mental health outcomes, and mortality. Other outcomes were grouped into overarching themes of patient-reported impacts and behaviors (treatment burden, self-rated health, self-management behavior, self-efficacy, adherence); physical activity and function (activities of daily living, physical function, physical activity); consultation related (communication, shared decision making, prioritization); and health systems (health care use, costs, quality of health care).

**CONCLUSIONS** This consensus study involved a wide range of international experts who identified a large number of outcomes for multimorbidity intervention studies. Our results suggest that quality of life, mental health outcomes, and mortality should be regarded as essential core outcomes. Researchers should, however, also consider the full range of outcomes when designing studies to capture important domains in multimorbidity depending on individual study aims and interventions.

*   multimorbidity
*   chronic disease
*   complexity
*   methodology
*   primary care
*   research
*   core outcome set

## INTRODUCTION

The Core Outcome Measures in Effectiveness Trials (COMET) initiative aims to develop consensus-based standardized sets of outcomes, known as core outcome sets ([http://www.comet-initiative.org/](http://www.comet-initiative.org/)).1 Core outcome sets represent the minimum that should be measured and reported in all clinical trials of a specific condition or conditions.1

Multimorbidity is commonly defined as the coexistence of 2 or more chronic conditions in an individual.2 Its effects include reduced health-related quality of life, increased psychological distress, functional difficulties, increased health care use, and heightened mortality risk.3–8 Current randomized controlled trials tend to adopt a single-disease focus, resulting in a paucity of relevant evidence for the management of patients with multimorbidity.9 A growing number of trials are examining the effectiveness of interventions to address the specific experiences of patients with multimorbidity.10 Systematic review of these studies highlights challenges of evidence synthesis due to differences between studies including methodologic choice of outcomes. The 2016 UK National Institute of Health and Care Excellence (NICE) Guidance on Multimorbidity stresses the need to design interventions and transform health services by addressing multimorbidity in both clinical guidelines and clinical practice.11 This NICE Guidance and related systematic and clinical reviews9–11 highlight the need for consensus regarding outcomes relevant for multimorbidity, so that evidence can be synthesized and is based on outcomes reflecting the priorities of all stakeholders, particularly patients.

The aim of this study was to identify a core outcome set for multimorbidity research studies using a Delphi consensus process with an international panel of experts. The scope of this core outcome set included studies of all intervention types targeting adults with multimorbidity. Studies of heterogeneous populations defined by an index condition plus comorbid conditions are conceptually different from studies of multimorbidity. In the former, outcomes are evaluated in the context of one specific condition, whereas in the latter, outcomes are interpreted in the context of the burden of multimorbidity. We therefore excluded research studies including a named index condition (eg, hypertension, diabetes) plus another condition, referred to as comorbidity studies, as the aim was to develop a core outcome set reflecting the heterogeneous nature of multimorbidity.

We had 3 specific objectives: (1) to identify outcomes and outcome metrics that had previously been used in intervention studies for multimorbidity; (2) to develop a consensus-based set of core outcomes for intervention studies in multimorbidity; and (3) to describe related outcome metrics used for outcomes for multimorbidity studies.

## METHODS

A protocol of this project was registered with the COMET initiative and is available online ([http://www.comet-initiative.org/studies/details/822](http://www.comet-initiative.org/studies/details/822)).12 We used the Core Outcome Set-STAndards for Reporting (COS-STAR) Equator Network guidelines for the reporting of this core outcome set.1 Core outcome sets can be considered in terms of what to measure, which we refer to as outcomes, and how to measure it, which we refer to as metrics.13

### Steering Group

A steering group oversaw the development of the core outcome set and consisted of academic family practitioners and primary care researchers with a specific interest in multimorbidity. All members are authors on this article.

### Development of a Preliminary Set of Core Outcomes for Multimorbidity: Information Sources

The steering group developed a list of relevant outcomes for multimorbidity intervention studies using a comprehensive strategy of workshop discussions and review of outcomes in existing peer-reviewed publications. This list of potential outcomes was developed over several years at academic primary care meetings and workshops attended by the author group and other experts in multimorbidity research.14,15 (See Supplemental Appendix 1, available at [http://www.annfammed.org/content/16/2/132/suppl/DC1](http://www.annfammed.org/content/16/2/132/suppl/DC1).) Members of the steering group also reviewed studies in the recently updated Cochrane review of the effectiveness of interventions for patients with multimorbidity and identified outcomes and related metrics reported in this review.10 We did so because the Cochrane systematic review involved a comprehensive systematic literature search across multiple databases and had been conducted relatively recently. The preliminary lists of outcome and related metrics were compiled into an online questionnaire using Survey Monkey (Supplemental Appendix 2, [http://www.annfammed.org/content/16/2/132/suppl/DC1](http://www.annfammed.org/content/16/2/132/suppl/DC1)).16

### Consensus Process

The Delphi process is a commonly used consensus technique.17 The steering group identified a panel of international experts with broad stakeholder representation. Panelists were selected based on their interest in multimorbidity research or through their existing membership on patient panels supporting multimorbidity research in Ireland, the United Kingdom, and Canada. This approach has been used several times previously in consensus studies examining prescribing indicators.18 The final panel included a multidisciplinary range of experts and stakeholders (Supplemental Appendix 3, [http://www.annfammed.org/content/16/2/132/suppl/DC1](http://www.annfammed.org/content/16/2/132/suppl/DC1)).

The questionnaire containing the preliminary core outcome set was first pilot tested (to assess usability) by members of academic staff in the departments of the steering group and modified accordingly. After formation of the Delphi panel, all potential members received the first-round questionnaire via SurveyMonkey.16 A full copy of this questionnaire, including instructions given to participants, is provided in Supplemental Appendix 2.

Panelists were asked to rate the importance of each potential outcome for inclusion in the final set and were presented with a range of potential metrics. They were also given an opportunity to suggest additional outcomes or metrics if desired. A web link to the outcome metrics presented was also included to support the panel’s decision making (Supplemental Appendix 2).

### Panel Size

On the basis of previous studies showing the range in sizes of Delphi panels, we anticipated that a purposively selected panel of at least 15 experts would adequately cover issues in the proposed outcome set.19 Only panelists responding to the first round were included in the second round. On the basis of previous Delphi studies, we invited a total of 30 panelists, assuming an approximate response rate of 50%.

### Round 1 of Delphi Process

Our Delphi process had 2 rounds. In round 1, panelists were asked to indicate their level of agreement for the inclusion of each outcome in the final core set using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = ambivalent, 4 = agree, 5 = strongly agree). For each outcome, we calculated the median score and interquartile range (IQR). When the lower limit of the IQR was greater than 3, the outcome was included in the core set; when the upper limit of the IQR was less than 3, the outcome was excluded. If the IQR of an outcome included 3, the language used in the outcome was revised following recommendations from the panel and included in the second round. All members of the steering group were involved in reviewing outcomes and in refining the core outcomes included.

### Round 2 of Delphi Process

In round 2 of the Delphi process, the outcomes not meeting the threshold for inclusion during round 1 were again presented with the same definition of consensus as used for that round. Participants were not informed of the previous round scores but would have been aware that the remaining outcomes had not achieved consensus in round 1. The wide range of potential outcome metrics included in round 1 indicated that it was beyond the scope of the current study to reach agreement on metrics, so metrics were not presented to the panelists for consideration in round 2. All statistical analyses were performed using Stata version 13.0 (StataCorp LLC).

### Ethical Considerations

Ethical approval for this study was granted by the Human Research Ethics Committee at the Royal College of Surgeons in Ireland (RCSI) medical school. All participants were contacted by e-mail with detailed information regarding the study. Patient and public representatives were contacted initially with a separate e-mail invitation that included a leaflet detailing specific information for this group. All panelists signing a consent form were sent another e-mail with a link to the SurveyMonkey questionnaire.

## RESULTS

An overview of the process used to develop the core outcome set for multimorbidity research is provided in Figure 1. Of the 30 individuals invited to the Delphi panel, 26 responded (87% response rate), including 5 patient or public representatives, with representation of 13 countries (Supplemental Appendix 3). Two individuals formally declined and 2 did not respond to e-mails.

![Figure 1](http://www.annfammed.org/https://www.annfammed.org/content/annalsfm/16/2/132/F1.medium.gif)

[Figure 1](http://www.annfammed.org/content/16/2/132/F1)

Figure 1 
Overview of the Delphi process used to develop the core outcome set for multimorbidity research (COSmm).

IQR = interquartile range.

### Round 1

Of the 30 preliminary outcomes presented to the Delphi panel in round 1, we found that 11 had a median and IQR score of greater than 3, and they were therefore included them in the core outcome set for multimorbidity research (COSmm) after this round (Table 1). Scores were highest for health-related quality of life, mental health outcomes, and mortality. No outcomes were excluded after round 1, and no additional outcomes were identified. A descriptive list of the potential metrics for measuring outcomes identified during round 1 is presented in Supplemental Appendix 4 (available at [http://www.annfammed.org/content/16/2/132/suppl/DC1](http://www.annfammed.org/content/16/2/132/suppl/DC1)).

View this table:
[Table 1](http://www.annfammed.org/content/16/2/132/T1)

Table 1 
Results of Round 1 of the Delphi Process

### Round 2

In round 2 of the Delphi process, the remaining 19 outcomes (for which the IQR included 3) were again presented to the 26 panelists, all of whom completed this round. At the end of this round, an additional 6 outcomes had median and IQR scores of greater than 3 (Table 2); these outcomes were added to the preceding 11 outcomes for inclusion, giving a total of 17 final outcomes in the COSmm. The steering group divided these 17 core outcomes into groups by theme, in an effort to clarify the key areas for consideration when choosing outcomes for individual studies from the COSmm (Table 3).

View this table:
[Table 2](http://www.annfammed.org/content/16/2/132/T2)

Table 2 
Results of Round 2 of the Delphi Process

View this table:
[Table 3](http://www.annfammed.org/content/16/2/132/T3)

Table 3 
Core Outcome Set for Multimorbidity (COSmm): 17 Outcomes by Group

## DISCUSSION

A Delphi consensus panel with 26 experts from 13 countries agreed on a set of 17 core outcomes for multimorbidity research, the COSmm. The highest-ranked outcomes were health-related quality of life, mental health outcomes, and mortality. Given the number of outcomes, we grouped the remaining outcomes according to overarching themes of patient-reported activities and behaviors, physical activity and function, consultation-related outcomes, and outcomes of importance to health systems, including costs.

Although the COSmm contains a large number of outcomes for consideration, it reflects the broad nature of multimorbidity and is consistent with the range of outcomes in studies included in the Cochrane systematic review on multimorbidity interventions.10 Given the likely variation in intervention types for people with multimorbidity, individual study investigators will use outcomes that reflect their aims and underlying mechanisms.14 Our results suggest, however, that all studies should consider health-related quality of life, mental health outcomes, and mortality as key outcomes, and then consider others within the COSmm based on their own individual study. Although few individual studies will be designed or powered to detect changes in mortality, our panelists regarded it as an important outcome, and inclusion of mortality in studies would facilitate its inclusion in meta-analyses in future systematic reviews. We would also caution that inclusion of all COSmm outcomes in an individual study would likely lead to excessive burden on study participants and could increase risk of type 1 errors in interpreting results. We also acknowledge that many of the outcomes that were excluded could also be considered important, for example, poly-pharmacy in a study examining a medication management intervention. Although the level of agreement among panelists was not sufficiently strong to include these outcomes, some may still be relevant in multimorbidity trials depending on the interventions involved.

One of the objectives of this study was to consider metrics that have been used previously for each of the core outcomes. The responses in round 1 suggested that a consensus process for the use of these metrics would be very complex and challenging; however, the metrics identified during the process will likely be of interest to researchers and other stakeholders, and have been included (Supplemental Appendix 4). Methodology on the development of core outcome sets is evolving,20,21 and we acknowledge that our COSmm will need to be updated regularly as new outcomes and metrics are developed over time.

### Comparison With Other Core Outcome Sets

Our core outcome set was developed specifically for studies of interventions for patients with multimorbidity, but other studies have used consensus approaches to develop outcomes to measure the quality of care for people with multimorbidity using the electronic health record.22 A range of consensus process methods have been used to develop various core outcome sets.1 We chose to use a Delphi panel approach on the basis of our previous experience and a review of core outcome set literature.19,21,23 Other potential approaches include expert panel groups, nominal group techniques, semi-structured group discussions, and questionnaires.21 Face-to-face approaches allow greater discussion but are restricted by limited accessibility for international stakeholders. Some elements of other approaches were used by conducting workshops to identify the draft core outcome set before the consensus process. This is an evolving area of research, and there is no clear evidence suggesting benefit of one consensus approach over another.19 We chose an online Delphi method as it allowed for involvement of clinicians, researchers, and patients in multiple sites across continents. This strategy also allowed for purposive selection of a panel to ensure representation of a range of stakeholders from different countries and health care systems with both professional and patient representation.24

Panelists achieved a high level of consensus after the first round for 11 outcomes. The level seen is higher than might have been expected. There are clearly many important outcomes for evaluating care delivered to complex populations. After the first round, we consulted the COMET group regarding the number of outcomes, and they provided helpful feedback, which suggested that our core outcome set is not unusual given the broad scope of multimorbidity.

We used a 5-point Likert scale for outcome scoring. Other core outcome sets have been developed using alternative outcome-scoring methods, which may have been more appropriate.1 These alternatives include use of a higher number of Likert scale options, scoring based on average agreement scores, and a ranking system. The methodology for developing core outcome sets has been evolving rapidly over the last few years, and there is no clear evidence as yet on which is the best scoring method. The COS Star Guidance1 was produced after we had published our protocol and secured ethical approval. We considered changing our approach to outcome scoring after round 1, but decided not to deviate from our protocol at that point.

### Strengths and Weaknesses

Our new core outcome set for multimorbidity studies was developed over a period of several years by an international group of researchers working in this area. It built on a number of workshops held with a wide range of participants at different international primary care meetings. We formed a Delphi panel consisting of a broad range of experts with experience in multimorbidity research from 13 countries and with patient and public representation. Identifying appropriate patient and public representatives was a challenge given the broad nature of multimorbidity, but we worked with experienced research groups to identify appropriate panelists from 3 countries. A further strength was the very high response rate for the first round of the consensus process and 100% participation for the second round.

One potential limitation of our COSmm is the high number of outcomes identified by the panel. We already had a high number of included outcomes after round 1, and we provided a cover letter for the round 2 survey stressing to panelists that there was not necessarily a need to identify additional outcomes. Despite this note, a further 6 outcomes were added. This result may relate to our consensus methods, but a range of approaches have been described in the literature with no clear agreement on the best approach to date. Another limitation may relate to our decision not to provide panelists with feedback on scores from other panelists for round 2. We used this approach to ensure that panelists could make independent judgments and avoid moving toward a group average score.24 Guidance on the use of Delphi techniques recommends that a measure of distribution around final scores should be reported, which we have done. Reporting of the final scores alone can mask major disagreement within the group, which might be the case if there was a wide distribution around scores.23 Our results do not suggest that this was the case.

Panel size and selection are always key considerations and potential limitations for any Delphi process. We chose to identify a purposive sample of panelists with prior exposure to the concept of multimorbidity given the complexity of this topic. This strategy could be seen as a limitation in that it did not open up the COSmm to a wider group. The broad nature of multimorbidity and the frequent debate about its definition and construct,11 however, were repeated themes at all the workshops that led to the development of the preliminary COSmm, and these workshops were open to all attendees at the 2 largest primary care research meetings in North America and Europe. We therefore felt it was appropriate to limit the Delphi panelists to those with prior exposure to multimorbidity as they had already considered issues pertaining to whether multimorbidity itself represents a construct.

Other core outcome set studies have used larger panel sizes to increase representation; however, there tends to be a lower response rate for larger panels.25 Some studies use snowballing techniques allowing open access to an online survey, which will lead to larger numbers of participants. This approach does not necessarily ensure broad and balanced representation of key stakeholders, however. Another potential limitation of our study is that although we included panelists from 13 countries, all were based in high-income countries. In addition, although we included patient representatives as Delphi panel members, as for many previous core outcome set studies,24 it is a limitation that they were not directly involved in the development of the initial COSmm.

### Conclusions

In conclusion, this consensus study resulting in development of a core set of outcomes specifically for multimorbidity research, the 17-outcome COSmm, involved a wide range of international experts who identified a large number of core outcomes for intervention studies among populations having multiple medical conditions. Such studies are likely to vary depending on target populations and intervention types, so given the scope of multimorbidity, the large number of outcomes identified is not surprising. Our results suggest that quality of life, mental health outcomes, and mortality should be regarded as essential core outcomes. Investigators should, however, also consider the full range of outcomes in the COSmm depending on their individual study aims and interventions.

## Acknowledgments

The authors would like to acknowledge the input of the Delphi panel members. See Supplemental Appendix 3, [http://www.annfammed.org/content/16/2/132/suppl/DC1](http://www.annfammed.org/content/16/2/132/suppl/DC1), for their positions, disciplines, and affiliations.

## Footnotes

*   Conflicts of interest: E.W. is Senior Research Fellow in the HRB Centre for Primary Care Research in Ireland (HRB grant HRC-2014-1); M.F. holds the Research Chair on Chronic Diseases in Primary Care, Université de Sherbrooke, Canada.

*   **Author contributions:** S.M.S., C.S., L.B., and M.F. conceived the idea for this article. All authors were involved in writing the protocol and the main article, including analysis of the Delphi panel results and commenting on development of the core outcome set at all stages as steering group members.

*   **Supplementary materials:** Available at [http://www.AnnFamMed.org/content/16/2/132/suppl/DC1/](http://www.AnnFamMed.org/content/16/2/132/suppl/DC1/).

*   Received for publication May 20, 2017.
*   Revision received September 12, 2017.
*   Accepted for publication October 15, 2017.


*   © 2018 Annals of Family Medicine, Inc.

## References

1.  Kirkham JJ, Gorst S, Altman DG, et al. Core Outcome Set-STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016; 13(10): e1002148.
    
    

2.  Fortin M, Bravo G, Hudon C, Vanasse A, Lapointe L. Prevalence of multimorbidity among adults seen in family practice. Ann Fam Med. 2005;3(3):223–228.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiMy8zLzIyMyI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTYvMi8xMzIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Fortin M, Bravo G, Hudon C, et al. Relationship between multimorbidity and health-related quality of life of patients in primary care. Qual Life Res. 2006; 15(1): 83–91.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1007/s11136-005-8661-z&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=16411033&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000234601400008&link_type=ISI) 

4.  Fortin M, Bravo G, Hudon C, Lapointe L, Dubois MF, Almirall J. Psychological distress and multimorbidity in primary care. Ann Fam Med. 2006;4(5):417–422.
    
    [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiNC81LzQxNyI7czo0OiJhdG9tIjtzOjIzOiIvYW5uYWxzZm0vMTYvMi8xMzIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

5.  Marengoni A, Angleman S, Melis R, et al. Aging with multimorbidity: a systematic review of the literature. Ageing Res Rev. 2011; 10(4): 430–439.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.arr.2011.03.003&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21402176&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000295446300006&link_type=ISI) 

6.  Barnett K, Mercer SW, Norbury M, Watt G, Wyke S, Guthrie B. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet. 2012; 380(9836): 37–43.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/S0140-6736(12)60240-2&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22579043&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000306115900031&link_type=ISI) 

7.  Violan C, Foguet-Boreu Q, Flores-Mateo G, et al. Prevalence, determinants and patterns of multimorbidity in primary care: a systematic review of observational studies. PLoS One. 2014; 9(7): e102149.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1371/journal.pone.0102149&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=25048354&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 

8.  Salisbury C, Johnson L, Purdy S, Valderas JM, Montgomery AA. Epidemiology and impact of multimorbidity in primary care: a retrospective cohort study. Brit J Gen Pract. 2011; 61(582): e12–e21.
    
    

9.  Wallace E, Salisbury C, Guthrie B, Lewis C, Fahey T, Smith SM. Managing patients with multimorbidity in primary care. BMJ. 2015; 350: h176.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzNTAvamFuMjBfMi9oMTc2IjtzOjQ6ImF0b20iO3M6MjM6Ii9hbm5hbHNmbS8xNi8yLzEzMi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. Smith SM, Wallace E, O’Dowd T, Fortin M. Interventions for improving outcomes in patients with multimorbidity in primary care and community settings. Cochrane Database Syst Rev. 2016; 3: CD006560.
    
    

11. National Institute for Health and Care Excellence (NICE). Multimorbidity: clinical assessment and management. Published Sep 21, 2016. [http://www.nice.org.uk/guidance/ng56](http://www.nice.org.uk/guidance/ng56). Accessed Oct 3, 2017.
    
    

12. Smith SM, Wallace E, Bayliss EA, Salibury C, Sasseville M, Fortin M. Establishing a Core Outcome set for studies examining multimorbidity: protocol. COMET Initiative. [http://www.comet-initiative.org/studies/details/822](http://www.comet-initiative.org/studies/details/822). Published 2016. Accessed Oct 3, 2017.
    
    

13. Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version 1.0. Trials. 2017; 18(Suppl 3): 280.
    
    

14. Smith SM, Bayliss EA, Mercer SW, et al. How to design and evaluate interventions to improve outcomes for patients with multimorbidity. J Comorb. 2013; 3: 10–17.
    
    

15. Sasseville M, Stewart M, Bouhali T, Fortin M. Relevant outcomes for patient-centered interventions for persons with multimorbidity: experts’ discussion. Ann Fam Med. 2017;15(4):388–389.
    
    [FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTUvNC8zODgiO3M6NDoiYXRvbSI7czoyMzoiL2FubmFsc2ZtLzE2LzIvMTMyLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. SurveyMonkey Inc. San Mateo, California, USA. [http://www.surveymonkey.com](http://www.surveymonkey.com). Accessed Oct 3, 2017.
    
    

17. Brown BB. Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts. Santa Monica, CA: RAND Corporation, 1968. [https://www.rand.org/pubs/papers/P3925.html](https://www.rand.org/pubs/papers/P3925.html). Accessed Oct 3, 2017.
    
    

18. Cooper JA, Ryan C, Smith SM, et al.The PROMPT Steering Group. The development of the PROMPT (PRescribing Optimally in Middle-aged People’s Treatments) criteria. BMC Health Serv Res. 201430; 14: 484.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/s12913-014-0484-6&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=25410615&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 

19. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011; 8(1): e1000393.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1371/journal.pmed.1000393&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=21283604&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 

20. Boers M, Kirwan JR, Tugwell P, et al. The OMERACT Handbook. [https://omeract.org/resources](https://omeract.org/resources). Updated Jan 19, 2018. Accessed Feb 12, 2018.
    
    

21. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012; 13: 132.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/1745-6215-13-132&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22867278&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 

22. Bayliss EA, McQuillan DB, Ellis JL, et al. Using electronic health record data to measure care quality for individuals with multiple chronic medical conditions. J Am Geriatr Soc. 2016; 64(9): 1839–1844.
    
    

23. Sinha IP, Gallagher R, Williamson PR, Smyth RL. Development of a core outcome set for clinical trials in childhood asthma: a survey of clinicians, parents, and young people. Trials. 2012; 13: 103.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1186/1745-6215-13-103&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=22747787&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000301894100005&link_type=ISI) 

24. Jones JE, Jones LL, Keeley TJ, Calvert MJ, Mathers J. A review of patient and carer participation and the use of qualitative research in the development of core outcome sets. PLoS One. 2017; 12(3): e0172937.
    
    

25. Devane D, Begley CM, Clarke M, Horey D, OBoyle C. Evaluating maternity care: a core set of outcome measures. Birth. 2007; 34(2): 164–172.
    
    [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1111/j.1523-536X.2006.00145.x&link_type=DOI) 
    
    [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=17542821&link_type=MED&atom=%2Fannalsfm%2F16%2F2%2F132.atom) 
    
    [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000246801400008&link_type=ISI)