Research ArticleMethodology

Concept Mapping as a Method to Engage Patients in Clinical Quality Improvement

Marianna LaNoue, Geoffrey Mills, Amy Cunningham and Adam Sharbaugh
The Annals of Family Medicine July 2016, 14 (4) 370-376; DOI: https://doi.org/10.1370/afm.1929

Abstract

Patient engagement has become a primary care research and practice priority. Little guidance exists, however, on how best to engage patients in primary care practice improvement, or how to measure the impact of their engagement. We present an overview of group concept mapping as a method for engaging patients in primary care practice improvement. We detail the group concept mapping process as a tool for use in primary care practice improvement, research, and evaluation, and we present resources to enable researchers and practice leaders to use this tool in practice improvement. To illustrate the method, we present a practice-based quality improvement project conducted with patients and staff at a large urban academic primary care practice.

INTRODUCTION

Patient engagement has been described as a “coming revolution”1 and as a “blockbuster prescription”2 for improving health care. It is prioritized in research funding at the Patient-Centered Outcomes Research Institute (PCORI) and the Agency for Healthcare Research and Quality (AHRQ), and it is mandated by the National Committee for Quality Assurance for patient-centered medical home (PCMH) accreditation. Patient engagement may occur across the dimensions of direct care, organizational design and governance, and policy making according to a framework proposed by Carman et al.3 This framework posits that a continuum of engagement exists within each of these dimensions: at the highest level of the organizational design dimension, patients co-lead initiatives, including quality improvement initiatives.3 Research supports positive outcomes associated with patient engagement in quality improvement: a systematic review by Crawford et al4 showed that patient involvement in quality improvement efforts led to improved patient information resources and access to care. Furthermore, a recent AHRQ guide for patient and family advisory programs noted that patient involvement in improvement efforts can lead to better quality, safety, and patient and staff satisfaction.5

Increasingly, primary care practices, including PCMHs, attempt to engage patients across a continuum that includes research and practice improvement. Early evidence suggests that practices that have involved patients in medical home transformation efforts view the engagement positively.6,7 Attempts to engage patients in quality improvement efforts remain uncommon, however. A recent survey of 123 PCMHs found that fewer than one-third engaged patients in these efforts, although most of them reported soliciting patient input in other areas.6 Another survey of 249 small PCMHs found that only 15% included patients on quality improvement teams.8 One barrier to directly engaging patients may be the lack of guidance on a clear method of doing so.

Medical practices typically solicit patient input in practice improvement through suggestion boxes and surveys, and increasingly, with patient advisory boards.6,5 Interviews and focus groups are more common in research settings, but they are also used to solicit and organize patient involvement in practice settings.9,10 These approaches are time and labor intensive, however, and qualitative data collection and analysis require training. Furthermore, the products of these qualitative techniques are often text-heavy documents that do not translate easily to visual displays of complex ideas and patterns that may be more readily understood.11 Because of the clear importance of patient engagement, methods are needed to formalize engagement and measure its outcomes. In this article, we present a method of “structured conceptualization” called group concept mapping, a promising method to engage patients in primary care practice improvement. We illustrate its application through an example from our work involving patients in quality improvement in a large family medicine practice.

CONTEXT AND RESOURCES

The term concept mapping is not new. It describes an entire set of knowledge representation theories and applications in educational and cognitive psychology,12 where the concepts in a particular subject area and their interrelationships are visually represented in a concept or knowledge map. In these maps, the proximity of the concepts to each other serves as an index of how related they are, and the lines or arrows between the concepts describe or imply the nature of their relationships.

The group concept mapping method that we describe here was developed in the 1980s by William Trochim. Trochim extended the ideas and theories of knowledge representation and concept maps by fully conceiving of them as a group process: participants define the conceptual space through their responses to a question prompt, and define the relatedness of the ideas by sorting the responses into cluster groupings based on perceived similarity. This process yields a visual depiction, in the form of a concept map, based on the ideas’ relatedness. All the participants brainstorm and organize knowledge in a domain, and then interpret and use the resultant maps. An early article detailing the development of the method and its statistical techniques is available online.13 Many (if not most) group concept mapping research and evaluation projects use the software designed by Trochim (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

As a patient/consumer engagement method, Trochim’s concept map has been used primarily in public health community-based participatory research.14–16 A small number of medical studies have used this method to elicit patient and caregiver experiences with chronic illness management.17–20 In a review, we found only 3 articles using concept mapping to engage patients in practice improvement,21–23 only 1 of which was conducted in primary care.23 None of these works provides a methodologic treatment of concept mapping for use in primary care practice improvement.

METHODOLOGIC BASIS

Group concept mapping brings together qualitative techniques (brainstorming and idea or theme sorting) and quantitative techniques (multidimensional scaling and cluster analysis). In short, multidimensional scaling is a statistical method that uses aggregated similarity ratings derived in the sorting task to generate a visual representation of the data—called a point map—on which each point is an idea from brainstorming and the distances between points serves as an index of their relatedness. Cluster analysis is applied to the scaling results to produce clusters of these points, which minimize the distances of the points in each cluster from one another and to produce a visual cluster of related ideas. Statistical details to perform these analyses, including programming language for SAS (SAS Institute, Inc), are given by Trochim and Linton.24 Supplemental Figures 1 and 2, available at http://www.annfammed.org/content/4/4/370/suppl/DC1, show the point and cluster maps, respectively, that resulted from our quality improvement study. Although all of these techniques are analytically robust in themselves, direct research on the group concept mapping method has only recently been published. Rosas and Kane25 reviewed 69 group concept mapping studies and concluded that the reproducibility reliability of sorting and rating is very high.

MAPPING PROCESS

The concept mapping process we present here for practice improvement is a hybrid process mixing features of both a research application and a program planning one. In this section, we present the details of our process and the lessons that we learned from our quality improvement project (Table 1). A total of 41 individuals—16 patients and 25 clinic participants (physicians, a nurse practitioner, and several clinic and quality staff)—participated; patients were recruited from the clinic waiting area by asking all patients who entered the clinic during 6 half-days of data collection between May and July 2014 if they would participate in the study. Patients who agreed to participate were compensated for their time with a $10 gift card to a local retailer. Clinic participants were recruited from the practice at a monthly clinical research meeting and from the clinical quality improvement committee. Both patient and clinic participants gave verbal consent to participate in the study. No demographic or identifying information was collected. The university’s institutional review board approved the study.

View this table:
Table 1

Concept Mapping Steps and Lessons Learned

Our group concept mapping process consisted of 6 steps: preparation, generation of statements, structuring of statements, representation of statements, interpretation, and use of concept maps. We describe each below.

Preparation

The preparation step consists first of identifying stakeholders/participants. This group should consist of members who are related through their involvement in a specific organization or process, or around a specific problem. The number of participants can vary but should include a “wide variety of relevant stakeholders.”26 Although it is possible to engage a large number of participants, typically 10 to 40 are needed to achieve saturation in the brainstorming phase.26 For a project that will compare groups with each other in terms of differences in their statement ratings, 10 is likely to be the smallest appropriate size per group.

Next, the participant group and leaders create a focus prompt. This statement or question will serve to encourage (or constrain) the breadth of ideas generated by brainstorming. Careful attention should therefore be paid to this step, and should include a logical analysis of possible responses. The focus prompt should be pilot-tested before it is used, and principles of question design typical to survey design are appropriate. Such principles include avoiding double-barreled questions (those questions phrased in such a way that 2 questions are being asked simultaneously) and ambiguous questions. We used the following focus prompt: “What are some ways that patients can be involved at our practice in order to improve the care we provide?” We later realized that our focus prompt was double-barreled and therefore tended to generate responses in domains of general practice improvement (eg, increasing access) and of involving patients in practice improvement (eg, patients could develop educational materials). Use of a third-person stem (such as, “What are some reasons people might not want to be involved…”) can be especially helpful in allowing participants to offer ideas and responses that are generalizable to “most people” or “other people,” thereby easing any concerns that may arise for participants about sharing private opinions in a group setting.

Generation of Statements

At this stage, participants/stakeholders brainstorm responses to the focus prompt. They can participate in person individually or in groups, or via the Internet individually. Although there appears to be some benefit to group-based brainstorming in terms of idea generation, there are also potential benefits to allowing members to brainstorm individually, such as minimizing the possibility that strong voices will overrun the group process. The literature on brainstorming suggests that placing emphasis on the volume of ideas while deferring judgment on their quality maximizes creative idea generation.27

When we conduct brainstorming with staff at our practice, we ask first for written (private) responses to allow for the expression of unpopular ideas, and then proceed to group brainstorming. A skilled facilitator should conduct the brainstorming and should attend to the group dynamic (if in the group setting) to maximize responses and to determine that brainstorming is complete. In our experience, the facilitator should look for something similar to “saturation” in qualitative research—the point at which no new themes are emerging.

In our project, we invited patients to come to a conference room in the practice after their appointments, which resulted in a mix of individuals or small groups. Participants were asked to brainstorm in response to the focus prompt in a session that was facilitated by a member of the study team. We wrote all responses on a whiteboard and left them on the board for the day. This process produced a type of hybrid group brainstorming as everyone could see the responses others had made.

Two separate brainstorming sessions occurred with primary care clinicians and clinical staff in 2 separate group meeting settings.

Table 2 shows examples of the ideas that were brainstormed by both patients and staff, organized by cluster. We collected data from 16 patients and 25 clinic participants, with an approximately equal number of responses generated in each group, despite the different methods of elicitation used in brainstorming. We diverged from the fully participatory process of engaging the exact same participants for all phases; instead, subsets of clinic participants, and different groups of patients completed the tasks sequentially. This flexibility increased our ability to complete all the tasks and, in our experience, did not detract from the engagement of the participants at each step. General consensus suggests that it is acceptable to brainstorm with a larger group first, and then perform the sorting and rating tasks with increasingly smaller groups or subsets.28

View this table:
Table 2

Results of Statement Cluster Analysis With Examples of Idea Statements

Structuring of Statements

The next step consists of sorting and rating, tasks that can also be conducted either in person or on the Internet. If done in person, these tasks can also take place individually or in groups, although the tasks are individual ones, so there is no benefit to a group other than economy of time. For sorting, participants read each statement from the combined brainstorming session(s) and create groupings for the statements. The groupings should adhere to the following guidelines: (1) at least 2 groupings of statements are required, (2) “miscellaneous” or “other” groupings are not allowed, (3) each grouping must contain at least 2 idea statements, and (4) each idea statement can be placed into only 1 grouping. Depending on the volume of ideas generated, this task can be time-consuming for participants; sorting the approximately 100 ideas in our study took our stakeholders 20 to 50 minutes. In our project, all patients completed the sorting task in person, whereas clinic participants performed the task using an online interface for the concept mapping software. If performed by hand, sorting information can be input directly into the proprietary software or into a spreadsheet program for analysis (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

Because patients in our practice waiting room were consenting into the study and performing the task at the same time, we had a high completion rate. An analysis of group concept mapping in research studies found a completion rate for sorting and rating tasks of slightly more than 50%.27

At this step, there is an opportunity for participants to rate each statement. For our study, we chose 2 rating dimensions: “How much of an impact would (this idea) have in the practice if it were implemented?” and “How feasible would (this idea) be to implement?” Each had a scale ranging from 1 (least impact or lowest feasibility) to 5 (most impact or highest feasibility). Using the concept mapping software, the rating data can be visualized as an average across a cluster, an average within each statement, or an average within clusters or within items, across a group. Collecting demographic data on participants allows for a comparison of ratings between groups, such as patients vs clinic participants. In our project, patients and clinic participants prioritized some clusters of ideas differently: for example, clinic participants believed that “Technology” and “Empowering Patient Proactivity and Self-Care” were most feasible as practice improvements, whereas patients rated “Better Patient-Provider Communication” and “Access and Navigation” as more feasible. The rating data can be displayed in a “go-zone” diagram showing the rating dimensions along the x and y axis and producing a quadrant map of statements; 1 quadrant contains all statements rated as having both high feasibility and high impact. Finally, the degree of agreement between groups can be quantified and visualized in a pattern match diagram (Supplemental Figures 2 and 3).

Representation of Statements

Once participants have individually sorted the statements, the results are aggregated across all individuals. First, a point map is created by aggregating all participants’ sorting data and analyzing it using multidimensional scaling. Next, a suggested set of boundaries is drawn on the point map to form the cluster map. Statements in the point map and cluster analysis are numbered so that participants can easily identify each point as a statement.

Many different clustering strategies are possible within a single project; user judgment is required to decide on meaningful clusters. Near outliers to clusters can be moved to adjacent clusters; this type of visual adjustment can partition the points into more interpretable clusters and is an important opportunity to engage participants in the group concept mapping process. A numerical diagnostic can also be used to guide the choice of final cluster map solution—the stress index—which indicates the degree to which the map accurately represents the grouping data. The program uses repetition of words in each cluster grouping to suggest names for the clusters, and then users or participants can modify the names. The cluster maps can be decided within the group in a fully participatory manner, or by a smaller group, or by an individual doing the analyses. In our study, we decided on a 6-cluster solution (Supplemental Figure 1).

We used the concept mapping software to create rating maps within each cluster by overlaying participants’ impact and feasibility ratings of ideas onto the cluster maps. In addition, we created go-zone diagrams to show the ideas that participants perceived to have both high feasibility and high impact (Supplemental Figure 2). Finally, the software can generate pattern matches, which allow comparisons of the patients’ and clinic participants’ feasibility and impact ratings (Supplemental Figure 3).

For a practice improvement project, the maps may be secondary to the visualization of the rating data. For example, in our case, finding that a cluster of ideas corresponded to “increasing technologic capacity in the practice” was of less interest than the feasibility and impact rating of the practice change ideas in this cluster, including “Increase opportunities to communicate with health professionals using portal or website.”

Interpretation and Use of Maps

In a fully participatory process, interpreting the maps takes place within the group. Final decisions about the use of the maps and other data would be driven by the initial reasons for the process and its desired outcomes. For program planning applications, having feasibility and impact ratings of ideas for program implementation would help with decision making. In our project, the research team generated a final cluster solution, rating data (separately for clinicians and patients), pattern matching, and go-zone diagrams showing ideas rated as having both high feasibility and high impact.

We shared these data with our practice’s Patient and Family Advisory Council. The council’s response was unanimously positive, and the members expressed that it would be a reasonable method for them to use for generating and prioritizing their own group activities, a process that is under way. (Our council also participated in another group elicitation and mapping project as participants, responding to the prompt “What does whole-person orientation in primary care mean?”) A draft version of this article was shared with our department chairperson, who forwarded it to faculty. Simultaneously, we used a modified version of the group concept mapping process for brainstorming and mapping both faculty and staff transformation ideas when our practice transitioned into an accountable care organization. These data were presented at a department-wide “town-hall” meeting. We also used a modified concept mapping process in our departmental strategic planning. Each of these examples attests to the power of concept mapping as an efficient method of eliciting, organizing, and visualizing ideas among stakeholders.

DISCUSSION

To summarize, group concept mapping is a promising method with several potential applications in primary care. As more primary care practices formally integrate patients into practice improvement, concept mapping offers a feasible patient engagement technique that can also illustrate and quantify the convergence and divergence of patient and other stakeholder ideas, highlight the unique perspective that patients bring to practice improvement, and provide a basis for patient-centered practice improvements. Concept mapping may also be a powerful method for stakeholder engagement in other types of clinical research because it demonstrates and quantifies the effect of patient involvement in the process. Our experience demonstrated the feasibility and utility of concept mapping to engage patients and clinicians in a busy urban primary care practice in practice improvement research. Although multiple sessions at the practice were needed to recruit a sufficient number of patient-participants, the brainstorming sessions ultimately yielded more than 100 quality improvement ideas from both patients and clinic participants at our practice. The pattern matching and go-zone diagrams will aid in prioritizing improvement ideas.

Group concept mapping is adaptable to many practice improvement and research purposes. It is flexible enough to allow users to pick the steps and outputs that are most important and feasible. The cost of using proprietary concept mapping software or the services of a statistician to program SPSS (IBM Corporation) or SAS may be a barrier for some practices or practice groups. If this is the case, a practice may choose just to brainstorm, have participants rank statements on paper, and use a spreadsheet program to generate average ranking order, to compare groups of participants, or both, if needed (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

We suggest maintaining the participatory spirit of the methodology, regardless of how it is used, and we recommend considering the visual cluster analysis for organizing the group concept mapping results into a meaningful action plan.

More research is needed on the concept mapping process and outcomes. Future studies could compare group concept mapping with other patient engagement techniques for efficacy and efficiency. Understanding patient acceptability of group concept mapping is also important—it may necessitate a greater patient time commitment than interviews or focus groups, but it may also allow for more active patient involvement in multiple steps of the process. The group concept mapping process, as we have presented it here, appears to be a viable method of engaging stakeholders, and we suggest that research is needed that measures engagement as an outcome of group concept mapping.

Acknowledgments

We would like to acknowledge Jennifer Fisher Wilson at the Writing Center at the Center for Teaching and Learning, Thomas Jefferson University, for her review of the final version of this manuscript.

Footnotes

  • Conflicts of interest: authors report none.

  • Funding support: This project was supported in part by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) under grant D56HP20783, The Jefferson PCMH Predoctoral Education Project (Award Total: $830,397; 0% financed with nongovernmental sources).

  • Disclaimer: This views and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by HRSA, HHS, or the US Government.

  • Previous presentations: Portions of the content of this article were presented at the 2015 North American Primary Care Research Group (NAPCRG) Annual Meeting, October 24–28, 2015; Cancun, Mexico.

  • Supplementary materials: Available at http://www.AnnFamMed.org/content/14/4/370/suppl/DC1/.

  • Received for publication April 2, 2015.
  • Revision received January 11, 2016.
  • Accepted for publication January 26, 2016.

References

  1. ↵
    1. Laurance J,
    2. Henderson S,
    3. Howitt PJ,
    4. et al
    . Patient engagement: four case studies that highlight the potential for improved health outcomes and reduced costs. Health Aff (Millwood). 2014; 33(9): 1627–1634.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Dentzer S
    . Rx for the ‘blockbuster drug’ of patient engagement. Health Aff (Millwood). 2013;32(2):202.
    OpenUrlFREE Full Text
  3. ↵
    1. Carman KL,
    2. Dardess P,
    3. Maurer M,
    4. Sofaer S,
    5. Bechtel C,
    6. Sweeney J
    . Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). 2013;32(2):223–231.
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Crawford MJ,
    2. Rutter D,
    3. Manley C,
    4. et al
    . Systematic review of involving patients in the planning and development of health care. BMJ. 2002;325(7375):1263.
    OpenUrlAbstract/FREE Full Text
  5. ↵
    1. Scholle SH,
    2. Torda P,
    3. Peikes D,
    4. Han E,
    5. Genevro J
    . Engaging Patients and Families in the Medical Home. Rockville, MD: Agency for Healthcare Research and Quality; 2010. AHRQ Publication No. 10-0083-E.
  6. ↵
    1. Han E,
    2. Hudson Scholle S,
    3. Morton S,
    4. Bechtel C,
    5. Kessler R
    . Survey shows that fewer than a third of patient-centered medical home practices engage patients in quality improvement. Health Aff (Millwood). 2013;32(2):368–375.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Fontaine P,
    2. Whitebird R,
    3. Solberg LI,
    4. Tillema J,
    5. Smithson A,
    6. Crabtree BF
    . Minnesota’s early experience with medical home implementation: viewpoints from the front lines. J Gen Intern Med. 2015;30(7):899–906.
    OpenUrlCrossRefPubMed
  8. ↵
    1. Scholle SH,
    2. Asche SE,
    3. Morton S,
    4. Solberg LI,
    5. Tirodkar MA,
    6. Jaén CR
    . Support and strategies for change among small patient-centered medical home practices. Ann Fam Med. 2013;11(Suppl 1):S6–S13.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    1. Nilsen ES,
    2. Myrhaug HT,
    3. Johansen M,
    4. Oliver S,
    5. Oxman AD
    . Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev. 2006;3(3):CD004563.
    OpenUrlPubMed
  10. ↵
    1. Mockford C,
    2. Staniszewska S,
    3. Griffiths F,
    4. Herron-Marx S
    . The impact of patient and public involvement on UK NHS health care: a systematic review. Int J Qual Health Care. 2012;24(1):28–38.
    OpenUrlAbstract/FREE Full Text
  11. ↵
    1. Dansereau DF,
    2. Simpson DD
    . A picture is worth a thousand words: the case for graphic representations. Prof Psychol Res Pr. 2009;40(1):104–110.
    OpenUrlCrossRef
  12. ↵
    1. Ruiz-Primo MA,
    2. Shavelson RJ,
    3. Schultz SE
    . On the validity of concept map-based assessment interpretations: an experiment testing the assumption of hierarchical concept maps in science. National Center for Research on Evaluation, Standards and Student Testing. https://www.cse.ucla.edu/products/reports/TECH455.pdf. Published Jul 1997. Accessed Jul 23, 2015.
  13. ↵
    1. Trochim WMK
    . Pattern matching, validity, and conceptualization in program evaluation. Eval Rev. 1985;9(5):575–604. http://www.soci-alresearchmethods.net/research/Pattern%20Matching,Validity,%20and%20Conceptualization.PDF. Accessed Jun 24, 2016.
    OpenUrlAbstract/FREE Full Text
  14. ↵
    1. Burke JG,
    2. O’Campo P,
    3. Peak GL,
    4. Gielen AC,
    5. McDonnell KA,
    6. Trochim WMK
    . An introduction to concept mapping as a participatory public health research method. Qual Health Res. 2005;15(10):1392–1410.
    OpenUrlAbstract/FREE Full Text
    1. Kelly CM,
    2. Baker EA,
    3. Brownson RC,
    4. Schootman M
    . Translating research into practice: using concept mapping to determine locally relevant intervention strategies to increase physical activity. Eval Program Plann. 2007;30(3):282–293.
    OpenUrlCrossRefPubMed
  16. ↵
    1. Windsor LC
    . Using concept mapping in community-based participatory research: a mixed methods approach. J Mix Methods Res. 2013; 7(3):274–293.
    OpenUrlCrossRefPubMed
  17. ↵
    1. De Ridder D,
    2. Depla M,
    3. Severens P,
    4. Malsch M
    . Beliefs on coping with illness: a consumer’s perspective. Soc Sci Med. 1997; 44(5): 553–559.
    OpenUrlCrossRefPubMed
    1. Butler SF,
    2. Budman SH,
    3. Fernandez K,
    4. Jamison RN
    . Validation of a screener and opioid assessment measure for patients with chronic pain. Pain. 2004;112(1–2):65–75.
    OpenUrlCrossRefPubMed
    1. De Ridder DT,
    2. Schreurs KM,
    3. Bensing JM
    . Adaptive tasks, coping and quality of life of chronically ill patients: the cases of Parkinson’s disease and chronic fatigue syndrome. J Health Psychol. 1998;3(1):87–101.
    OpenUrlAbstract/FREE Full Text
  20. ↵
    1. Williams MT,
    2. Donnelly JP,
    3. Holmlund T,
    4. Battaglia M
    . ALS: family caregiver needs and quality of life. Amyotroph Lateral Scler. 2008; 9(5): 279–286.
    OpenUrlCrossRefPubMed
  21. ↵
    1. Brunings P,
    2. Klar S,
    3. Butt G,
    4. Nijkamp MD,
    5. Buxton JA
    . “It’s a big part of our lives”: A qualitative study defining quality of hepatitis C care from the patient’s perspective. Gastroenterol Nurs. 2013;36(4):249–57.
    OpenUrlCrossRefPubMed
    1. Snider CE,
    2. Kirst M,
    3. Abubakar S,
    4. Ahmad F,
    5. Nathens AB
    . Community-based Participatory Research: Development of an Emergency Department–based Youth Violence Intervention Using Concept Mapping. Acad Emerg Med. 2010;17(8):877–885.
    OpenUrlCrossRefPubMed
  23. ↵
    1. Southern DM,
    2. Batterham RW,
    3. Appleby NJ,
    4. Young D,
    5. Dunt D,
    6. Guibert R
    . The concept mapping method. An alternative to focus group inquiry in general practice. Aust Fam Physician. 1999;28:S35–40.
    OpenUrlPubMed
  24. ↵
    1. Trochim WM,
    2. Linton R
    . Conceptualization for planning and evaluation. Eval Program Plann. 1986;9(4):289–308.
    OpenUrlCrossRefPubMed
  25. ↵
    1. Barki H,
    2. Pinsonneault A
    . Small group brainstorming and idea quality: is electronic brainstorming the most effective approach? Small Group Res. 2001;32(2):158–205.
    OpenUrlAbstract/FREE Full Text
  26. ↵
    1. Rosas SR,
    2. Kane M
    . Quality and rigor of the concept mapping methodology: a pooled study analysis. Eval Program Plann. 2012; 35(2): 236–245.
    OpenUrlCrossRefPubMed
  27. ↵
    1. Trochim W,
    2. Kane M
    . Concept mapping: an introduction to structured conceptualization in health care. Int J Qual Heal Care. 2005;17(3):187–191.
    OpenUrlAbstract/FREE Full Text
  28. ↵
    1. Trochim WMK,
    2. Cabrera D
    . The complexity of concept mapping for policy. E:CO. 2005;7(1):11–22.
    OpenUrl
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Get Permissions

Jump to section

Subjects

Keywords