Abstract
PURPOSE Addressing important but complex research questions often necessitates the creation of innovative mixed methods designs. This report describes an approach to developing research designs for studying important but methodologically challenging research questions.
METHODS The Methodological Think Tank has been held annually in conjunction with the Primary Care Research Methods and Statistics Conference in San Antonio since 1994. A group of 3 to 4 methodologists with expertise balanced between quantitative and qualitative backgrounds is invited by the think tank coordinators to serve on a 2-day think tank to discuss a research question selected from those submitted in response to a call for proposals. During the first half-day, these experts explore the content area with the investigator, often challenging beliefs and assumptions. During the second half-day, the think tank participants systematically prune potential approaches until a desirable research method is identified.
RESULTS To date, the most recent 7 think tanks have produced fundable research designs, with 1 being funded by a K award and 4 by R01 grants. All participating investigators attributed much of their success to think tank participation. Lessons learned include (1) the importance of careful selection of participating methodologists, (2) all think tank communities of inquiry must go through 4 stages of development from pseudocommunity to community, and (3) the critical importance of listening by the investigator.
CONCLUSION Researchers and academic departments could use this process locally to develop innovative research designs.
INTRODUCTION
Important but complex research issues have emerged that defy direct application of most available research designs and methods in which investigators have been trained. Most investigators have been trained in a particular discipline or a rather narrowly-focused set of research methods and designs. Often, researchers are limited by forcing projects into existing designs, resulting in studies that are driven by the available methods. It may be necessary to create innovative mixed methods designs that have never been used.1–4 It is generally accepted that collaborative teams of researchers are needed to develop fundable new and innovative designs. In fact, NIH has recently called for proposals for producing these innovative methods. The NIH Roadmap emphasizes the need for “new pathways to discovery” (http://nihroadmap.nih.gov). Its R21 grant program supports research and development of new technologies, techniques, and methods. Thus, one challenge facing the primary care research establishment is the development of new research approaches to address research questions of ever-increasing complexity and importance.
The Creative Process
Figure 1⇓ presents the process of creativity from motivation and preparation, through the combination of manipulation of ideas and approaches with incubation time to produce illumination (the creative idea), and ultimately to evaluation.5 Specific steps can be taken to encourage success at each stage in the creative process. At the preparation stage, opportunities for adequate learning and communication help; however, overpreparation can diminish creativity. Activities that encourage manipulation, incubation, and illumination include cross-fertilization, communication, intuition, and time. Finally, commitment and expectations of criticism support evaluation.5 Divergent thinking promotes the manipulation-incubation-illumination of creativity.5,6 To bring these divergent explorations into useful focus, however, convergent thinking must be applied to limit possibilities.6
The creative process.
Reprinted with permission from Ridkey J, Sheldon GF. Managing in Academics. St. Louis, Mo: Quality Medical Publishing; 1993.
Thus, the creative process is a balance between idea generation and idea evaluation.7 In other words, creativity exists because of tension—tension between flexibility and control, between focused and unfocused goals, between the means and the ends, and between over- and underevaluation.5 These tensions are seen in learning organizations and business innovations.8,9
Building Communities of Inquiry
Peck identified 4 stages in the development of a community of inquiry.10 Pseudocommunity is the initial stage of conflict avoidance, in which the group uses polite rules and limits on emotions to minimize differences. As the group process advances, chaos results as a stage of open conflict, typified by obvious differences and attempts to convert others to your perspective; chaos is resolved by either returning to pseudocommunity or progressing to emptiness. Emptiness is the stage in which group members empty themselves of barriers to communication, such as expectations, ideology, and need for control. Only after emptiness can community develop, characterized by commitment, shared leadership, graceful disagreement, perceived safety, and community spirit.
Using Peck’s stages, Miller identified the process of changing multidisciplinary groups into transdisciplinary teams.11 The processes of acceptance and shared expectations of multidisciplinary pseudocommunities progress to declaring group process with its acknowledgment of differences and attempts at conversion (chaos). Only then can an action consensus develop, characterized by continuing tension but without the barriers to communication (emptiness). Then, by identifying shared common space, the team moves into community with its shared framework and language.
The purpose of this case study is to describe a methodological think tank, an approach that uses team-building concepts to develop quickly a shared common space for identifying innovative ways of integrating qualitative and quantitative methods to study important but challenging research questions. These methods may have broader applications.
Think Tank Experience
In 1993, the Agency for Healthcare Research and Quality (AHRQ) funded the Conference on Multi-method Research (R13 HS07291), which was held in Omaha, Neb, with the purpose of developing a collaborative, cross-disciplinary research community in primary care.12 This 2½-day brainstorming conference was at times antagonistic and anything but collaborative, but the group of established investigators were eventually able to come together to find common ground as they began to listen to each other and be more reflective. The lessons learned from the Omaha conference experience include (1) the importance of careful selection of participating methodologists, (2) successful communities of inquiry go through 4 stages of development from pseudocommunity to community, and (3) the critical importance of listening by the investigator. These lessons have been repeatedly validated during the think tank experience.
Based upon these lessons, the authors established a methodological think tank that would bring together content and methods experts to participate in a process of developing a creative learning space over a very short period of time. The first methodological think tank was held during the 1994 Primary Care Research Methods and Statistics (PCRMS) Conference, meeting over 2 half-days. In the first 2 years in our role as think tank coordinators, we selected a topic—nature of caring—and recruited both an investigator and a diverse group of methodological experts. After reflecting on the experience, we realized that, because the investigator was recruited for but did not seek out think tank involvement, little was done to implement think tank results between meetings; it was only an exercise. Hence, beginning in 1996, the selection process for content experts (investigators) was changed so that proposals were solicited from investigators who had a methodological challenge that would require the judicious mixing of methods or creation of new methods. Accordingly, since 1996, only 1 study proposal has been selected each year for think tank participation from among those submitted by investigators.
Because of the authors’ previous experience at the Omaha conference that groups can easily become mired in either pseudocommunity or chaos,12 the think tank has sought to pass quickly through these stages and move rapidly into community. To accomplish a rapid move to community, careful selection of methodologists, including some with previous think tank experience, has been used to minimize tendencies for conflict avoidance and attempts to convert others. Although attendees of the larger Primary Care Research Methods and Statistics Conference frequently sought to observe the process, we continually refuse such observation, recognizing the need for a safe place if community is to exist. A key factor in the rapid evolution of community during the think tank is the use of tactics identified by Miller11 as conducive to development of transdisciplinary teams—brainstorming, storytelling, humor, and silence when stuck. The think tank is a time of exploration and brainstorming, often including storytelling and humor. Breaks are taken when participants become stuck and after extensive exploration to permit synthesis and refocusing. Thus, through selection of methodologists and group tactics, the think tank is rapidly moved into community to foster open exploration of the content area and methods.
Selection of Studies and Methodologists
In response to a call for research problems, experienced investigators submit think tank proposals consisting of a research question, a brief literature summary emphasizing the importance of the research question, and a discussion about the methodological dilemma—no proposed research design is submitted. Based upon the perceived significance of the research question and its methodological dilemma, 1 proposal is selected by the coordinators for think tank involvement. The think tank coordinators have backgrounds in multimethod research, with contacts both inside and outside the primary care research community, thus facilitating the selection of diverse methods experts.
This selection process has led to a diverse group of investigators and projects. Of the 10 investigators participating in the think tank, 4 had doctoral degrees in philosophy or public health and 6 were doctors of medicine. Although 7 investigators came from departments of family medicine, there was also 1 internist, 1 pediatrician, and 1 nurse. Table 1⇓ lists the topics addressed by the think tank since its inception in 1994.
Think Tank Topics
Once the proposal is selected, a group of 3 or 4 methodologists with quantitative and qualitative backgrounds equally represented are identified by the coordinators to serve on the think tank. Methodologists are selected based on the following criteria:
-
Methodological expertise, based on their record around particular method(s)
-
Successful research experience, based on their record of publication and funding
-
Flexibility and open-mindedness (capable of thinking outside the box), based on their record of challenging established truths
-
Collaborativeness (play well with others), based on their record of successful interdisciplinary collaboration
Preferably, each think tank panel includes some methodologists who have experience serving on previous think tanks. The investigator submitting the proposal serves as the content expert, although we also try to find methodological experts with some relevant content experience. Although no particular mix of backgrounds is essential, the desired think tank panel has the diverse research backgrounds and skills necessary to development of richer designs, but it is capable of innovative group process with minimal time needed for group development.
Think Tank Format
To make the most of the usable time available to the think tank, the investigator provides the methodologists with information before the think tank convenes. This information includes a literature review on the content area and descriptions of previous research approaches, with intent to ensure that methodologists begin the think tank firmly grounded in the content area and its research methods. This information is shared with the methodological experts, who may request additional information from the investigator before the meeting.
After the introductions, the think tank begins when the investigator is asked to provide an overview of the research problem and an update of recent developments. Based on experience, the member most likely committed to ideology, need for control, and the insolubility of the research question is the experienced investigator. Thus, to minimize the barriers that the investigator may seek initially to impose, coordinators begin the think tank by establishing ground rules that include abandonment of expectations, ideology, and control, and by restricting investigator involvement after his or her initial summary and update; this tactic is particularly important with less experienced investigators, because these initial deliberations are often quite technical and free ranging, and input from the investigator has a tendency to narrow the discussion. The process is allowed to proceed without interference until community is established. Once this initial exploratory phase is over, the investigator is allowed to join the methodologists in a brainstorming of ideas, issues, and methodological approaches. Although the coordinators take the lead initially, shared leadership quickly emerges as a sign of community. Thus, the first half-day is a period of exploration, often challenging beliefs and assumptions in the content area.
Typically, the first half-day ends with a sense of loss of direction and uncertainty; the investigator is often given the message that the original research question is either premature or not answerable in its current form. Nevertheless, the investigator has also been presented with a wealth of creative ideas concerning possible new directions and methods and is left energized by the experience. Hence, the investigator must use the time before the next half-day session to make some decisions about the research question.
The second half-day begins with a summary of the first half-day’s discussion and the investigator’s overnight reflections. After identifying the desired outcome based on the investigator’s input, the coordinator leads the think tank in a guided discussion to systematically prune the potential approaches until a desirable method is identified, often involving a series of studies.
Thus, the think tank represents an analogy to this innovative process—a balance between idea generation and idea evaluation—as the first half-day devotes time and energy to an unbridled exploration of the content and methods focused on process, whereas the second half-day limits time and energy to prune the diverse possibilities of the first half-day to reach a tangible research design focused on outcome. Once community has been established, the think tank provides opportunities for communication, cross-fertilization, and expectations of criticism, supporting the creative process.
RESULTS
Case Example
One think tank proposal was to develop a typology of health habit advice. The think thank began with a discussion of the importance of the research question and the methodological problems. Participants also explored the complexity of a typology that would recognize the importance of the content of the advice, when the advice was presented, characteristics of the clinician providing the advice and in what setting, the strategies used to present advice, and the outcomes of the advice. All these elements were considered in the context within the clinical encounter, family, practice, community, and health care system. Ultimately, it was decided by the investigator to focus on the provision of advice within the clinical encounter. The approach would use data from an existing practice-based database to develop an initial typology that would be reviewed and altered by the practitioners contributing to that database. Input of a panel of national experts in preventive medicine reviewing the typology would then be used in conjunction with a review of the research and theoretical literature to produce a revised typology. This typology would again be reviewed by the expert panel. The resultant typological framework would first be tested using existing databases and, if robust, could be used in future studies. This formative work led to a refinement in the investigator’s thinking about the process of health behavior change discussions, and the investigator was successful in securing National Cancer Institute funding for this line of inquiry.
Think Tank Track Record
Table 1⇑ displays the topics and resultant approaches from each think tank. Of the 8 recent think tanks, the discussions in one led to completely changing the research question. Two of the think tanks focused on measurement issues. In another 2, the discussion generated a process for typology development, in one case using existent data and in another case using newly collected data through the creation of a new instrument. Other think tanks produced comprehensive research designs, such as whole system case studies, serial studies using a large database that reflexively used qualitative and quantitative data to create and describe patient clusters, and a pilot design to integrate encounter data for demonstrating progression in patients’ stage of change.
Success of the think tank is measured by its ability to produce novel but fundable research designs. Investigators are contacted periodically to assess the outcomes of participation in the think tank process. In addition to evaluating the perceived impact of think tank participation, they are asked to provide descriptions of grants, collaborations, and implemented studies that resulted directly or indirectly from their think tank experience. Excluding the coordinator-initiated think tanks (1994, 1995) and the most recent think tank (2003), all think tank participants valued their think tank participation and attributed much of their eventual research success to this participation. Currently, these 7 think tanks have resulted in fundable research designs, including one through a K award and 4 through an R01 mechanism. The think tank process has also led to 3 external collaborations for investigators.
DISCUSSION
Although we cannot measure the innovativeness of methodological approaches produced, the Methodological Think Tank has a track record of success. This outcome suggests that a small, diverse group of methodologists can rapidly evolve into a transdisciplinary team in collaboration with an experienced researcher to develop innovative designs to answer important questions that are not easily addressed with conventional research designs. To do so, the team of methodological experts and the investigator must progress through stages of team development.
Guidelines for Applying the Think Tank Process Locally
When attempting to recreate the think tank process locally, begin with the selection of the think tank coordinator. This person should be able to take a social, flexible, hands-off approach to leading the group. Although he or she must establish and enforce ground rules, the coordinator must also allow group exploration while being capable of directing the focusing process of day 2. The next step is to find a cadre of local and external potential methodologists. They need the qualities mentioned above and can be found through searches and recommendations by others. Before selecting methodologists, be sure to review their curriculum vita and talk with their collaborators. Although the methodologists need not have worked together before, it is advantageous if they have. Accordingly, if the selected group of methodologists is local, have them convene before the think tank to meet each other and consider having them engage in team-building exercises to build community. The investigator for the think tank should be selected by the methodologists and coordinator based on their research experience and the potential for their research to benefit from the think tank experience.
Initially, do not allow coinvestigators to participate until the coordinator is confident with his or her role; observers must be excluded. The think tank process should be based on a format similar to the one described in this article. We therefore recommend a 2-day process that begins with a period of exploration by the methodologists while the investigator listens. Only after community group dynamics are established can the investigator participate. Table 2⇓ lists barriers to creative thinking and participation.7 Most of these limits are dealt with through the setting of ground rules and the open tradition of the think tank. During the overnight period separating day 1 and day 2, the local research team can review day 1 developments with the investigator and assist in preparing for day 2.
Barriers to Creative Thinking and Participation
Implementation of a think tank process by institutions or departments represents an investment of faculty time and money, especially if external methodologists are brought in. This modest investment, however, can meaningfully increase creativity of research designs.
Limitations
There are obvious potential limitations to the process as described. First, our results may simply reflect selection bias. The success of the investigators in obtaining funding may reflect our selection of experienced investigators who were already on the cusp of funding, although the value of the think tank as attributed by the investigators would tend to refute that. In addition, the selected methodologists may be unique, representing a special group of scholars that could not be recreated by local institutions. Another potential limitation is that investigators must not be so entrenched in their methods before the think tank that they are unwilling to explore other options. Finally, the think tank process does not develop overnight; it requires time and support from the department chair or dean to develop a cadre of methodologists.
A collaborative process involving researchers with diverse methodological expertise engaged in a creative discussion over a relatively short period of time has promise for facilitating progress on complex research questions. One strength of the think tank process lies in its ability to produce innovative designs; not only has the think tank consistently lead to novel approaches, but it has a track record of producing fundable designs. Nevertheless, the process does require a group of highly selected methodologists who have sufficient uninterrupted time and are able to tolerate ambiguity and nonlinearity. Although this article describes an approach for implementing think tanks in the context of a national conference, the think tank process may have broad applications for primary care departments.
Footnotes
-
Conflicts of interest: none reported
-
Funding support: This project is supported by a grant from the Agency for Healthcare Research and Quality (R13 HS08775).
- Received for publication October 27, 2004.
- Revision received December 16, 2005.
- Accepted for publication January 26, 2006.
- © 2006 Annals of Family Medicine, Inc.
