Publishing Multimethod Research

Like Deborah Cohen and Kurt Stange who recently posted track comments based on their participation in the NAPCRG workshop on publishing mixed methods, I also “hosted” a break-out group of approximately 10 researchers. We took our chairs outside and met on the veranda to explore participants’ experiences with publishing mixed methods. Most of the participants were experienced researchers who had considerable experience in publishing both qualitative and quantitative research manuscripts. Participants had earlier listened to the overview of publishing options covered in this Annals of Family Medicine editorial and sensed that ideally mixed methods studies would be published in a single manuscript, although they also like the option of publishing simultaneous manuscripts in the same journal like the Solberg/Hroscikoski combination published in July/August 2006 issue of Annals. Either of these options was seen as one of the ways for qualitative research to better reach clinicians who would not find the qualitative portions that are often published in social science journals.

A number of challenges to publishing a single comprehensive manuscript quickly emerged. A junior faculty member felt that to be considered for promotion and tenure, it was important to get more publications, so separation of qualitative and quantitative manuscripts seemed more attractive. They also felt that this would require a team of researchers, but only the first author would be given credit for the publication. This was confirmed by a group member who serves on a P&T committee who emphasized that P&T committees look at numbers and impact values. The group also worried about which journals would publish manuscripts that are longer, with one group member noting that most journals are constrained by the print format, seeing the length being determined in the same way that older songs were constrained by the number of grooves on a record. While online options seemed like a potential solution, one of the participants noted that P&T committees wanted a hardcopy version, with online journals still not being valued.

After deciding that reality seemed to favor the publication of simultaneous or consecutive qualitative and quantitative manuscripts, a number of ideas surfaced for enhancing the linkage of these. The group felt that the current approaches to alerting readers that there was a parallel manuscript could become stronger, particularly in journals with online versions. They felt it would be particularly helpful if the online version of an article that was previously published could be updated to include a “look forward” link to find other manuscripts from the same study that are published later than the initial publication (currently journals can put in a jump link to access other articles published in the same issue and could also put in a link to access past publications in the same journal).

Other ideas that were considered included: putting in “pop-up” text boxes on online versions; adding in information in the “information for authors” that indicates that the journal encourages simultaneous publication; and having a summary section of bullet points for multiple manuscripts from a line of research.

I would like to thank the members of the “outside veranda” group for such a free-ranging and stimulating conversation and invite them to post their impressions of publishing mixed methods.

Competing interests:   Annals Associate Editor

I facilitated one of the groups at the Workshop on publishing mixed method research at NARCRG (October, 2006). The group was comprised of 10- 15 researchers, most of whom where experts in doing mixed method research. Below, I attempt to summarize their comments. Our discussions focused on barriers to publishing mixed method research in medical journals.

Barrier: Many of the participants in the group found journal limits regarding length of a manuscript prohibitive when attempting to publish mixed methods research. Authors have at least three options when publishing a mixed method study: publish qualitative only, publish quantitative only, and/or attempt to publish a mixed methods manuscript. With the latter, describing the methods and the findings is more complex and requires more space.

Suggestion: Journals should have a more generous word or page limit for manuscripts that use and report on mixed methods research.

Barrier: Doing and reporting studies that aim to synthesize qualitative research. I am not sure I will recount this discussion adequately and urge the expert in this area to log on to the Annals TRACK and do so. My understanding is the participant was interested in developing several different ways to conduct and publish qualitative meta-syntheses. She felt editors might not be sympathetic to such syntheses.

Barrier: Medical audiences seem to be less interested in the description of theoretical and conceptual frameworks that inform the research being reported. When manuscripts include theory and conceptual development as well as empirical evidence then the manuscript is too long and complicated.

Suggestions: It was suggested that one solution to this is to publish theory and philosophy separate from empirical findings. Most weren’t satisfied with this suggestion, remarking that it leads to publication of findings that are decontextualized.

My notes: It seemed to me at this point that participants and others would benefit from examples of published mixed methods pieces. The examples from Social Science and Medicine are nice, but this journal has a more generous page limit and is read by researchers more than clinicians. There is the edition of Annals where the theme is multimethod research. This might be useful to hyperlink to the TRACK discussion.

Question: How should one format a single paper that involves mixed methods?

Responses from the group: There needs to be a research question that pulls together the analysis and results reported in the paper.

Telling the story is important – how did the use of one method lead to the use of another method.

Define the limits of your research question. One mixed method dataset may provide the data to publish multiple papers. Researchers need to define and limit the scope of their research questions so that a mixed method paper can be published that is relevant to the medical audience and small enough to meet stringent word limits.

This advice doesn’t apply well when trying to publish the ‘big’ paper or central findings paper from a study with a multi-method design.

When a big paper doesn’t lend itself to a smaller word limit, is there a way to keep the analysis and results whole?

- can we use the web to do this (putting some methods information on the web or placing some supporting materials on the web)

- can we develop a way to integrate multiple publications from one research study? For example, one large study has multiple publications – some quantitative, some qualitative, some mixed methods. A number of these publications may be complementary to the manuscript being reviewed. Is there a way to ask authors in advance of publication to list other publications from this project or grant that they’d like readers to be aware of and perhaps linked to? Then the website could offer hyperlinks to these other complementary publications.

- It was pointed out that BMC puts the description of trial methods on the website. This might be an option when publishing mixed methods work (methods layout on web)

One participant said (and I paraphrase) -- we need to recognize that the questions we ask in family medicine research aren’t questions that lend themselves to the use of single methods. Family medicine is about context and this requires mixed methods.

In response to the concern about page limits that are too restrictive, there were others who did recognize that the process of revision and refining a manuscript to meet a tight page limit often got the author to recognize the kernel or message of a paper. There was the recognition that we can often be enamored with our own work (in all of its vastness) and winnowing it down to the key message for a particular audience is often a useful and rewarding process. It also seemed like this was a process that required outside collaborators because the author isn’t always in the best position to identify how to cut-down his or her own work.

Participants also talked about the importance of having multiple quotes included in a manuscript to support a particular theme. Multiple quotes are important because they can show that an experience was ‘common’ occurred across a number of respondents and multiple quotes can be selected to bring out the nuances of an experience. However, participants reported being in a catch-22 type of situation with many manuscript and journal reviewers/editors. If they included too many quotes then the paper was reviewed as being too long. If they included only one quote, then the manuscript might be critiqued for being too opaque and not including enough support for claims or even questioning how wide-spread a theme reported in the manuscript might be. This seemed to be a balancing act.

Many participants joked about the ways they found to manage these problems within the constraints of journal page limits and rules (rules which, by the way, lend themselves to the reporting of quantitative research). Many laughed about how small they could make the font if they put quotes in a table or put them on the web or in the appendix. [Note: tables are often not counted in page limits or word counts and it might be argued that tables lend themselves better to the reporting of quantitative data than qualitative data. Perhaps quotes shouldn’t be counted in page limits either???] Use of tables and the web for reporting qualitative data were typically not ideal options because the quote would often be removed or distanced from the point the author was making in the text.

It seemed that under ideal circumstances, authors would carefully select multiple quotes that bring out the key features of an experience they are describing, and the manuscript and analysis would undergo several iterations in order to refine the key points, and authors would carefully select, identify and place quotes in the text. At the same time, these key qualitative data would not be relegated to a table, appendix or webpage. They would be part of the manuscript for readers to examine, evaluate and understand as they read the findings of the report.

One participant asked this – if the manuscript is initially accepted do you have an intermediary to help you cut it down?

One participant asked: Is this a point in evolution for mixed method research? Is there enough good mixed method research published that we can examine what has been reported in order to identify a format authors can follow to get to the ‘kernel?’ The comparison was made to consortium criteria for reporting RCTs. Could there be something analogous for reporting mixed methods research?

Another participant reported a recent experience with JNCI (not sure this is correct journal) where he/she was given a word limit but the word limit did not include the methods section. There was some joking about this – one could find all kinds of ways to integrate qualitative data (quotes and the like) into the methods section.

Participants talked about the many functions of publication. Publication in peer-reviewed journals is linked with advancement (promotion) as well as ability to get future grant funding. Publication is not necessarily connected to dissemination. Peer review is also associated with validation of the excellence of one’s research and the one’s excellence as a researcher. This was something the group thought about as they were discussing how and whether or not it was important to try to publish their mixed method work in places where clinicians (and even patients) might see it. This is an especially difficult decision if there is a tight page limit for peer reviewed journals or if these publications were not peer reviewed.

One participant cautioned the others about attempting to publish two separate papers simultaneously in the same journal. The report (probably from personal experience) was that if they (editors) don’t like one of the papers they may turn both down. Also, it will be important for each paper to stand alone because both papers may not be reviewed by the same people.

Others responded by saying – if publishing two papers simultaneously in the same journal is going to be an approach to publishing mixed method research then the opportunity needs to be ‘real’ – the journal needs to assign reviewers experienced in using and reviewing mixed method research and reviewers would need to read and review both manuscripts.

Respectfully submitted on behalf of my group,

Deborah Cohen

PS - Here's a shameless plug for the qualitative website I'm working on - www.qualres.org. Please visit!!

Competing interests:   None declared

On October 16, 2007, at the 34th Annual meeting of the North American Primary Care Research Group, the Annals editors led a workshop on publishing multimethod research. It was attended by approximately 50 participants from at least 6 different countries, the large majority of whom were experienced in performing mixed methods research. After Ben Crabtree and Will Miller gave an overview of multimethod research, Debbie Cohen provided a new website with useful information on performing qualitative research: www.qualres.org.

Workshop participants divided into 4 small groups to explore their experience with communicating the findings of multimethod research in ways that maximize the learning and cross-talk between the two methods. At the end, we invited participants to use the Annals online discussion to contribute their insights. We look forward to a lively discussion, and hope to be able to synthesize the resulting new ideas in the On TRACK feature in a future issue of the Annals.

Below are my brief notes about ideas from my small group at the workshop. These are summarized as PROBLEMS related to publishing multimethod research, and potential SOLUTIONS.

I look forward to hearing your additional ideas.

Kurt Stange

PROBLEMS • Word/space limit • Challenge of portraying exploratory work to a solution-oriented audience • Challenge of portraying complex and iterative - Methods (design & analysis) - Results - Interventions - Participatory approach • Different language and world views of many quantitative and qualitative reviewers & readers • Different audiences • Expert peer review for mixed methods • Timing (findings from quantitative and qualitative analyses that could inform each other often are not done at the same time.)

SOLUTIONS • Quality work • Focusing the main message to draw readers in to being willing to read the complexity • Use hyperlinks for details - Write a very focused main report - Provide details online (in appendices or hot links to other sources) - Results • Use a logic model in a diagram to convey complex processes • Separate quantitative and qualitative papers with a 3rd paper that conveys integrative cross-cutting findings or new theory that develops as a result • Journal editors could try to come up with a consensus statement and/or templates for different approaches to integrating quantitative and qualitative methods. (Note the editorial on which this eletter is commenting is an attempt to do this.) - Methods (design & analysis) - Results

Competing interests:   Annals Editor