Critical incident technique and explicitation interviewing in studies of information behavior

https://doi.org/10.1016/S0740-8188(02)00166-4Get rights and content

Abstract

This article discusses two related techniques, critical incident technique (CIT) and explicitation, which are used in a variety of social science research settings, and critically reviews their application to studies of information behavior. The current application of both techniques is compared with Flanagan's early guidelines on the CIT and is discussed in relation to recent experience in the use of (1) the CIT in the JUSTEIS and VIVOS projects and (2) explicitation in projects concerned with text entering on interactive Web sites. The JUSTEIS project identifies trends, and reasons for those trends, in the uptake and use of electronic information services in higher education in the United Kingdom; this article examines experience gained over the first two cycles—1999 to 2000 and 2000 to 2001. The VIVOS project evaluated virtual health library services. Comparison of the experiences gained on the various projects suggests that critical incident methods could usefully be extended and enriched by some explicitation methods, to elicit the degree of evocation required for current and future studies of Internet use.

Introduction

The critical incident technique (CIT), which developed from work in the U.S. Army Air Forces Aviation Psychology Program by Flanagan (1954), is regarded as a flexible set of principles, to be modified for the situation under study. Explicitation, however, is relatively unknown beyond its country of origin, France.

The CIT was originally used to assess performance in professional practice; a recent example of this usage includes the performance of health professionals—for example, the prescribing decisions of physicians (Bradley, 1992) and the quality of care for cancer patients (Bjorklund, 1999). More complex sets of behavioral intentions can be examined, such as the factors that affect patient compliance (Stromberg, Bostrom, Dahlstrom, & Fridlund, 1999), customer perceptions and reaction across a range of service industries (de Ruyter, Wetzels, & van Birgelen, 1999), and the quality of nursing care (Redfern & Norman, 1999). Other applications include supported reflection for student learning (Naidu & Oliver, 1999). The aim in health services research is often a better understanding of the interaction between patient and professional, and therefore the behaviors (patient, professional, or both) that appear to lead to effective care outcomes. Similarly, the studies that examine service quality in hospitality research or other service industries are generally concerned with the dimensions of service quality and identification of the key factors that affect customers' positive and negative perceptions.

Flanagan (1954) advocated five steps: (1) determine the general aim of the activity; (2) develop plans and specifications for collecting factual incidents regarding the activity; (3) collect the data (either through interview or written up by the observer); (4) analyze, as objectively as possible, and (5) interpret and report on the requirements, particularly those that make a significant contribution to the activity. According to these steps, studies of quality of care, or quality of service, can be categorized as follows:

  • •

    General aims. These require a care or service episode, which may be discrete, or what Strauss, Fagerhaugh, Suczek, and Wiender (1985) termed a more extended “trajectory of care” for chronic conditions, for example. These situations are nevertheless bounded, with some purposes, agreed to by the customers or clients and the service providers, and therefore they fall within Flanagan's specification. Some studies have included all stakeholders; others concentrated on provider or user; and one study, unusually, considered the impact of other customers on the service experience (Grove & Fisk, 1997).

  • •

    Plans and specifications. Flanagan indicated that for direct observation of behavior, observers should be familiar with the activity, the groups whose behavior is being studied should be specified, and the behaviors should be categorized, or some criteria developed. Health services researchers are frequently from the same profession as the group studied, but in other organization studies, researchers are more likely to be academic social scientists than business professionals.

  • •

    Observational and retrospective data collection. Flanagan implied that observational data collection is preferable, although retrospective data collection of incidents fresh in the mind of the study subjects is acceptable, if informed consent procedures assure the participants that they are not being singled out, and that they will remain anonymous. Flanagan noted (without reference to the specific studies) that slight changes in the question can produce substantial differences in the responses. He suggested that proper piloting is necessary, with checking of the interpretation of the question and instructions. The question should have a brief specification of the type of relevant behavior, and subjects should be restricted to recall of certain incidents, such as the most recent. In the actual interview, the subject should be allowed to do most of the talking to permit an unbiased account.

  • •

    Variations on data collection. Flanagan also mentioned group interviews (recent examples include Jackson & Stevenson, 2000, Liefooghe & Olafsson, 1999) and questionnaires.

  • •

    Scope of studies. Most recent studies examined between 50 to 100 incidents, although more complex activities may require several thousand incidents. Few studies can afford the latter level of research effort, and very few studies exceed 300 incidents for analysis. Each research subject usually contributes one or two incidents, although Janson and Becker (1998) used three incidents per subject. Several studies do not detail how piloting was achieved, and more emphasis seems to be put on the data analysis, whether through content analysis, phenomenonological approaches Chell, 1998, Johnson & Fauske, 2000, or grounded theory. In several health care studies Luker et al., 2000, Redfern & Norman, 1999, the emphasis is on delineation of the positive and negative quality indicators. A similar emphasis is evident in some educational studies (e.g., good and bad supervisory behaviors; Curtis, Helion, & Domsohn, 1998) and business research (e.g., attribution of positive and negative tourism experiences; Jackson, White, & Schmierer, 1996). The depth of detail that can be provided by the subject appears important (Kemppainen, 2000).

  • •

    Data analysis. This includes the frame of reference, or what the underlying purpose or purposes might be; the formulation of categories (acknowledged to be a subjective process); and general behaviors (the appropriate level of specificity). Several of the studies (e.g., Callan, 1998) could be viewed as extended pilot studies to refine the research methods and define provisional classification of categories. Several studies Runeson et al., 2001, Vettor & Kosinski, 2000, Von Post, 1998 described the research approach as the analysis of “stories” of care. In others, the quality of the dilemma itself is of interest, as in Bendtsen, Hensing, McKenzie, and Stridman's (1999) study of prescribing of analgesics in primary care. The organizational behavior studies often use a quantitative approach, to study the components of a transaction (O'Driscoll & Cooper, 1996) or citizenship behavior (Skarlicki & Latham, 1995).

  • •

    Interpreting and reporting, with full acknowledgment of limitations (e.g., researcher bias and groups that are not representative). Typically, the difficulty is one common to many qualitative research studies, that of coder interreliability, a problem acknowledged by Perry (1997) and addressed by Kemppainen, Levine, Mistal, and Schmidgall (2001).

The preponderance of critical incident studies in the health services research area is notable. Bradley (1992) suggested that the perennial popularity of anecdotes in medicine is because they make interesting stories and that as “stories” they offer insights to a social anthropologist about the social and conceptual world inhabited by the group. Some of the studies, as indicated previously, capitalize on this approach to understanding the nature of care. More particularly, perhaps, despite the bandwagon of evidence-based practice, the continuing popularity of the case report, the medical case presentation, or the nursing care study as formats for disseminating knowledge suggests that health professionals structure their knowledge according to story schemata (Urquhart, 1998) or illness scripts (Schmidt, Norman, & Boshuizen, 1990). Health-related incidents and stories are perhaps more memorable to all concerned than is a visit to a supermarket.

The CIT relies on recall of an actual event. In contrast, vignettes (Urquhart, 1999) may be used to examine the likely behavior of subjects in certain situations. The technique is useful when it is likely that attitudes or behavior would be less likely to be revealed using a direct approach. Subjects may be unwilling to be truthful when the situation is controversial.

Section snippets

Explicitation

The explicitation data collection technique (Vermersch, 1994) has some similarities with the retrospective data collection use in the CIT. What emerges from studying explicitation is a thorough and specific set of guidelines that are largely compatible with the goals of Flanagan and others, but which, in emphasizing how data are gathered and in providing theoretical grounding for why this should be so, also offers an interesting tool for examining purposes in gathering qualitative data on

Explicitation and Flanagan's five vital steps

For comparison with the CIT, Flanagan's (1954) five vital steps are used to explore the scope of the explicitation method:

  • •

    General aims. Any event, phenomenon, activity, or process can be investigated using the technique, but participants in an explicitation study only give an account of one instance of experience. This agreement forms part of the contract made with interviewees, and interviewers may remind interviewees of this in the interview if the interviewer suspects a more general

Review of the use of cit in studies of information seeking and use

CIT information behavior studies frequently occur in the health sector. Similar to an earlier small-scale study by Northup, Moore-West, Skipper, and Teaf (1983), many of these studies examine the motivations for information seeking, urgency of the request, type of information required, sources used, and reasons for selecting those sources (see Table 1).

Larger-scale studies include a National Library of Medicine study of the use of MEDLINE among 552 health professionals (mostly physicians), who

Use of the CIT in value and impact studies

The CIT has been developed in different ways for various studies of the effectiveness, value, and impact of service provision. Unlike the market research literature, most of these studies do not attempt to determine the dimensions of an information service encounter in terms of what was good or bad, but more often the aim is to determine the outcome of information use, and hence, by implication, the value ascribed to the service providing the information. Examples of this approach include

The JUSTEIS project

The JUSTEIS project Armstrong et al., 2000, Armstrong et al., 20011

Problems of integrating CIT interviews and questionnaires

The variations between the questionnaire and interview responses concerning particular EIS used for the critical incident seem to reflect Flanagan's (1954) observation that subtle changes in the wording of the question can produce large changes in response. If respondents are unclear about their own classification and description of EIS, then it is vital that interviews form a substantial component of any survey work, to clarify the terminology used by users to describe particular EIS. It would

Adapting the CIT to an evaluation project (VIVOS)

The Value and Impact of Virtual Outreach Services (VIVOS) project3 Yeoman et al., 2001a, Yeoman et al., 2001b used the CIT in its evaluation of outreach services in health libraries. The VIVOS research team worked closely with librarians at

Explicitation: Text-entering project

The explicitation technique recently was used in a study of the mental processes of experienced users when performing the types of Web-based task required on dynamic pages (Light & Wakeman, 2001). The study examined how people responded to entering text into interactive components on Web sites, such as comment boxes, search fields, and order forms. The explicitation technique was used to describe the following: (1) thoughts that went through the minds of the users as they approached and started

Conclusion

Explicitation has developed along parallel tracks to some of the critical incident studies, but there is a far stronger emphasis on the processes required for evocation to encourage full recall. What explicitation offers is an approach that might disentangle some of the deeper-seated perceptions from the information user. The approach appears to encourage a better recall of the episode and some of the steps taken. The particular difficulties encountered in the JUSTEIS project suggests that the

Acknowledgements

The authors thank the Joint Information Systems Committee of the Higher Education Funding Councils for their support of JUSTEIS and Resource (formerly the Library and Information Commission) for their funding of VIVOS. The authors also wish to thank the JISC Scientific Adviser Professor Jenny Rowley for her support on the JUSTEIS project and the School of Cognitive and Computer Sciences at the University of Sussex for its support for the text-entering project. Other members of the research team

References (74)

  • Armstrong, C. J., Everitt, J., Fenton, R., Lonsdale, R. E., McDermott, E. A., Phillips, R., Spink, S., Thomas, R. E., &...
  • Armstrong, C. J., Lonsdale, R. E., Stoker, D. A., & Urquhart, C. J. (2000). JUSTEIS: JISC usage surveys: Trends in...
  • R. Bandler et al.

    Frogs into princes

    (1979)
  • L. Banwell et al.

    JUBILEE (JISC user behaviour in information seeking: longitudinal evaluation of EIS)

    (2001)
  • L. Banwell et al.

    JUBILEE (JISC user behaviour in information seeking: longitudinal evaluation of EIS)

    (2000)
  • M. Bjorklund

    Cancer patients' experiences of nurses' behaviour and health promotion activities: A critical incident analysis

    European Journal of Cancer Care

    (1999)
  • P. Borlund

    Experimental components for the evaluation of interactive information retrieval systems

    Journal of Documentation

    (2000)
  • C.P. Bradley

    Uncomfortable prescribing decisions: A critical incident study

    British Medical Journal

    (1992)
  • E. Chell

    Critical incident technique

  • D.G. Covell et al.

    Information needs in office practice: Are they being met?

    Annals of Internal Medicine

    (1985)
  • N. Curtis et al.

    Student athletic trainer perceptions of clinical supervisor behaviours: A critical incident study

    Journal of Athletic Training

    (1998)
  • D.M. D'Alessandro et al.

    Barriers to rural physician use of a digital health sciences library

    Bulletin of the Medical Library Association

    (1998)
  • Davies, R., Urquhart, C. J., Massiter, C., & Hepworth, J. B. (1997). Establishing the value of information to nursing...
  • K. de Ruyter et al.

    How do customers react to critical service encounters? A cross-sectional perspective

    Total Quality Management

    (1999)
  • D.M. Edwards et al.

    “Lost in hyperspace”: Cognitive mapping and navigation in a hypertext environment

  • K.A. Ericsson et al.
  • J.C. Flanagan

    The critical incident technique

    Psychological Bulletin

    (1954)
  • Graphics. Visualization, and Usability Center. (1998). GVU WWW user surveys, 10th—1998. Retrieved September 20, 2000,...
  • G. Hripcsak et al.

    User comments on a clinical event monitor

    Proceedings Annual Symposium Computer Applications Medical Care

    (1994)
  • S. Jackson et al.

    What do people need psychiatric and mental health nurses for?

    Journal of Advanced Nursing

    (2000)
  • S. Janson et al.

    Reasons for delay in seeking treatment for acute asthma: The patient's perspective

    Journal of Asthma

    (1998)
  • B.L. Johnson et al.

    Principals and the political economy of environmental enactment

    Educational Administration Quarterly

    (2000)
  • E.M. Keen et al.

    Visual processing in information retrieval searching: A collection of research papers

    (1980)
  • J.K. Kemppainen

    The critical incident technique and nursing care quality system

    Journal of Advanced Nursing

    (2000)
  • J.K. Kemppainen et al.

    HAART adherence in culturally diverse patients with HIV/AIDS: A study of male patients from a Veterans Administration hospital in northern California

    AIDS Patient Care and STDS

    (2001)
  • R. Lamb et al.

    Informational environments: Organizational contexts of online information use

    Journal of the American Society for Information Science and Technology

    (2003)
  • S. Levinson

    Pragmatics

    (1987)
  • Cited by (0)

    View full text