Abstract
The requirements of research become more complex and demanding in international collaborations. The opportunity to study naturally occurring variation in treatment prompted networking primary care research networks in the United Kingdom, the Netherlands, and North America to study acute otitis media. Additional challenges faced and addressed in this study included (1) differing national requirements for protecting human subjects; (2) variation in data collection processes in primary care practices; (3) data transmission among participants; (4) duties and tariffs on necessary instruments; (5) fluctuation in currency exchange rates; (6) incapacitation of coinvestigators; (7) complex administration of funds; (8) financing the additional, legitimate costs of collaboration; (9) sustaining strong personal relationships among coinvestigators; and (10) accepting longer time frames than would otherwise be expected. Overall, international practice-based research can be productive, affect millions of people, and be extremely rewarding to investigators. It is not, however, for the faint-hearted.
- International research
- international cooperation
- practice-based research/methods
- research design
- otitis media
INTRODUCTION
When viewed retrospectively through the lens of scientific papers with an emphasis on methods and results, studies typically appear to have been done in a logical and orderly manner as planned by the investigators. In reality, successful researchers are called upon to address the unexpected and overcome a host of challenges that are not usually elaborated in their reports. The purpose of this article is to illustrate how international research can be important and rewarding while confessing a set of challenges that beset an international study, thereby reminding investigators that international primary care research is extremely valuable but infused with additional requirements.
Research becomes even more complex and demanding when investigators unite to study phenomena across national boundaries. One of the common reasons for international studies is to identify and examine the consequences of variation in illness, disease, health care, and health system arrangements.1 Whereas scientific research principles are accepted globally, their expression and implementation may vary when specific methods and strategies are applied in different countries.2 These variations can be simultaneously a source of rewarding stimulation, new knowledge, and frustration for investigators.3 Such was the case for an international study of acute otitis media (AOM) funded by the US Agency for Healthcare Research and Quality (AHRQ) and administered by The Ambulatory Sentinel Practice Network (ASPN), a pioneering, national primary care practice-based research network in the United States.4
OPPORTUNITY
The attractive opportunity that prompted this international study of AOM in children 6 months to 18 years of age was grounded in naturally occurring variation in how children with AOM were treated in the United Kingdom (UK), the Netherlands, and the United States and Canada (North America).4 In the UK and North America usual care included routine prescribing of antibiotics, but for different lengths of time (5 to 7 vs 10 days, respectively). In the Netherlands a wait-and-see approach was applied for most children, and this strategy resulted in prescribing antibiotics for a minority of children. At the time of the study, 1994 to 1996, a placebo-controlled trial of antibiotic vs no antibiotic was not acceptable to human subjects review committees, but a study in which the requirement was to deliver to all enrolled subjects the care usually rendered by the nation’s clinicians was considered ethical. Thus, a concurrent observational study of children with acute otitis media receiving usual care in the Netherlands, the UK, and North America was undertaken by experienced, practice-based research networks5 comprised of frontline clinicians caring for people living in their communities in Europe and North America.6
The value of such research is illustrated in Figures 1⇓ and 2⇓. Despite different treatment practices, there was little variation in the time from initial treatment to the last report of pain or fever for children either younger than 2 or 2 to 18 years of age, based on children who had both initial visit data and a 10-day parent diary (85% of enrolled children), adjusted for sex of the child, breast feeding (ever), exposure to household tobacco smoke, and a parental history of otitis media. Results from this study were cited in subsequent randomized trials7 and were part of an evolution of care for acute otitis media during the 1990s toward current guidelines that overall encourage restrained use of antibiotics.8
Pain resolution for children with acute otitis media.
Fever resolution for children with acute otitis media.
Results such as these are important, verify the feasibility of practice-based research in networks of frontline clinicians, and reward the work and effort of international collaboration. Underlying these results, however, were a number of challenges that threatened the viability of the study, diminished its potential impact, and from which important lessons can be identified that might inform future international research efforts.
CHALLENGES
Challenge 1: Human Subjects Review
Each of the participating countries had established mechanisms to assure the ethical conduct of research and the protection of human subjects, but the processes differed. Because AHRQ was the funding agency, compliance with its rules and regulations was a necessity. Thus, the investigators outside the United States had to meet the dual expectations of their own review processes and also modify or augment them to accommodate the requirements from the United States.
For example, the organization of this process in the UK required the UK network to submit the study to 9 different institutional review boards because of the geographic dispersion of their practices. The work and vacation schedules of the chairpersons of these 9 review boards delayed approvals that had to occur in a defined sequence. There was considerable reluctance to accommodate the needs of this particular project, given that these review boards were being asked to do something different from what they required and expected. When the UK process was completed, the UK network launched the study only to learn that the US Office of Protection from Research Risks (OPRR) had not signed off on their review. An untimely change of staff at OPRR created further delays. ASPN, the administrator of the research funding, based on an interpretation of OPRR regulations, insisted that the UK network stop enrollment until the OPRR clearance was obtained. The UK suspended enrollment to the consternation of their clinicians.
There were substantial consequences of the multiple human subject reviews and the resulting delays. Data collection was extended into the final year of the project. The UK missed much of the wintertime enrollment period when acute otitis was more common, and never caught up with the other networks’ enrollment. Analysis and comparative feedback to the networks and their physicians were delayed, disappointing clinicians. Project analysis and publication of results could not be accomplished as planned.
Challenge 2: Instrumentation
This project was one of the first practice-based research projects to incorporate handheld computers as a means of data collection by the participating clinicians. The UK network was responsible for the development of this system, and after creating the software for what was thought to be a suitable platform, the Dutch clinicians simply said, “No!” The keyboard requirements and sequencing of data collection processes did not make sense in the Dutch practice environment. Consequently, an alternative computer, a Newton, was selected, and the software to support the study was completely rebuilt.
The new equipment had to be distributed across national boundaries and became subject to tariff and duty regulations. Grant management regulations forbade the use of grant funds for taxes and tariffs, and there were no alternative funds to cover this cost of several thousands of dollars. This problem was solved by preprogramming all of the handheld computers with the study materials, thereby converting them into research instruments that were exempt from duties. Handheld tympanometers were also required for the study, and a similar problem was overcome by importing the tympanometers into Europe as research instruments.
These instrumentation challenges resulted in relocation of programming from one country to another, duplication of effort, unfunded work, and delayed implementation of data collection.
Challenge 3: Data Transmission
Once in place, the handheld computer system functioned well, connected through a satellite system called eWorld. The database was assembled and coordinated at Brown University, with the study grant paying the costs of data transmission. Being able to see the database build in real time and being able to respond to problems promptly was a splendid, successful development.
Inadvertently, however, in each of the 2 years of data collection, ASPN exceeded limits on the credit card accounts to which the charges for eWorld were applied, resulting in the immediate discontinuation of eWorld services. This appeared to the research networks as a failure of the data transmission system. Clinicians and study coordinators suddenly could not transfer their data and would scramble to identify what had gone wrong, only to discover not a technical computing problem, but that a billing problem in the United States had “shut them down.”
Months later, the owner and operator of eWorld, decided to discontinue the service as part of a comprehensive new business plan. This decision took effect in the middle of the second winter enrollment season. Follow-up of enrolled patients required a 2-month visit with data to be transmitted via eWorld, and a few patients required a later visit.
Consequently, an alternative mechanism was required, immediately, not just in the vast North American continent, but across the Atlantic. The result was the invention, near the end of data collection, of another data collection system for each network, including a paper, pencil, and mail method. Data acquisition, closing of the data sets, and analysis were delayed, and all the networks and the central coordinating office had to adapt and absorb the additional expenses.
Challenge 4: Financial Administration
This acute otitis media study was designed and funded as a 3-year project, with each of the collaborating networks having its own subbudget, financial regulations, indirect cost requirements, and cash flow needs. From the perspective of each network, the processes of grant review and budget negotiations resulted in work that was more complex and a budget that was reduced. Subsequently, in the middle of the study, the US Congress cut the funding for AHRQ (then known as the Agency for Health Care Policy and Research), which led to a further 25% reduction in the budget that had already been reduced by negotiations at the start of the project. Because of delays in implementation, study enrollment did not conclude until month 33, requiring in some instances follow-up 9 months later, necessitating an unfunded extension on the heels of the US-imposed 25% reduction in budget. Also, during the study international monetary policies evolved, resulting in a devaluation of the US dollar against European currencies, compounding the effects of other budget reductions by an additional 25% reduction in Europe.
As a result of all of these situations, the networks were faced with the expected workload and the additional workloads associated with study delays and modifications—with a recurrently shrinking budget. These reductions had to be absorbed, somewhere, but preferably “not in my country’s budget.” Also, mismatches between when grant payments were made to networks and when costs were accrued by the project required skillful adaptation by local managers to cope with cash flow requirements. Other fiscal regulations, such as use of US airlines for travel, were reduced to being viewed as a mere nuisance by the European collaborators.
Challenge 5: Death and Despair
Midway into the project, the Dutch project director, in his mid-40s, suffered a fatal myocardial infarction while ice-skating with his daughter. There was no equally qualified person to assume his role. More importantly, his intellectual command of the project and his very strong and productive relationships with Dutch practices and the international collaborators could not be simply reinvented. Even though his colleagues ably stepped into the gap and continued the project, the hole he left was never completely filled.
After the project succeeded in completing enrollment and a robust data set was clean and established, the funding and time intended for analysis were long gone. Principled principal investigators persisted with analysis and reporting, but progress was inevitably slow because of competing requirements and loss of interest in an old project. Nonetheless, as the months passed, much of the analytic plan was achieved, and further reports were in draft stages when the principal investigator died.
SOME PRACTICAL LESSONS
Accomplishing research in primary care practice-based networks is complicated, and international research adds even more complexities that can be expected to prolong investigations. The story of the international acute otitis media study describes lessons that are probably most accurately viewed as old lessons, relearned. Some of these lessons are highlighted in Table 1⇓.
Practical Lessons in International Primary Care Research
Compared with routine single-site or single-country research, international research, particularly in practice-based research networks, requires additional effort, administrative skill, and patience. Pilot testing is even more important in international studies, but no amount of pilot testing can foresee or preclude environmental changes that may destabilize international research. International projects almost certainly will take longer than expected and should include more generous time frames than would otherwise be required. Strong, trusting relationships among coinvestigators can be the basis of overcoming much adversity; and the practical means of cultivating these relationships through conference calls, meetings, and social engagements is a legitimate and necessary research expense.
Trivial questions do not justify the effort, but for important, compelling questions, such as comparisons of treatments for conditions that afflict millions of people, the yield of findings can be worth the additional effort and expense. Overall, international practice-based research can be productive and extremely gratifying, but it is not for the fainthearted.
Acknowledgments
Ruut A. deMelker, Paul Grob, Max Jacobs, Frank van Balen and the practices in the research networks as named in Froom et al5 were indispensable to the international study on which this article is based.
Dr. Jack Froom (1923–2002) inspired a generation of primary care investigators, and this essay is dedicated to him and his lifetime partner, Selma, with our deepest admiration and sincere thanks—for everything.
Footnotes
-
↵† Deceased.
-
Conflicts of interest: none reported
-
Funding support: This research was supported by the Agency for Healthcare Research and Quality.
- Received for publication April 1, 2004.
- Revision received June 15, 2004.
- Accepted for publication June 27, 2004.
- © 2004 Annals of Family Medicine, Inc.
Jump to section
Related Articles
Cited By...
- Acute sore throat
- Data Collection Outcomes Comparing Paper Forms With PDA Forms in an Office-Based Patient Survey
- The Creation and Development of the Dental Practice-Based Research Network
- Electronic Data Collection Options for Practice-Based Research Networks
- In this Issue: The Patient-Clinician Relationship and Practice-Based Network Research