Development and Validation of Search Filters to Identify Articles on Family Medicine in Online Medical Databases

David H.J. Pols; Wichor M. Bramer; Patrick J.E. Bindels; Floris A. van de Laar; Arthur M. Bohnen

doi:10.1370/afm.1780

Published eLetters

If you would like to comment on this article, click on Submit a Response to This article, below. We welcome your input.

Submit a Response to This Article

Compose eLetter

Title *

Author Information

Contributors
First Name and Middle Initial * First or given name, e.g. 'Peter'. Last Name * Your last, or family, name, e.g. 'MacMoody'. Email Address * Your email address, e.g. higgs-boson@gmail.com Occupation * Your role and/or occupation, e.g. 'Orthopedic Surgeon'. Affiliation * Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.

Statement of Competing Interests

Competing interests? *

Yes

Please describe the competing interests

CAPTCHA

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

What code is in the image? *

Enter the characters shown in the image.

Vertical Tabs

Jump to comment:

Response from the authors
David H.J. Pols

Published on: 28 August 2015
New search filters for articles most welcome
Johannes C. van der Wouden

Published on: 03 August 2015

Published on: (28 August 2015)
Page navigation anchor for Response from the authors
Response from the authors
David H.J. Pols, GP
Other Contributors:
We appreciate the comments of J.C. van der Wouden et al. They make a valid point regarding the use of the 'reference standard'. Part of this set was used for the development of the filters and therefore implies a risk of overestimation of the performance when using that same standard for validating the filters. For this reason also a lot of effort was put into the creation of other datasets that were used for validation....
Show More

We appreciate the comments of J.C. van der Wouden et al. They make a valid point regarding the use of the 'reference standard'. Part of this set was used for the development of the filters and therefore implies a risk of overestimation of the performance when using that same standard for validating the filters. For this reason also a lot of effort was put into the creation of other datasets that were used for validation. To get insight in a possible bias, we excluded in an additional analysis the articles from the reference set that were used for the development of the filters and recalculated the overall performance of these filters. The sensitive filter gets a sensitivity of 98,0% and a specificity of 75,0% and the specific filter gets a specificity of 99,0% and a sensitivity of 92,3%. This indicates that excluding articles from the reference standard does not change the performance of our filter in a relevant way.
Indeed a search with only MeSH terms will be very specific, as skilled indexers have read the article and have determined that the contents are indeed related to Family Medicine. However, it is not only the time lag (for indexing articles in Pubmed) that reduces the sensitivity of 'MeSH-only-filters', but also the failure of the database or indexers to assign thesaurus terms for the setting. Therefore, a sensitive filter like ours always consists of controlled vocabulary (MeSH terms) and free text words in order to create better performance.
We have not tested our filter in other review test sets as suggested by J.C. van der Wouden et al. However, to do this properly the reviewer's searches should be performed without the filter, screening the references manually for relevance on Primary Care, which is a very time consuming task. Indeed, in that case the validation of our filter would have been even more thoroughly.
Finally, creating either 100% specific or 100% sensitive filters is feasible, only sensitivity and specificity, respectively, will be poor as a consequence. The challenge when creating a search filter is creating a filter with an overall good performance and that was our goal when creating these filters.
Competing interests: None declared
Show Less

Competing Interests: None declared.
Published on: (3 August 2015)
Page navigation anchor for New search filters for articles most welcome
New search filters for articles most welcome
Johannes C. van der Wouden, Associate Professor
Other Contributors:
This paper is most welcome. Searching online databases for relevant literature is a task performed daily by many people, from time-restricted physicians to devoted systematic reviewers. The search filters developed by Pols and co-authors were validated in multiple ways, which increases the confidence in their ability to sort the wheat from the chaff.
A few remarks may help to place their findings in context. Th...
Show More

This paper is most welcome. Searching online databases for relevant literature is a task performed daily by many people, from time-restricted physicians to devoted systematic reviewers. The search filters developed by Pols and co-authors were validated in multiple ways, which increases the confidence in their ability to sort the wheat from the chaff.
A few remarks may help to place their findings in context. The test characteristics that were found in the so-called reference set may be overoptimistic, as this was the set that was used to iteratively develop the filters. It can be questioned whether the calculation of an overall average across the test sets should include the reference set.
In terms of specificity, some of the previously developed search filters perform better in the test sets. With a specificity of 99.7-99.9% a search strategy solely based on relevant Mesh terms is superior to all others (supplementary table 4). There is one caveat: the time lag between records being added to databases and the assignment of Mesh terms will result in 0% specificity for recently added records when using only Mesh terms.
Of course, sensitivity and specificity should both be taken into account. However, it is also important to have an idea of the "signal to noise" ratio. The search filter that finds all relevant studies may do so at the cost of having to screen thousands of irrelevant titles and abstracts. The example given in the discussion section looks promising, but it would be helpful to see similar data for other test sets.
A future challenge is to disseminate these findings to the primary care research community. The Cochrane Primary Health Care Field is an obvious stakeholder to take up this task. Perhaps the filters could also be added to the Clinical Queries set that PubMed offers.
Competing interests: None declared
Show Less

Competing Interests: None declared.