The process of gathering literature for a systematic review is expected to follow certain, stringent guidelines, which often rule out the use of modern literature tools like Litmaps.
However, no tool or method should be overlooked that can help find relevant papers! Here, we cover how you can use Litmaps for your systematic review while still following the formal best practices for reproducibility and transparency.
Jump to...
What is a systematic review?
A systematic review effectively summarizes the state of research on a particular topic using reproducible techniques. These reviews often relate various studies to one another, correcting for biases between them in order to draw conclusions about their combined results.
Due to the high volume of publications, it's practically impossible for any single researcher to keep abreast of all developments. For this reason, systematic reviews exist to coalesce and efficiently summarize the state of research. These reviews exist across all domains, and they are particularly important in the health and medical spaces where they impact clinical decisions.
Given the significant impact of these reviews, certain guidelines exist by which to evaluate their quality and reliability. AMSTAR (A Measurement Tool to Assess Systematic Reviews) developed in 2007 and since updated, provides clear guidelines to assess the methodological quality of systematic reviews. The evidence within reviews can be further evaluated for quality using other tools, like GRADE (Grading of Recommendations Assessment, Development and Evaluation).
Systematic review vs literature review
Systematic reviews differ from basic literature reviews in that they are reproducible in how they collect and interpret studies. Systematic reviews tend to focus on a more precise question than a literature review, which generally covers several aspects of a topic.
For example, this systematic review focuses on whether Kava, a medicinal plant, is an effective treatment for anxiety. This literature review on Kava, on the other hand, summarizes our current understandings of kava in the range of therapeutic applications it has.
According to AMSTAR, systematic reviews should rely on a methodical and reproducible literature search. For example, reviewers should record what databases, search parameters and filters were used to discover relevant publications. This is not necessarily the case for all literature reviews.
Collecting literature for a systematic review
One of the key characteristics of a systematic review is that the method of searching for and collecting papers is as important as the findings themselves. The method should be reproducible, meaning that researchers in the future may re-discover the same set of papers using original methods. Replication may be necessary to clarify methods used in the original review, especially those that were subjective choices. For example, future researchers may question what was originally used as inclusion criteria in the search.
According to best practices (AMSTAR guidelines), a comprehensive literature search should be performed. This means:
At least two electronic sources (or databases) are searched
Metadata about the search is recorded (database type and years)
Search parameters are recorded (keywords, or in the case of medical literature, MESH terms)
Search strategy is recorded
Search is supplemented by consulting a range of sources including experts in the field, textbooks, reviews, references of studies found, etc.
Notice that these guidelines imply that papers are predominantly collected directly from existing corpuses of literature by direct look-up. This will be important when we talk about how to use Litmaps to gather literature for your systematic review. Litmaps is a keyword-less search, meaning one can't report what keywords are used, because of the very nature of how Litmaps discovers papers.
However, it's also important to note that the underlying goals of these guidelines is not designed to limit or inhibit the researcher. Consult the last guideline, which recommends even consulting experts in the field. At its core, these guidelines insist on a comprehensive search, using the best resources available, while maintaining transparency and reproducibility of process.
How to use Litmaps for a systematic review?
Litmaps can aid in the search for relevant papers for a systematic review, as well as finding any papers overlooked in traditional search. Additionally, Litmaps Monitor can be used to stay up-to-date on the field, throughout the review process.
In order to use these tools for a systematic review it's important to 1) record the collection methodology, and 2) consult a secondary set of electronic sources. This is true no matter what tool you're using, based on the best practices across systematic reviews.
1. Use Litmaps to search for articles for your systematic review
Use Litmaps to explore and collect papers for your review. Here's how to search for articles in Litmaps. Remember, the more inputs to your Litmap you have, the more targeted and comprehensive your search results will be.
To make your work reproducible, record relevant information about your search process. For transparency, be sure to include these search parameters:
Input (Set of input articles in your Litmap)
Date of lookup
Filters used
However, keep in mind that Litmaps algorithms nor its source databases are static. Databases may be updated with new entries, or lose entries, over time. This is entirely normal – all traditional systematic review databases are updated over time as well. A known issue exists with the reproduction of systematic reviews based on the natural evolution of software and data storage. Nonetheless, by recording the search metadata and relevant information, you can provide the most reliable information for future researchers to reproduce your search.
2. Use Litmaps to find gaps in your systematic review
This is one of the best ways to use Litmaps to help support your systematic review. You can follow traditional review techniques of relying on database lookups. However, you can avoid missing relevant articles and enhance your coverage by using Litmaps.
Use Litmaps for your systematic review after you've collected all the relevant papers for your research topic. Input these into a new Litmap to find related papers. Cross-check the newly recommended papers against those in your existing collection to see if there are any relevant studies you may have overlooked.
One of the key pitfalls of traditional literature searches is the reliance on keywords. By searching for connected papers without keywords, Litmaps can provide a more diverse set of paper suggestions. Litmaps may suggest papers where the names of substances or topics are different from what you've been searching.
3. Use Litmaps Monitor to stay up-to-date
All reviews, whether traditional literature reviews or comprehensive systematic reviews, suffer from one main issue: they quickly fall out-of-date.
Since reviews themselves take months to complete, new research may come out while you're still working on your paper. To avoid missing anything relevant, use Litmaps Monitor. Each week when the Litmaps database is updated with 100,000s of newly published papers, Monitor will re-run your Litmaps Search to see if any new papers come out that connect to your topic. If they do, you'll get an email letting you know.
Why use Litmaps for a systematic review?
Systematic reviews are expected to comprehensively cover the literature on a certain topic. The key advantage of using Litmaps as a part of the discovery process is that it may reveal papers that wouldn't come up using traditional search techniques.
Because Litmaps uses keyword-less search, it’s able to find papers that wouldn’t necessarily show up under a direct keyword search. This provides an advantage, because otherwise these papers would go unreviewed, skewing the final results.
How not to use Litmaps for a systematic review
Although Litmaps can help discover papers and fill gaps in research, it shouldn’t be used alone to complete a systematic review. A systematic review involves methodically searching an entire domain for any relevant publications. Thus, any single database or tool is insufficient, and instead some combination should be used.
AMSTAR-defined best practices for systematic reviews involve using at least "two electronic sources". This means researchers are expected to pull from multiple large-scale databases. For example, in the medical and health space, certain databases are considered to be the most important sources to search for reports of clinical trials. According to The Cochrane Handbook for Systematic Reviews of Interventions these are CENTRAL, MEDLINE / PubMed and Embase.
Although a variety of tools may be used to conduct the search and discover papers, the reproducibility of systematic search comes from using databases directly. So, remember to follow best practices when searching these databases. Record metadata about your search such as filters and dates of retrieval.