IJD® MODULE ON BIOSTATISTICS AND RESEARCH METHODOLOGY FOR THE DERMATOLOGIST - MODULE EDITOR: SAUMYA PANDA
|Year : 2016 | Volume
| Issue : 6 | Page : 602-607
|Methodology series module 6: Systematic reviews and meta-analysis
Maninder Singh Setia
Epidemiologist, MGM Institute of Health Sciences, Navi Mumbai, Maharashtra, India
|Date of Web Publication||9-Nov-2016|
Maninder Singh Setia
MGM Institute of Health Sciences, Navi Mumbai, Maharashtra
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Systematic reviews and meta-analysis have become an important of biomedical literature, and they provide the "highest level of evidence" for various clinical questions. There are a lot of studies - sometimes with contradictory conclusions - on a particular topic in literature. Hence, as a clinician, which results will you believe? What will you tell your patient? Which drug is better? A systematic review or a meta-analysis may help us answer these questions. In addition, it may also help us understand the quality of the articles in literature or the type of studies that have been conducted and published (example, randomized trials or observational studies). The first step it to identify a research question for systematic review or meta-analysis. The next step is to identify the articles that will be included in the study. This will be done by searching various databases; it is important that the researcher should search for articles in more than one database. It will also be useful to form a group of researchers and statisticians that have expertise in conducting systematic reviews and meta-analysis before initiating them. We strongly encourage the readers to register their proposed review/meta-analysis with PROSPERO. Finally, these studies should be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis checklist.
Keywords: Database, meta-analysis, systematic reviews
|How to cite this article:|
Setia MS. Methodology series module 6: Systematic reviews and meta-analysis. Indian J Dermatol 2016;61:602-7
| Introduction|| |
Systematic reviews and meta-analysis have become an important of biomedical literature, and they provide the "highest level of evidence" for various clinical questions. Thus, it is important to understand the key components of conducting a systematic review or meta-analysis (though it may highlight over here that statistical methods are advanced and require training/consultation). The dictionary of epidemiology has defined a systematic review as "A review of the scientific evidence which applies strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on the specific topic" (Porta 2014). The dictionary has defined meta-analysis as "statistical analysis of results from separate studies, examining sources of differences in results among studies, and leading to a quantitative summary of the results if the results are judged sufficiently similar or consistent to support such synthesis" (Porta 2014). Thus, from the above definitions, it can be stated that when statistical methods are used to combine the findings of studies that have been identified and included in a systematic review, it is a meta-analysis.
What is the need for systematic reviews and meta-analysis?
Sometimes, review articles are published in journals. For example, the September--October 2016 issue of the Indian Journal of Dermatology published two review articles (Ghosh 2016, Pradhan 2016). These two manuscripts may be considered as narrative reviews. These provide the latest information on the topic, usually written by authors who are considered to be experts in the area and are often commissioned by the editor. However, these are usually not done systematically, and thus the information in these reviews may not represent the available literature for the topic. Systematic reviews, however, as the name suggest follow a system and a rigid methodology - designed to reduce the bias of narrative reviews. A systematic review and meta-analysis may be initiated by an individual researcher and submitted to journal for publication.
There are a lot of studies - sometimes with contradictory conclusions - on a particular topic in literature. For example, some studies may show that drug A is better than drug B in the management of psoriasis, whereas others may report that drug B is better than drug A. Some other studies may demonstrate that there is no significant difference between drug A and B. Hence, as a clinician, which results will you believe? What will you tell your patient? Which drug is better? A systematic review or a meta-analysis may help us answer these questions. In addition, it may also help us understand the quality of the articles in literature or the type of studies that have been conducted and published (example, randomized trials or observational studies).
We will discuss some aspects of systematic reviews and meta-analysis in this article.
| Examples of Systematic Reviews and Meta-analysis|| |
Bacillus Calmette-Guérin and leprosy (Setia 2005)
The authors did a meta-analysis to understand the role of Bacillus Calmette-Guérin (BCG) in prevention of leprosy. They reviewed studies (experimental and observational) till February 2005 and statistically combined the statistical results from each study to present a final protective effect. They combined results from seven experimental studies and found that the protective effect of BCG was 26% (95% confidence intervals [CIs]: 14%-37%). Furthermore, the authors also combined 19 observation studies (five cohort studies and 14 case-control studies) and found that the protective effect of BCG was 61% (95% CI: 51%-70%). Thus, one of their conclusions was that the observational studies had overestimated the protective effect of BCG. They also found that an additional dose of BCG was more protective compared with a single dose. Thus, they also suggested that an additional dose of BCG should be considered for contacts of leprosy patients.
Interventions for melasma (Rajaratnam 2010)
The authors included 20 studies to compare 23 different interventions used for melasma. They have reported that due to the heterogeneity of studies, statistical pooling was not possible. They found that triple-combination cream was significantly more effective compared with a single drug or two drugs. Furthermore, they also found that even though azelaic acid was more effective when compared with 2% hydroquinone, it was less effective compared with 4% hydroquinone. An important conclusion from this study was that the quality of studies was poor and high-quality randomized controlled trials are needed.
Rifampicin, ofloxacin, and minocycline and leprosy (Setia 2011)
The authors performed a systematic review and meta-analysis to compare the effect of rifampicin, ofloxacin, and minocycline (ROM) and multidrug therapy in the treatment of leprosy. They reviewed six studies that compared ROM therapy with multidrug therapy. They found that single-dose ROM therapy was less effective compared with multidrug therapy in the management of paucibacillary leprosy. However, they found that only two studies had assessed the role of multiple dose ROM therapy in multibacillary leprosy. Thus, they concluded that more studies on the role of ROM therapy in multibacillary leprosy patients were necessary.
As seen in the above three examples, the conclusions were not only about the formal estimate or the protective role but also about the quality of studies. For example, in the first study, one of the conclusions was that observational studies had overestimated the protective effect of BCG. In the second study, the authors concluded that high-quality randomized trials are needed to compare the interventions in melasma. Finally, in the third study, one of the conclusions was that more trials of ROM therapy in multibacillary leprosy patients may be warranted to come to some useful conclusions about the role of ROM in leprosy - particularly multibacillary leprosy.
| Selection of a Research Question for Systematic Reviews and Meta-analysis|| |
The first step it to identify a research question for systematic review or meta-analysis. For example, in the above-mentioned example on role of ROM in leprosy, the population (P) was patients with leprosy (paucibacillary or multibacillary), intervention (I) was ROM therapy, comparison or control group (C) was multidrug therapy, and the outcome (O) was clinical improvement. Thus, this research question may be framed based on the PICO format.
After the research question has been identified, the researcher should perform an extensive search of the databases (discussed in later sections). If there is an existing review or meta-analysis of the topic that one would like to work on, kindly read the whole manuscript before initiating the study. If there are no new studies or just a couple of studies that have been published after this review or meta-analysis, it may be worthwhile to wait for some more literature before initiating a new systematic review or meta-analysis on the topic.
The researcher may also want to search PROSPERO (http://www.crd.york.ac.uk/PROSPERO/). In this website, prospective systematic reviews and meta-analysis are registered by researchers. Thus, it may be useful to find out if any other group is conducting a systematic review and meta-analysis on the same subject.
| Search the Database|| |
The next step is to identify the articles that will be included in the study. This will be done by searching various databases. It is important that the researcher should search for articles in more than one database. We have listed some common databases that may be used for literature search.
- PubMed: It is a common database used for search for biomedical research (http://www.pubmed.gov/)
- EMBASE: This is another important database for biomedical research. However, one needs to register to access resources from this database. It includes literature from peer-reviewed journals, in-press publications, and conferences (https://www.elsevier.com/solutions/embase-biomedical-research)
- Cochrane Database: This is a huge repository of systematic reviews and meta-analysis. The reviews and meta-analysis in this database are usually not limited to 4000-5000 words as required by some published journals.(http://www.cochranelibrary.com/)
- LILACS: It is an important database of scientific and technical literature of Latin America and the Caribbean region. However, many of the articles in this database may be in Spanish or Portuguese (http://lilacs.bvsalud.org/en/)
- Although these are some of the common databases used for biomedical literature, the researchers should also be aware of some other databases that may also be useful
- PsycINFO: It is useful database of documents (journal articles, book chapters, reviews, and editorials) on psychology, behavioral and social sciences, and related interdisciplinary articles (http://www.apa.org/pubs/databases/psycinfo/index.aspx)
- CINAHL: It is a useful database for searching nursing articles and related allied health literature (https://www.ebscohost.com/nursing/products/cinahl-databases/cinahl-complete).
| Identify the Articles|| |
After the databases have been selected, the next step is to use specific search terms. These terms should be similar across all the databases. These terms should be mentioned in the manuscript clearly. In the above-mentioned example on BCG and Leprosy (Setia 2005), the authors used the terms "BCG and Leprosy," "Leprosy vaccines," "BCG and Hansen's disease," and "BCG and hanseniasis." Kindly note, in this data search, the authors have used different terms for the same disease. This is particularly important in dermatology where the same condition can be identified by different terms. Another common example is tuberculosis - the researcher can use term "Tuberculosis" or "Koch's" or "TB."
It is also important to identify the time period of search. Most of the databases will allow the researcher to set up time frame. In PubMed, the researcher can custom the dates for search of potential articles (set up a start date as well as an end date). In any case, it is very important to state the end point of the search. In the above example (meta-analysis on ROM and leprosy), the authors have mentioned in the manuscript that they included articles up to October 2010.
Sometimes, relevant articles may not be captured after any electronic search. They may be identified from the reference list of the articles that have been included or reviewed for the meta-analysis. If these articles are included in the systematic review, it should be explicitly mentioned in the methods section.
It is quite likely that some researchers may want to include randomized clinical trials or observational studies that have not yet been published but have been presented in conferences. Some researchers argue that it is important to include these studies to ensure complete literature search. Others, however, argue that data presented in conferences are often preliminary analysis and the final results in the published articles may differ from those presented at the conference.
It is important to keep a record of all the abstracts/articles that have been identified from each database. After the initial search, some abstracts/articles may be common to many databases - these are called duplicate articles. The researcher has to report the number of the duplicate articles that were identified in each database.
Example: The researcher has identified 30 articles in PubMed and 31 articles in EMBASE using the same search terms (example: "Psoriasis" and "interventions"). However, of the 31 articles identified in EMBASE, 10 were already present in the PubMed search.
Thus, the final search result should be:
| Inclusion of Articles for the Systematic Review or Meta-analysis|| |
After identifying the abstracts, the next step to identify the complete studies/articles that will be included for the current systematic review or meta-analysis. As with other protocols, the inclusion and exclusion criteria have to be decided "a priori" and cannot be an ad hoc judgment. Some authors may decide to include only trials for the meta-analysis, whereas others may decide to include trials and observational studies (such a cohort or case-control studies) for the meta-analysis.
In the above-mentioned example - BCG and Leprosy - the authors excluded any studies that described the protective effect of BCG along with other therapies (such as other vaccines or medication in the same patient). In the other example - ROM and leprosy - the authors included human studies and excluded any study that assessed only the immunological changes after ROM therapy. The researcher may exclude some studies from the formal systematic review or meta-analysis. However, these studies should be read and reviewed for any additional information on the subject, description of potential reasons and hypothesis, and/or identification of additional studies that may not have been retrieved by the electronic search.
What if some of the abstracts are not in English? Will you include these articles?
The researchers have to decide "a priori" on the inclusion of non-English articles. If they decide to include only English language articles, then the review may not be complete. This may be a potential limitation of the study. If a large number of studies are not in English, then it may be important to include non-English Language articles.
However, the consequences of including non-English articles in the review should be weighed carefully. How many languages will be considered? It may be important to remember that many European studies, Korean studies, Japanese studies, and Chinese studies are published in the local language. These articles are usually accompanied by an English abstract. However, the manuscript will have to be translated for data abstraction.
Example - "psoriasis and interventions."
This is a hypothetical example:
| Data Abstraction|| |
Once the articles have been identified, the next step is to abstract the data from these articles. Data should be abstracted on a preformed sheet. It may be considered the equivalent of "Clinical Record Form." The data abstraction sheet may include the following information: Title of the manuscript, authors, year of publication, country of study, study population, inclusion criteria/exclusion criteria for individual study as described by the authors, randomization procedure (if described as randomized trial), the blinding procedures (if described as a blinded trial), intervention (dose, frequency, etc.), control (present/absent, dose, frequency, etc.), outcomes including the point estimates, subgroup estimates (if included in the study), and limitations. Additional data may be abstracted as per the requirement of the meta-analysis.
It is important to identify the individuals who will abstract the data. Usually, the primary author and another author should be responsible for independently abstracting data from the manuscripts. The names of these authors (as initials) should be mentioned in the meta-analysis. After these two authors have abstracted the data independently, they should discuss the findings. If there is a disagreement in the findings, then they can be resolved by consulting a senior author (who also should be identified in the manuscript).
Quality of trials
It is important to mention about the quality of trials that have been reviewed for the present meta-analysis. There are multiple scales for assessing the quality of trials. Some of the common scales used for scoring the trials are - Jadad scale, quality criteria from the Cochrane Back Review Group, scales based on Delphi methods. These scales score the trials based on the reporting by the authors (Was the study randomized or not? If randomized, was it appropriate? Was the study blinded or not? If the study was blinded, was it appropriate? What was the type of analysis? Was there discussion about drop outs?).
Sometimes, the researcher may require more data than that have been presented in the manuscript. In this situation, kindly write to the authors directly and request additional findings. Of course, it will be important to introduce the subject and the meta-analysis. Many authors respond to these queries. Some will provide additional data and others may not. In any case, if the authors have been contacted for additional data, it should be explicitly stated in the manuscript. The outcomes of these requests (data received, no response, etc.) should also be documented in the methods section.
| Statistical Analysis|| |
The statistical methods used for meta-analysis are complex and require extensive skills. Thus, it is important to include a statistician (who has experience using these methods) in the group. Some of these methods are "Fixed effects models" and "Random effects models." These are different statistical methods for pooling the data that have been abstracted from the manuscripts. It is also important to assess the statistical heterogeneity in these studies. If the studies are not heterogeneous, then "Fixed effects models" may be used. However, if there is statistical heterogeneity, then "Random effects models" may be appropriate. If the researchers include randomized trials and observational studies in the meta-analysis, it will be prudent to pool the data from these studies separately. Sometimes, the researchers may pool only prevalence data and not the odds ratios or relative risks.
This is as special type of graph used in meta-analysis. In this graph, the estimates and the CIs from individuals and the pooled estimate with the CIs are shown.
This is a hypothetical example of a forest plot:
The estimates and the CIs for each study are presented in the plot. The size of the box represents the "weight" of the study in the meta-analysis. Thus, study number four, seven, and nine have received the highest weights in the meta-analysis. The first study has received the lowest weight in the meta-analysis. The diamond at the bottom represents the "pooled estimate" (risk ratio: 1.04) and the two ends of the diamond represent the CIs (1.00-1.08).
The researcher should also assess the publication bias in a meta-analysis. It is likely that direction or significance of results may influence the publication of studies in literature - this is "Publication Bias." Some of the statistical methods to assess this bias are Begg's test and Egger's test. It can also be assessed visually by a funnel plot.
This is a hypothetical example of funnel plot:
The researcher has noted the asymmetry in the funnel plot. The studies appear to be present on either side of the central line. The statistical test (Begg's test) found a P = 0.066 (i.e., not significant). Thus, there is little evidence of publication bias.
| Reporting of Systematic Reviews and Meta-analysis|| |
As with other types of studies, guidelines have been developed for reporting of systematic reviews and meta-analysis. These guidelines are called Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) (http://prisma-statement.org/). We strongly encourage the readers to go through these guidelines.
Some of the components of this checklist are title, rationale, objectives, search, study selection, data collection process (data abstraction sheets), synthesis of results, summary of evidence, limitations, conclusions, and source of funding.
| Summary|| |
Systematic reviews and meta-analysis have become an important component of biomedical literature. It will be useful to form a group of researchers and statisticians that have expertise in conducting systematic reviews and meta-analysis before initiating them. We strongly encourage the readers to register their review/meta-analysis with PROSPERO. Finally, these studies should be reported according to the PRISMA checklist.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| Bibliography|| |
- Berger VW, Alperson SY. (2009). A general framework for the evaluation of clinical trial quality. Rev Recent Clin Trials 4, 79-88.
- Evidence Based Practice Toolkit. http://libguides.winona.edu/c.php?g=11614&p=61584.
- Ghosh A, Coondoo A. (2016). Lichen planus pigmentosus: The controversial consensus. Indian J Dermatol 61, 482-486.
- Greenland S. (1987). Quantitative methods in the review of epidemiologic literature. Epidemiol Rev 9, 1-30.
- Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. (1996). Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17, 1-12.
- KJ Rothman, S Greenland, TL Lash. (2008). Modern Epidemiology (Third ed.). Philadelphia, USA: Lippincott Williams & Wilkins.
- M Egger, GD Smith, DG Altman (Eds.). (2001). Systematic Reviews in Health Care: Meta-analysis in Context. London, UK: BMJ Publishing Group.
- M Porta (Ed). (2014). A Dictionary of Epidemiology (Sixth ed.): International Epidemiological Association, Oxford University Press, New York, USA.
- Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. (2008). Scales to assess the quality of randomized controlled trials: A systematic review. Phys Ther 88, 156-175.
- Pradhan S, Madke B, Kabra P, Singh AL. (2016). Anti-inflammatory and immunomodulatory effects of antibiotics and their use in dermatology. Indian J Dermatol 61, 469-481.
- DB Petitti. (2000). Meta-analysis, Decision Analysis, and Cost-effective Analysis. New York, NY, USA: Oxford University Press.
- PRISMA http://prisma-statement.org/documents/PRISMA%202009%20checklist.pdf
- Rajaratnam R, Halpern J, Salim A, Emmett C. (2010). Interventions for Melasma. Cochrane Library.
- Setia MS, Steinmaus C, Ho CS, Rutherford GW. (2006). The role of BCG in prevention of leprosy: A meta-analysis. Lancet Infect Dis, 6, 162-170.
- Setia MS, Brassard P, Jerajani HR, Bharat S, Gogate A, Kumta S, et al. (2008). Men who have sex with men in India: A systematic review of the literature. J LGBT Health Res, 4, 51-70.
- Setia MS, Shinde SS, Jerajani HR, Boivin JF. (2011). Is there a role for rifampicin, ofloxacin and minocycline (ROM) therapy in the treatment of leprosy? Systematic review and meta-analysis. Trop Med Int Health, 16, 1541-1551.
- Shams T, Setia MS, Hemmings R, McCusker J, Sewitch M, Ciampi A. (2010). Efficacy of black cohosh-containing preparations on menopausal symptoms: A meta-analysis. Altern Ther Health Med, 16, 36-44.
- Sjogren P, Halling A. (2002). Quality of reporting randomised clinical trials in dental and medical research. Br Dent J, 192, 100-103.
- Slim K, Nini E, Forestier D, Kwiatkowski, Panis Y, Chipponi J. (2003). Methodological index for non-randomized studies (minors): Development and validation of a new instrument. ANZ J Surg, 73, 712-716.
- JAC Sterne. (Ed). (2009). Meta-analysis in Stata: An Updated Collection from the Stata Journal. College Station, Texas, USA: Stata Press Publication.
| Article Access Statistics|
| Viewed||2304 |
| Printed||69 |
| Emailed||0 |
| PDF Downloaded||224 |
| Comments ||[Add] |