Document Detail

Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.
Jump to Full Text
MedLine Citation:
PMID:  24558353     Owner:  NLM     Status:  In-Data-Review    
Julian Elliott and colleagues discuss how the current inability to keep systematic reviews up-to-date hampers the translation of knowledge into action. They propose living systematic reviews as a contribution to evidence synthesis to enhance the accuracy and utility of health evidence.
Julian H Elliott; Tari Turner; Ornella Clavisi; James Thomas; Julian P T Higgins; Chris Mavergames; Russell L Gruen
Related Documents :
23100963 - Recombinant factor viia: use in fatal post partum hemorrhage - indian experience case s...
25378853 - Endobronchial ultrasound guided needle aspiration of a paraspinal mass with prior faile...
7894103 - Malignant hidroacanthoma simplex. a case report and literature review.
Publication Detail:
Type:  Journal Article     Date:  2014-02-18
Journal Detail:
Title:  PLoS medicine     Volume:  11     ISSN:  1549-1676     ISO Abbreviation:  PLoS Med.     Publication Date:  2014 Feb 
Date Detail:
Created Date:  2014-02-21     Completed Date:  -     Revised Date:  -    
Medline Journal Info:
Nlm Unique ID:  101231360     Medline TA:  PLoS Med     Country:  United States    
Other Details:
Languages:  eng     Pagination:  e1001603     Citation Subset:  IM    
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): PLoS Med
Journal ID (iso-abbrev): PLoS Med
Journal ID (publisher-id): PLoS
Journal ID (pmc): plosmed
ISSN: 1549-1277
ISSN: 1549-1676
Publisher: Public Library of Science, San Francisco, USA
Article Information
Download PDF
Copyright: 2014 Elliott et al
collection publication date: Month: 2 Year: 2014
Electronic publication date: Day: 18 Month: 2 Year: 2014
Volume: 11 Issue: 2
E-location ID: e1001603
PubMed Id: 24558353
ID: 3928029
Publisher Id: PMEDICINE-D-13-02601
DOI: 10.1371/journal.pmed.1001603

Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Julian H. Elliott12*
Tari Turner23
Ornella Clavisi4
James Thomas5
Julian P. T. Higgins67
Chris Mavergames8
Russell L. Gruen49
1Department of Infectious Diseases, Alfred Hospital and Monash University, Melbourne, Australia
2School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
3World Vision Australia, Melbourne, Australia
4National Trauma Research Institute, Alfred Hospital, Melbourne, Australia
5EPPI-Centre, Institute of Education, University of London, London, England
6School of Social and Community Medicine, University of Bristol, Bristol, England
7Centre for Reviews and Dissemination, University of York, York, England
8Informatics and Knowledge Management Department, The Cochrane Collaboration, Freiburg, Germany
9Department of Surgery, Monash University, Melbourne, Australia
Correspondence: * E-mail:
[conflict] All authors are contributors to the Cochrane Collaboration. JHE is a Senior Research Fellow at the Australasian Cochrane Centre, which supports the conduct and use of systematic reviews. The views expressed in this paper are JHE's own and not necessarily those of the Australasian Cochrane Centre or the Cochrane Collaboration. TT has an unpaid adjunct position as a Senior Research Fellow at the Australasian Cochrane Centre, which supports the conduct and use of systematic reviews. The views expressed in this paper are TT's own and not necessarily those of the Australasian Cochrane Centre or the Cochrane Collaboration. CM is an Academic Editor for PLOS ONE and is a full-time employee of The Cochrane Collaboration. JHE and RG co-lead the development of a systematic review workflow management tool (Covidence). Covidence has been developed with funding from competitive government grants and is provided as a not-for-profit service to the systematic review community.
Contributed by footnote: Wrote the first draft of the manuscript: JHE. Contributed to the writing of the manuscript: JHE TT OC JT JH CM RLG. ICMJE criteria for authorship read and met: JHE TT OC JT JH CM RLG. Agree with manuscript results and conclusions: JHE TT OC JT JH CM RLG.
The Policy Forum allows health policy makers around the world to discuss challenges and opportunities for improving health care in their societies.

  • The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action.
  • Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed.
  • We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence.
  • Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication.
  • Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.

The Bridge from Evidence to Practice

Health research promises societal benefit by making better health possible. However, there has always been a gap between research findings (what is known) and health care practice (what is done), described as the “evidence-practice” or “know-do” gap [1]. The reasons for this gap are complex [2], but it is clear that synthesising the complex, incomplete, and at times conflicting findings of biomedical research into forms that can readily inform health decision making is an essential component of the bridge from “knowing” to “doing.”

Systematic reviews (SRs) and meta-analyses have provided incalculable benefit for human health by contributing to the bridge from knowing to doing, but this benefit is limited by characteristics of the current SR enterprise [3]. The methods of SR and meta-analysis are well developed [3][5], but less progress has been achieved with the other essential component of accuracy—currency. The time from the date of the last search to SR publication is commonly over a year [6], and in an analysis of the time taken for primary study results to be incorporated into an SR, the median time from primary study publication to SR publication ranged from 2.5 to 6.5 years (Figure 1) [7]. Once published, only a minority of reviews are updated within 2 years of publication [8], and this inability to maintain currency leads to significant inaccuracy. By 2 years post-publication 23% of SRs that have not been updated will have failed to incorporate new evidence that would substantively change conclusions about the effectiveness or harms of therapies [9].

Current approaches to updating SRs focus on detecting SRs most in need of updating [10],[11]. While these methods contribute to the currency of SRs, they cannot adequately reduce the inaccuracy caused by out-of-date SRs. It is often difficult to assemble an authorship team to complete prioritised updates; publication of updates takes many months, during which time the SR remains out of date and therefore potentially inaccurate; and many SRs are not designated as high priority and therefore remain out of date and inaccurate. Despite the availability of these methods and the concerted efforts of many individuals, much of the global corpus of SRs remains out of date.

Incremental advances in traditional SR updating is unlikely to lead to substantial and sustained improvements in the currency of SRs in the context of exponential growth in primary research and SRs [12]. In order to address this considerable source of SR inaccuracy, and produce evidence summaries that are both methodologically rigorous and up to date, a new approach is needed. This innovation has previously been unachievable because rigorous reviews are demanding of time and resources, and “up to date” required rapid processes, which limited the feasibility of rigorous methods. New technologies create the opportunity to resolve this trade-off and enable answers to health questions that are both methodologically rigorous and up to date. We have named this new approach to the updating of SR “living systematic review.” Our aim in proposing this approach is to address the challenges facing contemporary evidence synthesis, while retaining the strengths of SR that have been a critical underpinning of knowledge for health over the last quarter century. This approach to SR adds to rather than replaces existing methods and is particularly relevant for reviews of controlled trials in fast moving topic areas.

Living Systematic Reviews

Living systematic reviews are high quality, up-to-date online summaries of health research that are updated as new research becomes available, consistent with the vision of the pioneering Oxford Database of Perinatal Trials to “include a library of trial overviews, which will be updated when new data become available” [13]. There are four fundamental differences between conventional SR and living systematic review: publication format, work processes, author team management, and statistical methods.

The essential difference between living systematic review and conventional SR is the publication format. Instead of a conventional static SR report or update, living systematic reviews are dynamic, persistent, online-only evidence summaries, which are updated rapidly and frequently. The corollary of a living publication is three changes to SR production. First, work processes must be adapted. A literature search strategy is maintained and outputs fed continuously into an SR workflow, including continuous updating of identified studies, assessment of study quality, data extraction, meta-analysis, and SR report. Instead of the intense, sporadic effort of conventional SRs and SR updates, living systematic review requires moderate, ongoing contribution. Second, author team management must be responsive to a continuous workflow, coordinating effort over long periods of time and allowing evolution in the author team, while maintaining institutional memory.

Third, updating meta-analyses entails a reanalysis of data and, as with repeated analysis of accumulating primary trial data, an inflated rate of false-positive findings is likely if statistical tests are repeated naively [14]. Also, effect size estimates can be unstable, especially in the early stages of evidence generation [15]. These issues are relevant to all updates to meta-analyses, but are particularly important for living systematic reviews given the potential frequency of updates. Statistical monitoring of meta-analyses using formal sequential methods can control the risk of spurious findings, while achieving pre-specified power to identify a pre-specified clinically relevant magnitude of effect [16],[17]. Sequential methods are controversial in meta-analysis since they are based on testing rather than estimation, and an estimate that is widely disseminated on the basis of a significance test result may be biased. As an alternative, a Bayesian approach provides a natural framework for monitoring accumulating evidence in which prior distributions can be used to reduce the probability of falsely concluding superiority of an intervention in the early stages of the review [18], and to stabilize the meta-analysis by exploiting external information about the likely degree of statistical heterogeneity across the studies [19]. Furthermore, a Bayesian framework feeds naturally into decision making.

Production of Living Systematic Reviews

In addition to the three essential changes to review production described above, a key enabler of living systematic review is SR production efficiency. Improvements in efficiency can have profound effects on the application of health technologies. For example, a 4-log decrease in the cost of sequencing a human genome, from US$100 million to US$10,000 over 10 years [20], has catalysed a genomics revolution with profound benefits for health. In contrast, rising methodological expectations have led to an increase in the complexity and cost of SR and production timelines often in excess of 1–2 years [21]. We describe below several recent developments that have the potential to improve dramatically the efficiency of conventional SR and enable the widespread production of living systematic reviews.

Workflow and Collaboration Tools

Despite the availability of some specific tools, the efforts of most SR authors are fragmented across generic word processing, spreadsheet, email, reference management, and statistical analysis tools [22],[23]. This fragmentation hampers the production and updating of SRs, undermines the experience and engagement of SR authors, and limits the availability of process data. Growing innovation in tools and platforms [22] will enable more efficient SR production, but the right incentives and partnerships need to be in place for these innovations to translate into broadly available applications [24].


Text-mining technologies are currently being developed to improve the efficiency of SR production [25]. While experimental work encompasses many stages of the review process the most refined techniques are currently concerned with study identification [26]. Here, machine learning technologies have the potential to reduce the manual “screening” of titles and abstracts by up to 50% in new reviews [27] and more than 90% in review updates [28], greatly enhancing the efficiency of review production. Other initiatives are developing semi-automation technologies to assist with the development of search strategies, assessment of study quality, extraction of data from documents, and production of SR protocols and reports.

Data Repositories and Linked Data

Important health care questions are often the subject of evidence syntheses by multiple independent groups isolated from each other in redundant effort. The value embedded in the process and output of these parallel activities is only partially captured in discrete, static, and unstructured document-based outputs. Efforts to encourage registration of SRs can help minimise unnecessary duplication [29]. Initiatives that enable SR process and output data to be prospectively stored and reused by others are important developments that will reduce redundant effort [30],[31]. When these data are stored in structured formats using shared ontologies [32],[33], following W3C formats for linked data (RDF, OWL), unnecessary duplication can be avoided, but opportunities will also arise to draw from and contribute to the rapidly expanding world of linked open data.

Participation and the Crowd

Despite the fact that SRs are important, resource intensive, and time critical, most SRs are conducted by small academic teams, working part-time over many months. Larger authorship groups increase the efficiency of SR production [34] and the expertise available to each review group [3], but remain underutilised, particularly for high priority questions in which both the demand for evidence and engagement are high. In clinical and laboratory research, high priority questions are often addressed by collaborations of dozens or hundreds of researchers working together, but similar undertakings do not currently exist in evidence synthesis. Increasing the involvement of end users in SR production improves the outputs of SR [35][37], and “citizen science” approaches in which citation screening is crowd-sourced from a network of non-expert contributors have also been tested [38]. Efforts to identify smaller units of scholarly effort for dissemination and attribution [39][41] may be applicable to SR and encourage broader participation. These approaches to participation need to be evaluated in comparison to conventional approaches and to manage the risk of bias associated with contributors' potential competing interests.

Publication of Living Systematic Reviews

The shift to a persistent, dynamic online-only publication format will be enabled by at least two other developments. First is efficient peer and editorial review of a living document. When an ongoing search strategy identifies no new studies for inclusion, the review can be updated with the date of last search without further review. When new studies or data are identified for inclusion, but these make negligible difference to summary estimates and have no effect on review conclusions, these data can be incorporated into the review with a modest form of review (e.g., editorial only). If new studies or data are identified that result in significant changes to summary estimates or the review's conclusions, these should undergo rapid, but nonetheless robust, peer and editorial review. In the latter situations post-publication peer review can contribute to the accuracy of published reviews.

Second, living systematic reviews should be compatible with the norms of scholarly communication. Attribution of contribution to the living publication can follow existing norms, such as ICMJE criteria. Once authors' contributions no longer fulfil criteria for authorship, they can be removed from the author list and acknowledged as former contributors. Citation can also follow existing practices, including version number/date and date accessed. Finally, current practice can also inform listing in bibliographic databases with minor updates appended to an existing entry and major updates listed as a new publication.

Living Systematic Review and a New Evidence Ecosystem

Living systematic review contributes to the translation of knowledge into practice, primarily because of the contribution currency makes to accuracy and utility, but this approach to evidence synthesis also supports, and is enhanced by, associated upstream and downstream innovations.

In the growing deluge of research the noble science of systematic review resembles archeology: academic teams searching for buried artifacts and working tirelessly to reveal their true meaning. The growth in primary research and availability of diverse research outputs—protocols, trial registration, clinical study reports, and individual patient data—will continue to challenge current SR models. New methods are needed to identify datasets relevant to specific health questions and enable synthesis and insight. For example, annotation of research outputs with richer meta-data will increase the efficiency of review-specific search and screening. More importantly, publication and verification of research outputs in structured forms (e.g., using semantic technologies) will transform review-specific quality assessment and data extraction [42]. Living systematic review, together with these upstream innovations, will ensure that the potential rich insights from large datasets such as clinical study reports, individual patient data, and health system “big data” are made available for health decision making in a rigorous, efficient, and timely manner.

Living systematic review also enhances the efficiency and opportunities for knowledge translation. First, living systematic review enables “living knowledge translation,” including living guidelines, standards, policies, and decision support systems. Second, the value inside the “container” of an SR can be unlocked when the associated data are made available, particularly as open access to linked data formats. The availability of these data opens up opportunities for integration with guideline development platforms and clinical decision support systems [43] to create a new evidence “ecosystem” (Figure 2).


An essential link between health research and societal benefit is the transformation of millions of published research studies into accurate and usable summaries for health decision making. Many individuals and organisations are committed to this task and their efforts have improved the health of our societies, but increasing demand for accurate evidence—methodologically rigorous and up to date—is not being met. We propose living systematic review as a contribution to the methods of evidence synthesis that addresses these challenges by combining currency with rigour to enhance the accuracy and utility of health evidence. Living systematic review involves modifications to review production and publication, enabled by improved production efficiency and adherence to the norms of scholarly communication. The approach is widely applicable and although challenges remain (Table 1), feasible responses to these challenges exist. Together with emerging innovations in the reporting of primary research and in the creation and use of evidence in health systems, living systematic review contributes to a new evidence ecosystem in which health knowledge and practice are efficiently and rigorously entwined.


Provenance: Not commissioned; externally peer reviewed.

1. (2005) Bridging the “know-do” gap: meeting on knowledge translation in global health. Geneva: World Health Organization.
2. Grimshaw JM,, Eccles MP,, Lavis JN,, Hill SJ,, Squires JE, (Year: 2012) Knowledge translation of research findings. Implement Sci7: 5022651257
3. Institute of Medicine (2011) Finding what works in health care: standards for systematic reviews. Washington (D.C.): National Academies Press.
4. Moher D,, Liberati A,, Tetzlaff J,, Altman DG, (Year: 2009) Group P (Year: 2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med6: e100009719621072
5. Methodological Expectations of Cochrane Intervention Reviews (MECIR) (2013) Available: Accessed 8 November, 2013.
6. Sampson M,, Shojania KG,, Garritty C,, Horsley T,, Ocampo M,, et al. (Year: 2008) Systematic reviews can be produced and published faster. J Clin Epidemiol61: 531–53618471656
7. Bragge P,, Clavisi O,, Turner T,, Tavender E,, Collie A,, et al. (Year: 2011) The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol11: 9221682870
8. Jadad AR,, Cook DJ,, Jones A,, Klassen TP,, Tugwell P,, et al. (Year: 1998) Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA280: 278–2809676681
9. Shojania KG,, Sampson M,, Ansari MT,, Ji J,, Doucette S,, et al. (Year: 2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med147: 224–23317638714
10. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al. . (2008) When and how to update systematic reviews. Cochrane Database Syst Rev: MR000023.
11. Pattanittum P,, Laopaiboon M,, Moher D,, Lumbiganon P,, Ngamjarus C, (Year: 2012) A comparison of statistical methods for identifying out-of-date systematic reviews. PLoS ONE7: e4889423185281
12. Bastian H,, Glasziou P,, Chalmers I, (Year: 2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?PLoS Med7: e100032620877712
13. Chalmers I, (Year: 1986) Electronic publications for updating controlled trial reviews. Lancet328: 287
14. Berkey CS,, Mosteller F,, Lau J,, Antman EM, (Year: 1996) Uncertainty of the time of first significance in random effects cumulative meta-analysis. Control Clin Trials17: 357–3718932970
15. Ioannidis J,, Lau J, (Year: 2001) Evolution of treatment effects over time: empirical insight from recursive cumulative metaanalyses. Proc Natl Acad Sci U S A98: 831–83611158556
16. Wetterslev J,, Thorlund K,, Brok J,, Gluud C, (Year: 2008) Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol61: 64–7518083463
17. Higgins JP,, Whitehead A,, Simmonds M, (Year: 2011) Sequential methods for random-effects meta-analysis. Stat Med30: 903–92121472757
18. Spieglehalter D, (Year: 2004) Incorporating Bayesian ideas into health-care evaluation. Statist Sci19: 156–174
19. Turner RM,, Davey J,, Clarke MJ,, Thompson SG,, Higgins JP, (Year: 2012) Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol41: 818–82722461129
20. Wetterstrand K (2013) DNA Sequencing costs: data from the NHGRI Genome Sequencing Program (GSP). Available: Accessed 11 July 2013.
21. Tricco AC,, Brehaut J,, Chen MH,, Moher D, (Year: 2008) Following 411 Cochrane protocols to completion: a retrospective cohort study. PLoS ONE3: e368418997866
22. Elliott J (2012) Exploiting innovations in technology to improve the efficiency of review production. 20th Cochrane Colloquium. Auckland.
23. Ciapponi A, Glujovsky D (2012) Survey among Cochrane authors about early stages of systematic reviews. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012 Auckland, New Zealand
24. Elliott J,, Mavergames C,, Becker L,, Meerpohl J,, Thomas J,, et al. (Year: 2013) The efficient production of high quality evidence reviews is important for the public good. BMJ346: f84623407729
25. Thomas J,, McNaught J,, Ananiadou S, (Year: 2011) Applications of text mining within systematic reviews. Research Synthesis Methods2: 1–14
26. Thomas J, Brunton J (2010) EPPI-Reviewer 4.0: software for research synthesis. London: Social Science Research Unit, Institute of Education.
27. Wallace B,, Trikalinos T,, Lau J,, Brodley C,, Schmid C, (Year: 2010) Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics11: 5520102628
28. Wallace BC,, Small K,, Brodley CE,, Lau J,, Schmid CH,, et al. (Year: 2012) Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med14: 663–66922481134
29. Booth A,, Clarke M,, Dooley G,, Ghersi D,, Moher D,, et al. (Year: 2013) PROSPERO at one year: an evaluation of its utility. Systematic Reviews2: 423320413
30. (2013) SRDR: Systematic Review Data Repository. Available: Accessed 8 November 2013.
31. (2013) Cochrane Register of Studies. Available: Accessed 8 November 2013.
32. (2013) Cochrane Reviews Ontology. Available: Accessed 8 November 2013.
33. (2013) OCRe: The Ontology of Clinical Research. Available: Accessed 8 November 2013.
34. McGee R, Willis N, Jones A, Higgins G, Craig J, et al. . (2012) Factors associated with the timely completion of a Cochrane review. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand
35. Sung NS,, Crowley WF,, Genel M,, Salber P,, Sandy L,, et al. (Year: 2003) Central challenges facing the national clinical research enterprise. JAMA289: 1278–128712633190
36. Lavis J,, Davies H,, Oxman A,, Denis JL,, Golden-Biddle K,, et al. (Year: 2005) Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy10 Suppl 135–4816053582
37. Colby DC,, Quinn BC,, Williams CH,, Bilheimer LT,, Goodell S, (Year: 2008) Research glut and information famine: making research evidence more useful for policymakers. Health Aff (Millwood)27: 1177–118218607053
38. Noel-Storr A, Struthers C, Huckvale C, Dooley G, McShane R (2012) ALOIS, ScreenToGo and the Cochrane Register of Studies: using mobile technology to engage and involve consumers in the work of the Cochrane Collaboration. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand.
39. What is the Human Variome Project?Nat Genet39: 42317392793
40. Giardine B,, Borg J,, Higgs DR,, Peterson KR,, Philipsen S,, et al. (Year: 2011) Systematic documentation and analysis of human genetic variation in hemoglobinopathies using the microattribution approach. Nat Genet43: 295–30121423179
41. Mons B,, van Haagen H,, Chichester C,, Hoen PB,, den Dunnen JT,, et al. (Year: 2011) The value of data. Nat Genet43: 281–28321445068
42. US Food and Drug Administration (2013) Study data standards for regulatory submissions: position statement. Available: Accessed 8 November 2013.
43. Brandt L, Kristiansen A, Guyatt G, Vandvik P (2012) Providing likeable and understandable guidelines using GRADE in the EMR with direct links to individual patient data. Phase 1. In: Proceedings of the 9th Guidelines International Network Conference; 22–25 August 2012Berlin, Germany.


[Figure ID: pmed-1001603-g001]
doi: 10.1371/journal.pmed.1001603.g001.
Figure 1  Time from primary study publication to incorporation in systematic review.

Analysis of 792 study reports incorporated into 73 systematic reviews across 28 high priority topics in the field of neurotrauma. Study reports were included in the analysis if they were incorporated into a systematic review relevant to one of the high priority topics and published in the period 2001–2009. Systematic reviews were included in the analysis if they were relevant to one of the high priority topics and published in the period 2001–2012. Bars represent medians and interquartile range.

[Figure ID: pmed-1001603-g002]
doi: 10.1371/journal.pmed.1001603.g002.
Figure 2  Current and emerging health knowledge ecosystems.

The current health knowledge ecosystem (inner circle) is characterized by inefficiencies that hamper the flow of knowledge from health practice through primary research, systematic review and guidelines, and finally back to impacts on health practice. The new health knowledge ecosystem that is emerging (outer circle) is characterized by a continuous flow of knowledge between efficient, living components, including the growing importance of learning health care systems, which together with traditional primary research will populate common data repositories. Living evidence services derived from these repositories, supporting living guidance and decision support systems will close a “living” health knowledge loop.

[TableWrap ID: pmed-1001603-t001] doi: 10.1371/journal.pmed.1001603.t001.
Table 1  Living systematic review: modifications to conventional systematic review and key challenges.
Category Item Description Key Challenges
Essential modifications to conventional processes
Production Work processes Search strategy maintained and fed continuously into SR workflow Shift to continuous work processes
Author team management Coordinated and continuous effort Coordination of ongoing review outputs and author team turnover
Statistical methods Updating of meta-analysis Building consensus regarding appropriate statistical methods
Publication Publication format Persistent, dynamic online-only publication Updatable online publication formats with authorship and version control
Production Workflow and collaboration tools Tools and platforms for SR authoring Incentives and partnerships for innovation and translation into widely available applications
Semi-automation Machine-assisted SR production processes Demonstration of performance in large scale real world implementation; extension of applications beyond citation screening
Data repositories and linked data Repositories of structured SR data Development of common data formats, controlled vocabularies and SR ontologies
Participation and the crowd Larger, diverse author groups; citizen and crowd participation; nanopublication Demonstration of performance in large scale real world implementation; refining incentives, training, and quality assurance
Publication Peer and editorial review Adaptations to conventional peer and editorial review Validation and acceptance by academic community
Norms of scholarly communication Attribution, citation, and listing in bibliographic databases Ensuring conventional academic incentives are maintained
Associated developments
Production Research output annotation Annotation of research outputs with rich, structured meta-data Development of meta-data, processes, and incentives
Structured research datasets Publication and verification of structured research data Standards, models, and incentives for data sharing
Publication Living knowledge translation Living guidelines, standards, policies, and decision support systems Development, demonstration of feasibility, and evaluation of benefits and risks
Integration with guideline development and decision support systems Structured data moving “nimbly” from SR to guideline to decision support systems Development, validation, and evaluation of integrated evidence systems

Article Categories:
  • Policy Forum
Article Categories:
  • Medicine
    • Clinical research design
      • Meta-analyses
      • Systematic reviews

Previous Document:  Cecum lymph node dendritic cells harbor slow-growing bacteria phenotypically tolerant to antibiotic ...
Next Document:  Quantitative measurement of melanoma spread in sentinel lymph nodes and survival.