Document Detail

Fool's gold: Why imperfect reference tests are undermining the evaluation of novel diagnostics: a reevaluation of 5 diagnostic tests for leptospirosis.
Jump to Full Text
MedLine Citation:
PMID:  22523263     Owner:  NLM     Status:  MEDLINE    
Abstract/OtherAbstract:
BACKGROUND: We observed that some patients with clinical leptospirosis supported by positive results of rapid tests were negative for leptospirosis on the basis of our diagnostic gold standard, which involves isolation of Leptospira species from blood culture and/or a positive result of a microscopic agglutination test (MAT). We hypothesized that our reference standard was imperfect and used statistical modeling to investigate this hypothesis.
METHODS: Data for 1652 patients with suspected leptospirosis recruited during three observational studies and one randomized control trial that described the application of culture, MAT, immunofluorescence assay (IFA), lateral flow (LF) and/or PCR targeting the 16S rRNA gene were reevaluated using Bayesian latent class models and random-effects meta-analysis.
RESULTS: The estimated sensitivities of culture alone, MAT alone, and culture plus MAT (for which the result was considered positive if one or both tests had a positive result) were 10.5% (95% credible interval [CrI], 2.7%-27.5%), 49.8% (95% CrI, 37.6%-60.8%), and 55.5% (95% CrI, 42.9%-67.7%), respectively. These low sensitivities were present across all 4 studies. The estimated specificity of MAT alone (and of culture plus MAT) was 98.8% (95% CrI, 92.8%-100.0%). The estimated sensitivities and specificities of PCR (52.7% [95% CrI, 45.2%-60.6%] and 97.2% [95% CrI, 92.0%-99.8%], respectively), lateral flow test (85.6% [95% CrI, 77.5%-93.2%] and 96.2% [95% CrI, 87.7%-99.8%], respectively), and immunofluorescence assay (45.5% [95% CrI, 33.3%-60.9%] and 96.8% [95% CrI, 92.8%-99.8%], respectively) were considerably different from estimates in which culture plus MAT was considered a perfect gold standard test.
CONCLUSIONS: Our findings show that culture plus MAT is an imperfect gold standard against which to compare alterative tests for the diagnosis of leptospirosis. Rapid point-of-care tests for this infection would bring an important improvement in patient care, but their future evaluation will require careful consideration of the reference test(s) used and the inclusion of appropriate statistical models.
Authors:
Direk Limmathurotsakul; Elizabeth L Turner; Vanaporn Wuthiekanun; Janjira Thaipadungpanit; Yupin Suputtamongkol; Wirongrong Chierakul; Lee D Smythe; Nicholas P J Day; Ben Cooper; Sharon J Peacock
Related Documents :
22369033 - Current and future techniques in the evaluation of dysphagia.
22337493 - Psychometric functions for shortened administrations of a speech recognition approach u...
2708863 - A comparison of two techniques for the quantitative determination of oocysts in faecal ...
Publication Detail:
Type:  Evaluation Studies; Journal Article; Research Support, Non-U.S. Gov't     Date:  2012-04-20
Journal Detail:
Title:  Clinical infectious diseases : an official publication of the Infectious Diseases Society of America     Volume:  55     ISSN:  1537-6591     ISO Abbreviation:  Clin. Infect. Dis.     Publication Date:  2012 Aug 
Date Detail:
Created Date:  2012-07-12     Completed Date:  2012-11-07     Revised Date:  2014-02-24    
Medline Journal Info:
Nlm Unique ID:  9203213     Medline TA:  Clin Infect Dis     Country:  United States    
Other Details:
Languages:  eng     Pagination:  322-31     Citation Subset:  IM    
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms
Descriptor/Qualifier:
Adolescent
Adult
Diagnostic Errors / statistics & numerical data*
Diagnostic Tests, Routine / methods*,  standards
Humans
Leptospira / isolation & purification
Leptospirosis / diagnosis*
Microscopy / methods
Models, Statistical
Randomized Controlled Trials as Topic
Reference Standards*
Young Adult
Grant Support
ID/Acronym/Agency:
089275//Wellcome Trust; 090219/Z/09/Z//Wellcome Trust
Comments/Corrections

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): Clin Infect Dis
Journal ID (iso-abbrev): Clin. Infect. Dis
Journal ID (publisher-id): cid
Journal ID (hwp): cid
ISSN: 1058-4838
ISSN: 1537-6591
Publisher: Oxford University Press
Article Information
Download PDF
© The Author 2012. Published by Oxford University Press on behalf of the Infectious Diseases Society of America. All rights reserved. For Permissions, please e-mail: journals.permissions@oup.com.
creative-commons:
Received Day: 4 Month: 12 Year: 2011
Accepted Day: 21 Month: 3 Year: 2012
Print publication date: Day: 1 Month: 8 Year: 2012
Electronic publication date: Day: 20 Month: 4 Year: 2012
pmc-release publication date: Day: 20 Month: 4 Year: 2012
Volume: 55 Issue: 3
First Page: 322 Last Page: 331
ID: 3393707
PubMed Id: 22523263
DOI: 10.1093/cid/cis403
Publisher Id: cis403

Fool's Gold: Why Imperfect Reference Tests Are Undermining the Evaluation of Novel Diagnostics: A Reevaluation of 5 Diagnostic Tests for Leptospirosis
Direk Limmathurotsakul12
Elizabeth L. Turner7
Vanaporn Wuthiekanun2
Janjira Thaipadungpanit2
Yupin Suputtamongkol5
Wirongrong Chierakul23
Lee D. Smythe6
Nicholas P. J. Day28
Ben Cooper2
Sharon J. Peacock249
1Department of Tropical Hygiene
2Mahidol-Oxford Tropical Medicine Research Unit
3Department of Clinical Tropical Medicine
4Department of Microbiology and Immunology, Faculty of Tropical Medicine
5Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand
6WHO/FAO/OIE Collaborating Centre for Reference and Research on Leptospirosis, Centre for Public Health Sciences, Queensland Health Scientific Services, Brisbane, Australia
7Department of Medical Statistics, Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine
8Centre for Clinical Vaccinology and Tropical Medicine, Nuffield Department of Clinical Medicine, Churchill Hospital, University of Oxford,
9Department of Medicine, Cambridge University, Addenbrooke's Hospital, United Kingdom
Correspondence: Correspondence: Direk Limmathurotsakul, MD, PhD, 420/6 Mahidol-Oxford Tropical Medicine Research Unit, Faculty of Tropical Medicine, Mahidol University, Rajvithee Rd, Bangkok, 10400 Thailand (direk@tropmedres.ac).

The clinical manifestations of leptospirosis range from a mild influenza-like illness to multiorgan failure and death. The common signs and symptoms of this infection fail to discriminate leptospirosis from a range of other infectious diseases that occur in the tropics, including dengue fever, malaria, and rickettsial infection [1]. Although laboratory diagnosis has the potential to guide the management of patients with leptospirosis, this is not currently achieved in most regions of the world because of a lack of point-of-care tests. Once such tests become available, however, it will be important to ensure that the diagnostic reference standard against which they are compared during clinical evaluation is robust.

We have undertaken several therapeutic and diagnostic evaluation studies involving patients presenting to hospitals in Thailand with suspected leptospirosis [25]. The reference tests used were a combination of culture for Leptospira species and the microscopic agglutination test (MAT). Definite leptospirosis was defined as the isolation of Leptospira species from a normally sterile site and/or a 4-fold increase in the MAT titer between acute- and convalescent-phase serum samples or a single MAT titer of ≥1:400. This is consistent with the published recommendations of the World Health Organization Leptospirosis Burden Epidemiology Reference Group [6].

We began to question the accuracy of the recommended approach after becoming increasingly aware that some patients with suspected leptospirosis who had positive results of alternative tests, such as a real-time polymerase chain reaction (PCR) targeting the gene encoding the 16S ribosomal RNA (rRNA) subunit, a lateral flow test, and/or an immunofluorescence assay (IFA), were negative for leptospirosis on the basis of our reference standard. This was a consistent finding across different studies in which we had obtained a convalescent-phase specimen from the majority of cases. We hypothesized that the reference standard was imperfect and that the accuracy of alternative diagnostic tests, estimated using the gold standard, was biased. We sought an appropriate statistical model with which to determine the true accuracy of alternative tests in this situation.

Bayesian latent class models have been increasingly used to evaluate the true accuracy of diagnostic tests and do not require the assumption that any test or combination of tests is perfect [7, 8]. The objective of this study was to use Bayesian latent class models to reanalyze individual-level data from 4 existing data sets gathered during studies of patients presenting to the hospital with suspected leptospirosis. On the basis of these findings, we estimated the true accuracy of a range of serological and molecular diagnostic tests for leptospirosis and determined the impact of an imperfect gold standard on the reported accuracies of alternative tests. Information from all studies was combined using Bayesian random-effects meta-analysis to further support the observations from individual studies.


METHODS

We followed a standard protocol for meta-analyses [9], together with the methods recommended by the Cochrane Diagnostic Test Accuracy Working Group, the STARD (Standards for the Reporting of Diagnostic Accuracy) statement, and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement [10].

Search Strategy and Study Selection

Our aim was to reanalyze complete individual-level data sets created by us during hospital-based studies of suspected leptospirosis conducted in northeast Thailand between 2000 and 2010 (Table 1). All studies had undertaken prospective enrollment of adult patients (age, >14 years) with suspected leptospirosis and had used a combination of blood culture for Leptospira species and MAT (hereafter, “culture plus MAT”) as the diagnostic reference standard. Studies were selected if individual-level data sets were available for analysis. For patients included in >1 study, only data from the first study were used in the analysis.

Ethics Statement

Ethical approval for all studies included in the analysis was obtained from the Ministry of Public Health, Thailand, and the Oxford Tropical Research Ethics Committee, United Kingdom. Written inform consent was obtained from each subject enrolled into these studies [25].

Diagnostic Tests

All diagnostic tests that were used in each study were evaluated. In each study, blood was collected on the day of admission and cultured for Leptospira organisms, as described previously [3]. Serum samples (5 mL) collected on admission and, if available, at a 2-week follow-up visit were used for serological testing. Serum was stored at −80°C between collection and serological testing. The MAT was performed at the World Health Organization/United Nations Food and Agriculture Organization/World Animal Health Organization Collaborating Center for Reference and Research on Leptospirosis in Brisbane, Australia. The panel of serovars used in the MAT included representative serovars from all serogroups known to cause leptospirosis in Thailand. A real-time PCR assay targeting the 16S rRNA subunit, a lateral flow test (Leptotek, BioMerieux, the Netherlands), and an in-house IFA were performed as described previously [2, 11, 12]. The MAT detects crude antibodies against Leptospira organisms, the lateral flow test detects immunoglobulin M (IgM), and the IFA detects immunoglobulin G, immunoglobulin A, and IgM [11, 12]. All tests were performed by experienced technicians. The readers of results of culture, MAT, and other diagnostic tests in each study were blinded to the results of the other tests and any clinical information.

Statistical Analysis
Culture Plus MAT as Gold Standard Model

Five diagnostic tests (culture, MAT, IFA, lateral flow test, and PCR) were analyzed for each of the studies, using culture plus MAT as the gold standard. Prevalence, sensitivity, specificity, and positive and negative predictive values for each of the 6 tests were calculated using Stata 11.1 (Stata, College Station, TX).

Bayesian Latent Class Models

Use of latent class models and Bayesian latent class models to determine the accuracy of diagnostic tests when the accuracy of the gold standard is imperfect or unknown has been described in detail elsewhere [1315]. Figure 1 illustrates how the imperfect gold standard model estimates unbiased accuracies of diagnostic tests in one example scenario: application of 3 diagnostic tests to 1 study population. In brief, use of Bayesian latent class models does not assume that any test or a combination of any tests is perfect but considers that each test could be imperfect in diagnosing the true disease status. The true disease status of the patient population is then defined on the basis of the overall prevalence (the probability that a patient with suspected leptospirosis is truly infected) [1315]. Latent class models estimate the prevalence and accuracy of each test on the basis of the observed frequency of the possible combinations of test results.

To estimate the accuracy of a diagnostic test by use of latent class models, the best-fitting model, as determined by the presence or absence of correlation between diagnostic tests in the model, should be used [14, 15]. Possible correlations we evaluated were based on existing knowledge and external evidence. Therefore, correlation among antigenic tests (culture and PCR) and correlation among serological tests (MAT, IFA, and the lateral flow test) were considered. All models assumed that no prior information (noninformative priors) about the unknown parameters (ie, prevalence, sensitivities, and specificities) was available, except that the specificity of culture was fixed at 100%. For multicenter studies [4, 5], the models also assumed that sensitivities and specificities of culture and MAT were consistent over different study sites. All parameters and associated 95% credible intervals (CrIs) were estimated using WinBUGS 1.4 [16].

To estimate the overall accuracy of culture alone, MAT alone, and culture plus MAT across all studies, the information obtained from the best-fitting Bayesian latent class model for each data set was combined using a Bayesian meta-analysis model [17]. Random effects were used to account for differences in the sensitivities of culture and MAT and the specificity of MAT between studies. The ranges of each parameter from the Bayesian latent class model with the best fit obtained for individual data sets were used as informative priors in the meta-analysis model [17]. Appendixes 1 and 2 (Supplementary Materials) provide full data sets and all of the models used, respectively. Appendix 3 (Supplementary Materials) provides details about the method and the results for the best-fitting model selection.

Sensitivity Analyses

Sensitivity analyses were performed in which patients without convalescent-phase samples were excluded and also in which different prior information were used [18].


RESULTS

The flow diagram in Figure 2 provides an overview of the study. Five studies conducted by us between 2000 and 2010 that used or evaluated diagnostic tests to investigate patients presenting to the hospital with suspected leptospirosis were searched for the presence of complete individual-level data. One study was excluded because individual test results were not available [19]. Of 2455 records from the 4 remaining data sets, 803 were excluded because they were duplicated in ≥1 of the other studies (n = 641) or because not all of the intended tests were undertaken (n = 162). Therefore, 4 data sets involving 1652 patients were included in the analysis (Table 1) [25]. Studies A and B were prospective observational studies conducted at a single hospital in Udon Thani, in northeast Thailand, in which culture, MAT, and at least 1 additional test (PCR, lateral flow test, or IFA) was undertaken in all cases. Studies C and D were prospective multicenter studies in which culture and MAT were undertaken in all cases. All serological tests had been performed using paired samples, when available. The median duration of illness prior to admission was 4 days, and the median duration of illness prior to obtaining the convalescent-phase serum sample was 14 days.

We first assumed that the combination of culture plus MAT was a perfect reference test (100% sensitivity and 100% specificity). This gave an estimated prevalence for leptospirosis of 36.9%, 24.0%, 24.8%, and 16.8% for studies A, B, C, and D, respectively. The sensitivity of culture alone was low and varied significantly among studies (28.5%, 28.6%, 15.4%, and 6.3% for studies A, B, C, and D, respectively). The sensitivity of MAT was high but also varied significantly among studies (86.9%, 85.7%, 91.0%, and 96.2% for studies A, B, C, and D, respectively). PCR, the lateral flow test, and IFA were found to have either low sensitivity (55.5%, 87.6%, and 64.7%, respectively) or low specificity (82.5%, 70.5%, and 95.2%, respectively).

Bayesian latent class models were then used to obtain an estimate of the accuracy of each diagnostic test, without the assumption that the reference test was perfect. In the first stage, we defined the best-fitting Bayesian latent class model for each data set by determining the presence of correlations between antigenic tests (culture and PCR) and serological tests (MAT, IFA, and the lateral flow test) (Appendix 3). The best-fitting model for study A was the model that included correlations between culture and PCR and between MAT and the lateral flow test. The best-fitting model for study B was the model that included correlation between MAT and IFA. The model without correlation between diagnostic tests was selected for studies C and D because we did not expect a correlation between culture and MAT. Sensitivities and specificities of culture, MAT, and culture plus MAT, estimated by Bayesian latent class model for each study, are shown in Figure 3. These findings formed the basis for choosing the model with correlation between culture and PCR, between MAT and the lateral flow test, and between MAT and IFA for the meta-analysis.

Data across all 4 studies were then combined and analyzed using a Bayesian latent class random-effects meta-analysis model, which demonstrated that all of the estimated parameters were considerably different from those estimated when culture plus MAT was assumed to be perfect. The meta-analysis model indicated that culture, MAT, and culture plus MAT had very low sensitivities of 10.5% (95% CrI, 2.7%–27.5%), 49.8% (95% CrI, 37.6%–60.8%), and 55.5% (95% CrI, 42.9%–67.7%), respectively (Figure 3 and Table 2). The specificity of both MAT alone and of culture plus MAT was 98.8% (95% CrI, 92.8%–100%). This means that the prevalence of leptospirosis estimated in each data set using the model was much higher than if relying on culture plus MAT (eg, 57.4% vs 36.9% for study A). Of the 2 antigenic tests, PCR had the highest sensitivity, at 52.7% (95% CrI, 45.2%–60.6%). Among the 3 serological tests, the lateral flow test had the highest sensitivity, at 85.6% (95% CrI, 77.5%–93.2%). Because it is possible that a combination of PCR and a serological test could be used as point-of-care diagnostic tests for leptospirosis, using positive results of either test, the sensitivity and specificity of PCR plus the lateral flow test were calculated by the model; these were 93.2% (95% CrI, 88.8%–96.9%) and 93.1% (95% CrI, 84.1%–98.5%), respectively (Table 2).

Sensitivity Analysis

Sensitivity analysis was performed in which 555 of 1652 patients (33.6%) without convalescent-phase samples were excluded. By use of a random-effects meta-analysis model, the sensitivities of MAT, the lateral flow test, and IFA were estimated to be 70.3% (95% CrI, 44.1%–91.5%), 89.7% (95% CrI, 80.3%–97.2%), and 71.2% (95% CrI, 52.4%–89.8%), respectively, for patients with leptospirosis who had a convalescent-phase sample. The accuracies of other diagnostic tests were not substantially different from the above values, although all CrIs were wider as a consequence of the reduced information in the sample sizes. There was no substantial change when different prior information was used (Appendix 4 [Supplementary Data]).


DISCUSSION

The key findings of this study are that the true sensitivities of culture, MAT, and culture plus MAT are low. The finding that culture has a low sensitivity is neither novel nor surprising, since Leptospira organisms are only present in the blood during the first week of untreated infection, and isolation of this bacterium from clinical samples is technically demanding. Furthermore, our patient population may have consumed over-the-counter antibiotics prior to hospital presentation, which may have resulted in false-negative culture results. The proportion of patients who had taken antibiotics by the time of presentation in our studies is not known, but a study of patients presenting to a hospital in neighboring Laos showed that 57% of patients who were admitted to Mahosot Hospital and underwent investigations, including lumbar puncture, had antimicrobial activity detected in their urine [20]. Other possible explanations for the low sensitivity of culture are that viable Leptospira species are difficult to recover from clinical samples, using the existing culture methods. Of note, cultures were maintained for at least 6 months before results were deemed to be negative. More worrying is the low sensitivity of MAT, since this is central to the case definition of definite leptospirosis and is widely used.

There are several potential reasons why MAT had low sensitivity in our setting. Antibodies to Leptospira may take several weeks to become detectable by MAT [21]. In the studies described here, the second (ie, convalescent-phase) serum sample was collected at least 10 days after the start of symptoms that were attributed to leptospirosis, but it is possible that this interval was too short in some cases [22]. In common with other research studies and reflecting real life, we also failed to obtain a convalescent-phase serum specimen from 34% of patients, either because they died or because they were discharged and were lost to follow-up. The results from our sensitivity analysis also show that the sensitivity of MAT was 70.3% in the ideal situation, in which convalescent-phase samples were obtained from all patients. This increase in sensitivity is consistent with existing knowledge and is comparable to a previous estimate (76%), in which only patients with culture-confirmed leptospirosis were considered [22]. However, this also suggests that a number of patients with leptospirosis have a false-negative test result by MAT even if a convalescent-phase sample is available. Other possible explanations for the low sensitivity of MAT are that lipopolysaccharide from different Leptospira serovars induces a variable level of immune response and that the Leptospira serovars used in the MAT did not include 1 or more locally important strains, although we have no evidence that this is the case. The poor sensitivity of culture plus MAT in the real clinical setting, as shown by Bayesian latent class modeling, suggests that improvement of both tests or development of a new gold standard test is required.

Our data supported a positive correlation between both antigenic tests (ie, culture and PCR), a finding that could be interpreted as meaning that results of both culture and PCR are more likely to be positive if the burden of Leptospira organisms in blood is high and to be negative if the burden is low [2, 23]. A positive correlation was also found between serological tests (ie, between MAT and IFA and between MAT and the lateral flow test). These findings are consistent with existing knowledge.

A major effect of poor sensitivity of the reference test is that the prevalence of leptospirosis is underestimated. The estimated prevalence of leptospirosis for each study separately, as determined by Bayesian latent class modeling (32.6%–57.4%), was around double that of previous estimates that used culture plus MAT as the gold standard (16.8%–36.9%). This is credible, since 50% of our study patients with suspected leptospirosis left the hospital without a definite diagnosis following a test panel that included bacterial culture; other serological tests, including those for rickettsial infections (which are also common in our setting); radiological tests; and detailed clinical evaluation [25]. Our study suggests that leptospirosis may have been the cause of fever in a proportion of these cases.

Evaluation of diagnostic tests when the accuracy of the gold standard is unknown is an active area of biostatistical research, since the use of an imperfect gold standard to evaluate the accuracy and clinical usefulness of an alternative test is flawed and leads to biased results [8, 14, 17, 24]. Our study has demonstrated that culture plus MAT represents a relatively poor gold standard against which to compare alternative diagnostic tests for leptospirosis and has shown the usefulness of statistical models under such circumstances. For example, PCR had a sensitivity and specificity of 55.5% and 82.5%, respectively, when compared with culture plus MAT. When recalculated using Bayesian latent class modeling, the sensitivity and specificity of PCR were 52.7% and 97.2%, respectively, representing a test with a high degree of specificity.

Our study had several strengths, including the use of large and individual-level data sets rather than summary estimates from the published literature. Disparity of study characteristics and risk of bias were comparatively low, since all studies were conducted prospectively and by the same research unit. None of the studies were supported by diagnostic companies. In addition, we used a random-effects model, a rigorous statistical method that has been recommended by the Cochrane Diagnostic Test Accuracy Working Group as the method of choice for diagnostic meta-analyses [10].

This study also has several limitations. Use of basic Bayesian latent class models to estimate the sensitivity and specificity of each test in a population does not allow us to determine the effects that symptom duration, antimicrobials received prior to presentation, and timing of convalescent-phase samples at the level of individual patients have on these parameters. These effects could be evaluated in advanced Bayesian latent class models [25]. Correlation between IFA and the lateral flow test could not be evaluated because these tests were not performed together in any study included in the analysis. The small number of studies included also meant that important study characteristics, including differences in the prevalence of leptospirosis, and the level of reproducibility between studies of the finding of high accuracy for PCR and the lateral flow test were not assessed. The commercial lateral flow test evaluated here had a published specificity that was underestimated, compared with the gold standard, but it is not currently available [26, 27]. The currently available rapid serological tests for leptospirosis include a latex agglutination test and an IgM enzyme-linked immunosorbant assay [2830], neither of which has been evaluated by us to date.

We conclude that our current reference testing strategy is imperfect. As a result, both the prevalence of leptospirosis in northeast Thailand and the accuracy of alternative diagnostic tests have been underestimated. There is an urgent need for rapid serological tests for leptospirosis. Our findings support the use of latent class models to evaluate such a new test against an imperfect gold standard.


Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online (http://cid.oxfordjournals.org). Supplementary materials consist of data provided by the author that are published to benefit the reader. The posted materials are not copyedited. The contents of all supplementary data are the sole responsibility of the authors. Questions or messages regarding errors should be addressed to the author.

Supplementary Data
Click here for additional data file (supp_cis403_cis403supp1.doc)

Click here for additional data file (supp_cis403_cis403supp2.doc)

Click here for additional data file (supp_cis403_cis403supp3.doc)

Click here for additional data file (supp_cis403_cis403supp4.doc)


Notes

Acknowledgments. We gratefully acknowledge the support provided by staff at the Mahidol-Oxford Tropical Medicine Research Unit and participating hospitals. The lateral flow test was provided during 2000 at no cost by BioMerieux (The Netherlands).

Disclaimer. BioMerieux had no involvement in any part of the work described, nor the writing of the manuscript.

Financial support. This study was supported by the Wellcome Trust. D. L. is supported by a project grant awarded by the Wellcome Trust (090219/Z/09/Z). S. J. P. is supported by the National Institute for Health Research Cambridge Biomedical Research Centre.

Potential conflict of interest. All authors: No reported conflicts.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.


References
1. Suttinont C,Losuwanaluk K,Niwatayakul K,et al. Causes of acute, undifferentiated, febrile illness in rural Thailand: results of a prospective observational studyAnn Trop Med ParasitolYear: 20061003637016762116
2. Thaipadungpanit J,Chierakul W,Wuthiekanun V,et al. Diagnostic accuracy of real-time PCR assays targeting 16S rRNA and lipl32 genes for human leptospirosis in Thailand: a case-control studyPLoS OneYear: 20116e1623621283633
3. Wuthiekanun V,Chierakul W,Limmathurotsakul D,et al. Optimization of culture of Leptospira from humans with leptospirosisJ Clin MicrobiolYear: 2007451363517301285
4. Phimda K,Hoontrakul S,Suttinont C,et al. Doxycycline versus azithromycin for treatment of leptospirosis and scrub typhusAntimicrob Agents ChemotherYear: 20075132596317638700
5. Wuthiekanun V,Sirisukkarn N,Daengsupa P,et al. Clinical diagnosis and geographic distribution of leptospirosis, ThailandEmerg Infect DisYear: 200713124617370525
6. World Health Organization (WHO)Report of the first meeting of leptospirosis burden epidemiology reference groupYear: 2010GenevaWHO134
7. Speybroeck N,Praet N,Claes F,et al. True versus apparent malaria infection prevalence: the contribution of a Bayesian approachPLoS OneYear: 20116e1670521364745
8. Limmathurotsakul D,Jamsen K,Arayawichanont A,et al. Defining the true sensitivity of culture for the diagnosis of melioidosis using Bayesian latent class modelsPLoS OneYear: 20105e1248520830194
9. Deeks JJ. Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening testsBMJYear: 20013231576211463691
10. Leeflang MM,Deeks JJ,Gatsonis C,Bossuyt PM. Systematic reviews of diagnostic test accuracyAnn Intern MedYear: 20081498899719075208
11. Smits HL,Eapen CK,Sugathan S,et al. Lateral-flow assay for rapid serodiagnosis of human leptospirosisClin Diagn Lab ImmunolYear: 20018166911139212
12. Appassakij H,Silpapojakul K,Wansit R,Woodtayakorn J. Evaluation of the immunofluorescent antibody test for the diagnosis of human leptospirosisAm J Trop Med HygYear: 19955234037741173
13. Zhou XM,Mcclish DK,Obuchowski NA. Methods for correcting imperfect standard bias. In: Statistical methods in diagnostic medicineYear: 20021st edNew York: Wiley-Interscience40
14. Joseph L,Gyorkos TW,Coupal L. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standardAm J EpidemiolYear: 1995141263727840100
15. Dendukuri N,Joseph L. Bayesian approaches to modeling the conditional dependence between multiple diagnostic testsBiometricsYear: 2001571586711252592
16. Lunn D,Spiegelhalter D,Thomas A,Best N. The BUGS project: evolution, critique and future directionsStat MedYear: 20092830496719630097
17. Chu H,Chen S,Louis TA. Random effects models in a meta-analysis of the accuracy of two diagnostic tests without a gold standardJ Am Stat AssocYear: 20091045122319562044
18. Spiegelhalter D,Abrams K,Myles J. Bayesian approaches to clinical trials and health-care evaluationYear: 2004West Sussex, United Kingdom: Wiley
19. Suputtamongkol Y,Niwattayakul K,Suttinont C,et al. An open, randomized, controlled trial of penicillin, doxycycline, and cefotaxime for patients with severe leptospirosisClin Infect DisYear: 20043914172415546074
20. Moore CE,Sengduangphachanh A,Thaojaikong T,et al. Enhanced determination of Streptococcus pneumoniae serotypes associated with invasive disease in Laos by using a real-time polymerase chain reaction serotyping assay with cerebrospinal fluidAm J Trop Med HygYear: 201083451720810803
21. Faine S. Guidelines for the control of leptospirosisYear: 1982Geneva: WHO offset publication no 67.
22. Cumberland P,Everard CO,Levett PN. Assessment of the efficacy of an IgM-ELISA and microscopic agglutination test (MAT) in the diagnosis of acute leptospirosisAm J Trop Med HygYear: 199961731410586903
23. Ahmed A,Engelberts MF,Boer KR,Ahmed N,Hartskeerl RA. Development and validation of a real-time PCR for detection of pathogenic Leptospira species in clinical materialsPLoS OneYear: 20094e709319763264
24. Banoo S,Bell D,Bossuyt P,et al. Evaluation of diagnostic tests for infectious diseases: general principlesNat Rev MicrobiolYear: 20064S203217366684
25. Bernatsky S,Lix L,Hanly JG,et al. Surveillance of systemic autoimmune rheumatic diseases using administrative dataRheumatol IntYear: 2011315495420665025
26. Wagenaar JF,Falke TH,Nam NV,et al. Rapid serological assays for leptospirosis are of limited value in southern VietnamAnn Trop Med ParasitolYear: 2004988435015667716
27. Sehgal SC,Vijayachari P,Sugunan AP,Umapathi T. Field application of Lepto lateral flow for rapid diagnosis of leptospirosisJ Med MicrobiolYear: 20035289790112972585
28. Smits HL,Chee HD,Eapen CK,et al. Latex based, rapid and easy assay for human leptospirosis in a single test formatTrop Med Int HealthYear: 20016114811251907
29. Obregon AM,Fernandez C,Rodriguez I,Balbis Y,Martinez B,Rodriguez J. Latex agglutination system for the rapid diagnosis of leptospirosis in CubaRev Panam Salud PublicaYear: 2004162596515615600
30. Bajani MD,Ashford DA,Bragg SL,et al. Evaluation of four commercially available rapid serologic tests for diagnosis of leptospirosisJ Clin MicrobiolYear: 200341803912574287

Figures

[Figure ID: CIS403F1]
Figure 1. 

Schematic illustration of the use of Bayesian latent class model to obtain unbiased estimates of accuracy of diagnostic tests. A, Overview of all possible outcomes of diagnostic tests, based on true disease status, if 3 diagnostic tests are applied to 1 study population. Broken lines represent unknown parameters. Patients under evaluation could be either diseased or nondiseased, and prevalence represents the probability that a patient is diseased. Solid lines represent the application of all 3 diagnostic tests (t1, t2, and t3) to every patient in the study. Test results are conditional on the sensitivity and specificity of each test. True disease status (diseased or nondiseased) is a latent variable, as it is not directly observed but can be estimated as the prevalence in the Bayesian latent class model. B and C, Comparison of how to estimate the accuracy of diagnostic tests, using the gold standard model (B) and the imperfect gold standard model (C).



[Figure ID: CIS403F2]
Figure 2. 

Study flow diagram.



[Figure ID: CIS403F3]
Figure 3. 

Forest plot of sensitivity and specificity of culture (A), microagglutination test (MAT; B), and the combination of culture and MAT (culture + MAT; C and D) for leptospirosis estimated by Bayesian latent class models (LCMs) and random-effects meta-analysis models. Squares represent median estimates of the sensitivities and the specificities, and the size of the square represents the size of the study. Horizontal lines represent 95% credible intervals of the estimates. Bayesian LCM assuming conditional dependence between culture and PCR and between MAT and lateral flow (LF) was used for study A, and Bayesian LCM assuming conditional dependence between MAT and immunofluorescence assay was used for study B. Bayesian LCM assuming conditional independence between tests and consistency of test accuracies between study sites was used for studies C and D. Meta-analysis was performed by application of random-effects variables into the combined data set of all 4 studies, assuming conditional dependence between culture and PCR, between MAT and LF, and between MAT and IFA. Abbreviations: IFA, immunofluorescence assay; LCM, latent class model; LF, lateral flow; MAT, microagglutination test; PCR, polmerase chain reaction.



Tables
[TableWrap ID: CIS403TB1] Table 1. 

Characteristics of Populations and Studies in Thailand Included in the Analyses


Studya Authors Province(s) Year Study Design Sample Sizeb Diagnostic Tests
A Thaipadungpanit et al [2] Udon Thani 2000–2001 Prospective, observational 371 Culture, MAT, PCR, and LF
B Wuthiekanun et al [3] Udon Thani 2000–2002 Prospective, observational 496 Culture, MAT, IFA
C Phimda et al [4] Udon Thani, Nakorn Rachasima, Chaiyapoom, Chumphon 2003–2005 Multicenter, randomized controlled trial 314 Culture, MAT
D Wuthiekanun et al [5] Udon Thani, Maha Sarakarm, Yasothorn, Chainut, Rayong, Chanthaburi, Prachuap Khiri Khun, Phattalung 2003–2004 Multicenter, prospective, observational 471 Culture and MAT

Abbreviations: IFA, immunofluorescence assay; LF, lateral flow test; MAT, microscopic agglutination test; PCR, polymerase chain reaction.

a Studies are ordered chronologically.

b Records of patients for whom not all of the intended tests were performed and records duplicated among studies are excluded.


[TableWrap ID: CIS403TB2] Table 2. 

Prevalence of Leptospirosis and Accuracy of Diagnostic Tests, Determined Using Culture Plus Microagglutination Tests as the Gold Standard or Bayesian Latent Class and Random-Effects Meta-analysis Models


Parameters Culture Plus MAT as Gold Standard (95% CI)a Bayesian Model (95% CrI)b
Prevalence
 Data set A 36.9 (32.0–41.9) 57.4 (49.8–64.3)
 Data set B 24.0 (20.3–28.0) 38.1 (27.4–52.2)
 Data set C 24.8 (20.2–30.0) 45.9 (34.4–55.5)
 Data set D 16.8 (13.5–20.5) 32.6 (21.5–46.9)
Culture
 Sensitivity 28.5, 28.6, 15.4, and 6.3* 10.5 (2.7–27.5)
 Specificity 100 100
 PPV 100 100
 NPV 70.5, 81.6, 78.2, and 84.1* 45.6 (37.8–54.7)
MAT
 Sensitivity 86.9, 85.7, 91.0, and 96.2* 49.8 (37.6–60.8)
 Specificity 100 98.8 (92.8–100)
 PPV 100 98.3 (88.4–100)
 NPV 92.9, 95.7, 97.1, and 99.2* 59.2 (49.8–68.4)
Culture plus MATc
 Sensitivity 100 55.5 (42.9–67.7)
 Specificity 100 98.8 (92.8–100)
 PPV 100 98.5 (89.6–100)
 NPV 100 62.0 (52.1–72.2)
PCRd
 Sensitivity 55.5 (46.7–64.0) 52.7 (45.2–60.6)
 Specificity 82.5 (77.0–87.1) 97.2 (92.0–99.8)
 PPV 65.0 (56.2–73.7) 96.2 (88.5–99.8)
 NPV 76.0 (70.7–81.3) 60.4 (51.5–69.2)
LFd
 Sensitivity 87.6 (80.9–92.6) 85.6 (77.5–93.2)
 Specificity 70.5 (64.2–76.3) 96.2 (87.7–99.8)
 PPV 63.5 (56.6–70.4) 96.9 (88.9–99.9)
 NPV 90.7 (86.4–94.9) 83.3 (72.2–92.5)
PCR plus LFc,d
 Sensitivity 92.7 (87.0–96.4) 93.2 (88.8–96.9)
 Specificity 66.2 (59.8–72.3) 93.1 (84.1–98.5)
 PPV 61.7 (55.0–68.3) 94.9 (86.8–99.0)
 NPV 93.9 (90.3–97.6) 91.1 (83.4–96.2)
IFAd
 Sensitivity 64.7 (55.4–73.2) 45.5 (33.3–60.9)
 Specificity 95.2 (92.5–97.1) 96.8 (92.8–99.8)
 PPV 81.1 (71.7–88.4) 95.2 (87.8–99.7)
 NPV 89.5 (86.1–92.3) 57.1 (47.3–68.2)

Abbreviations: CI, confidence interval; CrI, credible interval; IFA, immunofluorescence assay; LF, lateral flow test; MAT, microscopic agglutination test; NPV, negative predictive value; PCR, polymerase chain reaction; PPV, positive predictive value.

*For studies A, B, C, and D, respectively.

a Values were estimated on the basis of the observed proportion, which was determined by assuming that culture plus MAT is the gold standard (ie, 100% sensitive and 100% specific). The 95% confidence intervals were obtained using Stata 11.1 (Stata).

b Values were estimated using Bayesian latent class and random-effects meta-analysis models, assuming that culture plus MAT is imperfect. Posterior estimates and 95% credible intervals of each parameter were obtained in WinBUGs from 10 000 iterations of each of 2 chains, starting from different initial values following a burn-in period of 5000 iterations.

c Positive results of one or both tests is diagnostic for leptospirosis infection.

d Values calculated by assuming that culture plus MAT is the gold standard were based on data from studies A and B, whereas values calculated by the Bayesian model were estimated from a meta-analysis model, using the data set for all 4 studies combined.



Article Categories:
  • Articles and Commentaries


Previous Document:  Variation in reported neonatal Group B Streptococcal disease incidence in developing countries.
Next Document:  Impact of USA300 Methicillin-Resistant Staphylococcus aureus on Clinical Outcomes of Patients with P...