Document Detail

Credible knowledge: a pilot evaluation of a modified GRADE method using parent-implemented interventions for children with autism.
Jump to Full Text
MedLine Citation:
PMID:  21426564     Owner:  NLM     Status:  MEDLINE    
BACKGROUND: Decision-making in child and youth mental health (CYMH) care requires recommendations that are developed through an efficient and effective method and are based on credible knowledge. Credible knowledge is informed by two sources: scientific evidence, and practice-based evidence, that reflects the "real world" experience of service providers. Current approaches to developing these recommendations in relation to CYMH will typically include evidence from one source or the other but do not have an objective method to combine the two. To this end, a modified version of the Grading Recommendations Assessment, Development and Evaluation (GRADE) approach was pilot-tested, a novel method for the CYMH field.
METHODS: GRADE has an explicit methodology that relies on input from scientific evidence as well as a panel of experts. The panel established the quality of evidence and derived detailed recommendations regarding the organization and delivery of mental health care for children and youth or their caregivers. In this study a modified GRADE method was used to provide precise recommendations based on a specific CYMH question (i.e. What is the current credible knowledge concerning the effects of parent-implemented, early intervention with their autistic children?).
RESULTS: Overall, it appeared that early, parent-implemented interventions for autism result in positive effects that outweigh any undesirable effects. However, as opposed to overall recommendations, the heterogeneity of the evidence required that recommendations be specific to particular interventions, based on the questions of whether the benefits of a particular intervention outweighs its harms.
CONCLUSIONS: This pilot project provided evidence that a modified GRADE method may be an effective and practical approach to making recommendations in CYMH, based on credible knowledge. Key strengths of the process included separating the assessments of the quality of the evidence and the strength of recommendations, transparency in decision-making, and the objectivity of the methods. Most importantly, this method combined the evidence and clinical experience in a more timely, explicit and simple process as compared to previous approaches. The strengths, limitations and modifications of the approach as they pertain to CYMH, are discussed.
J Michael Van Adel; Jennifer Dunn Geier; Adrienne Perry; Jo-Ann M Reitzel
Related Documents :
21255904 - Determining advanced recycling fees and subsidies in "e-scrap" reverse supply chains.
21341834 - Electron correlation effects on the first hyperpolarizability of push-pull π-conjugate...
21170904 - Meta-analysis of diagnostic test data: a bivariate bayesian modeling approach.
21086524 - The new biological anthropology: bringing washburn's new physical anthropology into 201...
23535194 - Parameter uncertainty in biochemical models described by ordinary differential equations.
15183324 - Applying data mining techniques to library design, lead generation and lead optimization.
Publication Detail:
Type:  Evaluation Studies; Journal Article; Research Support, Non-U.S. Gov't     Date:  2011-03-22
Journal Detail:
Title:  BMC health services research     Volume:  11     ISSN:  1472-6963     ISO Abbreviation:  BMC Health Serv Res     Publication Date:  2011  
Date Detail:
Created Date:  2011-04-08     Completed Date:  2011-06-22     Revised Date:  2013-06-30    
Medline Journal Info:
Nlm Unique ID:  101088677     Medline TA:  BMC Health Serv Res     Country:  England    
Other Details:
Languages:  eng     Pagination:  60     Citation Subset:  IM    
The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO, Ottawa, ON, Canada.
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms
Autistic Disorder / diagnosis,  therapy*
Child, Preschool
Decision Making
Evidence-Based Practice / standards*
Mental Health Services
Parent-Child Relations*
Pilot Projects
Review Literature as Topic

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): BMC Health Serv Res
ISSN: 1472-6963
Publisher: BioMed Central
Article Information
Download PDF
Copyright ©2011 Michael Van Adel et al; licensee BioMed Central Ltd.
Received Day: 23 Month: 12 Year: 2009
Accepted Day: 22 Month: 3 Year: 2011
collection publication date: Year: 2011
Electronic publication date: Day: 22 Month: 3 Year: 2011
Volume: 11First Page: 60 Last Page: 60
ID: 3072313
Publisher Id: 1472-6963-11-60
PubMed Id: 21426564
DOI: 10.1186/1472-6963-11-60

Credible knowledge: A pilot evaluation of a modified GRADE method using parent-implemented interventions for children with autism
J Michael Van Adel1 Email:
Jennifer Dunn Geier2 Email:
Adrienne Perry3 Email:
Jo-Ann M Reitzel4 Email:
1The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO, Ottawa, ON, Canada
2Autism Intervention Program - Eastern Ontario, Children's Hospital of Eastern Ontario, Ottawa, ON, Canada
3Department of Psychology, York University, Downsview, ON, Canada
4McMaster Children's Hospital, Hamilton-Niagara Regional Autism Intervention Program, Hamilton, ON, Canada


An essential ingredient in effective clinical practice is that clinicians benefit, in a timely way, from current research knowledge in their fields. Those responsible for the organization and delivery of child and youth mental health (CYMH) care require such knowledge, which is rapidly changing and ever increasing. Moreover, clinicians benefit from knowledge that is credible. This is especially critical in the development of clinical recommendations, which are intended to achieve the linked objectives of improvements in patient or client outcome and the efficient use of resources. To be credible, these recommendations must consider input from two sources: scientific evidence and the practice-based evidence, reflecting the "real world" expertise and experience of service providers.

Clearly, it is less than practical to expect that individual clinicians can continuously survey and evaluate available knowledge and adapt clinical decisions accordingly. Despite this recognition, however, there are still obstacles that need to be overcome in improving the access to and practical application of credible knowledge in CYMH. Primarily, there is a lack of agreement about the best methods for organizations to identify, assess, and synthesize relevant studies [1-7]. Without a useful methodology for identification and evaluation, published research results may not be adequately assessed for their scientific validity. Poor research, which can suffer from serious methodological flaws, may then be relied upon to make critical treatment decisions. On the other hand, when recommendations are based on rigorous evaluation of only high quality evidence, this evidence may not be assessed for its relevance and practical application to specific care and may not be suitable for generalization to clinicians' practice. These flaws are present in the current approaches to synthesizing evidence-based practice in CYMH.

Numerous professional bodies and scientific organizations have taken the approach of developing clinical practice guidelines. These guidelines include recommendations to aid patient and practitioner decision-making in specific clinical situations. A strength of this approach is that a panel of experts typically develops these recommendations. Panel members have access to the evidence, an understanding of the clinical problems and research methods, and sufficient time for deliberation. However, the recommendations are often formulated without systematically identifying and summarizing the evidence [8]. A proper evaluation of the evidence is critical in ensuring the validity of guidelines. Otherwise, relevant studies may be missed and recommendations made which contradict the current best available evidence. Furthermore, when more rigorous and transparent methods are used, guideline development can take two years or more. As a result, these guidelines can require considerable financial and human resources and the timeline may result in a delay in getting pertinent information to health care providers, thereby having a potential impact on clinical practice [4,9,10].

Systematic reviews take a different approach by using explicit search strategies and inclusion criteria to identify and judge evidence of the effectiveness of clinical interventions [11]. The relative comprehensiveness of these reviews is a key strength. Null or even negative findings are often included, whereas in other approaches such research has been largely ignored in the accumulation of supportive findings [12]. However, health professionals, physicians, and policy makers infrequently refer to these reviews in decision-making [13-15]. So, despite the increasing summaries of the evidence in systematic reviews and clinical practice guidelines, as many as 30 to 45 per cent of patients fail to receive evidence-based treatment, leading to less than optimal outcomes [16-18]. This is due to a number of limitations in these reviews [11]; perhaps the most significant one being the lack of a well-defined, transparent process for including the input of relevant evidence users that have practice-based knowledge. This limitation results in the omission of input from those intended to implement and benefit from the recommendations.

Input from practicing health professionals can make recommendations more practical and clinically- relevant. This can be accomplished through an approach that integrates scientific evidence with the knowledge of patients' values and preferences as well as the implications for treatment planning and decision-making in clinical practice.

A New Approach

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach [19] incorporates the strengths of previous approaches and accounts for their limitations. It may be the more effective method for providing credible knowledge that is essential to CYMH.

GRADE employs a sequential assessment of 1) the quality of the evidence, followed by 2) an evaluation of the balance between benefits and harms and 3) a subsequent judgment about the strength of recommendations [20]. The separation of judgments on the quality of evidence and the strength of recommendations is a critical and defining feature of this method.

GRADE was developed based on a systematic review that identified the fundamental elements of an effective evidence grading system. These elements include: the rating of the quality of individual studies; the quantity in terms of magnitude of effect, number of studies and sample size; and the consistency of the findings between similar and different study designs [21].

The current approaches to synthesizing evidence-based practice research in the CYMH field should be more appropriately described as clinical treatment efficacy research [22]. In the past, these approaches have relied on a one-to-one fit between the research on effective practices and the implementation of these practices in routine care. One clear solution to this problem would see research conducted that attends to service delivery issues at the outset. However, until that sort of evidence becomes available existing evidence-based practice research would be left collecting dust on academic shelves. Instead, the GRADE method may provide a method that can make use of the wealth of existing research. The GRADE approach attempts to merge research-based knowledge of efficacy with practice-based knowledge of effectiveness through the input of the expert panel.

The process begins with the selection of a focused question that can lead to action and the establishment of an expert panel with knowledge related to the question. Ideally, and for the sake of efficiency, relevant published and unpublished scientific evidence relating to the question is gathered from an existing systematic review. A list of potentially important outcomes is then created which panel members must independently rate according to importance ranging from unimportant to critical. The improvement of outcomes with high patient importance (i.e., considered critical) should lead to stronger recommendations [23].

Following the rating of the importance of outcomes, the quality of the evidence is then rated. A number of factors can increase or decrease the quality rating and this should reflect the confidence in the studies' estimates of benefits and harms.

Finally, the evidence, its quality rating and the outcome information is synthesized succinctly and transparently in evidence profiles. These profiles guide the panel in deciding on the strength of recommendation according to the tradeoffs between benefits and harms of the various alternatives.

Based on conceptual arguments and past experience with the process, many developers and users of GRADE believe that this process can be applied across all forms of intervention [24]. Others have argued that a single system cannot adequately address evidence across a wide range of contexts without being too complex to be useful [25,26]. Until now, GRADE has been used primarily to address topics in the field of medicine [e.g., [23,27,28]]. CYMH is an area in need of a more efficient and effective approach to producing credible knowledge. Furthermore, it is an area in which the potential of the GRADE method has not been evaluated.

Current Study

Against this backdrop, the Provincial Centre of Excellence for Child and Youth Mental Health at the Children's Hospital of Eastern Ontario completed a pilot project to investigate the use of a streamlined version of the GRADE approach. Autism intervention research was identified as a suitable and interesting area in which to pilot test this methodology. Specifically, this streamlined method was used to determine current credible knowledge concerning the effects of parent-implemented, early intervention for children with autism. Implementing a methodology such as GRADE has considerable implications for improving efficiency, communication, and clinical decision-making. GRADE may facilitate the progression of the CYMH field toward greater use of evidence-based practice by service providers. This pilot study is the first step in evaluating the feasibility of this methodology as applied to CYMH research and determining whether any modifications are necessary in its application to this field. The findings related to autism as well as the strengths and limitations of the approach will be discussed.

Autism Research

The history of controversy surrounding autism makes this area suitable for this pilot project and is certainly illustrative of the importance of an established methodology for determining credible knowledge in the field of CYMH. Autism research has, for example, seen the persistence of a dispute surrounding evidence concerning causal linkages associated with the measles-mumps-rubella (MMR) vaccine or the vaccine preservative thimerosal. Notwithstanding, there has been no evidence to support an association between an MMR vaccine and autism, nor is there a plausible biological mechanism [as reviewed by 29]. Valuable time and resources were used to conduct as many as twenty studies to definitively respond to a study [30], whose arguments had many flaws that need not be explored here [29,31,32].

There has also been intense debate surrounding the evidence base and suitability of the variety of educational strategies and interventions that have been developed for individuals with autism [33-37]. For instance, controversy exists regarding the necessary intensity of intervention to achieve positive outcomes, the use of certain outcomes and the efficacy of one program compared to another [38]. Some approaches have also been popular with parents, but many professionals have not embraced them enthusiastically, again leading to considerable controversy [39]. While debate is clearly healthy in furthering the scientific research in relation to these practices, the level of uncertainty in this area has resulted in professionals, parents, and policy makers receiving confusing and sometimes unreliable information about the relative effectiveness of interventions [6].

Narrative reviews of interventions in this area have sought to address this problem by identifying evidence-based treatments. Unfortunately, many of these reviews have not taken a systematic approach, limiting their comprehensiveness and validity [33,35,40-44]. Alternatively, systematic reviews have included a wider range of evidence but most have also been found to suffer from methodological weaknesses that make them susceptible to bias [45]. Other reports have taken a systematic approach to the scientific evidence along with expert clinical opinion and took many years to complete. For instance, the National Autism Centre [46] recently completed a review of effective treatments for autism. This review took four years and required substantial resources. Still, others using this approach could not provide precise recommendations regarding specific interventions.

The National Research Council [36] and New York State Department of Health [47] each conducted a systematic review and developed clinical practice guidelines intended to identify effective practices. These reports did not recommend specific interventions. The reports cited a lack of well-controlled studies and an inability to compare interventions between studies. This was stated to be a result of varying descriptions of interventions, participants, treatment fidelity and other methodological considerations. Instead, both reports identified several key features of successful approaches to the education of children with autism, including early intervention after diagnosis, intensive instructional programming, as well as parental involvement in educating their child.

Parent-implemented interventions have been a particular focus in recent reviews, as increased parental skills allow for continual opportunities for children's learning across a range of situations [48,49]. Even in using a more narrowly defined scope of interventions for children with autism, the authors of these reviews were unable to provide recommendations as to specific interventions and offered few implications for practice. This study aims to show that a modified GRADE approach can improve on these attempts at developing recommendations when applied to a CYMH topic.

The Grade Approach

The process used for this pilot project was adapted from that described by the GRADE working group and is more fully described elsewhere [19,24,50-53]. The approach represents an explicit assessment of the quality of evidence, the balance between benefits and risks and the strength of recommendations. Separating the judgments of the quality of the evidence and the strength of recommendations is a critical and defining aspect of GRADE.

Adjustments to the process were made throughout the course of the pilot project to adapt the method in answering a CYMH question. Any changes to the process are noted. The question selected for this pilot project was: What is the current credible knowledge concerning the effects of parent-implemented, early intervention for their children with autism?

A detailed literature search was performed for existing systematic reviews that addressed this question. Unfortunately, systematic reviews by Diggle, McConachie & Randle [48] and McConachie & Diggle [49] related to this question used eligibility criteria that may have been too narrow. A limited number of heterogeneous studies were identified that were not comparable, and did not lead to specific and precise recommendations [54]. Specifically, the former excluded studies that used non-random allocation of groups, while both studied children aged only one to six years with any diagnosis on the autism spectrum. Broader criteria could lead to more informative recommendations by including non-randomized studies and a wider age range, resulting in a broader range of interventions reviewed. As a result, eligibility criteria were specified independent of the systematic reviews, as follows:

(1) Families had to have at least one child who received a formal diagnosis of autism.

(2) Parent-implemented: At least one parent was formally trained to deliver an intervention for their child with autism, which was different in some way from any intervention(s) delivered by service providers.

(3) Early intervention: At the time of study initiation, within each study group at least one recipient of a parent-implemented intervention had been under eight years of age.

(4) Research designs: prospective and controlled.

(5) All publication dates considered.

(6) Effects: any outcomes (e.g., child, interaction, parent, system-related), which includes benefits and harms.

Based on these criteria, an independent information specialist searched the following databases: ERIC, The Cochrane Library, MEDLINE, EMBASE, PsycInfo, CINAHL, and Dissertation Abstracts International. Other sources of information were examined including the bibliographies of systematic and narrative reviews and reference lists of key articles identified through the search strategy. Screening by three independent screeners of the results yielded 11 primary studies to be included [55-65]. This deviates from the original GRADE method in that the evidence-base is typically independently established by an existing systematic review in a GRADE review. Given the issues with the existing systematic reviews on this topic (noted above), the procedure was modified in the current study to have an independent information specialist search for individual studies. This subsequently modifies the procedure for appraising the evidence, as described below.

A panel, which included three clinical and methodological experts in autism research, was then formed. In this pilot project, one role of the panel was to serve as a proxy for end users in determining the values and preferences of those end users in rating the importance of outcomes. The inclusion of additional members was beyond the resources of this work.

Outcome Ratings

In addition to using the panel's experience, an extensive literature search was performed to identify preference estimates from population-based studies in order to gain additional, evidence-based, perspectives in rating the importance of outcomes. These studies are intended to provide information that reflects the values, beliefs and preferences of those affected by autism. The panel could use these studies in rating the importance of outcomes. The panel then created a list of potentially important outcomes to rate, independent of the primary studies. Outcomes are rated on a scale from 1-9: a rating of 7-9 indicating that the outcome is critical for a decision or recommendation; a rating of 4-6 indicating it is important; and, a rating of 1-3 indicating it is not important.

Quality of Evidence

Quality of the evidence was then assessed using the GRADE four-category system (high, moderate, low and very low quality). The quality of the evidence reflects the extent to which one can be confident that the estimates of the benefits and harms of a researched intervention are accurate. Evidence based on randomized controlled trials begins with a high quality rating, observational studies with a low rating and any other evidence with a very low rating. However, studies may then be upgraded or downgraded. Five limitations--related to study quality, consistency, directness, precision, and reporting bias--may lead to downgrading. Upgrading the quality of the evidence may result from large effects, dose-response gradient and if all plausible confounding variables or biases would decrease the demonstrated effect, such as only sicker patients receiving an experimental intervention, yet still faring better. It is important to note that evidence that is downgraded for any reason cannot be upgraded. To ensure transparency and objectivity, the panel provided explicit reasons for downgrading and upgrading by including footnotes in the evidence profiles; this is a requirement of the GRADE system.

Due to the heterogeneity of the evidence base for the present application, namely that each study evaluated a different intervention using a different set of outcomes along with variations in the specific populations studied and those implementing the interventions, study results could not be combined nor directly compared. This diverged from the typical GRADE method where studies evaluating the same outcomes are combined and the quality of each outcome is then evaluated separately. In the present modified version of GRADE, studies were individually appraised for quality on a number of dimensions (see Table 1). The labels high, moderate, low and very low quality evidence were then applied to individual studies as opposed to a synthesized body of evidence, as in the original GRADE method. For each intervention considered, the panel then formulated a consensus recommendation based on the panel members' judgments regarding the balance between the benefits, harms, and values and preferences (i.e., the desirability or preference that individuals exhibit for a particular outcome) of the intervention. Due to the limited resources of this pilot-project, information on costs (i.e., resource utilization) were not acquired, and they were not considered when making a recommendation.

Strength of Recommendations

The strength of recommendations was then determined according to the tradeoffs between benefits and harms of the various alternatives. The strength of a recommendation reflects the extent to which one can be confident that the desirable effects of an intervention outweigh the undesirable effects. Although the level of confidence is a continuum, the GRADE system offers two levels of recommendations for or against treatments (i.e., strong and weak), according to the extent of difference between benefits and harms. The strength of a recommendation is also influenced by the values and preferences of end users, in that critical outcomes are the primary factors for establishing a recommendation. This allows the GRADE system to emphasize that weak recommendations in the face of high quality evidence and vice versa, can occur. This is because other factors, beyond the quality of the evidence, including the values and preferences of the relevant population and the importance of the outcomes, impact the strength of a recommendation [20]. Given that the heterogeneity of the evidence did not allow for a synthesis of the evidence, the labels strong and weak were applied to individual interventions, as opposed to the body of the evidence as in the original GRADE method.

Recommendations included details about the population and intervention. This should aid in the interpretation of the strength of the recommendations and help clinical decision-makers deal with individual patients. Given that some of the judgments leading to the recommendations are subjective, the decisions made are explicit and transparent in the remarks that go along with the recommendations. Specific suggestions for new studies to be undertaken were also noted (see example in Additional file 1).


It took approximately four months to complete this pilot project. A clear answer to the original question as to which parent-implemented intervention is the most effective, was not reached due to the heterogeneity of the studies included (see Table 2 for a description of the interventions). The content of the interventions evaluated differed in almost all studies, while different sets of outcome measures were also used. Any combination or meta-analysis of the data was simply not possible. As a result, specific recommendations were made for each intervention studied. These detailed recommendations included a summary of each piece of evidence with emphasis on the key characteristics (e.g., age group studied, outcomes showing improvement, etc.) influencing a recommendation identified by the panel (see example in Additional file 1).

Overall, the quality of the evidence ranged from high to very low, based on seven randomized controlled trials and four observational, or quasi-experimental, studies (see Figure 1). The factors that led to downgrading the quality of some studies included a lack of blinding, inappropriate matching of control groups and/or lack of clarity in describing the population, intervention and/or study design. A "weak positive" recommendation was made for each of the eight interventions compared to a no intervention or local services control group. A "weak positive" recommendation was also made for one intervention in each of the four studies comparing interventions; one study compared an intervention to both a group receiving local services and a group receiving a different intervention. All of the studies provided some evidence in favour of the studied intervention, as each demonstrated some positive effects and the remaining showing no significant change. Notably, no study found evidence of any harmful effects of the interventions (see Table 3 for a summary of the evidence and recommendations). Given that costs were not considered and each study demonstrated positive effects and no harms, our results suggest that parent-implemented interventions for children with autism are likely to be beneficial. The exclusion of costs is very likely to have had an effect on the strength of recommendations, as studies were of varying quality, though evidence of benefits on an outcome were required in order to recommend an intervention. By completing this process, gaps and weaknesses in this specific area of research were also identified.


This pilot project provided evidence of the potential utility and feasibility of the streamlined GRADE process in the field of CYMH. Being the first implementation of this method in this area, it provides a unique contribution, though modifications to the process were required and certain limitations will have to be addressed in future studies in order for the GRADE method to achieve its potential in this field. A synthesis of the evidence relating to parent-implemented interventions for their child with autism was completed, along with an objective evaluation of this evidence. Following the evaluation of the evidence, recommendations were developed in detail to facilitate interpretation by decision makers for whom the recommendations are intended. This reporting should ensure that decision makers have all of the information required to judge the quality of a recommendation, determine its applicability, and adapt it, if necessary. Overall, it appeared that early, parent-implemented interventions for autism result in positive effects that outweigh any undesirable effects (though publication bias was not assessed and could have impacted the results of this study). However, as opposed to an overall recommendation for these interventions as a whole, specific recommendations had to be made based on the question of whether a particular intervention's benefits outweighed its harms, a necessary departure from the original GRADE method.


The first limitation may also be considered one of the strengths of this study. The GRADE approach was being implemented in a new field of research. Yet, as a result, there were no previous detailed applications or standards to draw from in the literature and modifications to the process were required. Some of the lessons learned from the current application of GRADE should inform future implementations of this approach in CYMH. This may also suggest that another panel may not reproduce the same results. In fact, others may disagree with aspects of our application to GRADE in this area of research and the subsequent grading of the quality of evidence and strength of recommendations. However, the transparency maintained by this process should allow others to understand the decisions made by this panel. For example, studies may be downgraded due to a lack of precision when there are very few patients or events, but there are no defined standards for these terms. Consequently, this panel followed the GRADE principle that a study may require greater precision in support of a recommendation when there is a close balance between the advantages and disadvantages of an intervention [66]. Given the lack of harmful events in the studied interventions, this was not the case here. Furthermore, the current panel consistently recommended the use of interventions for which there was no evidence of harm and some demonstrated benefit on one of the important outcomes. This is a particular judgment on the part of the current panel that has impacted the recommendations. Taken together, future iterations of this process may use panels with different values and judgments that would raise the threshold for demonstrated benefit as well as decide on a minimum sample size, which, if not met, would be downgraded for a lack of precision. Regardless of these decisions, they should continue to be made explicit and applied consistently.

The lack of variability in the composition of the panel may also be considered a limitation in the present application of GRADE. This panel was composed of clinical and methodological experts in the area of autism research. This may have reduced the variability of opinion in establishing the values and judgments and created a conflict of interest in reviewing and appraising evidence in their own field. This became a more prominent factor when the panel was required to appraise individual studies and provide individual recommendations for each intervention as part of the modified GRADE approach. However, these experts also had unique insight into the values and preferences of end-users, for which they were used as a proxy, as well as previous experience in this area of research that facilitated the timely completion of this pilot project. Moreover, these modifications to the process were necessary due to the evidence base and, again, all decisions were made transparent.

Examining the cause of the somewhat unsatisfactory finding, it is seemingly a product of the evidence and not due to any shortcomings in the GRADE method itself. The results of the studies could not be combined or compared due to the high degree of heterogeneity in the evidence base. Two interventions were rarely comparable in terms of the content of the intervention, outcomes measured, specific population studied or those implementing the interventions. Therefore, it was not always possible to construct a consistent account of the effects of an intervention. This is a common finding in mental health, and specifically autism research [33,36,45,48,49,67,68].

Research in this area has been identified as containing numerous gaps and being too heterogeneous to allow for direct comparisons of interventions [36,68]. A number of directions need to be pursued in order to allow for these comparisons. Specific areas to be pursued include: studying a wider range of outcomes; lengthening the time to follow-up assessments; assessing moderators and mediators of treatment outcomes; and, testing varied approaches to treatment delivery or even replicating past research [12,68,69]. An in-depth search of studies addressing these factors in relation to autism interventions was conducted for this pilot project and returned few results [e.g., [70,71]]. Studies addressing these factors would allow for mental healthcare providers and users to make the most informed decisions possible for each individual. Furthermore, the use of common, agreed upon outcome measures including child, parent, family and interactions variables, would facilitate the comparison of studies. Moreover, this research would produce more valuable information by using minimal standards in design and reporting on the interventions and participants. To this end, guidelines could be followed to improve the reporting of how studies were planned, conducted and analyzed [e.g., [72,73]].

The current approaches to synthesizing and evaluating this evidence cannot overcome the limitations in the research. An assumption of these approaches is that the development of the evidence-base has considered the fit between the intervention and its use in clinical practice. In reality, this fit has rarely been taken into account. Moreover, many make the assumption that all randomized control trials are of a high quality since they are generally considered the gold standard in research. The difficulty with this assumption is that there are situations where randomized control trials are impractical or unethical [31,74]. This is particularly true in some areas of CYMH research, where for a variety of logical reasons these designs are infrequently applied, such as autism [6]

In effect, the evidence-base in CYMH is primarily comprised of efficacy studies for which development is based on theory, methods and models to evaluate treatments that do not correspond to the demands of clinic and community-based care [75]. As a result, clinical research, and notably randomized controlled trials, does not provide practitioners with definitive answers for dealing with the individual-at-hand [76]. This may account for the discrepancy between the scientific support of interventions and the lack of their use in these settings [22,77-79].


The GRADE approach explicitly addresses the fit between the research and its use in practice through the functions of the panel, a clear strength. While research should still attend to the factors noted above, GRADE can focus on the clinical relevance of the considerable body of evidence already in existence. Another strength of this approach is that it does not reduce the evidence underlying an intervention to a simple Yes/No conclusion, but it includes information about key variables from the studies in the recommendations. This should aid in the interpretation of the strength of the recommendations and may affect clinical decision-making due to diverse values and preferences of individuals implementing the interventions. The short amount of time required to prepare the recommendations was also a strength of this process. It took four months to complete the process, whereas guideline development and systematic reviews can take up to two years or more [4,9,10].

This approach also benefits from an explicit methodology, transparency, consistency in judgments about the quality of evidence, and linking evidence to recommendations. Though evidence is linked to recommendations, the judgments about the quality of the evidence and the balance of benefits and harms leading to the recommendations are made separately. Not all grading systems separate decisions regarding the quality of the evidence from the strength of recommendations [19]. Those that fail to do so may create confusion. High quality evidence does not necessarily imply strong recommendations, and strong recommendations can arise from low quality evidence. GRADE also compels the panel to consider all relevant outcomes and to adhere to a consistent and comprehensive process in evaluating the evidence. This is a key advantage of this process, especially when considering that guideline developers have been found to lack objectivity due to excessive influence from industry and experts participating in the guideline panels, and to rarely abide by guidelines for the preparation of guidelines [8,80]

It should also be noted that GRADE has now been adopted by a number of organizations including the Cochrane Collaboration and the World Health Organization [28,53]. The increased usage of this methodology and presentation of its results, typically in summary of findings tables, may result in more rapid, clear and consistent translation of evidence. Furthermore, the results of one implementation of GRADE could be used in subsequent reviews quickly and transparently. For example, evidence from a GRADE review of interventions for other disorders on the autism spectrum could potentially be added to this review, while being downgraded for a lack of directness.

These strengths contribute to our experience that GRADE is a feasible approach in CYMH. Its methodology promotes reconciliation between clinicians', methodologists', administrators' and patients' perspectives along with the scientific evidence. Eliminating this traditional separation, seen as a significant limitation in clinical practice guidelines and systematic reviews, will help the evolution of the CYMH field toward the use of evidence-based practice by service providers.

Future Directions

Future implementations of the GRADE approach should include a broader representation of stakeholders in the panel than was used in this pilot project. Other members of the panel could include those of varying levels of service administration such as policy makers, parents, and other practitioners. They would provide more varied perspectives and reduce the gap between the evidence and practice. For example, considering practical barriers to implementation, such as funding restrictions or gaps in clinical expertise could be especially helpful in making the most useful recommendations. In addition, including stakeholders of varying levels would help to drive the knowledge exchange of the findings. Knowledge exchange is an area in and of itself that should be explored to provide guidance on the best procedure for translating results in order to produce meaningful change when using the GRADE method [81]. Other considerations in future uses of GRADE include having the details of the manualization of the intervention as part of the evaluation of methodological quality. Manuals serve the important objective of making the recommended treatments available for clinical training and practice. The specification of sample characteristics in the study could also be evaluated in the GRADE process. These details reflect the recognition that specific treatments may be efficacious only within a limited range of individuals depending on such factors as their age, sex, problem severity, socioeconomic status, and ethnicity [82].


In conclusion, it was determined that it is feasible to develop more timely recommendations in CYMH, specifically early parent-implemented autism interventions, using a modified version of the systematic and transparent GRADE approach. A definitive answer as to which is the best intervention was not produced. However, the GRADE approach appeared capable of providing a specific answer had the evidence been less heterogeneous and directly comparable. Due to the heterogeneity of the evidence a modified GRADE approach was presently evaluated. As research in this area increases and fills in the gaps identified above, it is important to have a method to evaluate the benefits and risks of new interventions, in an efficient manner, as they arise. The GRADE process meets this need and establishes credible knowledge that leads directly to recommendations for effective organization and delivery of CYMH care. Most importantly, GRADE and the recommendations developed from it, will help CYMH service providers and users to make treatment decisions that are informed by the scientific and practice-based evidence.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JMVA conceived of the study, and participated in its design and coordination and drafted the manuscript. JDG, AP, and JAMR were expert panel members and as such contributed significantly to the findings and also contributed to the study design, and edited the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

Supplementary Material Additional file 1

Appendix - Sample Recommendation. Example of recommendation from the pilot-project

Click here for additional data file (1472-6963-11-60-S1.DOC)


The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO is funded by the Ontario Ministry of Children and Youth Services. The views expressed in this paper are those of the author and do not necessarily represent the views of the Ministry of Child and Youth Services. We would like to thank Dr. Ian Manion for his review and assistance in preparation of the manuscript and Dr. Howard Schachter for his work in developing this project. Finally, thanks to the GRADE Working Group for free access to their software.

Barwick MA,Boydell KM,Stasiulis E,Ferguson HB,Blasé K,Fixsen D,Research utilization among children's mental health providersImplement SciYear: 200831910.1186/1748-5908-3-1918400090
Harbour R,Miller J,A new system for grading recommendations in evidence based guidelinesBMJYear: 2001323334610.1136/bmj.323.7308.33411498496
McAlister FA,van Diepen S,Padwal RS,Johnson JA,Majumdar SR,How Evidence-Based Are the Recommendations in Evidence-Based Guidelines?PLoS MedYear: 20074e25010.1371/journal.pmed.004025017683197
National Institute for Health and Clinical ExcellenceThe Guideline ManualYear: 2007London
Scottish Intercollegiate Guidelines NetworkA Guideline Developer's HandbookYear: 2001Edinburgh
Simpson RL,Evidence-based practices and students with autism spectrum disordersFocus Autism Other Dev DisablYear: 20052014014910.1177/10883576050200030201
U.S Department of EducationIdentifying and implementing educational practices supported by rigorous evidence: A user friendly guideYear: 2003Washington, DC
Shaneyfelt TM,Mayo-Smith MF,Rothwangl J,Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literatureJAMAYear: 19992811900190510.1001/jama.281.20.190010349893
Kunz R,What's in the Black Box?ChestYear: 200612971010.1378/chest.129.1.716424405
Raine R,Sanderson C,Black N,Developing clinical guidelines: A challenge to current methodsBMJYear: 200533163163310.1136/bmj.331.7517.63116166137
Grimshaw JM,Hutchinson A,Clinical practice guidelines-do they enhance value for money in health care?Br Med BullYear: 199651927940
Weisz JR,Hawley KM,Finding, evaluating, refining and applying empirically supported treatments for children and adolescentsJ Clin Child PsycholYear: 19982720621610.1207/s15374424jccp2702_79648037
Dawes M,Sampson U,Knowledge management in clinical practice: a systematic review of information seeking behavior in physiciansInt J Med InformYear: 20037191510.1016/S1386-5056(03)00023-612909153
Lavis JN,Ross SE,Hurley JE,Hohenadel JM,Stoddart GL,Woodward CA,Abelson J,Examining the role of health services research in public policymakingMilbank QYear: 2002801255410.1111/1468-0009.0000511933791
Oxman AD,Lavis JN,Fretheim A,Use of evidence in WHO recommdendationsLancetYear: 20073691883188910.1016/S0140-6736(07)60675-817493676
McGlyn EA,Asch SM,Adams J,Keesey BA,Hicks J,DeCristofaro A,Kerr EA,The quality of health care delivered to adults in the United StatesN Engl J MedYear: 20033482635264510.1056/NEJMsa02261512826639
Leatherman S,Sutherland K,Quality of care in the NHS of EnglandBMJYear: 2004328E28829010.1136/bmj.328.7445.E28815087367
Seddon M,Marshall M,Campbell S,Roland MO,Systematic reviews of studies of quality of clinical care in general practice in the UK, Austrialia and New ZealandQual Health CareYear: 200110315215810.1136/qhc.0100152
Atkins DD,Best D,Briss PA,Eccles M,Falck Ytter Y,Flottorp S,Guyatt GH,Harbour RT,Haugh MC,Henry D,Hill S,Jaeschke R,Leng G,Liberati A,Magrini N,Mason J,Middleton P,Mrukowicz J,O'Connell D,Oxman AD,Phillips B,Schunemann HJ,Edejer TT,Varonen H,Vist GE,Williams JW Jr,Zaza S,Grading quality of evidence and strength of recommendationsBMJYear: 20043281490149810.1136/bmj.328.7454.149015205295
Swiglo BA,Murad MH,Schunemann HJ,Kunz R,Vigersky RA,Guyatt GH,Montori VM,A Case for Clarity, Consistency, and Helpfulness: State-of-the-Art Clinical Practice Guidelines in Endocrinology Using the Grading of Recommendations, Assessment, Development, and Evaluation SystemJ Clin Endocrino MetabYear: 20089366667310.1210/jc.2007-1907
West S,King V,Carey TS,Lohr KN,McKoy N,Sutton SF,Lux L,Systems to Rate the Strength of Scientific EvidenceIn Evidence Report/Technology Assessment Volume AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and QualityYear: 2002119911979732
Weisz JR,Weiss B,Donenberg GR,The lab versus the clinic: effects of child and adolescent psychotherapyAm PsycholYear: 1992471578158510.1037/0003-066X.47.12.15781476328
Guyatt G,Schünemann HJ,Cook D,Jaeschke R,Pauker S,Bucher H,Grades of Recommendation for Antithrombic AgentsChestYear: 2004119Suppl 137
Schünemann HJ,Fretheim A,Oxman AD,Improving the use of research evidence in guideline development: 1. Review of guidelines for guidelinesHealth Res Pol SysYear: 200641321
Roddy E,Zhang W,Doherty M,Arden NK,Barlow J,Birrell F,Carr A,Chakravarty K,Dickson J,Hay E,Hosie G,Hurley M,Jordan KM,McCarthy C,McMurdo M,Mockett S,O'Reilly S,Peat G,Pendleton A,Richards S,Evidence-based clinical guidelines: A new system to better determine true strength of recommendationJ Eval Clin PractYear: 20041234735210.1111/j.1365-2753.2006.00629.x
Baumer JH,Guideline Review: Autism spectrum disorders, SIGNArch Dis Child Educ Pract EdYear: 20089316316610.1136/adc.2008.14593818809695
Brozek JL,Baena-Cagnani CE,Bonini S,Canonica GW,Rasi G,van Wijk RG,Zuberbier T,Guyatt G,Bousquet J,Schunemann HJ,Methodology for development of the Allergic Rhinitis and its Impact on Asthma Guideline 2008 updateAllergyYear: 200863384610.1111/j.1398-9995.2007.01560.x18053015
De Palma R,Liberati A,Ciccone G,Bandieri E,Belfiglio M,Ceccarelli M,Leoni M,Longo G,Magrini N,Marangolo M,Roila F,Developing Clinical Recommendations for Breast, Colorectal, and Lung Cancer Adjuvant Treatments Using the GRADE System: A Study From the Programma Ricerca e Innovazione Emilia Romagna Oncology Research GroupJ Clin OncolYear: 2008261033103910.1200/JCO.2007.12.160818309939
Gerber JS,Offit PA,Vaccines and Autism: A Tale of Shifting HypothesesClin Infect DisYear: 20094845646110.1086/59647619128068
Wakefield AJ,Murch SH,Anthony A,Linnell J,Casson DM,Malik M,Dhillon AP,Thomson MA,Harvey P,Valentine A,Davies SE,Walker-Smith JA,Ileal-lymphold-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in childrenLancetYear: 199835163764110.1016/S0140-6736(97)11096-09500320
Davidson KW,Trudeau KJ,Ockene JK,Orleans CT,Kaplan RM,A Primer on Current Evidence-Based Review Systems and Their Implications for Behavioral Medicine. Ann Behav MedYear: 20042822623815576261
Offit PA,Quarles J,Gerber MA,Hackett CH,Marcuse EK,Kollman RT,Gellin BG,Landry S,Addressing parents' concerns: do multiple vaccines overwhelm or weaken the infant's immune system? PediatricsYear: 2002109124911773551
Dawson G,Osterling J,Guralnick MJEarly intervention in autismThe Effectiveness of Early InterventionYear: 1997Baltimore, MD: Paul H Brookes307324
Eikeseth S,Outcome of comprehensive psycho-educational interventions for young children with autismRes Dev DisabilYear: 20093015817810.1016/j.ridd.2008.02.00318385012
Gresham FM,Beebe-Frankenberger ME,MacMillan DL,A selective review of treatments for children with autism: Description and methodological considerationsSchool Psychol RevYear: 199928559575
National Research CouncilEducating Children with AutismYear: 2001Washington, DC: National Academy Press
Smith T,Outcome of early intervention for children with autismClin Psycho: Res-PractYear: 19996334910.1093/clipsy/6.1.33
Ospina MB,Seida JK,Clark B,Karkhaneh M,Hartling L,Tjosvold L,Vandermeer B,Smith V,Behavioural and Developmental Interventions for Autism Spectrum Disorder: A Clinical Systematic ReviewPlos OneYear: 20083e375510.1371/journal.pone.000375519015734
McWilliam RA,Controversial Practices: The Need for a Reacculturation of Early Intervention FieldsTop Early Child SpecYear: 19991917718810.1177/027112149901900310
Connor M,A Review of Behavioural Early Intervention Programmes for Children with AutismEduc Psychol PractYear: 19981410911710.1080/0266736980140206
Green G,Maurice C, Green G, Luce S. Austin TXEarly behavioral intervention for autism. What does research tell us? Behavioral intervention for young children with autismA manual for parents and professionalsYear: 1996Pro-Ed2944
Harris SL,Autism and Pervasive Developmental DisordersYear: 1998Cambridge: Cambridge University Press
Perry A,Condillac RA,Evidence-based practices for children and adolescents with Autism Spectrum Disorders: Reviews of the literature and practice guideYear: 2003Toronto, ON: Children's Mental Health Ontario
Probst P,Parent Training in a rehabilitation setting for autistic children: Concepts and outcomesZ Klin Psychol Psychiatr PsychotherYear: 200149132
Seida JK,Ospina MB,Karkhaneh M,Hartling L,Smith V,Clark B,Systematic reviews of psychosocial interventions for autism: an umbrella reviewDev Med Child NeurolYear: 2009519510410.1111/j.1469-8749.2008.03211.x19191842
National Autism CentreNational Standards ReportYear: 2009Randolph, Mass
New York State Department of Health Early Intervention ProgramClinical practice guideline: The guideline technical report autism/pervasive developmental disordersYear: 1999New York: Author
Diggle T,McConachie H,Randle V,Parent implemented early intervention for young children with autism spectrum disorderCochrane Database Syst RevYear: 200312535477
McConachie H,Diggle T,Parent implemented early intervention for young children with autism spectrum disorder: a systematic reviewJ Eval Clin PractYear: 20071312012910.1111/j.1365-2753.2006.00674.x17286734
Atkins D,Eccles M,Flottorp S,Guyatt GH,Henry D,Hill S,Liberati A,O'Connell D,Oxman AD,Phillips B,Schünemann HJ,Edejer TT,Vist,Williams JW Jr,the GRADE Working GroupSystems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approachesBMC Health Serv ResYear: 200443810.1186/1472-6963-4-3815615589
Atkins D,Briss PA,Eccles M,Flottorp S,Guyatt GH,Harbour RT,Hill S,Jaeschke R,Liberati A,Mangrini N,Mason J,O'Connell D,Oxman AD,Phillips B,Schünemann HJ,Edejer TT,Vist GE,Willams JW Jr,the GRADE Working GroupSystems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new systemBMC Health Serv ResYear: 200552510.1186/1472-6963-5-2515788089
Guyatt G,Gutterman D,Baumann MH,Addrizzo-Harris D,Hylek EM,Phillips B,Raskob G,Lewis SZ,Schünemann HJ,Grading strength of recommendations and quality of evidence in clinical guidelines: report from an American College of Chest Physicians Task ForceChestYear: 200612917418110.1378/chest.129.1.17416424429
Guyatt G,Vist G,Falck-Ytter Y,Kunz R,Magrini N,Schunemann H,An emerging consensus on grading recommendations?Evid Based MedYear: 2006112410.1136/ebm.11.1.2-a17216711
Wallen M,Stagnitti S,There was insufficient evidence to conclude whether parent-mediated early intervention was effective for children with autismAust Occup Therap JYear: 20065313713910.1111/j.1440-1630.2006.00557.x
Aldred C,Green J,Adams C,A new social communication intervention for children with autism: a pilot randomised controlled treatment study suggesting effectivenessJ Child Psychol PsycYear: 2004451420143010.1111/j.1469-7610.2004.00338.x
Drew A,Baird G,Baron-Cohen S,Cox A,Slonims V,Wheelwright S,Swettenham J,Berry B,Charman T,A pilot randomised controlled trial of a parent training intervention for preschool children with autismEur Child Adoles PsyYear: 20021126627210.1007/s00787-002-0299-6
Howlin P,Rutter M,Berger M,Hemsley R,Hersov L,Yule W,Treatment of Autistic ChildrenYear: 1987New York: John Wiley & Sons
Koegel RL,Bimbela A,Schreibman L,Collateral effects of parent training on family interactionsJ Autism Dev DisordYear: 19962634735910.1007/BF021724798792265
McConachie H,Randle V,Hammal D,Le Couteur A,A Controlled Trial of a Training Course for Parents of Children with Suspected Autism Spectrum DisorderJ PediatrYear: 200514733534010.1016/j.jpeds.2005.03.05616182672
Neef NA,Pyramidal Parent Training by PeersJ Appl Behav AnalYear: 19952833333710.1901/jaba.1995.28-3337592149
Ozonoff S,Cathcart K,Effectiveness of a home program intervention for young children with autismJ Autism Dev DisordYear: 199828253210.1023/A:10260068183109546299
Ricards AL,Walstab JE,Wright-Rossi A,Simpson J,Reddihough DS,A Randomized, Controlled Trial of a Home-Based Intervention Program for Children with Autism and Developmental DelayJ Dev Behav PediatrYear: 20072830831610.1097/DBP.0b013e318032792e17700083
Schreibman L,Kaneko WM,Koegel RL,Positive affect of parents of autistic children: a comparison across two teaching techniquesBehav TherYear: 19912247949010.1016/S0005-7894(05)80340-5
Tonge B,Brereton A,Kiomall M,MacKinnon A,King N,Rinehart N,Effects on Parental Mental Health of an Education and Skills Training Program for Parents of Young Children with Autism: A Randomized Controlled TrialJ Am Acad Child PsyYear: 20064556156910.1097/01.chi.0000205701.48324.26
Wang P,Effects of a Parent Training Program on the Interactive Skills of Parents of Children with Autism in ChinaJ Pol Pract Intel DisabilYear: 200859610410.1111/j.1741-1130.2008.00154.x
Guyatt GH,Oxman AD,Kunz R,Vist GE,Falck-Ytter Y,Schunemann HJ,Rating Quality of evidence and strength of recommendations. What is "quality of evidence" and why is it important to clinicians?BMJYear: 200833699599810.1136/bmj.39490.551019.BE18456631
Canadian Paediatric SocietyEarly intervention for children with autismPaed Child HealthYear: 20049267270
Jensen PS,Weersing R,Hoagwood KE,Goldman E,What is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft UnderbelllyMent Health Serv ResYear: 20057537410.1007/s11020-005-1965-315832693
Weisz JR,Jensen AL,Child and adolescent psychotherapy in research and practice contexts: Review of the evidence and suggestions for improving the fieldEur Child Adolesc PsychiatryYear: 200110121810.1007/s007870170003
Lechago SA,Carr JE,Recommendations for Reporting Independent Variables in Outcome Studies of Early and Intensive Behavioral Intervention for AutismBehav ModifYear: 20083248950310.1177/014544550730903418525063
Rogers S,Vismara L,Evidence-Based Comprehensive Treatments for Early AutismJ Clin Child Adoles PsycholYear: 20083783810.1080/15374410701817808
von Elm E,Altman DG,Egger M,Pocock SJ,Gøtzsche PC,Vandenbroucke JP,Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studiesBMJYear: 200733580680810.1136/bmj.39335.541782.AD17947786
Begg C,Cho M,Eastwood S,Horton R,Moher D,Olkin I,Pitkin R,Rennie D,Schulz KF,Simel D,Stroup DF,Improving the quality of reporting of randomized controlled trials: the CONSORT statementJAMAYear: 199627663763910.1001/jama.276.8.6378773637
Cartwright N,Are RCTs the Gold Standard?BiosocietiesYear: 20072112010.1017/S1745855207005029
Hoagwood K,Burns BJ,Kiser L,Ringeisen H,Schoenwald SK,Evidence-based practice in child and adolescent mental health servicesPsychiatr ServYear: 2001521179118910.1176/
Feinstein AR,Horwitz RI,Problems in the 'evidence' of 'evidence-based medicine'Am J MedYear: 199710352953510.1016/S0002-9343(97)00244-19428837
Schoenwald SK,Hoagwood K,Effectiveness, transportability, and dissemination of interventions: What matters when?Psychiatr ServYear: 2001521190119710.1176/
Southam-Gerow MA,Some reasons that mental health treatments are not technologies: Toward treatment development and adaptation outside labsClin Psychol Sci PractYear: 20041118618910.1093/clipsy/bph070
Weisz JR,Lab-clinic differences and what we can do about them: I. The clinic-based treatment development modelClin Child Psychol NewsYear: 20001513
Grilli R,Magrini N,Penna A,Mura G,Liberati A,Practice guidelines developed by specialty societies: The need for a critical appraisalLancetYear: 200035510310610.1016/S0140-6736(99)02171-610675167
Grol R,Grimshaw J,From best evidence to best practice; about effective implementation of change in patient careLancetYear: 200336212253010.1016/S0140-6736(03)14546-114568747
Kraemer HC,Frank E,Kupfer DJ,Moderators of treatment outcomes: clinical, research, and policy importanceJAMAYear: 20062961286128910.1001/jama.296.10.128616968853


[Figure ID: F1]
Figure 1 

Quality Ratings. Frequency count of the quality of the evidence as rated by the panel

[TableWrap ID: T1] Table 1 

Factors assessed in critically appraising the evidence

Factors that might decrease quality of evidence Factors that might increase quality of evidence
• Lack of allocation concealment
• Lack of blinding (particularly for subjective outcomes)
• Failure to adhere to an intention to treat analysis
• Stopping early for benefit
• Selective outcome reporting
• Poor matching of groups in nonrandomized studies or inappropriate comparison group (e.g., children with more serious behavioural problems placed in control group)
• Absence of treatment fidelity assessment
• Inadequate description of intervention and those delivering intervention
• Insufficient follow-up period or lack of reporting on losses to follow-up
• Large magnitude of effect
• Plausible confounding, which would reduce a demonstrated effect
• Dose-response gradient (increased intensity of treatment leads to enhanced benefits)

[TableWrap ID: T2] Table 2 

Descriptions of the studied interventions

Study Intervention Sample Size
Aldred et al. (2005) Therapist delivered, manual-based program for training and educating parents in adapted communication. Treatment = 14
Control = 14

Drew et al. (2003) Speech and language therapist delivered program aimed to train parents to develop their child's joint attention skills and joint action routines. Treatment = 12
Control = 12

Howlin & Rutter (1987) A behavioural, linguistic and social training program used to teach parents language modification techniques, management training and behavioural procedures. Treatment = 16
Control = 16

Koegel et al. (1996) Pivotal Response Training, a naturalistic paradigm emphasizing motivational procedures and responsivity to multiple cues to improve parent-child interaction. Treatment = 7
Control = 10

McConachie et al. (2005) "More Than Words" course aimed at improving parents' understanding of autism and facilitating communication. Treatment = 26
Control = 25

Neef (1997) A pyramidal model of parent training by peers to instruct parents in making use of skill training opportunities during daily routines. Treatment = 14
Control = 12

Ozonoff & Cathcart (1998) Individualized TEACCH-based home program focusing on cognitive, academic, and prevocational skills. Treatment = 11
Control = 11

Rickards et al. (2000) Specialist pre-school teacher delivered, home-based program using discussion, direct skills training and adaptation of the home environment to the child. Treatment = 18
Control = 21

Schreibman et al. (1995) Pivotal Response Training, a naturalistic paradigm emphasizing motivational procedures and responsivity to multiple cues to improve parent-child interaction. Treatment = 10
Control = 7

Tonge et al. (2001) Special educator or psychologist delivered, manual-based, parent education and behaviour management intervention. Treatment = 35
Control = 35

Wang (2008) The author delivered a program aimed at improving the interactive skills of parents using applied behaviour analysis and naturalistic teaching principles Treatment = 14
Control = 11

[TableWrap ID: T3] Table 3 

Summary of the Evidence

Study Quality Rating Factors affecting quality Strengths (+)/Weaknesses(-) Recommendation* Best Estimates
Aldred et al. (2005) High + RCT
+ Independent randomization
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
Weak + Improved symptom severity, social communication skills and parent perception of child's communication skills

Drew et al. (2002) Moderate + RCT
+ Appropriate randomization
+ Clear outcome reporting
+ Low risk of bias
+Valid outcome measures
- poorly matched groups (gender)
- lack of blinding
Weak + Improved Communication Skills

Howlin & Rutter (1987) Low + Appropriately matched control group
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
- Quasi-Experimental design
Weak + Improved mother/child speech and interaction

Koegel et al. (1996) High + RCT
+ Appropriate randomization
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
Weak + Improved parents' global style of interaction with their child

McConachie et al. (2005) Low + RCT
+ Appropriately matched control group
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
- Quasi-Experimental design
Weak + Improved children's vocabulary and parents' use of facilitative strategies

Neef (1997) Very Low + Low risk of bias
- Quasi-Experimental design
- Inappropriately matched control group
- Allocation Concealment/blinding
- Incomplete outcome reporting
- Undetermined validity of outcome measures
Weak + Improved parent teaching skills and child skills development

Ozonoff & Cathcart (1998) Very Low + Appropriately matched control group
+ Low risk of bias
+ Valid outcome measures
- Quasi-Experimental design
- lack of blinding
- Incomplete outcome reporting
Weak + Improved overall development, imitation, fine and gross motor skills and nonverbal conceptual skills

Rickards et al. (2000) High + RCT
+ Appropriate randomization
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
Weak + Improved children's IQ and behaviour problems

Schreibman et al. (1995) Moderate + RCT
+ Appropriate randomization
+ Allocation Concealment/blinding
+ Low risk of bias
+ Valid outcome measures
- Incomplete outcome reporting
Weak + Improved parental affect while interacting with their child

Tonge et al. (2001) High + RCT
+ Independent randomization
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
Weak + Improved self-reported parent mental health

Wang (2008) High + RCT
+ Appropriate randomization
+ Allocation Concealment/blinding
+ Clear outcome reporting
+ Low risk of bias
+ Valid outcome measures
Weak + Improved the quality of parent's interactions with their child

* Recommendations are either Strong or Weak and For (+) or Against (-) the studied interventions; RCT = Randomized Controlled Trial

Article Categories:
  • Research Article

Previous Document:  Level of daily physical activity in individuals with COPD compared with healthy controls.
Next Document:  Anti-T cell immunoglobulin and mucin domain-2 monoclonal antibody exacerbates collagen-induced arthr...