Response to Barnett et al. Small N designs for rehabilitation research. J Rehabil Res Dev. 2012;49(1):175-86.
Article Type: Letter to the editor
Subject: Medicine, Physical (Research)
Medicine, Physical (Laws, regulations and rules)
Rehabilitation centers (Laws, regulations and rules)
Author: McMillan, Garnett
Pub Date: 03/01/2012
Publication: Name: Journal of Rehabilitation Research & Development Publisher: Department of Veterans Affairs Audience: Academic Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2012 Department of Veterans Affairs ISSN: 0748-7711
Issue: Date: March, 2012 Source Volume: 49 Source Issue: 3
Topic: Event Code: 310 Science & research; 930 Government regulation; 940 Government regulation (cont); 980 Legal issues & crime Advertising Code: 94 Legal/Government Regulation Canadian Subject Form: Rehabilitation centres Computer Subject: Government regulation
Product: Product Code: 8069200 Rehabilitation Facilities NAICS Code: 62231 Specialty (except Psychiatric and Substance Abuse) Hospitals
Geographic: Geographic Scope: United States Geographic Code: 1USA United States
Accession Number: 293667706
Full Text: Response to Barnett et al. Small N designs for rehabilitation research. J Rehabil Res Dev. 2012;49(1):175-86.

Dear Editor:

I am writing to congratulate Barnett et al. for their article on small N designs, as well as the Journal of Rehabilitative Research and Development for publishing it. Their article fosters an opportunity to formalize the role of pilot studies in rehabilitation research. I believe that there will always be room to improve the efficiency with which novel rehabilitation therapies are developed and advocated. Adequate or perhaps even overwhelming evidence for efficacy is necessary to promote a new therapy if it costs time or money, proposes to replace existing treatments, or causes undesirable side effects. Efficacy is traditionally demonstrated with the randomized controlled trial (RCT), but difficulties in recruitment and retention, problems shared with the pharmaceutical industry [1], often lead to underpowered studies. Furthermore, the background of many rehabilitation interventions do not approach the theoretical foundations of drug studies in which causal mechanisms understood at the molecular level allow one to show drug efficacy in spite of weak design and analysis [2]. These challenges, and others identified by Barnett et al. [3], result in inefficient rehabilitation therapeutic development and promotion in the Veterans Health Administration.

Barnett et al. recognize that, in general, small N or single-case designs alone may not provide sufficient evidence to promote a new therapy. However, small N designs in the context of pilot studies may greatly enhance the efficiency with which new therapies are discovered before testing in a conventional, large-scale, randomized trial.

One might think of a pilot study as an attempt to identify the one therapy among a set of competitors that is most likely to deliver positive results in a large-scale randomized trial. Consider the following design: subjects recruited into the pilot study are randomly assigned to competitors and consistently measured according to the single-case protocol described in Barnett et al. with clearly defined outcome measures. After an arbitrary period of recruitment, the collection of single-case recruitments is analyzed using meta-analysis. Based on the analysis, competitors are ranked in terms of average outcome so that each is assigned a probability of being the most effective among therapies considered. The pilot study continues until the investigator is sufficiently convinced that the best competitor has been identified and, perhaps most importantly, believes that the results of the study will convince reviewers of the value of a large-scale randomized trial. I would recommend a Bayesian hierarchical modeling approach that allows one to use simple, easily interpretable estimates of the relevant parameters [4]. This approach also allows one to add new competitors and reanalyze the accumulated data during the course of the pilot study.

There is a great deal of brain power in the pharmaceutical industry that is being expended on improving the efficiency with which new drugs are approved. However, for a variety of reasons, much of rehabilitation research is not obligated to meet the same regulatory requirements as the pharmaceutical industry. Looser standards in rehabilitation development should be exploited to improve the well-being of rehabilitation patients. The goal of the approach outlined here is to employ the small N design in a pilot study framework to have a more productive, prerandomized trial phase of therapeutic development. Even so, there is more methodological work to do in this area, such as clarifying the roles of self-reported outcomes and unblinded treatment allocation, both of which are common in rehabilitation research. I believe that the work of Barnett et al. is an important step toward more efficient therapeutic development and evaluation.

Garnett McMillan

National Center for Rehabilitative Auditory Research, Portland, OR



[1]. Ledford H. Translational research: 4 ways to fix the clinical trial. Nature. 2011; 477(7366):526-28. [PMID:21956311]

[2]. Briggs M. Why do statisticians answer silly questions that no one ever asks? Significance. 2012;9(1):30-31.

[3.] Barnett SD, Heinemann AW, Libin A, Houts AC, Gassaway J, Sen-Gupta S, Resch AR, Brossart DF. Small N designs for rehabilitation research. J Rehabil Res Dev. 2012; 49(1):175-86. JRRD.2010.12.0242

[4.] Speiglehalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation. Chichester (England): Wiley and Sons; 2004.
Gale Copyright: Copyright 2012 Gale, Cengage Learning. All rights reserved.