Debating the utility of computerised neurocognitive testing in the sports concussion arena.
Abstract: The purpose of this article was to contribute to an argument regarding the utility of computerised baseline and follow-up neurocognitive testing within the sports concussion arena. Heated debate around this issue via a number of contributions has appeared recently in the journal Current Sports Medicine Reports, with its use being roundly condemned by one party as 'scientifically unfounded' and therefore 'financially irresponsible'. It is proposed that this vehemently negative viewpoint is located in a 'smoke and mirrors' portrayal of the validity of such neurocognitive screening, being substantiated on questionable extrapolations from laboratory-type group research to the clinical situation. The stance runs counter to the tenets of modern clinical neuropsychology, and is incompatible with more rigorous scientific pointers from current research. Abreast of the latest concussion in sport consensus recommendations, it is concluded that there is compelling support for the burgeoning use of computerised neurocognitive evaluation in the sports concussion arena as the optimal and most responsible healthcare currently available in this arena.
Article Type: Report
Subject: Sports injuries (Diagnosis)
Diagnosis (Methods)
Diagnosis (Technology application)
Neurology (Methods)
Neurology (Technology application)
Computer-aided medical diagnosis (Methods)
Brain (Concussion)
Brain (Diagnosis)
Author: Shuttleworth-Edwards, Ann B.
Pub Date: 12/01/2011
Publication: Name: South African Journal of Sports Medicine Publisher: South African Medical Association Audience: Academic Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2011 South African Medical Association ISSN: 1015-5163
Issue: Date: Dec, 2011 Source Volume: 23 Source Issue: 4
Topic: Computer Subject: Technology application
Geographic: Geographic Scope: South Africa Geographic Code: 6SOUT South Africa
Accession Number: 280557371
Full Text: Introduction

In response to Randolph's recent article published in Current Sports Medicine Reports (CSMR), in which he argues against the utility of computerised neurocognitive baseline testing in the sports concussion context, (1) the present author submitted some critical commentary regarding the underpinnings of his argument, including material to suggest that his stance is counter-productive in terms of optimal healthcare and unwarranted. (2) In turn, Randolph and McGrew have presented a number of challenging responses, (3,4) including the statement that 'to advocate for the addition of baseline neuropsychological testing for athletes at this time is scientifically unfounded, financially irresponsible and should be roundly condemned for confusing and raising false expectations in the athletes that are subjected to it as well as adding additional burden to an already overtaxed medical system'. (4) However, considerable concern remains about what appears to be biased and uncritical dismissal of the use of neurocognitive screening in this forum generally by these authors (Randolph and McGrew), and an unwarranted demeaning of the ImPACT test in particular. (1)

In this regard, the present author's previous commentary (2) drew attention to the limitations of evidence that is derived exclusively from group research to facilitate optimal health service, rather than incorporating other crucial factors such as theoretical pointers and clinical case observations given the multitude of genetic, biologic and co-morbid variations as has been eloquently argued elsewhere. (5-7) Further, the author's prior commentary (2) draws attention to Randolph's surprising criticism of the ImPACT test because it produces a multifunction test report rather than yielding a single indicator of brain dysfunction, whereas this can be considered to be one of its most significant strengths. A 'single entity' assessment approach has long) been thoroughly discredited in modern clinical neuropsychology. (8,9) Finally, in terms of the various contributions to the debate in question, Kutcher, (10) in more balanced fashion, warns that Randolph's target in his most recent article is 'conspicuously narrow', and that the practitioner is advised to take cognizance of the complexity of the issue, being 'as leery of fool's gold as dirty bathwater'. (10)

Accordingly, the purpose of the present commentary is to expand on the intricacies involved in this issue, with particular reference to Randolph's most recent article published in the CSMR. (1) The argument to be presented accords with sentiments expressed in a rebuttal to Randolph's earlier article in the JAT, (11) in which it was stated that 'the article does not present a balanced and representative review of the literature ... and does nothing to move our understanding of concussion forward ... In addition, the authors' opinions are inconsistent with current thought within the field of neuropsychology'. (12)

Statistical misrepresentation in light of clinical applications

Extracted from a study of Broglio et al., (13) data are presented by Randolph to bolster an argument about the poor test-retest reliability and relative sensitivity of the ImPACT test. (1,3) Much is made of a 40% false positive rate of impairment relative to baseline that occurred on at least one of the four ImPACT test composites as reported in that study. (1,3) Using the analogy of a thermometer, it is suggested that one would not wish to use such an instrument if it were so unreliable as to 'classify 40% of healthy individuals as pyretic and 40% of pyretic individuals as healthy'. (3) However, there are a number of serious difficulties inherent in these extrapolations, as follows.

Firstly, in the Broglio et al. study, three different computerised neurocognitive tests were administered to testees at baseline, day 45 and day 50. However, in the regular clinical situation, it is only a single neurocognitive test that is administered at baseline, and likewise on each follow-up test occasion. Therefore, uncritical extrapolation of research outcome from the Broglio study to document clinically applicable test-retest reliability, false positive indications of impairment, etc., in respect of any of these three programmes in isolation, has seriously compromised validity on methodological grounds, due to the confounding effect on each other of the multiple test administrations.

Further, from a neuropsychological perspective, to draw an analogous comparison between the single score measurement of a thermometer and a psychometric instrument is inappropriate. Psychometric test scores do not have stand-alone positivistic meaning in their own right (as with a temperature thermometer), rather being a relativistic type of measurement requiring highly contextualised interpretation in the hands of those who have been trained in their use. (14) A single change score in isolation on a test such as ImPACT, that incorporates at least four core composite scores in the clinical profile, (15) would be viewed sceptically in terms of its implications, and interpreted closely in relation to: the site of the injury; the symptom reports; the educational/ occupational background and special abilities and disabilities of the athlete; and, importantly, norms that are applicable to the testee in terms of the demographic influences of age, education and cultural affiliation. (16)

Essentially, the point that is being made here is that false positive indications derived from any scores, and particularly a single score, in the de-contextualised group research context such as pertains to the study of Broglio et al., have minimal relevance in the appropriately conducted clinical neuropsychological examination. Accordingly, Randolph's 40% false positive challenge regarding the use of the ImPACT test is entirely without substance on clinical interpretive grounds, as well as on the most fundamental of methodological grounds.

In contrast to these dubious implications derived from the study of Broglio et al. are indications from many more rigorous studies, and some recent studies in particular, that serve to demonstrate an entirely opposite picture in terms of the discriminatory ability of the ImPACT programme. (17,18) In these studies a high level of predictive value on the length of recovery is established for the ImPACT programme of 73% taking both the cognitive tests and symptom reports into account, with an increase of 24% in sensitivity over the use of the symptom reports alone. (17) Early postconcussion neurocognitive assessment on ImPACT in an emergency department detected neurocognitive deficits that clinical grading could not, and correlated with the severity of deficit at 3- and 10-day post-injury follow-up. (18)

Benefits of neuropsychological evaluation and concluding comments

Research outcome aside, there are non-debatable clinical advantages in having neuropsychological post-concussion evaluations available in the hands of the trained neuropsychologist. Such benefit includes the ability to demonstrate the extent of cognitive malfunction during the recovery process in order to provide guidelines for scholastic or occupational purposes. Even fairly brief periods of unrecognised and/ or unsubstantiated cognitive dysfunction in the wake of a concussion may have quite serious negative consequences within such settings, and calls for highly individualised recommendations supported by objective criteria. These goals are massively facilitated by the availability of follow-up computerised test data perse, and especially in comparison with baseline data.

In sum, taking all the above into consideration, in direct opposition to the expressed sentiments of Randolph and McGrew, the ImPACT test appears to be exactly the kind of tool that the present author would like to add to an investigative armamentarium with a view to enhancing medical management in respect of the sports concussive injury. It has clearly been endorsed by hundreds of clinicians worldwide in the burgeoning use of this particular tool.

However, as cautioned by Kutcher, (10) whether using ImPACT or some other instrument of this type such as ANAM, CogState Sport or Headminder, this procedure should not be followed blindly as part of a marketing ploy. Nor should the process be seen as a shortcut to ruling brain dysfunction in or out. (14,19) Rather it should take place via discretionary clinical evaluation of the optimal mechanism to employ in order to meet the health needs within the particular sports concussion arena, by a team of suitably qualified medical and neuropsychology practitioners. As such, the practice is fully in accordance with the recommendations of the most recent consensus on concussion in sport. (20) Used appropriately within the confines of legitimate practice in clinical neuropsychology, the incorporation of computerised neurocognitive screening can be considered to have positively revolutionised the management of the sports concussive injury in the direction of becoming massively more refined than it was previously, thereby allowing for more optimal and responsible recovery and rehabilitative healthcare.

A note on the ImPact programme

The ImPACT programme was developed within a research context approximately ten years ago, (15) and is currently in its 4th updated web-based and technologically sophisticated version. It consists of a user-friendly series of tests and a symptom check list that take about 25 minutes to complete, and produces an automated report that collates the neurocognitive and symptom outcome, for multiple test-taking occasions. There is a spectrum of functional modalities assessed based on traditional well-researched neuropsychological stimuli, including visual and verbal memory (immediate and delayed), visual motor speed, reaction time, as well as a cognitive efficiency index, and a test of impulse control that provides an indicator of test-taking validity. Restricted web-based access to the report by designated clinicians enables high level neuropsychological interpretation of the outcome and feedback within a matter of hours, regardless of location of the testee. The test is acknowledged as the most widely used of all computer programmes that have been especially devised for use in the sports concussion arena. (1)


Acknowledgements for funding of the author's concussion research programme are due to the National Research Foundation (NRF) and the Joint Research Council of Rhodes University, Grahamstown, South Africa.

Declaration of conflict of interests

The author has been involved in concussion management in South Africa and the UK for the past ten years, using the ImPACT programme for clinical and research purposes.


(1.) Randolph C. Baseline neuropsychological testing in managing sport-related concussion: does it modify risk? Curr Sports Med Rep. 2011;10(1):21-26.

(2.) Shuttleworth-Edwards A. Response to the article on baseline testing: Throwing away clinical gold with the statistical bathwater. Curr Sports Med Rep 2011;10(6):391-392.

(3.) Randolph C. Letter to Editor-in-Chief. Curr Sports Med Rep. 2011;10(6):393.

(4.) McGrew CA. Letter to Editor-in-Chief. Curr Sports Med Rep. 2011; 10(6):394.

(5.) Miller DW, Miller CG. On evidence medical and legal. J Am Phys Surg 2005;10(3):7075.

(6.) Upshur EG. Looking for rules in a world of exceptions. Reflections on evidence-based practice. Perspect Biol Med 2005;48:477-489.

(7.) Dattilio F, Edwards D, Fishman D. Case studies within a mixed methods paradigm: Toward a resolution of the alienation between researcher and practitioner in psychotherapy research. Psychotherapy, Theory, Research, Practice, Training 2010;47(4):427-441.

(8.) Walsh K. Understanding brain damage: A primer of neuropsychological evaluation. 2nd ed. Melbourne: Churchill Livingstone, 1991.

(9.) Lezak MD, Howieson DB, Loring DW. Neuropsychological Assessment. 4th ed. Oxford: Oxford University Press, 2004.

(10.) Kutcher, JS. Letter to Editor-in-Chief. Curr Sports Med Rep. 2011; 10(6): 395.

(11.) Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train 2005;40(3):139-154.

(12.) Lovell MR. Letter to Editor. J Athl Train 2006;41:137.

(13.) Broglio SP, Macciocchi SN, Ferrarra MS. Sensitivity of the concussion assessment battery. Neurosurgery 2007;60(6):1050-1058.

(14.) Shuttleworth-Edwards AB, Border MA. Computer based screening in concussion management: use versus abuse. Br J Sports Med 2002;36:473.

(15.) Iverson GL, Lovell MR, Collins MW. Immediate Postconcussion Assessment and Cognitive Testing (ImPACT) normative data version 2.0. Pittsburgh, USA: ImPACT Applications Inc.; 2002.

(16.) Shuttleworth-Edwards AB, Whitefield-Alexander VJ, Radloff SE, Taylor AM, Lovell MR. Computerized neuropsychological profiles of South African versus United States athletes: A basis for commentary on cross-cultural norming issues in the sports concussion arena. Physician and Sports Med 2009;37(4):45-51.

(17.) Lau B, Collins M, Lovell M. Sensitivity and specificity of subacute computerized neurocognitive testing and symptom evaluation in predicting outcomes after sports-related concussion. Am J Sports Med 2011;39(11):2311-231n Epub ahead of print.

(18.) Thomas D, Collins M, Saladino R, Frank V, Raab J, Zuckerbraun N. Identifying neurocognitive deficits in adolescents following concussion. Acad Emerg Med 2011;18(3):246-254.

(19.) Echemendia R, Herring S, Bailes J. Who should conduct and interpret the neuropsychological assessment in sports-related concussion. Br J Sports Med 2009;43:i32 i35.

(20.) McCrory P, Meeuwisse W, Johnston K, et al. Consensus statement on concussion in sport: the 3rd International Conference on Concussion in Sport held in Zurich, November 2008. Br J Sports Med 2009;43:i76-i84.

Ann B Shuttleworth-Edwards, PhD

Professor of Psychology, Director National Sports Concussion Initiative (NSCI), Department of Psychology, Rhodes University, Grahamstown, South Africa

Correspondence to: Professor Ann Edwards (
Gale Copyright: Copyright 2011 Gale, Cengage Learning. All rights reserved.