Interrater reliability of the Road Law and Road Craft Test as part of the OT-DORA Battery for off-road driver assessment.
Background: The Occupational Therapy Driver Off Road Assessment
(OT-DORA) Battery is in development as a comprehensive off-road
evaluation of fitness to drive. Although several psychometric properties
of the Road Law and Road Craft Test (RLRCT) have been previously
studied, interrater reliability must also be established to include this
test in the battery.
Aim: The aim of the study was to determine the interrater reliability of the RLRCT.
Method: Three occupational therapy driver assessors independently scored 20 client responses on the 14-item RLRCT.
Results: The interrater reliability for all items, except item 6, was found to be between 0.57 and 1.00. The scoring instructions for item 6 were revised and the interrater reliability rose from 0.42 to 0.77.
Conclusion: The RLRCT has been found to be reliable when administered by different raters and is thus suitable for inclusion in the OT-DORA Battery. Ongoing research to confirm the psychometric properties of the RLRCT is required.
Automobile driving, intraclass correlation coefficient.
Item response theory
Occupational therapy (Laws, regulations and rules)
Automobile driving (Analysis)
Motor vehicle driving (Analysis)
Highway law (Analysis)
Unsworth, Carolyn A.
Pallant, Julie F.
|Publication:||Name: British Journal of Occupational Therapy Publisher: College of Occupational Therapists Ltd. Audience: Academic Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2011 College of Occupational Therapists Ltd. ISSN: 0308-0226|
|Issue:||Date: August, 2011 Source Volume: 74 Source Issue: 8|
|Topic:||Event Code: 930 Government regulation; 940 Government regulation (cont); 980 Legal issues & crime Advertising Code: 94 Legal/Government Regulation Computer Subject: Government regulation|
|Geographic:||Geographic Scope: Australia Geographic Code: 8AUST Australia|
Driving provides an important means of independence for many people, by allowing easier access to social supports and medical and retail services, and also fosters a sense of self-worth and autonomy for those who would otherwise find difficulty in accessing these services (Yassuda et al 1997, McGregor 2002). Increases in the standards of living and medical developments have contributed to an ageing population and, in turn, to an increase in the number of drivers aged 65 years or older (Michalik 1990, Hakamies-Blomqvist 1996, Stansfeld et al 1996). As older drivers have an increased likelihood of health-related concerns that may have an impact on driving ability (such as medical conditions, medication and overall health status), they may need an assessment of their fitness to drive (Unsworth et al 2007). In addition, many younger people with medical problems or disabilities are seeking to obtain a driving licence (Stav 2004, Unsworth 2007, Hunter et al 2009, Unsworth 2010) and may also require an assessment of fitness to drive. Hence, there is growing research interest in occupational therapy on driver assessment and community mobility, as evidenced by recent special issues in the American Journal of Occupational Therapy (2010, 64), Canadian Journal of Occupational Therapy (2011, 78) and a forthcoming issue of the Australian Occupational Therapy Journal (scheduled for 2012, 59).
International procedures for driver assessment processes vary. In some countries such as the United Kingdom (UK), non-specialist occupational therapists may refer clients to a mobility centre, where driver assessor occupational therapists work. These non-specialist occupational therapists need an assessment to help determine if a client requires referral to a mobility centre for on-road assessment. In other countries, such as Australia or the United States, driver assessor trained occupational therapists (DATOTs) work in both community and hospital environments and may conduct an off-road assessment followed by an on-road assessment (Stav 2004, Unsworth 2007, Hunter et al 2009, Stapleton and Connelly 2010) and make recommendations to the licensing authorities concerning the fitness to drive of their clients. However, there is no consistent and agreed approach to off-road testing for fitness-to-drive evaluations. Non-specialist occupational therapists, as well as DATOTs, currently use a variety of standardised and unstandardised assessments resulting in inconsistent testing experiences for older drivers, with the potential for fitness-to-drive recommendations that are not evidence based (Unsworth 2007, Stapleton and Connelly 2010). Errors in determining fitness to drive could deprive an individual of independent transport or place the individual, as well as other road users, at risk of trauma or death.
The need for an off-road battery containing the best selection of valid and reliable assessments is clear. In countries such as the UK, an off-road assessment battery would enable non-specialist occupational therapists to make evidence-based referrals for their clients to mobility centres. In addition, DATOTs in the UK and internationally could use such an assessment to help make optimal licensing recommendations to licensing authorities, as all occupational therapists can contribute to promoting community mobility independence (Brown 2009).
Twenty existing assessments either designed for off-road driver assessment or commonly used in such assessments were reviewed (Unsworth et al 2005), as well as the pilot off-road assessment developed by Eby et al (2007). However, none met the selection criteria: being suitable for use internationally by occupational therapists, focused on obtaining information about the driver's potential strengths and limitations prior to an on-road assessment, considering physical, sensory and cognitive skills, taking less than 2 hours to administer and being priced under 100 [pounds sterling]. To remedy this situation, the Occupational Therapy Driver Off-Road Assessment (OT-DORA) Battery has been developed as a standardised assessment that is appropriate for use internationally and can be conducted in approximately 60-90 minutes (Unsworth et al 2010a). The OT-DORA Battery is being published by AOTA Press.
OT-DORA is a comprehensive battery that screens a client's physical, sensory and cognitive/perceptual skills, as guided by competency standards for practice (Occupational Therapy Australia--Victoria 1998). Table 1 provides a summary of all the components of the battery. The cognitive/ perceptual screening section of the battery includes the Occupational Therapy Drive Home Maze Test (Krishnasamy and Unsworth 2011) and the Road Law and Road Craft Test (RLRCT; unpublished, La Trobe University 1990, Unsworth et al 2010b). The 14-item RLRCT assesses road law (including signs) and road craft (concerning how to drive and factors that affect this) knowledge. This test provides insight into the client's problem-solving ability, sustained attention and visual perceptual skills when interpreting the diagrams and pictures included in the test (Unsworth et al 2010b). The test was designed so that it can be used by occupational therapists internationally, with only minor modifications needed to tailor the test to the laws of different countries. For example, item 4 can be adjusted to specify the maximum legal blood alcohol content for driving in each country.
Unsworth et al (2010b) examined the psychometric properties of the RLRCT using a variety of statistical techniques, including Rasch analysis. The distribution of scores on the person-item threshold plot from the Rasch analysis indicated that the RLRCT was reasonably well targeted for the sample, and that there was a good spread of items representing the lower and middle range of scores. There were few difficult items on the test that only people with excellent skills would get correct. This is appropriate because the main focus of the RLRCT is to identify clients with low levels of road law and road craft knowledge. In other words, the test is appropriately targeted at picking up clients who have weaknesses, rather than identifying those with strengths in this area.
Examination of the reliability of the test is also necessary to ensure that not only is the test accurate at identifying clients with weaknesses, but that multiple assessors using the same tool would identify these same clients. In other words, can the RLRCT provide consistent scores for an individual when administered by different therapists (Portney and Watkins 2009)? The reliability of a measure is important to ensure the accuracy of the scores and to support the tool's validity (Nunnally 1970). While both interrater and intrarater reliability are important, asking clients to perform the same test a week after their initial performance poses practical problems.
The aim of the present study, therefore, was to examine the interrater reliability of the RLRCT (Part 1), and to make adjustments to the tool where indicated by the results (Part 2). With interrater reliability established, there can be increased confidence in including the RLRCT in the OT-DORA Battery.
Occupational therapists: Three occupational therapists (referred to as A, B and C in the Method), who were a convenience sample, consented to participate in the study through one large health care network in Melbourne, Australia (population 3.9 million; Australian Bureau of Statistics 2010). Although many clinical interrater reliability studies are performed with only two raters, which is deemed adequate (Shrout and Fleiss 1979, Portney and Watkins 2009), three therapists were included in this study to increase confidence that the test items could be scored in the same manner by different therapists.
Clients: The clients were selected as consecutively consenting adults in a larger study researching the development of the OT-DORA Battery, who were English speaking and could attempt to complete the written RLRCT. Data from a total of 80 clients who had all been assessed by the same therapist were collected, with 20 clients randomly selected to be included in the interrater reliability study.
Demographic information, such as age, gender and relevant medical diagnosis, was collected, as well as the client's scores on the RLRCT (Unsworth et al 2010b). The RLRCT has 14 items and the client receives a total score out of 37, with higher scores indicating greater knowledge. The assessment is presented as a paper booklet and requires the person to read and write in English, or to make use of an interpreter. It is also possible for the therapist to read the questions and record the responses if the client is unable to read and write.
The assessment takes about 10 minutes. It requires the client to respond to eight questions, and then to apply knowledge of road laws and to indicate which vehicle proceeds first in a series of six diagrams that show two or three cars at various intersections. For example, one of the written questions asks the client what he or she should do if involved in an accident, and another asks what hazards should be looked for when driving through a shopping centre car park. One of the diagrams depicts two cars (labelled A and B) approaching each other at an unsigned crossroad intersection and asks the client to indicate in which order the vehicles should proceed.
Ethics approval was sought and obtained from the La Trobe University Human Ethics Committee and the Austin Health Human Research Ethics Committee. The RLRCT was administered to all clients by therapist A.
Part 1: The completed (but unscored) RLRCTs were then photocopied by the researchers and scored independently by therapists A, B and C at separate locations.
Part 2: Once the results from Part 1 were analysed, it was determined that any test items with an intraclass correlation coefficient of lower than 0.50 would be examined; that instructions for such items would be revised; and that the therapists would be asked to re-score 'unmarked' copies of client assessments for this item. Again, this task was undertaken independently by the therapists in separate locations. A 4-week gap between Parts 1 and 2 was deemed adequate to ensure that raters did not remember their original scores for any items.
Intraclass correlation coefficients (ICCs) (Shrout and Fleiss 1979) were used to determine both the correlation and the agreement in the scores given by a multiple number of raters. Shrout and Fleiss (1979) discussed six different forms of ICC, with ICC Type 2, 1 being identified as the most appropriate for the current investigation. In this form of ICC, the sample of raters (therapists) is selected from a larger pool, and each rater scores each target (client responses to each test item). The ICC can establish if scores between raters correlate and that there is agreement (Portney and Watkins 2009). This method has been used extensively in the allied health field to examine the interrater reliability of a variety of measures (Dawson et al 1994, Scott et al 2006, Classen et al 2009). The degree of reliability used in this study was proposed by Portney and Watkins (2009) to measure the strength of correlation coefficients: 0.00 to 0.25 = Little or no relationship; 0.25 to 0.50 = Fair relationship; 0.50 to 0.75 = Moderate to good relationship; and 0.75 and above = Good to excellent relationship.
The three therapist participants were female, with a mean age of 46.67 years (SD 8.50 years, range 37 to 53). All participants held at least a bachelor's degree in occupational therapy, with postgraduate certificate level training in driver assessment. The participants had a mean of 25.33 years of experience as occupational therapists (SD 8.15, range 16 to 31), and a mean of 17 years of experience as driver assessors (SD 7, range 9 to 22).
The client participants were 20 years or older and were functionally impaired drivers who had been referred to an occupational therapist for an assessment of their fitness to drive a motor vehicle. These participants had a mean age of 75 years (SD 9 years, range 55 to 87); 17 of the client participants were male (85%). Half of the participants (10 participants, 50%) had a diagnosis of a neurological condition; five participants (25%) had a diagnosis of Alzheimer's disease or dementia; and the remaining five participants had a variety of other diagnoses, including cancer, cognitive disorders and endocrine disorders.
The ICCs for each of the 14 items in the RLRCT are provided in Table 2. Ten of the items had an ICC of 0.75 or above, suggesting good to excellent reliability, with an additional three items having an ICC that indicated moderate to good reliability Only item 6 had an ICC below 0.50.
Item 6 of the RLRCT was identified as having only 'fair' interrater reliability, and the reason for this was thought to be ambiguity in the scoring description. The original scoring instruction was:
Score 1 point for each appropriate answer in the correct order: 4pts A. Check mirrors. B. Indicate when appropriate. C. Check blind spots. D. Move over when safe to do so.
The marking instruction for item 6 was then revised, with examples provided for raters, as shown in Fig. 1.
After a 4-week period, the three raters were given unmarked copies of each client's answers for item 6, and were asked to re-score using the revised scoring instruction. The ICC for item 6 using the new scoring instruction was calculated as ICC (2, 1) = 0.77, indicating a good to excellent relationship.
Previous investigations using Rasch analysis found the RLRCT to be a unidimensional assessment, with good internal consistency and no evidence of bias across gender or age (Unsworth et al 2010b). The aim of the present study was to determine if the test was also reliable when scored by different raters. Using the relationship measure of the strength proposed by Portney and Watkins (2009), all items except item 6 were found to have moderate to excellent agreement between raters. A review of the scoring for item 6 revealed ambiguities that could account for its 'fair' reliability. The subsequent rewording of the item-scoring description and the second scoring by the three therapists produced good to excellent reliability. Further research is required to confirm the interrater reliability of the RLRCT with larger, international samples of occupational therapists and drivers. Research is also required to establish the intrarater reliability of this test.
Research and clinical progress in the field of driver assessment is dependent on the development of standardised, comprehensive (covering physical, sensory and cognitive/perceptual screening) and inexpensive off-road assessment batteries, with excellent face validity and clinical utility The OT-DORA Battery (Unsworth et al 2010a) has been developed to meet this need Testing has been undertaken (and is continuing) of individual tests included in this battery, as well as of the battery as a whole, to ensure that occupational therapists internationally can administer it to their clients with confidence The research reported in this paper contributes to the development of this battery by commencing research to establish the interrater reliability of the RLRCT
This study has been supported by the La Trobe University Faculty of Health Sciences Grant Scheme and a Sir Edmund Herring Memorial Scholarship Royal Automobile Club of Victoria (RACV) 2007 for the first author.
Conflict of interest: None declared.
* With revised scoring instructions, the RLRCT has been found to possess moderate to excellent interrater reliability.
What the study has added
Combined with previous research validating the use of the RLRCT, the finding from this study that the test has adequate interrater reliability supports inclusion of the RLRCT in the OT-DORA Battery.
Australian Bureau of Statistics (2010) Australian social trends. ABS Catalogue No. 4102.0. Canberra: ABS.
Brown Y (2009) Promoting participation: addressing transport needs within occupational therapy. (Editorial.) British Journal of Occupational Therapy, 72(11), 471.
Classen S, Shechtman O, Stephens B, Davis E, Lanford D, Mann W (2009) The impact of intersection design on the driving performance of adults in the recovery phase of a turn. British Journal of Occupational Therapy, 72(11), 472-81.
Dawson D, Chan R, Kaiserman E (1994) Development of the Power-mobility Indoor Driving Assessment for residents of long-term care facilities: a preliminary report. Canadian Journal of Occupational Therapy, 61(5), 269-76.
Eby DW, Molnar LJ, Shope JT, Dellinger AM (2007) Development and pilot testing of an assessment battery for older drivers. Journal of Safety Research, 38(5) 535-43.
Hakamies-Blomqvist L (1996) Research on older drivers: a review. IATSS Research, 20(1), 91-101.
Hunter J, de Vries J, Brown Y, Hekstra A (2009) Handbook of disabled driver assessment. Ljubljana: Portare Working Group.
Krishnasamy C, Unsworth CA (2011) Normative data, preliminary inter-rater reliability and predictive validity of the Drive Home Maze Test. Clinical Rehabilitation, 25(1), 88-95.
McGregor D (2002) Driving over 65: proceed with caution. Journal of Gerontological Nursing, 28(8), 22-26.
Michalik C (1990) Development and evaluation of measures to reduce accident risk of elderly drivers. IATSS Research, 20(1), 25-31.
Nunnally JC (1970) Introduction to psychological measurement. New York: McGraw Hill.
Occupational Therapy Australia--Victoria (1998) Competency standards for occupational therapy driver assessors. Melbourne: Occupational Therapy Australia -Victoria.
Portney L, Watkins M (2009) Foundations of clinical research: applications to practice. Upper Saddle River, NJ: Pearson/Prentice Hall.
Scott F, Unsworth CA, Fricke J, Taylor N (2006) Reliability of the Australian Therapy Outcome Measures for Occupational Therapy (AusTOMs-OT) Self-care scale. Australian Occupational Therapy Journal, 53(4), 265-76.
Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-28.
Stansfeld S, Gallacher J, Babisch W, Shipley M (1996) Road traffic noise and psychiatric disorder: prospective findings from the Caerphilly study. British Medical Journal, 313(7052), 266-67.
Stapleton T, Connelly D (2010) Occupational therapy practice in predriving assessment post stroke in the Irish context: findings from a nominal group technique meeting. Topics in Stroke Rehabilitation, 17(1), 58-68.
Stav W (2004) Driver rehabilitation: a guide for assessment and intervention. San Antonio, TX: PsychCorp.
Unsworth CA (2007) Development and current status of occupational therapy driver assessment and rehabilitation in Victoria, Australia. Australian Occupational Therapy Journal, 54(2), 153-56.
Unsworth CA (2008) Development of the OT Drive Home Maze Test. Unpublished manual. Melbourne: LaTrobe University.
Unsworth CA (2010) Issues surrounding driving and driver assessment for people with mental health problems. Mental Health Occupational Therapy, 15(2), 41 -44.
Unsworth CA, Lovell R, Terrington N, Thomas S (2005) Review of tests contributing to the occupational therapy off-road driver assessment. Australian Occupational Therapy Journal, 52(1), 57-74.
Unsworth CA, Wells Y, Browning C, Kendig H, Thomas SA (2007) To continue, modify or relinquish driving: findings from a longitudinal study of healthy ageing. Gerontology: International Journal of Experimental, Clinical and Behavioral Gerontology, 53(6), 423-31.
Unsworth CA, Pallant JF, Russell KJ, Odell M (2010a) Draft: The Occupational Therapy Driver Off-Road Assessment (OT-DORA) Battery Manual. Melbourne: La Trobe University.
Unsworth CA, Pallant JF, Russell KJ, Germano C, Odell M (2010b) Validation of a test of road law and road craft knowledge with older or functionally impaired drivers. American Journal of Occupational Therapy, 64(2), 306-15.
Yassuda MS, Wilson JJ, von Mering O (1997) Driving cessation: the perspective of senior drivers. Educational Gerontology, 23(6), 525-38.
Carolyn A Unsworth, (1) Julie F Pallant, (2) Kay Russell, (3) Morris Odell (4) and Melissa Coulson (5)
(1) Associate Professor, Department of Occupational Therapy, LaTrobe University, Bundoora, Victoria, Australia; Adjunct Professor, School of Health Sciences, Jonkoping University, Jonkoping, Sweden; and Visiting Professor, Faculty of Health and Social Care, London South Bank University, London, UK.
(2) Associate Professor, Rural Health Academic Centre, University of Melbourne, Shepparton, Victoria, Australia.
(3) Senior Occupational Therapist, Austin Health Heidelberg Repatriation Hospital, Melbourne, Victoria, Australia.
(4) Medical Adviser to VicRoads, Victorian Institute of Forensic Medicine, Monash University, Melbourne, Victoria, Australia.
(5) Research Assistant, Department of Occupational Therapy, LaTrobe University, Bundoora, Victoria, Australia.
Associate Professor Carolyn A Unsworth, Department of Occupational Therapy, LaTrobe University, Bundoora, Victoria 3086, Australia. Email: email@example.com
Table 1. Tests included in the OT-DORA Battery Recommended administration (administer to all clients): Section A. Initial interview Section B. Medical history Section C. Medication screen Section D. Sensory assessment (including vision, and proprioception) Section E. Physical assessment (including balance, and motor skills) Section F. Cognitive assessments (including the OTDHMT and RLRCT) Section G. Summary of issues identified during the assessment Optional administration (administer if clinically indicated): Section H. Further assessments (including assessments of hearing, unilateral neglect, detailed assessments of tone, range of movement, muscle strength, and pain). Key: OTDHMT--Occupational Therapy Drive Home Maze Test (Unsworth 2008, Krishnasamy and Unsworth 2011); RLRCT--Road Law and Road Craft Test (Unsworth et al 2010b). Table 2. Road Law and Road Craft Test interrater reliability intraclass correlation coefficients (ICCs) ICC (2,1) 95% CI Question 1 1.00 1.00-1.00 (a) Question 2 0 86 0.74-0.94 (b) Question 3 0.80 0.63-0.91 (b) Question 4 0.96 0.93-0.98 (b) Question 5 0.79 0.56-0.91 (b) Question 6 Original 0.42 0.09-0.70 (b) Question 6 Revised 0.77 0.58-0.98 (b) Question 7 0.92 0.85-0.97 (b) Question 8 0.70 0.43-0.86 (b) Question 9 0.75 0.56-0.88 (b) Question 10 0.92 0.84-0.93 (b) Question 11 0.57 0.32-0.78 (b) Question 12 0.57 0.31-0.78 (b) Question 13 0.91 0.82-0.96 (b) Question 14 0.84 0.69-0.93 (b) (a) The p value for this ICC could not be calculated because there was no discrepancy between the raters' marks. (b) p<0.001. Fig. 1. Revised marking instructions for item 6. Score 1 point for each correct answer in sequential order. Points should still be awarded for any correct answers even if incorrect answers are given amongst them, provided that the correct answers are given in the correct sequence. The maximum score is 4. The correct answers are (1) Check mirrors, (2) Indicate when appropriate, (3) Check blind spots, and (4) Move over when safe to do so. Examples: 7. How do you change lanes? List the four steps involved Example 1 in order. Score: 3 1) Look in side and rearvision mirrors [check] 2) Put indicator on [check] 3) Check you aren't going too slow X 4) Change lanes [check] 7. How do you change lanes? List the four steps involved Example 2 in order. Score: 2 1) Indicate [check] 2) Check mirrors X 3) Go into lane [check] 4) (Left blank)
|Gale Copyright:||Copyright 2011 Gale, Cengage Learning. All rights reserved.|