Assessment of medical students by OSPE method in pathology.
Objective structured practical examination (OSPE) is a practical
exam system wherein there is a series of stations at which students work
through tasks designed to test various skills and are tested using
agreed check lists with observers sitting at some of the stations. It
has been found to be objective, valid and reliable. It also eliminates
examiner bias. The purpose of this study is to introduce OSPE both as an
evaluation and a teaching tool and to draw attention to its advantages
and disadvantages. We designed an OSPE that tested all the above
objectives satisfactorily. The OSPE exam was conducted in the department
of pathology during the second internal assessment of the students. This
method was the compared with the conventional method of practical exam
conducted during the first internal assessment of the same batch of
students. The results showed that OSPE tests different desired
components of competence better. It is an objective, valid and reliable
method and gets rid of variation due to examiner bias. It has a better
discrimination index on merit. Interpretative exercises are the most
relevant part and deserve the lion's share of total marks. Students
feel more comfortable and less stressed to perform the exam.
KEY WORDS: Objective structured practical examination; Medical education; Teaching methodology; Pathology
Medical education (Usage)
Bandi, Prasana K.
|Publication:||Name: Internet Journal of Medical Update Publisher: Dr. Arun Kumar Agnihotri Audience: Academic; Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2012 Dr. Arun Kumar Agnihotri ISSN: 1694-0423|
|Issue:||Date: Jan, 2012 Source Volume: 7 Source Issue: 1|
|Topic:||Event Code: 360 Services information|
|Geographic:||Geographic Scope: India Geographic Code: 9INDI India|
Undergraduate medical education is currently undergoing extensive re-evaluation. This is because the conventional clinical and practical examination is beset with several problems. Although marking should depend only on student variability, examiner variability significantly affects scoring. The marks awarded also reflect only the global performance of the candidate and are not based on demonstration of individual competencies. Problems in communication significantly affect the outcome. Attitudes are usually not tested at all by the conventional examination There is a conscious attempt to define new core educational objectives. These defects of clinical and practical examinations have been realized for long and have given rise to attempts at improving the current scenario. (1-3) It has led to the development of new exam systems which can test all the objectives systematically. (4,5) All these attempts are relatively new and are still in the process of being tried out. An earlier innovation in this regard is the objective structured clinical examination (OSCE) later extended to the practical examination (OSPE) described in 1975 and in greater detail in 1979 by Harden and his group from Dundee. (6,7) This method with some modifications has stood the test of time and has largely overcome the problems of the conventional clinical examinations mentioned earlier. Recently, the method was the subject of an international conference at Ottawa in 1985 when the worldwide experiences with OSCE and OSPE were exchanged. (8) OSPE was also proposed for medical undergraduates in Pathology by Ramnarayan in 1990. (9)
OSPE is a practical exam system wherein there is a series of stations at which students work through tasks designed to test various skills and are tested using agreed checklists with observers sitting at some of the stations. It has been found to be objective, valid and reliable. It also eliminates examiner bias.
The objectives, as far as Pathology practical are concerned, may be defined as (1) A range of practical skills that students should possess (2) Reading of certain basic and critical tests, (3) Analysis and interpretation of test results and its clinical correlation. The purpose of this study is to introduce OSPE both as an evaluation and a teaching tool and to draw attention to its advantages and disadvantages. We designed an OSPE that tested all the above objectives satisfactorily.
The OSPE exam was conducted in the department of pathology during the second internal assessment of the students. This method was then compared with the conventional method of practical exam conducted during the first internal assessment of the same batch of students. At the end of the exam (1) A feedback was taken from the students regarding their experience and (2) Item analysis was done for each station of the exam.
Conduct of OSPE
The model of OSPE we made can accommodate total 24 students divided in two batches of 12 each. Twelve students were further divided in two groups of 6 each. Each student will pass through 6 stations for 10 minutes each with 12 students taking on replicas of each of the 6 stations simultaneously (Chart 1).
It required 4 examiners and a total of 60 minutes per batch. The 6 stations were divided into 3 categories. They are (1) Clinical Pathology (CP) exercises here they are assessed according to a predetermined checklist and it primarily checks the psychomotor domain. Peripheral smear staining, hemoglobin estimation, blood grouping, total WBC count and chemical analysis of urine come under this category. (2) Spotters here spotters without any clinical details are kept. These test only recall of facts skill of cognitive domain. Normal and abnormal blood cells, histopathology slides, gross specimen and some instruments come under this. (3) Interpretative exercises here clinical situations along with relevant tests are provided. Candidates are expected to read the tests, analyze the results, interpret the data and correlate with the clinical information provided. These exercises test the interpretation of data skills of cognitive domain. Exercises that come under this category include reporting of peripheral smear and / or bone marrow smears provided with relevant clinical data and basic hemogram; reading, interpretation and correlation of ESR, microhematocrit, reticulocyte count, osmotic fragility. Interpretation of charts of CSF, blood investigations, coagulation tests, urine findings etc. of the 3 categories, only the CP exercise category is observed directly by the examiners. All the exercises kept in CP, spotters and interpretative exercises will be seen by all candidates. At the end of examination a questionnaire was given to the students after exam to get the feedback. (Table 1: A, B & C).
Comparison with conventional practical examination method
The above exam was compared with the conventional practical exam conducted during the first internal assessment of the same students in which the practical skills of CP exercises are not evaluated. Instead they were assessed based on questions asked at the end of performing the tests. An item analysis was done for each question which included calculation of difficulty index and discrimination index, asked during OSPE as well as the conventional practical exam and a comparison was made between the two.
In OSPE the students got a mean of 47.2% marks ranging from 27.7% to 71.25% while in conventional practical examination the students scored a mean of 45.1% ranging from 31.3% to 57%.
A scattergram of the marks obtained by two different methods was also prepared separately (Chart 2). This showed a distribution of marks over a wide range in OSPE method, while clustering of marks in conventional method.
[GRAPHIC 2 OMITTED]
An item analysis for each station in OSPE was done and difficulty index (p value)) and discrimination index (d value) were calculated. As shown by Table 2 in OSPE, except station 2 which showed a non acceptable p value (below 30%) and station 3 which showed a non acceptable p value ( below 30%) and a poor d value (below 0.2) rest all showed an acceptable p value and an excellent d value.
A similar analysis of questions of conventional practical exam showed an acceptable p value only in exercise 3 while d value was good in all except exercises 4 (Table 3).
Feedback from students
Based on the questionnaire given to the students a conclusion was made comparing the OSPE method and the conventional practical examination method. The results of the questionnaire (Table 4) showed that OSPE was a better accepted examination method and proved to be a better method for assessment of practical skills of students.
The feedback from the students showed that more than 90% agreed that they felt more comfortable with OSPE, it was less stressful to perform and that it was a more objective assessment. It assessed the practical and correlative skills better. For 65% of students OSPE was also a learning experience.
OSPE is one of the new exam systems designed to make an objective, valid and reliable assessment of different components of competence. The main features of OSPE are that both the process and the product are tested giving importance to individual competencies. The examination covers a broad range of clinical skills much wider than a conventional examination. The scoring is objective, since standards of competence are preset and agreed check lists are used for scoring. Where questions are asked in response stations, these are always objective. Examiners variability is eliminated thus increasing the validity of the examination. (10)
This study compared OSPE with the conventional practical exam conducted in the same batch. The conventional format showed an overall low difficulty index (which indicates the proportion of total students in the batch who have answered the item correctly). This may be because the practical skills were not observed or evaluated using a checklist, but assessed based on questions asked at the end of the session. In conventional method only exercise 3 showed an acceptable difficulty index and the same exercise, which was to perform a differential count on a given blood smear, was given in station 3 of OSPE where it showed a non acceptable difficulty index this may be because the limited time of 10 minutes given in OSPE was not sufficient to perform the exercise. Apart from this clustering of marks was also noted in conventional method which was due to the examiner's bias.
The exercise that most effectively separated the better students from the average and poorly performing students were the interpretative exercises. These exercises test the interpretative skills of the students and, according to us, are the most relevant part for a practicing medical professional. It raises the standard of the exam and gives scope for different types of exercises with varying difficulty. It is the key to the wide discrimination index. However, it is a challenge to the examiners and requires elaborate preparation and extreme dedication. We decided to have interpretative exercise for 35% of total marks of the practical examination.
Conduct of OSPE warrants a good rapport between the colleagues and is essentially team work. It needs commitment of time and personnel and elaborate preparation before exam. Discussion after each exam helps to overcome small practical hitches and evolve a better format. The feedback from students showed that it was well taken by them. They were convinced of the objectivity and the relevance of this exam system. They felt it more comfortable and less stressful to perform. A good majority also found it a learning experience.
A similar study was performed by Feroz et al and they also found encouraging results with a high rate of acceptance among the students. (11)
The process is, however, not without limitations. There is risk of observer fatigue if the observer has to record the performance of several candidates on lengthy check lists. All stations must invariably demand only equal time. Ensuring this, therefore, requires careful organization.
OSPE tests different desired components of competence better. It is an objective, valid and reliable method and gets rid of variation due to examiner bias. It has a better discrimination index on merit. Interpretative exercises are the most relevant part and deserve the lion's share of total marks. Students feel more comfortable and less stressed to perform the exam.
In conclusion, OSPE has several distinct advantages. However, in the current situation it may not be realistic to expect its inclusion in the formal summative evaluation schedule of universities. However, it is feasible in view of the tremendous advantages that it offers, to include the formative (day-to-day) assessment of students to improve their clinical competence and to derive an objective score for internal assessment. OSPE, from our experience, is a system that is to be evolved at local centers accommodating the views of participates, and not by copying a standard format used elsewhere.
(1.) Edelstein DR, Ruder HJ. Assessment of clinical skills using video tapes of the complete medical interview and physical examination. Med Teach. 1990;12(2):155-62.
(2.) Stiliman PL, Brown DR, Redfield DL, et al. Construct validation of the Arizona Clinical Interview Rating Scale. Educ Psychol Meas. 1977;37:1031-56.
(3.) Newbie DI. The observed long-case in clinical assessment. Med Educ. 1991 Sep;25(5):369-73.
(4.) Selby C, Osman L, Davis M, et al. Set up and run an objective structured clinical exam. BMJ. 1995 May;310(6988):1187-90.
(5.) Godfrey RC. Undergraduate examinations--a continuing tyranny. Lancet. 1995 Mar;345(8952):765-7.
(6.) Harden RM, Stevenson M, Downie WW, et al. Assessment of clinical competence using objective structured examination. Br Med J. 1975 Feb;1(5955):447-51.
(7.) Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979 Jan;13(1):41-54.
(8.) Hart IR, Honden RM, Walton HJ. Newer developments in assessing clinical competence. In: Hart IR, Honden RM, Walton HJ, editors. International Conference Proceedings. Ottawa: Congress Centre;1985.
(9.) Ramnaryan K. A more effective practical examination in Pathology. Indian J Pathol Microbiol. 1990 Oct;33(4):399-401.
(10.) Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med. 1993 Apr-Jun;39(2):82-4.
(11.) Feroze M, Jacob AJ. OSPE in pathology. Indian J Pathol Micribiol. 2002 Jan;45(1):53-7.
Kavita Munjal ([PSI]) * MD, Prasana K Bandi ** MD, Amit Varma * MD and Shirish Nandedkar ** MD
* Associate Professor, ** Professor, Department of Pathology, Sri Aurobindo Institute of Medical Sciences, Indore 452001 (Mp), India
(Received 18 January 2011 and accepted 03 March 2011)
([PSI]) Correspondence at: Department of Pathology, Sri Aurobindo Institute of Medical Sciences, Indore 452001 (MP), India; Phone: 00 91 731 2491848; Email: firstname.lastname@example.org
Table 1(A,B &C): OSPE FEEDBACK FROM STUDENTS Please tick any one alternative for each of these questions asked: A. OSPE evaluation: Question Agree Neutral Disagree No comment Exam was fair Wide knowledge area covered Needed more time at stations Exams well administered Exams well structured & sequenced Exam minimized chance of failing OSPE less stressful than other exams Highlighted areas of weakness Student aware of level of information needed Wide range of performance skills covered B. Quality of performance testing Question Not at Neutral To great all extent Fully aware of nature of exam Tasks reflected those taught Time at each station was adequate Setting and context at each station felt authentic Instructions were clear and unambiguous Tasks asked to perform were fair Sequence of stations logical and appropriate Exam provided opportunities to learn C. Student perception of validity and reliability Question Not at Neutral To great all extent OSPE exam scores provide true measure of essential clinical skills in pediatrics OSPE scores are standardized OSPE practical and useful experience Personality, ethnicity and gender will not affect OSPE scores Table 2: Item analysis of exercises of OSPE Exercises Difficulty index Discrimination index (p value) * (d value) ** 1 31.2 0.62 2 12.5 0.25 3 6.2 0.12 4 43.7 0.85 5 37.5 0.75 6 37.5 0.75 Table 3: Item analysis of exercises of conventional practical examination Exercises Difficulty index Discrimination index (p value) * (d value) ** 1 18.75 (1.37 2 12.5 0.50 3 31.25 0.62 4 0 0 5 25 0.75 * p value 30%-70%--Acceptable > 70%--Very easy exercise < 30%--Very difficult exercise ** d value > 0.35--excellent 0.25-0.35--good < 0.2--poor Table 4: Conclusion drawn from the questionnaire given to students Question OSPE Conventional Feel more comfortable 91% 9% Less stressful to perform 83% 17% More objective assessment 95% 05% Better assessment of practical skills 89% 11% Better assessment of correlative skills 83% 17% It was a learning experience 65% --
|Gale Copyright:||Copyright 2012 Gale, Cengage Learning. All rights reserved.|