If you want to know why students fail, just ask them: self and peer assessments of factors affecting academic performance.
Subject: Teaching (Analysis)
Academic achievement (Analysis)
Authors: DeVilbiss, Carita
Rice, Valerie J.
Laws, Linda
Alfred, Petra
Pub Date: 10/01/2010
Publication: Name: U.S. Army Medical Department Journal Publisher: U.S. Army Medical Department Center & School Audience: Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2010 U.S. Army Medical Department Center & School ISSN: 1524-0436
Issue: Date: Oct-Dec, 2010
Geographic: Geographic Scope: United States Geographic Code: 1USA United States
Accession Number: 253536856
Full Text: INTRODUCTION

Academic attrition is a challenge in all demanding education programs, regardless of the field. According to the US Army Accessions Command in 2004, academically demanding advanced individual training programs are experiencing high rates of academic attrition where many students must repeat courses or recycle into training programs. The Health Care Specialist (military occupational specialty 68W), commonly referred to as a combat medic, has been selected as the focus of this study. However, it is only one of the military occupational specialties that are experiencing high attrition and recycling.

As the changing nature of warfare yields an increasingly dispersed battlefield, the combat medic (hereinafter referred to as 68W) can be required to function while physically separated from a medical support unit. That is, they could be the only health care provider available during critical situations. The 68W will have to make independent decisions and be proficient in complex medical evaluation and intervention techniques. All graduates from 68W training are expected to be capable of performing emergency medical and lifesaving trauma care techniques, including critical skills in trauma assessment, advanced airway and shock management, intravenous therapy, and administration of specific, limited medications. These specialists may be required to provide independent care for casualties for as long as 72 hours.

To prepare for these stringent requirements, students attend an intensive 16-week advanced individual training program at Fort Sam Houston, Texas, in the Department of Combat Medic Training facility. The first 6 weeks focus on Emergency Medical Technician (EMT) training, followed by 10 weeks of combat medic training that includes classroom lecture, interactive computer training, and patient simulation. Between 2007 and 2009, the 68W attrition rate peaked as high as 45%, but on average was 31% during that period (average pass rate of 69%) (D. Whittaker and D. P. Parsons, unpublished data, January 2010). Toward the end of 2009, the attrition rate significantly dropped to approximately 25% after some major process changes in the 68W program (R. Brooks, written communication, March 17, 2010).

[FIGURE 1 OMITTED]

During initial focus group interviews, the 68W company commanders, drill sergeants, and academic instructors revealed their perception that approximately 50% of academic recycles are due to reasons internal to the student, including: life skills, motivation, cognitive ability, stress/resiliency/coping, inability to deal with failure, tendency to distraction, and symptoms of learning disorders. The other 50% were considered external to the student and included issues such the program of instruction (curriculum), scheduling, selection and training of instructors, and teaching proficiency (V. J. Rice, Y. Woods, and M. Bundy, unpublished data, 2004). An open question is whether students who were successful in the course differ significantly in either their personal "internal" dimensions, or their perceptions of "external" issues from students who were not successful. This paper addresses 4 areas considered external to the student: class structure and schedule, instructors, support systems, and sleep. Figure 1 shows the line of research conducted by the Army Research Laboratory Field Element and highlights the portion described in this paper.

Peer evaluations are often used to evaluate current performance and to predict future success among educators, law enforcement professionals, managers, medical professionals, and students. Some studies have shown that peer evaluations are better predictors of career success than evaluations by trainers or instructors. (1) Although seen less often in studies with medical students, peer evaluations have been used to:

Peer reviews of both clinical skills and social skills are related to later professional success in medicine (eg, referrals, hospital privileges). (3)

Within the 68W Advanced Individual Training (AIT) program, students are assigned a battle buddy* for the duration of the training program. During the first 72 hours following arrival at Fort Sam Houston, drill sergeants assign one student to another based on gender and basic combat training post. The program is consistent with the established US Army Training and Doctrine Command policies for enlisted training systems that includes guidance for a battle buddy system that pairs new recruits into teams to provide mutual support and assistance; teach teamwork; develop a sense of responsibility and accountability; and improve safety during training. (4) Hence the existing battle buddy system can be employed to compare peer evaluations with self evaluations for both passing and failing students.

RESEARCH OBJECTIVE

The purpose of this study was to identify potential reasons students fail (or do well) during 68W AIT, as reported by students who failed the course, passed with a B or above, or were a friend or battle buddy of either the failing or passing cohort.

METHOD

Participants

All participants were volunteers who were currently enrolled in a 16-week 68W training program at Fort Sam Houston. Volunteers were recruited in 3 categories: (a) students who had failed the course twice, (b) students who were very successful in the course (average grade of at least a B or above), and (c) the battle buddy or friend of the unsuccessful and successful students to provide a peer evaluation. A total of 189 students (28 failing; 101 passing; and 60 peers) volunteered to participate.

Instrument

A single questionnaire was developed and then broken into two formats. The first format addressed the student (either passing or failing). The second format was used for the peer evaluations where the battle buddies were asked to complete the questionnaire as they felt was most appropriate for their buddy, ie, either a passing or failing student. Both formats asked students how they felt on a number of topic areas, including motivation, sleeping patterns, study habits, stress perception, coping strategies, learning disabilities, health status, leadership abilities, unit cohesion, morale, indices of family status, demographics, and other situational and organizational factors that may contribute to attrition. The questionnaire included a number of statements that the students were asked to rate on a 5-point Likert scale (strongly agree, agree, neutral, disagree, and strongly disagree). Students were also asked to participate in an interview, but interview results are not included here. This paper reports only responses for the external factors, eg, class structure, teaching, support systems, sleep, etc.

Procedure

Students were briefed on the protocol and the voluntary nature of the study. If interested, they were asked to sign a volunteer consent form, after which they completed the questionnaire and an interview. Each student volunteer, whether passing or failing, was also asked to consent to the researchers giving a questionnaire to their battle buddy, or another trusted friend. The battle buddy or friend was instructed to complete the questionnaire as it applied to the volunteer, not themselves. All research complied with the human use guidelines of Army Regulation 70-25. (5)

Analysis

The first comparison was to contrast responses from failing students and passing students to 5-point Likert scale questions. Responses for each question were compressed into 3 categories (strongly agree and agree, neutral, strongly disagree and disagree). The resulting 2 by 3 matrix was analyzed with a [chi square] test. To analyze the peer evaluations, the questionnaires (35 for passing students and 25 for failing students) were matched (eg, the study volunteer and his/her battle buddy) and the difference between them analyzed with a t test to determine whether the difference varied significantly from zero. The Statistical Package for Social Sciences (SPSS Inc, Chicago, Illinois) was used for all analyses with a significance level of P=.05.

RESULTS

The results summarized here are for the external factors included in the extensive questionnaire. Positive results (strengths of the program) are presented first, followed by areas identified as difficulties.

Participants

A total of 189 students volunteered to participate; 28 failing and 101 passing students and 60 battle buddies (25 peers for the failing students and 35 for the passing students). The students were predominately Caucasian (78%), with the remainder comprised of 8% African American, 8% Hispanic, and 6% identified as other. The gender composition across groups was comparable, with male students comprising approximately three-quarters of the students (79%, 72%, and 82% for failing, passing, and battle buddy, respectively). However, the female students were younger than the male students (average age 22.4 years (female), 24.7 years (male)). The failing students were younger than passing students and the peers (average age 23.0 years, 24.7 years, and 23.8 years, respectively). Half of the students (50.0% of failing and 52.0% of passing) reported that they had studied at the college level, ie, they checked either "some college" or "college diploma" levels. However, more passing students reported having college diplomas than did failing students (16.0% and 7.1%, respectively).

Class Structure and Schedule

When asked whether the class sizes were too large to allow for individual attention, only one in 8 students (13% passing and 14% failing) agreed with that statement. Two-thirds of the students (66% passing and 61% failing) did not feel that class size prevented individual attention. Students also agreed (50% passing and 43% failing) that it is difficult to stay focused because the class days are too long. The students differed, not surprising, when asked whether they struggled with the fast pace of the course. Significantly more failing than passing students (57% and 19%, respectively) agreed that they struggled with the fast pace of the course ([chi square] = 16.48, P <.01). Although not significant, more failing students tended to agreed that there was not enough tutoring or mentoring available during the class (29% failing vs 20% passing students). The consolidated results are presented in the Table.

[FIGURE 2 OMITTED]

Sleep Patterns

There were no positive self-reports in regard to sleep. More than half of the students (57% of failing students, 67% of passing students) reported getting 5 to 6 hours of sleep each night (Figure 2).

A larger difference was seen when 43% of the failing students reported they were getting only 3 to 4 hours of sleep each night, as compared with 23% of the passing students.

However, when asked whether they get enough sleep, 33% of the failing students and 17% of the passing students report that they "seldom" or "never" get enough sleep at night. Consistently, more failing students than passing students (29% vs 16%) report that they "always" or "often" stay up and study after "lights out." Fifty-eight failing students and 37% of passing students report that they "always" or "often" have trouble staying awake in class (Figure 3). Battle buddies of passing students believed the passing students had less trouble staying awake in class (that is, they felt the passing students were able to stay awake more easily than the passing students themselves believed) (P<.05).

[FIGURE 3 OMITTED]

Instructors (including Drill Sergeants) and Teaching

Two out of 5 students (46% passing and 43% failing) reported that instructors were easily understood and taught at the appropriate level. When asked to comment on the teaching methods, significantly more failing students than passing students (54% and 25%, respectively) agreed that the teaching methods made it difficult to succeed. In the peer evaluations for this question, there was a significant difference in the response from the passing student and the response of the individual's battle buddy. The peer evaluations under estimated the difficulty that the passing students were having with the teaching methods. In fact, significantly more passing than failing students (37% vs 21%) reported that adjusting to new instructors frequently (as they shifted between subject areas) had a negative impact on grades ([chi square] =7.24, P<.05). When asked whether the drill sergeants affect a student's performance, more passing students than failing students (56% and 43%, respectively) agreed that drill sergeants affect academic performance, but this trend was not statistically significant (Figure 4).

[FIGURE 4 OMITTED]

[FIGURE 5 OMITTED]

Support Systems: Unit

There were no differences between the responses from the failing and passing students when asked whether they had bonded with their unit. More than half of the students (57% passing, 54% failing) reported that they had bonded well (Figure 5). More than half of the students (56% passing, 57% failing) reported that they felt their unit was "always" or "often" supportive of them. However, of note, a number of the students (25% failing and 14% passing) responded "N/A" to this question without further explanation.

Support Systems: Family & Friends

More than two-thirds of the students (68% passing, 71% failing) reported they had other people besides their battle buddy that they could turn to for support. Approximately 8 of 10 students (87% passing, 79% failing) reported their family and friends were supportive and were not pressuring them to find a way out of the course and out of the Army. However, 31% of passing students, compared with 11% of those failing, reported the wishes of their family affected their academic performance.

[FIGURE 6 OMITTED]

Support Systems: Battle Buddy

In addition to reporting support from their unit and the family and friends, the majority of the students report a positive relationship with their battle buddy. Three-fourths of the students (71% passing, 79% failing students) reported that felt they could approach their battle buddy to talk about personal matters or problems (Figure 6). More than 80% of all students (82% passing, 89% failing) agreed their battle buddy provides help and is supportive of them. Two-thirds of students (60% passing, 68% failing) reported they perceived they had a "close" relationship with their battle buddy.

COMMENT

These results of the students' assessments of external factors provide insight into the some of the issues surrounding academic performance of students attending combat medic training. They offer support for some organizational programs, dispel some myths, and identify some issues.

Ask Students

First, the results suggest that students are truthful in reporting their impressions. The assumption was that students who failed the program and will be assigned to attend alternate training in a different, nonmedical military occupational specialty might have something to lose (reputation, might receive less than fair treatment in regard to their new assignment) and therefore might be less truthful on a questionnaire. Their battle buddy would not have anything to lose and therefore might be expected to be more forthright with their responses. However, no differences were seen between failing student reports and the reports from their battle buddies. In fact, the peer evaluations did not differ significantly from the corresponding student responses except when battle buddies of passing students reported more positive responses than the individuals did themselves. For example, the battle buddies reported that passing students had less trouble staying awake in class than was reported by the passing students themselves.

Class Size

Second, class sizes have more than doubled since the onset of the war on terror from an average of approximately 200 to nearly 500. This data was collected approximately in the middle of that growth period, during which time the class sizes were approximately 350 per class. The growth in class size occurred without a commensurate increase in staff (instructors and supervisors). All are concerned about the affect this has on student learning and retention, as well as on burnout of instructors. Instructors have intensified their efforts to be available to students, set up study halls and review sessions, and deliver instruction in the best way possible. During late 2006 and early 2007, students reported that they received individual attention in spite of the class size. However, a trend was noticed in the impressions of failing students that they did not receive the mentoring they needed.

Teaching Methods

Third, it is often stated anecdotally that the instruction in the 68W program is perhaps taught at too high a level. The findings that less than half of students reporting that instructors are easily understood and teaching at an appropriate level would appear to give credence to the anecdotal statements. In addition, teaching methods appear to be an issue for failing students. Given that the primary method of teaching is through lecture accompanied by PowerPoint slides and written outlines, this issue could easily be addressed to the benefit of the students. Combining information from this study with additional research results from this population could assist in guiding improvements in teaching and presentation techniques. For example, research with this population, and with operating room technicians, revealed visual learners performed better academically than auditory learners. (6) On the other hand, passing students felt adjusting to changing instructors was more of an issue than did failing students. It is not clear whether failing students did not see this as an issue, or were so focused on other issues of concern (such as the pace of the course) that adjusting to changing instructors was a lesser priority.

Support Systems

Fourth, support systems appear to be readily available to the majority of students, primarily through friends and relatives. Furthermore, the battle buddy system appears to be working despite the criticism that appointing a battle buddy will not guarantee friendship or support. All students reported feeling they could approach and talk with their battle buddies, and that battle buddies were supportive. This seems to occur even when the relationship is not described as close.

Other information identified in this study is not new, but does substantiate items long thought to be issues. These issues include the long hours in the classroom (from 0900 until 1630) resulting in students having difficulty focusing on the subject matter, the fast pace of the course, and the perceived negative impact of drill instructors. Drill instructors are no longer used during 68W AIT, although supervising noncommissioned officers remain.

Perhaps one of the most pertinent issues corroborated by these findings is the lack of sleep. Student grades have previously been associated with lack of sleep (7) and abbreviated sleep times have been associated with classroom sleepiness and difficulties in concentration and focus. (8) Sleep is important for learning and consolidation of new memories, (9) which is the primary task for students attending AIT. While the recommended number of hours of sleep for young adults is approximately 8 to 9 hours per night, our students are getting 5 to 6 hours, and many failing students reported getting only 3 to 4 hours each night. It may be that the training program is effectively screening out students who require an average amount of sleep for their age.

Limitations

The primary limitation to this study is the size of the population, especially the failing student population. Also, this is a unique population (students) who may not be representative of the general population.

Recommendations

* Use carefully constructed questionnaires and interviews to gather information on students and student perceptions of barriers to learning, as well as programs that assist learning.

* Reinvestigate the effect of class size on students, now that it has risen to approximately 450.

* Review the program of instruction, including individual lectures and presentations, for grade level.

* Review teaching methods for those considered most effective for this age and subject matter. Consider matching teaching methods with learning styles of students.

* Provide information on positive impact of the battle buddy system to incoming instructors and students. Continue this program.

* Investigate teaching methods. Give students periodic breaks in the type and style of teaching/ learning throughout the day. Continue to provide study guides, outlines and other assistance to help counter the fast pace of the course. Consider lengthening the course or having at-risk students attend an early preparatory program before the course.

* Consider methods to decrease the need for students to continually adjust to changing instructors, such as increasing the uniformity of presentations or assigning instructors to remain with a class throughout their program.

* Investigate methods to increase time available to students so they can effectively study and learn their materiel, as well as get the needed amount of sleep. Educate administrators, educators, and students on the need for sleep and how adequate sleep enhances learning and memory.

ACKNOWLEDGEMENTS

This study was conducted under an approved ARLHRED Human Use Protocol (ARL-20098-05031) entitled: Personal Factors Related to Student Performance and Retention Among 68W Health Care Specialists at Fort Sam Houston, Texas: Self and "Battle-Buddy" Reports on Factors Affecting Academic Performance. It was funded by the US Army Research Laboratory, Human Research and Engineering Directorate.

REFERENCES

(1.) Hofstede G. Predicting managers' career success in an international setting: the validity of ratings by training staff versus training peers. Manag Int Rev. 1994;34:63-70.

(2.) McCormack WT, Steven CB, Small PA. Simplifying medical student summative peer evaluation of clinical competence. Paper presented at: Annual Meeting, American Association of Medical Colleges Southern Group on Educational Affairs; 2002; Charleston, South Carolina.

(3.) Wigen K, Holen A, Ellingsen O. Predicting academic success by group behavior in PBL. Med Teach. 2003;25(1):32-33.

(4.) TRADOC Regulation 350-6: Enlisted Initial Entry Training (IET) Policies and Administration. Fort Monroe, VA: US Dept of the Army, US Army Training and Doctrine Command; May 8, 2007:34. Available at: http://www.tradoc.army.mil/tpubs/regs/ r350-6.pdf. Accessed September 3, 2010.

(5.) Army Regulation 70-25: Use of Volunteers as Subjects of Research. Washington, DC: US Dept of the Army; January 25, 1990.

(6.) Alfred P, Rice V, Boykin G, Laws L. Auditory discrimination and academic performance. Paper presented at: Proceedings of the Applied Human Factors and Ergonomics Conference, Miami, FL, July 17-20, 2010.

(7.) Kelly WE, Kelly KE, Clanton RC. The relationship between sleep length and grade-point average among college students. Coll Student J. 2001;35(1). Available at: http://findarticles.com/p/articles/ mi_m0FCR/is_1_35/ai_74221510/?tag=content;col 1. Accessed September 3, 2010.

(8.) Sadeh A, Gruber R, Raviv A. The effects of sleep restriction and extension on school-age children: what a difference an hour makes. Child Dev. 2003;74, 2:444-455.

(9.) Ellenbogen JM, Hulbert JC, Stickgold R, Dinges DF, Thompson-Schill SL. Interfering with theories of sleep and memory: sleep, declarative memory, and associative interference. Curr Biol. 2006;16(13):1290-1294.

Carita DeVilbiss, PhD

Valerie J. Rice, PhD

Linda Laws

Petra Alfred

Dr DeVilbiss is the owner and president of Dilemma Doc, a statistics and human factors consulting company.

Dr Rice is Chief, Army Medical Department Field Element, Army Research Laboratory, Fort Sam Houston, Texas.

Ms Laws is the Director of Medical Surgical Nursing at Methodist Stone Oak Hospital, San Antonio, Texas. When this article was written, she was a consultant with BearingPoint, Inc.

Ms Alfred is a research psychologist, Army Medical Department Field Element, Army Research Laboratory, Fort Sam Houston, Texas.
... measure medical student achievement and predict
   future performance in terms of clinical competence,
   interpersonal skills, and professional behavior. (2)


Percentage of students (categorized by those either passing or
failing coursework) who agreed with statements describing
possible negative circumstances attributable to class structure
and schedule conditions.

Statement                               Failing    Passing
                                        Students   Students

Class size too large                      14%        13%

Class size did not prevent individual     61%        66%
attention

Difficult to stay focused with long       43%        50%
class days

Had to struggle with the fast pace *      57%        19%

Tutoring and mentoring inadequate         29%        20%

* Significant at P <.01
Gale Copyright: Copyright 2010 Gale, Cengage Learning. All rights reserved.