This study involved two comparable large general teaching hospitals located in the Netherlands, which have a similar web-based system for confidential and voluntary reporting of incidents. One hospital was used as the intervention hospital and the other one as the control hospital. In total, about 130 residents were working in the intervention hospital and about 90 residents were working in the control hospital at the start of the study. Residents of the intervention hospital were obliged to attend the patient safety course if their contract within the hospital continued for at least three months after the course ended. However for some residents it was not possible to attend, for example because of scheduled holidays, or maternity leave. If a resident was not able to attend the first course, we tried to let him/her attend the second course.

Controls were only recruited from disciplines comparable to those of the course participants and if their contract within the hospital continued for at least three months after the course ended. The national rules and regulations regarding health services research were followed. The Scientific Research Review Board of the VU University Medical Center provided a waiver for this study.

We organized two separate patient safety courses for residents from multiple specialties of the intervention hospital, both starting in September 2007. Both courses were identical in content and aimed at increasing knowledge of patient safety and skills to recognize and cope with unintended events and unsafe situations in an early stage. Each course spanned a period of three months and consisted of one plenary day, followed by two half-days in smaller groups. We used a mixture of educational methods to create an interactive learning environment.

The importance of incident reporting was stressed throughout the course. During the first course meeting participants practiced incident analysis by performing root cause analyses with the Prevention and Recovery Information System for Monitoring and Analysis (PRISMA medical version) [¹⁹]. This method for analyzing incidents distinguishes four main categories of causes: technical, organizational, human and patient related. To get used to noticing and reporting incidents and to become aware of the importance of reporting, each participant received ten pocket-size reporting cards at the first course meeting (Figure 1). Filling out these cards by health care workers had proved to be useful in previous research [²⁰]. We asked residents to fill out the cards during their clinical work and to analyze five incidents by investigating the underlying causes. Development and content of the course were described in more detail elsewhere [²¹].

We collected data in three different ways.

1] We distributed questionnaires before, directly after and three months after the course was given (M1-M3). These measured self-assessed changes in knowledge, skills, attitudes, intentions and behavior of respondents. The post-course measurement of the first course took place at about the same time (+/- two weeks) as the pre-course measurement of the second course, allowing for an internal cross-over comparison. At the same points in time we sent questionnaires to the external controls (hereafter referred to as "control group"). We sent follow-up measurements only to those controls who responded to the first measurement. To stimulate responsiveness, we raffled lottery tickets among the control group. We did not raffle lottery tickets among the intervention groups. All questionnaires were encoded and processed confidentially.

2] We gathered and processed incident reporting cards and root cause analyses that were filled out by course participants between the first and second course meeting. This provided additional insight into the skills and behavior of residents to notice incidents.

3] The hospital-wide digital incident reporting systems of both hospitals provided information about the factual reporting behavior of residents.

We developed a 17-item questionnaire (see Additional File 1). At each measurement we asked the residents about a selection of these items. We put the last three items only to the intervention groups, all other items were presented to the controls as well. The first two items referred to the experienced level of knowledge of the respondents concerning handling incidents and patient safety improvement in general, which was measured on a 4-point Likert scale (Strongly disagree/Disagree/Agree/Strongly agree). For item 3-11, three answer options were given (No/Cannot decide/Yes). Item 12 was open ended. Item 3-5 assessed course participants' changes in skills with regard to noticing and analyzing incidents. Item 6-11 consisted of vignettes that provided insight into residents' attitudes towards incident reporting in specific situations. Incident reporting in this setting meant voluntary reporting of incidents by filling out a digital registration form at the hospital incident reporting system. As the patient safety course stressed the importance of making a report of all incidents, it would have been correct to consider all the vignettes in the questionnaire worth a report. Item 12-17 focused on incident reporting attitudes, intentions and behavior and were based on a questionnaire of Coyle et al. (2005), who measured the impact of an educational program for graduate trainees as well. Item 1-11 were developed by content experts (ABB & CW) and based on their experience in health care and patient safety research. An overview of the different data collection methods that we used in this study is presented in table 1.

We processed data in SPSS and used a probability of p ≤ 0.05 (two-tailed) to establish statistical significance. We assessed comparability of residents' characteristics for nominal measures by Chi-square analysis, and for scale measures by the Independent-Samples T-test and the Mann-Whitney test. We analyzed questionnaires with Friedman's test for analysis of three related assessments. We used the Paired Samples T- test and the Wilcoxon Signed Ranks Test to analyze two related measurements. To be able to compare the questionnaire's items, we transformed the 4-point Likert scale variable into a 0-2 scale (0 = Strongly disagree; 2 = Strongly agree). We created a sum variable to analyze the six vignette items. The Mann-Whitney test was used for comparing answers between intervention groups and control group. We used logistic regression (enter and stepwise) to determine the relation between the items measuring incident reporting intentions or behavior and the items measuring knowledge, skills, attitudes or intentions. To do this, we first transformed the items measuring intentions and behavior into a binary variable by merging the answer categories 'No' and 'Cannot decide'. Data from the hospital reporting system were analyzed using the Chi-square test.

In total, 44 (34%) residents were eligible to attend and attended one of the two patient safety courses. Forty-three percent (n = 19) of these residents attended the first course, 57% (n = 25) participated in the second course. The response rates among these course participants were: M1: 100% (n = 44), M2: 98% (n = 43) and M3: 100% (n = 44). At the control hospital we approached 57 (63%) residents at the first measurement. The response rates of the control group were: M1: 63% (n = 32), M2: 73% (n = 22) and M3: 96% (n = 23). Eight controls left the hospital during the study period and therefore had to be excluded from some of the measures. Table 2 shows the respondents' characteristics.

Analyses of respondents' characteristics demonstrated that the intervention groups and the control group were comparable and formed representative samples of the hospitals' resident population. Questionnaires' outcomes at the first measurement did not show differences between the two intervention groups, nor between the control group and the intervention groups. Hospital incident reporting systems showed no significant differences in residents' reporting behavior during a period of eight months before the first course started.

At baseline, 60% (n = 26) of the course participants agreed or strongly agreed that they knew what to do if they were involved in an incident, and 48% (n = 20) agreed or strongly agreed to have sufficient knowledge to improve patient safety at the department. After the course they experienced a significant positive change in their level of knowledge. Post-course, 86% (n = 36) agreed or strongly agreed that they knew what to do if they were involved in an incident, and 88% (n = 36) agreed or strongly agreed to have sufficient knowledge to improve patient safety at their department. The control group demonstrated no significant changes in knowledge.

Three months after the course, almost three-quarters of the participants declared that the course had contributed to an improved signaling of unsafe situations. More than 90% of the participants acknowledged that because of the course they were able to recognize that multiple factors contribute to incidents and that the course had taught them how to analyze incidents systematically. In total, 165 incident reporting cards were filled out by 36 (82%) course participants (mean: 4) and 31 root cause analyses were performed by 14 (32%) of these residents (mean: 2), which demonstrated the ability of the residents to notice and analyze incidents. A selection of reported incidents, in which the variety in types and causes of the identified incidents is illustrated, is shown in table 4.

At the baseline measurement a minority of residents considered the vignettes worth reporting and a greater part of residents judged reporting by residents to be important for incidents with harm, as well as for incidents without harm. Analysis of vignettes at all three measurements showed significant positive changes (p < 0.001) in the intervention groups. Immediately after the course (p < 0.001), as well as when pre-course measurements were compared to follow-up measurements (p = 0.002), residents more often considered the incidents proposed in the questionnaire worth reporting. After participating in the course, more residents judged that reporting incidents without harm is necessary, but this change did not reach statistical significance. All but one or two residents thought it important to report incidents with harm for the patient. The control group showed no significant lasting changes in attitudes.

At baseline intentions to report incidents within the next six months and intentions to start reporting within the next month were present among the majority of the respondents. These intentions remained stable immediately after the course as well as during follow-up. The controls did not demonstrate significant changes over time, either. None of the independent variables (knowledge, skills or attitudes) made a significant contribution to the incident reporting intentions.

At the baseline measurement less than half of the residents declared to have reported an incident within the last six months. According to the answers to the questionnaire the number of reported incidents increased significantly after the course. No significant changes were found in the number of residents who reported incidents. The control group did not show any changes over time in the number of reporters or the number of reports. At the intervention hospital the reporting system registered an increase of 130% (from 19 to 44 reports, p = 0.001. Mean per resident: from 0.15 to 0.32 reports) of reported incidents by residents eight months after the first course had started compared to eight months prior to the first course meeting. This increase remained stable over another period of eight months. Increased reporting was not restricted to the course participants: other residents from the intervention hospital also reported more incidents. In the same period the incident reporting system of the control hospital registered a reduction in reports by residents (from 10 to 7 to 0, p = 0.003). None of the independent variables (knowledge, skills, attitudes or intentions) made a significant contribution to the incident reporting behavior.

Although the number of reported incidents by residents in the hospital reporting system remained low, this probably was not caused by an absence of incidents in their work, as research revealed that residents are regularly involved in incidents [^23-30] and residents filled out a large number of incident reporting cards during the course. The non-occurrence of actual incident reporting behavior might be explained by the Theory of Planned Behavior (TPB) [³¹]. This widely used social-psychological model distinguishes the following four aspects that influence behavior: attitudes, subjective norms, behavioral control and intentions. We found that the intentions and the attitudes of our residents concerning incident reporting were mainly positive. Therefore, it is likely that the incident reporting barriers that our residents experienced were related to the other two aspects of the TPB: the subjective norms and the behavioral control. In this study, we did not make an inventory of possible barriers that might discourage incident reporting among the residents. Nevertheless, other publications [¹⁵,¹⁶,³²] suggested several barriers, related to human as well as system factors, that could hinder incident reporting. For example, a lack of encouragement by faculty, a lack of timely and high-quality feedback on medical incident reports, and fear of compromising one's career or personal reputation might contribute to discouraging subjective norms. A low perceived behavioral control might be associated with time constraints, complex reporting systems and forgetfulness. Relating identified barriers to the aspects of the TPB can be useful as a guidance for the development and evaluation of interventions to bring on behavioral changes [³³].

Although the hospitals in this study can be considered comparable and general, one must be aware that a limited number of residents, of only one institution was trained, and only one other institution acted as control. Therefore the ability to generalize the outcomes for other settings may be restricted. Moreover, part of the results are based on the perception of the respondents, which might provoke social desirability bias. We tried to reduce this limitation by including objective data from the hospital digital reporting system and the specifically study-related reporting tool. The questionnaire that we used was not validated, but a major part of it had been used in other studies previously. We cannot be certain that the results are caused entirely by the course, as residents are continuously exposed to a diversity of stimuli which might influence their knowledge, skills, attitudes, intentions and behavior as well. Nevertheless, comparison with the controls justifies the assumption that the course did have positive effects. Although this study showed that it is feasible to measure positive effects concerning incident reporting after patient safety education, incident reporting is just one of the issues relevant for improving patient safety.

Thus, we found that knowledge, attitudes, intentions, and the presence of skills concerning incident reporting, were not sufficient to activate reporting behavior of residents who did not have previous incident reporting experience. To stimulate reporting behavior of these residents, it is necessary to identify and overcome the barriers that are perceived and that discourage the reporting of incidents. To maintain the positive effects on the behavior of residents that were already reporting incidents, another important step is to develop methods to anchor patient safety education throughout the educational career. Assessing the effectiveness of teaching other patient safety issues and other health professionals would also be interesting for future research.

*Not all residents who responded to the questionnaire filled out all items. The response rate per item varied between 40 (91%) and 44 (100%).

†Item 1&2: 0 = Strongly disagree; 2 = Strongly agree. Item 3-16: 0 = No; 2 = Yes.

‡Measured using the Wilcoxon Signed Ranks Test.

§Not applicable.

¶Measured over all three measurements at once using the Friedman Test.

#Answers included only those respondents who, at the first measurement, declared not to have reported an incident within the last six months (item 11).

**Number of incidents - number of residents.