|
The assessment of pathologists/laboratory medicine
physicians through a multisource feedback tool.
|
|
|
|
|
| Abstract: |
* Context.--There is increasing interest in ensuring that
physicians demonstrate the full range of Accreditation Council for
Graduate Medical Education competencies. Objective.--To determine whether it is possible to develop a feasible and reliable multisource feedback instrument for pathologists and laboratory medicine physicians. Design.--Surveys with 39, 30, and 22 items were developed to assess individual physicians by 8 peers, 8 referring physicians, and 8 coworkers (eg, technologists, secretaries), respectively, using 5-point scales and an unable-to-assess category. Physicians completed a self-assessment survey. Items addressed key competencies related to clinical competence, collaboration, professionalism, and communication. Results.--Data from 101 pathologists and laboratory medicine physicians were analyzed. The mean number of respondents per physician was 7.6, 7.4, and 7.6 for peers, referring physicians, and coworkers, respectively. The reliability of the internal consistency, measured by Cronbach [alpha], was [greater than or equal to] .95 for the full scale of all instruments. Analysis indicated that the medical peer, referring physician, and coworker instruments achieved a generalizability coefficient of .78, .81, and .81, respectively. Factor analysis showed 4 factors on the peer questionnaire accounted for 68.8% of the total variance: reports and clinical competency, collaboration, educational leadership, and professional behavior. For the referring physician survey, 3 factors accounted for 66.9% of the variance: professionalism, reports, and clinical competency. Two factors on the coworker questionnaire accounted for 59.9% of the total variance: communication and professionalism. Conclusions.--It is feasible to assess this group of physicians using multisource feedback with instruments that are reliable. (Arch Pathol Lab Med. 2009;133:1301-1308) |
|
|
|
| Article Type: | Report |
| Subject: |
Pathologists
(Surveys) Medical technologists (Surveys) Organizational behavior (Management) Work environment (Management) Medical care (Quality management) Medical care (Analysis) |
| Authors: |
Lockyer, Jocelyn M. Violato, Claudio Fidler, Herta Alakija, Pauline |
| Pub Date: | 08/01/2009 |
| Publication: | Name: Archives of Pathology & Laboratory Medicine Publisher: College of American Pathologists Audience: Academic; Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2009 College of American Pathologists ISSN: 1543-2165 |
| Issue: | Date: August, 2009 Source Volume: 133 Source Issue: 8 |
| Topic: | Event Code: 200 Management dynamics Canadian Subject Form: Organizational behaviour Computer Subject: Company business management |
| Geographic: | Geographic Scope: United States Geographic Code: 1USA United States |
| Accession Number: | 230247122 |
| Full Text: |
In the United States and Canada, the Accreditation Council for
Graduate Medical Education (ACGME) competencies (1) and the Royal
College of Physicians and Surgeons of Canada CanMEDS roles (2) are
established, key parameters for performance in practice. The ACGME
competencies are patient care, medical knowledge, practice-based
learning and improvement, interpersonal and communication skills,
professionalism, and system-based care. (1) The CanMEDS roles include
medical expert, scholar, communicator, collaborator, professional,
manager, and health advocate. (2) These competencies for practice have stimulated a new look within and across medicine for both practitioners and physicians in training, in laboratory medicine and in other disciplines of medicine. For example, the Education Committee of the College of American Pathologists defined pathology-specific competencies and used data from a survey of pathologists to create educational courses targeted to maintenance of each competence category. (3) Similarly, the Academy of Clinical Laboratory Physicians and Scientists proposed a curriculum to define goals and objectives for training, provide guidelines for instructional methods, and provide some examples of how outcomes can be assessed. (4) It suggested that assessment tools might include record reviews, checklists, global ratings, simulations, 360[degrees] global ratings, portfolios, standardized oral examinations, written examinations, and procedure/case logs. (4) Both education and assessment tools have begun to be developed in response. For example, the Mayo Clinic in Rochester, Minnesota, created a basic-competency leadership and management program for its residents and fellows. (5) The Henry Ford Hospital in Detroit, Michigan, has developed tools to assess "customer" satisfaction in anatomic pathology, specifically surveying physician-users about services available. (6) Tools that efficiently assess physicians across a broad range of competencies are needed. The 360[degrees] evaluation is one of the tools that the Academy of Clinical Laboratory Physicians and Scientists has recommended to assess ACGME competencies. (4) This evaluation is particularly helpful for assessing competencies related to patient care, interpersonal and communication skills, professionalism, and systems-based practice. In a 360[degrees] global rating, questionnaires completed by multiple individuals in the training sphere of influence assess performance of the trainee. Supervisory faculty, medical technologists, clinicians, residents with whom the resident has consultative interactions, and patients in whose care the resident participates during the course of his or her work are all appropriate respondents.4 Although the term 360[degrees] evaluation is very descriptive, the more common and accurate term is multisource feedback (MSF). (7) Multisource feedback is preferred in medicine and other organizations as a 360[degrees] assessment implies assessment by subordinates (direct reports) as well as supervisors and is inconsistent with the language and direction that team-based collaborative care is taking. Multisource feedback is a useful way to provide feedback about behaviors that can be addressed and changed. It is currently used as part of revalidation and quality improvement for practicing physicians with examples of MSF to be found for most specialties. (7) Multisource feedback can be designed to elicit feedback about all of the competencies, although, there are more objective tools for assessing patient care and medical knowledge. (7) If this type of tool is to be adopted by residency programs, it is helpful for faculty members to also have experience with it. Studies of MSF show that reliable and valid instruments (questionnaires) can be developed. (7-12) It appears feasible to develop quality improvement programs in which most of the physicians in the discipline can be assessed by 8-10 coworkers, 8-10 peers, and 25 patients. (6-11) This number of raters produces acceptable reliability for both the overall instrument and the physician being assessed. (6-11) Furthermore, given that the intent of MSF is to guide professional development, studies have shown that participating physicians will use their feedback data to guide the changes they make. (7,8,12) Although studies could not be located in which MSF was used to assess pathologists and laboratory medicine physicians (P/LMPs) by searching MEDLINE, MSF has been used in another investigative medicine area, namely, radiology. (11) 1. What is the feasibility of an assessment system for P/LMPs that provides feedback from a P/LMP's peer physicians, referring physicians, coworkers, and self? 2. What questions about a P/LMP's practice can assessors answer? 3. What are the score profiles for each of the items (ie, mean and standard deviation) on the surveys? 4. Do the items on a survey group together into meaningful scales to guide performance improvement? 5. Are the instruments reliable for both the practice of P/LMPs and for the individual physician who is assessed? MATERIALS AND METHODS The College of Physicians and Surgeons of Alberta, Physician Achievement Review Program (CPSA-PAR), began developing MSF instruments in 1996 (www.par-program.org). (13) This program is mandatory and requires that every physician participate on a 5-year cycle. The original goal of the program was to provide feedback to physicians about 6 broad categories of performance: medical knowledge and skills, attitudes and behavior, professional responsibilities, practice improvement activities, administrative skills, and personal health. Instruments have been developed and tested for several disciplines. (13) As part of this work, a set of instruments was created and tested psychometrically for P/LMPs. The instruments (questionnaires) were developed by a working group of P/LMPs. The working group comprised physicians from academic appointments; from tertiary, community, and regional settings; and from the disciplines of anatomic pathology, clinical pathology, general pathology, microbiology, clinical chemistry, hematopathology, neuropathology, pediatric pathology, and forensic pathology. The committee drew on previous instruments (13) and the CanMEDS competencies (2) to develop the items for each of the 4 questionnaires (P/LMP peers, referring physician, coworker, and self). The goal for each survey was to have a reasonable number of items to provide feedback about most aspects of practice across the 3 surveys. Similar to the decisions reached for the assessment of radiologists, but unlike other PAR instruments, there was not a patient questionnaire component. Having 2 instruments for physician peers (P/LMP peers and referring physician) recognized that the feedback provided about practice would be optimized by having 2 sources of input, each with a discrete set of questions. After the committee had developed the questionnaires, the questionnaires were mailed to every physician working in P/LMP in Alberta, Canada, for feedback about the items. Questionnaires were modified following that feedback. The final instruments for P/LMP peers, referring physicians, and coworkers (eg, technologists) consisted of 39, 30, and 22 items, respectively, as shown on Tables 1 through 4. The self-assessment section used the items from the peer survey but were written in the first person. Raters were asked to use a 5-point rating scale (from 1 [strongly disagree] to 5 [strongly agree]). All questionnaires provided respondents with the option of being able to indicate they were "unable to assess" the physician on the item. Each physician was responsible for completing a self-assessment and for identifying the 8 P/LMP peers, 8 referring physicians, and 8 coworkers who could answer the questions on the survey. Previous work had established that raters chosen by people being assessed do not provide significantly different evaluations than those selected by a third party. (14) Each P/LMP was asked to provide 8 assessors for each survey because previous studies (6,7,8,10-12) have shown that 8 to 12 surveys are likely to produce a generalizability coefficient ([Ep.sup.2]) [greater than or equal to] .70, suggesting the data provided to each physician is reliable (stable). (7,9,11) Pivotal Research Inc (Edmonton, Alberta, Canada), a private company that handles the CPSA-PAR program, recruited the physicians under the direction of the CPSA. These physicians had to be licensed to practice and have been in practice in the province for at least 3 years. Pivotal Research provided the assessors with copies of the questionnaire, and responses went directly to Pivotal Research. Pivotal Research initiated reviews with all 120 P/LMPs who met the eligibility criteria. The study was initiated January 31, 2007, and concluded December 4, 2007, when the data collection for the study was closed. Participation in the PAR program is mandatory for all physicians in the province, although physicians can be exempt or their participation deferred because they are no longer practicing in the province, have retired, or have been in their current location for less than 3 years, or by providing evidence of extenuating personal circumstances. Pivotal Research provided a data set for the physicians but removed identities from the data. No identifying data (eg, gender, age, year or school of graduation from medical school, or location) were provided in the data set. A number of statistical analyses were undertaken to address the research questions posed. Response rates were used to determine feasibility for each of the respondent groups (question 1). For each item on each survey, the percentage of the unable-to-assess answers, along with the mean and standard deviation, was computed to determine the viability of the items and the score profiles (questions 2 and 3, respectively). Items in which the unable-to-assess answer exceeded 15% on a survey might be in need of revision or deletion. We used exploratory factor analysis to determine which items on each survey belonged together (ie, becoming a factor or scale; research question 4). This analysis allowed us to identify the factors and the number of factors for each instrument to describe the relative variance accounted for by each factor and its coherence. In this study, using individual-physician data as the unit of analysis for each survey, the items were intercorrelated using Pearson product moment correlations. The correlation matrix was then decomposed into principal components, and these were subsequently rotated to the normalized varimax criterion. Items were considered to be part of a factor if their primary loading was on that factor. The number of factors to be extracted was based on the Kaiser rule (ie, eigenvalues >1.0). In this analysis, we eliminated items with unable-to-respond rates [greater than or equal to] 50% because they represented items requiring considerable review. The factors or scales established through exploratory factor analysis were used to establish the key domains (eg, clinical competence) for improvement, whereas the items within each factor provided more precise information about specific behaviors (eg, works at a reasonable pace, works to resolve conflict in the workplace, has effective verbal communication skills). Physician improvement could be guided by the scores on factors or items. This analysis made it possible to determine whether the instrument items were aligned into the appropriate constructs (factors) as intended. Instrument reliability (stability) was assessed (research question 5). Internal consistency reliability was examined by calculating the Cronbach a coefficient for each of the rater groups and for each of the scales or factors for each rater group. This calculation provided an assessment of the overall internal consistency for each instrument as well as for each factor within the instruments. This analysis was followed by a generalizability analysis to determine the [Ep.sup.2] to ensure there were sufficient numbers of items and raters to provide stable data for each individual physician on each instrument. Normally, an [Ep.sup.2] [greater than or equal to] .70 suggests data are stable. (7,9,11) If the [Ep.sup.2] is low, it suggests that more raters or more items are required to enhance stability. This study received approval from the University of Calgary, Conjoint Health Research Ethics Board, to undertake a psychometric analysis of the data collected. RESULTS We analyzed data for 101 of the 120 P/LMPs (84%). Of the physicians who participated, 63 were in anatomic pathology (of these 8 were also general pathologists), 23 were general pathologists, 6 were medical microbiologists, 3 were hematopathologists, 3 were infectious diseases specialists (of whom 2 were medical microbiologists), 2 were neuropathologists, and 1 had an internal medicine certificate. Of the 120 reviews initiated, 11 (9%) were deferred or exempt. Data from another 8 physicians were in process (ie, awaiting responses) and were not made available for analysis at the time data collection was closed. Data related to the physicians' sociodemographic background were not provided. Of the 808 questionnaires available, there were 738 peer questionnaires (91%), 730 referring physician questionnaires (90%), and 742 coworker questionnaires (92%) based on 8 surveys per source per physician. Almost all of the physicians (100/101 or 99%) completed the self-assessment. Most of the physicians had data from all 4 sources (peers, referring physicians, coworkers, and self); 94 had all 4 sources, 5 had 3 sources, and 2 physicians had 1 source. The mean number of responses per participant was 7.6, 7.4, and 7.6 for peers, referring physicians, and coworkers, respectively. Most items on the questionnaires could be answered by respondents. As presented in Tables 1, 3, and 4, the number of unable-to-assess items showed that 3 items (of 30; 10%) on the referring physician questionnaire and 4 items (of 22; 18%) on the coworker questionnaire had unable-to-assess rates [greater than or equal to] 15%. There were no items on the peer or self questionnaire with unable-to-assess rates [greater than or equal to] 15%. With the exception of 1 item on the self-assessment questionnaire, all items had means between 4 and 5 on the 5-point scale. The means on the self-assessment were lower than the means on the medical peer questionnaire. The factor analysis showed that the data on the medical peer questionnaire decomposed into 4 factors that accounted for 68.8% of the total variance: reports and clinical competence, collaboration, educational leadership, and professional behavior (Table 2). For the referring physician survey, the items with 50% or more unable-to-assess items were eliminated, and a pairwise deletion was used, resulting in 3 factors that accounted for 66.9% of the variance: professionalism, reports, and clinical competency (Table 3). For the coworker questionnaire, the items with 50% or more unable-to-assess items were eliminated, and a pairwise deletion was used, resulting in 2 factors that accounted for 59.9% of the variance: communication and professionalism (Table 4). Reliability analyses (Cronbach a reliability of internal consistency) indicated that all of the instruments' full scales had high internal consistency (Cronbach [alpha] > .95). The reliability for the factors (subscales) within each questionnaire had high internal consistency (Cronbach [alpha] > .87). The [Ep.sup.2] analysis showed that the medical peer, referring physicians, and coworker instruments resulted in an [Ep.sup.2] of .78, .81, and .80, respectively. COMMENT In this study, we developed and evaluated a set of MSF questionnaires to assess P/LMPs by peers, referring physicians, and coworkers to assess the feasibility and reliability of the instruments and to begin to develop evidence for validity. Physicians were assessed on a number of aspects of practice that the regulatory authority and the physicians themselves (through their participation on the committee and comments about the questionnaires) believed to be important. Although not designed to specifically assess ACGME (1) or CanMEDS (2) competencies, the items and the factors suggest that this instrument does assess some aspects of both sets of competencies. However, to develop a tool that actually assessed either set of competencies would require the addition of new items and retesting of the instrument and its factors. We believe we met the goals of this assessment. This type of assessment is feasible in our setting as demonstrated by our very high response rates. Although this is partly explained by the fact that it is mandatory for physicians to participate and failure to participate can result in loss of license to practice, there is no legislation requiring their respondents (physicians or coworkers) to participate. These rates are consistent with the response rates for other groups of Canadian physicians (7,9,11) who have participated in MSF in conjunction with a regulatory authority. The instruments are reliable at both an instrument and individual practitioner level in our setting. The reliability analysis (Cronbach [alpha]) suggests both the instrument and the scales are internally consistent. Furthermore, the [Ep.sup.2] data indicate that the data provided to each physician were also stable across raters. The instruments had sufficient numbers of items and assessors to provide reliable data. These findings are equivalent to, or higher than, those found in other studies. (7) We have some initial evidence for the validity of the instruments, recognizing that establishing validity is a process and not a one-time study. Almost all of the questions could be answered by the responding physicians and coworkers. However, there were items on all questionnaires that many of the respondents were unable to assess; these items need to be examined. Some may be amenable to modification. Others may need to be deleted. On the coworker instrument, items that asked coworkers about how the physician worked with patients could not be answered. On the referring physician questionnaire, the items querying leadership and ethical behaviors need to be reconsidered. Peers had difficulty with items in which they did not routinely observe the physician (eg, discussion about alternate laboratory tests, undertaking technical procedures, managing conflicts of interest and committee work). Although many respondents could and did answer these questions, there is a tendency with MSF, particularly, from assessed physicians who are not satisfied with their results, to disregard the entire data set when they see items on surveys that cannot be answered. The score profiles are negatively skewed. The range and the mean ratings were high, with most physicians receiving all of their ratings between 4 and 5. These profiles are similar to those of other groups. (7) Similarly, the self-ratings were lower than those provided by P/LMPs peers, a finding similar to those found in other studies of this nature. (7) Our exploratory factor analyses found that items did group together into factors in ways that are consistent with the intent of the PAR program. The CPSA, as a regulatory authority, is concerned about professionalism, collaboration, communication, and clinical performance. As such, the factors we identified provide the general direction for physician improvement, whereas the individual items provide more specific feedback. By providing data by source, the P/LMPs can assess and compare the information provided by each of the groups. Each physician received descriptive data (means and standard deviations) on the scales and individual items for himself or herself as well as for the group as a whole. Additional work will be required to examine the validity of the instruments. For example, it would be useful to determine whether physicians who are high performers on this assessment are high performers on other assessments that more objectively examine performance (eg, accuracy of laboratory reports). There are limitations to this study. This study focused on physicians in 1 province in Canada. All P/LMPs in the province were recruited because the program is mandatory. Data for 19 physicians were incomplete when the data collection was closed. However, in work of this nature, the pragmatics of providing feedback to physicians in a timely manner necessitated that comparator data be calculated so the physicians could receive their reports within a reasonable time frame. There is little reason to suspect that these physicians had different profiles than the 101 physicians whose data were available to us. We cannot be certain that P/LMPs in other parts of Canada or elsewhere in the world would have similar performance profiles. This study, like all but one MSF study, (14) permitted participating physicians to identify respondents. Although this may introduce bias, the CPSA wanted a system of administration that was feasible across all specialty groups. The results are dependent on the honesty of the professionals involved, and it is hoped that the assessors have a vested interest in the outcome. If future studies with these instruments, given the reality that P/LMPs are found in settings in which there is a hierarchy and known collaborations, it may be possible to assign raters for a further assessment of the validity of the instruments. Multisource feedback is relatively new. With the advent of the ACGME (1) and CanMEDS (2) competencies, MSF has a certain appeal as a way to inform physicians about professionalism, collaboration, and communication, so that they can improve in directed ways. Multisource feedback appears to be a relatively inexpensive way of assessing these competencies and assessing the changes physicians make based on the feedback received. Although the present study did not examine the use physicians made of their MSF data, this would be another legitimate scholarly inquiry. A follow-up study to determine how the physicians used their data, the changes they made as a result of the feedback, and their perceptions of this type of assessment is certainly warranted and has been undertaken in other MSF work. (7,8,12) At an institutional or provincial level, aggregate data for this group of physicians could be used as part of a needs assessment to guide educational programs or institutional policy or procedures. Some discussions have taken place to see how these data can be used to improve care within health regions. Although not possible in our setting, correlating MSF data to other more objective data would further establish the evidence for the validity of this approach to assessment. Although this tool was not developed for residents, it is likely that many of the questions and the approach could be adapted for resident assessment. We believe our MSF instruments for P/LMPs provide a viable method of assessing physicians comprehensively and providing guided feedback on a number of competencies and behaviors. The instruments were developed with a regulatory authority as a quality improvement program. Although the items focus on the needs of a regulatory authority, their breadth and scope may provide a base set of items on which to assess aspects of ACGME (1) and CanMEDS (2) competencies for other practicing physicians and residents. The authors thank the College of Physicians and Surgeons of Alberta, namely, Trevor Theman, Registrar; John Swiniarski, Assistant Registrar; and Bryan Ward, Associate Registrar, for permission to conduct the assessment and Steve Dennis, President, Pivotal Research Inc, for the anonymous data set on which this study was conducted. The support of the working group is also appreciated: Pauline Alakija, Hallgrimur Benediktsson, Richard Berendt, Valerie Boras, Gwen Clark, Maire Duggan, Valerie Boras, Ray Lewkonia, Tony Morris, Lakshmi Puttagunta, Nargis Rayani, and George Wood. References (1.) Accreditation Council for Graduate Medical Education. ACGME competencies: July 1, 2007, requirements Web site. http://www.acgme.org/outcome/ comp/GeneralCompetenciesStandards21307.pdf. Accessed May 16, 2008. (2.) Royal College of Physicians and Surgeons of Canada. The CanMEDS roles framework. http://www.rcpsc.medical.org/canmeds/index.php. Published 2005. Accessed May 16, 2008. (3.) Hammond MEH, Filling CM, Neumann AR, Homburger HA. Addressingthe maintenance of certification challenge, Arch Pathol Lab Med. 2005;12 9:666-675. (4.) Smith BR, Wells A, Alexander B, Bovill E, et al; for the Academy of Clinical Laboratory Physicians and Scientists. Curriculum content and evaluation of residency competency in clinical pathology (laboratory medicine): a proposal. Hum Pathol. 2006;37(8):934-968. (5.) Hemmer PR, Karon BS, Hernandez JS, Cuthbert C, Fidler ME, Tazelaar HD. Leadership and management training for residents and fellows: a curriculum for future medical directors, Arch Pathol Lab Med. 2007;131(4):610-614. (6.) Zarbo RJ. Determining customer satisfaction in anatomic pathology. Arch Pathol Lab Med. 2006;130(5):645-649. (7.) Lockyer JM, Clyman SG. Multisource feedback(360-degreeevaluation). In: ES Holmboe, RE Hawkins, eds. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby Elsevier; 2008:75-85. (8.) Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med. 2002;77(10)(suppl):S64-S66. (9.) Violato C, Lockyer J. Selfand peer assessmentofpediatricians, psychiatrists and medicine specialists: implications for self-directed learning. Adv Health Sci Educ Theory Pract. 2006;11(3):235-244. (10.) Archer J, Norcini J, Southgate L, Heard S, Davies H. mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? [published online ahead of print October 12, 2006]. Adv Health Sci Educ Theory Pract. 2006;13(2):181-192. doi: 10.1007/s10459-006-9033-3. (11.) Lockyer JM, Violato C, Fidler HM. Assessment of radiology physicians by a regulatory authority [published online ahead of print March 28, 2008]. Radiology. 2008;247(3):771-778. doi: 10.1148/radiol.2473071431. (12.) Fidler H, Lockyer J, Toews J, Violato C. Changing physicians' practices: the effect of individual feedback. Acad Med. 1999;74(6):702-714. (13.) College of Physicians and Surgeons of Alberta. Physician Achievement Review program Web site. http://www.par-program.org/PAR-Inst.htm. Published 2005. Accessed May 16, 2008. (14.) Ramsey PC, Wenrich MD, Carline JD, Inui TS, Larson EB, LeGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269(13):1655-1660. Jocelyn M. Lockyer, PhD; Claudio Violato, PhD; Herta Fidler, MSc; Pauline Alakija, MD, FRCPC Accepted for publication October 15, 2008. From the Departments of Community Health Sciences (Drs Lockyer and Violato), Continuing Medical Education and Professional Development (Dr Lockyer and Ms Fidler), and Pathology (Dr Alakija), Faculty of Medicine, University of Calgary, Alberta, Canada. The authors have no relevant financial interest in the products or companies described in this article. Reprints: Jocelyn M. Lockyer, PhD, University of Calgary, Continuing Medical Education and Professional Development, 3330 Hospital Dr NW, Calgary, AB T2N 0L1, Canada (e-mail: lockyer@ucalgary.ca). Table 1. Item Analysis of Pathologists and Laboratory Medicine
Physicians (P/LMPs) Peer and Self Questionnaires
P/LMPs Peers
Items n Mean SD UA, % (No.)
1. Reports are clear 708 4.66 0.499 3.9 (29)
2. Reports are concise 705 4.60 0.573 4.2 (31)
3. Reports are correct 704 4.67 0.488 4.5 (33)
4. Reports are timely 687 4.54 0.635 6.8 (50)
5. Reports are complete 702 4.68 0.487 4.6 (34)
6. Reports answer the 703 4.66 0.481 4.2 (31)
intended questions
7. Reports make
appropriate management 666 4.60 0.543 9.6 (71)
recommendations
8. Reports discuss the
limitations of the study 655 4.56 0.566 11.1 (82)
reported
9. Reports offer a
differential diagnosis, 693 4.65 0.490 6.1 (45)
where appropriate
10. Physician applies 670 4.70 0.470 9.2 (68)
safe laboratory practice
11. Physician discusses
alternatives to tests 619 4.56 0.540 16.1 (119)
that may be of limited
clinical usefulness
12. Physician consults
with laboratory 722 4.70 0.480 2.2 (16)
physicians
appropriately
13. Physician works
within the limits of 727 4.68 0.474 1.5 (11)
his/her expertise
14. Physician is able to 730 4.69 0.487 1.1 (8)
analyze complex problems
15. Physician is 717 4.64 0.550 2.8 (21)
reasonably available for
consultation
16. Physician
participates in 691 4.64 0.528 6.4 (47)
professional development
activities
17. Physician contributes
to the education of 697 4.48 0.630 5.6 (41)
laboratory physician
colleagues
18. Physician contributes
to the education of 652 4.49 0.631 11.5 (85)
clinical colleagues
19. Physician makes 719 4.61 0.500 2.6 (19)
relevant clinical
correlations
20. Physician contributes 667 4.57 0.547 9.6 (71)
to quality improvement
21. Physician uses
resources in an efficient 682 4.54 0.550 7.5 (55)
and appropriate way
22. Physician shares
workload responsibly with 674 4.55 0.580 8.5 (63)
colleagues
23. Physician 723 4.60 0.557 1.9 (14)
collaborates with
colleagues
24. Physician performs
technical procedures 527 4.63 0.498 19.4 (143)
competently
25. Physician maintains a
healthy balance between 595 4.49 0.615 9.2 (68)
professional and personal
responsibilities
26. Physician
demonstrates appropriate 669 4.47 0.608 14.5 (107)
behavior in stressful
situations
27. Physician works well
with residents and other 631 4.60 0.531 10.8 (80)
trainees
28. Physician works well
with nurses, 656 4.60 0.538 2.3 (17)
technologists,
transcriptionists, and
other support staff
29. Physician accepts
responsibility for 721 4.67 0.489 2.3 (17)
his/her professional
actions
30. Physician accepts 694 4.52 0.588 6.0 (44)
feedback constructively
31. Physician exhibits 721 4.66 0.495 2.2 (16)
ethical behavior toward
peer
32. Physician exhibits
professional behavior 722 4.66 0.515 1.9 (14)
toward peer
33. Physician
demonstrates effective 682 4.48 0.628 7.3 (54)
leadership where
appropriate
34. Physician manages
conflicts of interest 597 4.48 0.586 18.7 (138)
appropriately
35. Physician contributes 670 4.54 0.600 9.2 (68)
to departmental meetings
36. Physician
participates in facility 620 4.45 0.665 16.0 (118)
or regional committees
37. Physician prioritizes
professional duties
effectively and 631 4.51 0.571 14.5 (107)
appropriately when faced
with multiple patients
and work-related problems
38. Physician gives 688 4.48 0.558 6.6 (49)
constructive feedback
39. Physician works
effectively with other
professionals to prevent 667 4.41 0.645 9.6 (71)
and/or resolve conflict
P/LMPs Self
Items n Mean SD UA, % (No.)
1. Reports are clear 99 4.53 0.595 1.0 (1)
2. Reports are concise 99 4.51 0.612 1.0 (1)
3. Reports are correct 99 4.51 0.542 1.0 (1)
4. Reports are timely 99 4.19 0.752 1.0 (1)
5. Reports are complete 99 4.48 0.560 1.0 (1)
6. Reports answer the 99 4.51 0.560 1.0 (1)
intended questions
7. Reports make
appropriate management 97 4.23 0.621 3.0 (3)
recommendations
8. Reports discuss the
limitations of the study 96 4.33 0.627 4.0 (4)
reported
9. Reports offer a
differential diagnosis, 98 4.47 0.522 2.0 (2)
where appropriate
10. Physician applies 100 4.50 0.541 0
safe laboratory practice
11. Physician discusses
alternatives to tests 91 4.24 0.638 9.0 (9)
that may be of limited
clinical usefulness
12. Physician consults
with laboratory 100 4.52 0.559 0
physicians
appropriately
13. Physician works
within the limits of 100 4.52 0.522 0
his/her expertise
14. Physician is able to 100 4.45 0.557 0
analyze complex problems
15. Physician is 100 4.44 0.715 0
reasonably available for
consultation
16. Physician
participates in 99 4.41 0.589 1.0 (1)
professional development
activities
17. Physician contributes
to the education of 100 4.22 0.705 0
laboratory physician
colleagues
18. Physician contributes
to the education of 99 4.19 0.665 1.0 (1)
clinical colleagues
19. Physician makes 100 4.39 0.530 0
relevant clinical
correlations
20. Physician contributes 100 4.28 0.637 0
to quality improvement
21. Physician uses
resources in an efficient 99 4.36 0.579 0
and appropriate way
22. Physician shares
workload responsibly with 97 4.41 0.554 2.0 (2)
colleagues
23. Physician 98 4.46 0.540 1.0 (1)
collaborates with
colleagues
24. Physician performs
technical procedures 93 4.35 0.583 7.0 (7)
competently
25. Physician maintains a
healthy balance between 100 3.86 0.888 0
professional and personal
responsibilities
26. Physician
demonstrates appropriate 100 4.27 0.548 0
behavior in stressful
situations
27. Physician works well
with residents and other 94 4.36 0.602 6.0 (6)
trainees
28. Physician works well
with nurses, 100 4.51 0.522 0
technologists,
transcriptionists, and
other support staff
29. Physician accepts
responsibility for 100 4.59 0.514 0
his/her professional
actions
30. Physician accepts 100 4.34 0.555 0
feedback constructively
31. Physician exhibits 100 4.54 0.540 0
ethical behavior toward
peer
32. Physician exhibits
professional behavior 100 4.52 0.522 0
toward peer
33. Physician
demonstrates effective 100 4.24 0.638 0
leadership where
appropriate
34. Physician manages
conflicts of interest 96 4.24 0.557 4.0 (4)
appropriately
35. Physician contributes 100 4.19 0.706 0
to departmental meetings
36. Physician
participates in facility 100 4.04 0.803 0
or regional committees
37. Physician prioritizes
professional duties
effectively and 100 4.29 0.608 0
appropriately when faced
with multiple patients
and work-related problems
38. Physician gives 99 4.15 0.560 1.0 (1)
constructive feedback
39. Physician works
effectively with other
professionals to prevent 99 4.22 0.599 1.0 (1)
and/or resolve conflict
Abbreviation: UA, unable to assess.
Table 2. Peer Questionnaire Rotated Component Matrix
Reports and
Clinical
Items Competence Collaboration
1. Reports are clear 0.727
2. Reports are concise 0.658
3. Reports are correct 0.803
4. Reports are timely 0.512
5. Reports are complete 0.786
6. Reports answer the
intended questions 0.726
7. Reports make
appropriate management 0.644
recommendations
8. Reports discuss the
limitations of the study
reported 0.675
9. Reports offer a
differential diagnosis,
where appropriate 0.797
10. Physician applies
safe laboratory practice 0.574
11. Physician discusses
alternatives to tests
that may be of limited
clinical usefulness 0.560
12. Physician consults
with laboratory
physicians appropriately 0.588
13. Physician works
within the limits of
his/her expertise 0.607
14. Physician is able to
analyze complex problems 0.639
15. Physician is
reasonably available for
consultation 0.662
16. Physician
participates in
professional development
activities 0.560
17. Physician contributes
to the education of
laboratory physician
colleagues
18. Physician contributes
to the education of
clinical colleagues
19. Physician makes
relevant clinical
correlations 0.565
20. Physician contributes
to quality improvement
21. Physician uses
resources in an efficient
and appropriate way 0.440
22. Physician shares
workload responsibly with
colleagues 0.715
23. Physician
collaborates with
colleagues 0.683
25. Physician maintains a
healthy balance between
professional and
personal responsibilities
26. Physician
demonstrates appropriate
behavior in stressful
situations
27. Physician works well
with residents and other
trainees 0.526
28. Physician works well
with nurses,
technologists,
transcriptionists, and
other support staff 0.588
29. Physician accepts
responsibility for
his/her professional
actions 0.664
30. Physician accepts
feedback constructively 0.663
31. Physician exhibits
ethical behavior toward
peer 0.657
32. Physician exhibits
professional behavior
toward peer 0.663
33. Physician
demonstrates effective
leadership where
appropriate
34. Physician manages
conflicts of interest
appropriately
35. Physician contributes
to departmental meetings
36. Physician
participates in facility
or regional committees
37. Physician prioritizes
professional duties
effectively and
appropriately when faced
with multiple patients
and work-related problems
38. Physician gives
constructive feedback
39. Physician works
effectively with other
professionals to prevent
and/or resolve conflict
Variance accounted for
(overall 68.82), % 57.40 4.87
Cronbach [alpha]
(overall = .98) .96 .94
Educational Professional
Items Leadership Behavior
1. Reports are clear
2. Reports are concise
3. Reports are correct
4. Reports are timely
5. Reports are complete
6. Reports answer the
intended questions
7. Reports make
appropriate management
recommendations
8. Reports discuss the
limitations of the study
reported
9. Reports offer a
differential diagnosis,
where appropriate
10. Physician applies
safe laboratory practice
11. Physician discusses
alternatives to tests
that may be of limited
clinical usefulness
12. Physician consults
with laboratory
physicians appropriately
13. Physician works
within the limits of
his/her expertise
14. Physician is able to
analyze complex problems
15. Physician is
reasonably available for
consultation
16. Physician
participates in
professional development
activities
17. Physician contributes
to the education of
laboratory physician
colleagues 0.724
18. Physician contributes
to the education of
clinical colleagues 0.745
19. Physician makes
relevant clinical
correlations
20. Physician contributes
to quality improvement 0.587
21. Physician uses
resources in an efficient
and appropriate way
22. Physician shares
workload responsibly with
colleagues
23. Physician
collaborates with
colleagues
25. Physician maintains a
healthy balance between
professional and
personal responsibilities 0.626
26. Physician
demonstrates appropriate
behavior in stressful
situations 0.538
27. Physician works well
with residents and other
trainees
28. Physician works well
with nurses,
technologists,
transcriptionists, and
other support staff
29. Physician accepts
responsibility for
his/her professional
actions
30. Physician accepts
feedback constructively
31. Physician exhibits
ethical behavior toward
peer
32. Physician exhibits
professional behavior
toward peer
33. Physician
demonstrates effective
leadership where
appropriate 0.532
34. Physician manages
conflicts of interest
appropriately 0.485
35. Physician contributes
to departmental meetings 0.589
36. Physician
participates in facility
or regional committees 0.658
37. Physician prioritizes
professional duties
effectively and
appropriately when faced 0.568
with multiple patients
and work-related problems
38. Physician gives
constructive feedback 0.595
39. Physician works
effectively with other
professionals to prevent
and/or resolve conflict 0.590
Variance accounted for
(overall 68.82), % 3.86 2.69
Cronbach [alpha]
(overall = .98) .87 .91
Table 3. Referring Physician Descriptive Statistics and Rotated
Component Matrix
UA,
Items n Mean SD % (No.)
1. Reports are clear 726 4.71 0.500 0.4 (3)
2. Reports are concise 727 4.66 0.540 0.4 (3)
3. Reports are correct 720 4.72 0.507 1.1 (8)
4. Reports are clinically
relevant 726 4.72 0.488 0.4 (3)
5. Reports are complete 727 4.70 0.524 0.4 (3)
6. Reports are timely 725 4.55 0.681 0.5 (4)
7. Reports take into
account the clinical
information I provide 715 4.69 0.522 1.8 (13)
8. Reports make appropriate
management recommendations 654 4.49 0.682 10.0 (73)
9. Reports suggest
appropriate consultation/
additional testing 692 4.59 0.601 5.1 (37)
10. Reports discuss the
limitations of the report/
investigation 712 4.62 0.575 2.1 (15)
11. Reports offer a
differential diagnosis,
where appropriate 707 4.63 0.559 3.0 (22)
12. The physician performs
and communicates urgent
consultations promptly 700 4.74 0.504 4.1 (30)
13. The physician suggests
appropriate laboratory
tests 656 4.57 0.602 10.1 (74)
14. The physician discusses
alternatives to test
requests that may be of
limited clinical usefulness 609 4.55 0.647 16.2 (118)
15. The physician consults
physician colleagues
appropriately 633 4.63 0.545 13.3 (97)
16. The physician works
within the limits of
his/her expertise 681 4.66 0.510 6.7 (49)
17. The physician is able
to analyze complex
problems 659 4.66 0.530 9.6 (70)
18. The physician is
reasonably available for
consultation 710 4.71 0.502 2.6 (19)
19. The physician
demonstrates knowledge of 704 4.68 0.492 3.4 (25)
current medical practice
20. The physician
participates effectively in
educational rounds (eg,
CPD, M&M rounds) 496 4.65 0.541 32.1 (234)
21. The physician performs
technical procedures
competently (a) 316 4.72 0.504 56.7 (414)
22. The physician accepts
responsibility for his/her
professional actions 641 4.69 0.487 12.2 (89)
23. The physician accepts
feedback constructively 585 4.59 0.557 19.7 (144)
24. The physician exhibits
professional behavior
toward peer 703 4.75 0.454 3.6 (26)
25. The physician exhibits
ethical behavior toward
peer 667 4.77 0.431 8.4 (61)
26. The physician exhibits
professional and ethical
behavior (a) toward
patients and their
families 320 4.73 0.463 56.0 (409)
27. The physician
demonstrates effective
leadership where
appropriate 510 4.63 0.552 30.0 (219)
28. The physician manages
conflicts of interest
appropriately (a) 327 4.62 0.579 55.1 (402)
29. The physician
collaborates with
colleagues 675 4.69 0.510 7.5 (55)
30. The physician seeks
additional information
where appropriate 677 4.71 0.487 7.3 (53)
Cronbach [alpha] (.98
overall)
Variance accounted for
(overall 66.91), %
Items Professionalism Reports
1. Reports are clear 0.825
2. Reports are concise 0.762
3. Reports are correct 0.787
4. Reports are clinically
relevant 0.770
5. Reports are complete 0.789
6. Reports are timely 0.511
7. Reports take into
account the clinical
information I provide 0.582
8. Reports make appropriate
management recommendations
9. Reports suggest
appropriate consultation/
additional testing
10. Reports discuss the
limitations of the report/
investigation
11. Reports offer a
differential diagnosis,
where appropriate 0.561
12. The physician performs
and communicates urgent
consultations promptly
13. The physician suggests
appropriate laboratory
tests
14. The physician discusses
alternatives to test
requests that may be of
limited clinical usefulness
15. The physician consults
physician colleagues
appropriately 0.574
16. The physician works
within the limits of
his/her expertise 0.590
17. The physician is able
to analyze complex
problems 0.507
18. The physician is
reasonably available for
consultation 0.600
19. The physician
demonstrates knowledge of 0.565
current medical practice
20. The physician
participates effectively in
educational rounds (eg,
CPD, M&M rounds) 0.559
21. The physician performs
technical procedures
competently (a)
22. The physician accepts
responsibility for his/her
professional actions 0.692
23. The physician accepts
feedback constructively 0.714
24. The physician exhibits
professional behavior
toward peer 0.805
25. The physician exhibits
ethical behavior toward
peer 0.780
26. The physician exhibits
professional and ethical
behaviora toward patients
and their families
27. The physician
demonstrates effective
leadership where
appropriate 0.662
28. The physician manages
conflicts of interest
appropriately (a)
29. The physician
collaborates with
colleagues 0.669
30. The physician seeks
additional information
where appropriate 0.591
Cronbach [alpha] (.98
overall) .94 .94
Variance accounted for
(overall 66.91), % 57.48 5.66
Clinical
Items competence
1. Reports are clear
2. Reports are concise
3. Reports are correct
4. Reports are clinically
relevant
5. Reports are complete
6. Reports are timely
7. Reports take into
account the clinical
information I provide
8. Reports make appropriate
management recommendations 0.692
9. Reports suggest
appropriate consultation/
additional testing 0.696
10. Reports discuss the
limitations of the report/
investigation 0.573
11. Reports offer a
differential diagnosis,
where appropriate
12. The physician performs
and communicates urgent
consultations promptly 0.513
13. The physician suggests
appropriate laboratory
tests 0.763
14. The physician discusses
alternatives to test
requests that may be of
limited clinical usefulness 0.716
15. The physician consults
physician colleagues
appropriately
16. The physician works
within the limits of
his/her expertise
17. The physician is able
to analyze complex
problems
18. The physician is
reasonably available for
consultation
19. The physician
demonstrates knowledge of
current medical practice
20. The physician
participates effectively in
educational rounds (eg,
CPD, M&M rounds)
21. The physician performs
technical procedures
competently (a)
22. The physician accepts
responsibility for his/her
professional actions
23. The physician accepts
feedback constructively
24. The physician exhibits
professional behavior
toward peer
25. The physician exhibits
ethical behavior toward
peer
26. The physician exhibits
professional and ethical
behaviora toward patients
and their families
27. The physician
demonstrates effective
leadership where
appropriate
28. The physician manages
conflicts of interest
appropriately (a)
29. The physician
collaborates with
colleagues
30. The physician seeks
additional information
where appropriate
Cronbach [alpha] (.98
overall) .89
Variance accounted for
(overall 66.91), % 3.77
Abbreviations: UA, unable to assess; CPD, continuing professional
development; M&M, morbidity and mortality
(a) Item was not included in factor analysis because of low response
rate.
Table 4. Coworker Descriptive Statistics and Rotated Component Matrix
Item n Mean SD UA, % (No.)
1. Responds in a timely
fashion 738 4.36 0.758 0.5 (4)
2. Listens effectively 740 4.38 0.689 0.3 (2)
3. Treats me with respect 740 4.61 0.579 0.1 (1)
4. Contributes to the
teaching of coworkers 672 4.52 0.624 9.0 (67)
5. Respects the training
and knowledge of coworkers 709 4.50 0.600 4.3 (32)
6. Demonstrates appropriate
concern for coworker safety 627 4.35 0.631 15.2 (113)
7. Demonstrates appropriate
behavior in stressful
situations 683 4.22 0.776 7.4 (55)
8. Works to resolve and/or
prevent conflict in the
workplace 627 4.06 0.835 15.1 (112)
9. Responds professionally
and skillfully to urgent
situations 712 4.46 0.653 3.9 (29)
10. Demonstrates respect
for coworkers regardless of
sex, ethnicity, or
disability 731 4.53 0.641 1.5 (11)
11. Has effective verbal
communication skills 740 4.36 0.713 0.3 (2)
12. Has effective written
communication skills 678 4.37 0.643 8.6 (64)
13. Gives constructive
feedback 702 4.25 0.709 5.3 (39)
14. Is willing to take
responsibility for errors
in the laboratory 604 4.30 0.711 18.5 (137)
15. Is organized 702 4.21 0.797 5.4 (40)
16. Is knowledgeable 732 4.72 0.478 1.2 (9)
17. Is available for
questions 738 4.51 0.619 0.4 (3)
18. Introduces him/herself
appropriately to patients
and/or family members (a) 177 4.56 0.610 76.0 (564)
19. Gives patients enough
information to understand
what will occur during the
procedure (a) 97 4.58 0.734 86.8 (644)
20. Respects patient
confidentiality 605 4.67 0.497 18.5 (137)
21. Answers patient
questions appropriately (a) 133 4.58 0.642 81.9 (608)
22. Demonstrates respect
for patients regardless of
sex, ethnicity, or
disability (a) 310 4.67 0.541 58.2 (432)
Cronbach [alpha] (.95
overall)
Variance accounted for
(overall 59.93), %
Item Communication Professionalism
1. Responds in a timely
fashion 0.569
2. Listens effectively 0.711
3. Treats me with respect 0.587
4. Contributes to the
teaching of coworkers 0.610
5. Respects the training
and knowledge of coworkers 0.666
6. Demonstrates appropriate
concern for coworker safety 0.608
7. Demonstrates appropriate
behavior in stressful
situations 0.648
8. Works to resolve and/or
prevent conflict in the
workplace 0.740
9. Responds professionally
and skillfully to urgent
situations 0.567
10. Demonstrates respect
for coworkers regardless of
sex, ethnicity, or
disability 0.552
11. Has effective verbal
communication skills 0.828
12. Has effective written
communication skills 0.765
13. Gives constructive
feedback 0.669
14. Is willing to take
responsibility for errors
in the laboratory 0.619
15. Is organized 0.649
16. Is knowledgeable 0.726
17. Is available for
questions 0.708
18. Introduces him/herself
appropriately to patients
and/or family members (a)
19. Gives patients enough
information to understand
what will occur during the
procedure (a)
20. Respects patient
confidentiality 0.709
21. Answers patient
questions appropriately (a)
22. Demonstrates respect
for patients regardless of
sex, ethnicity, or
disability (a)
Cronbach [alpha] (.95
overall) .92 .87
Variance accounted for
(overall 59.93), % 54.18 5.76
Abbreviation: UA, unable to assess.
(a) Item was not included in factor analysis because of low response
rate. |
| Gale Copyright: | Copyright 2009 Gale, Cengage Learning. All rights reserved. |