The assessment of pathologists/laboratory medicine physicians through a multisource feedback tool.
Abstract: * Context.--There is increasing interest in ensuring that physicians demonstrate the full range of Accreditation Council for Graduate Medical Education competencies.

Objective.--To determine whether it is possible to develop a feasible and reliable multisource feedback instrument for pathologists and laboratory medicine physicians.

Design.--Surveys with 39, 30, and 22 items were developed to assess individual physicians by 8 peers, 8 referring physicians, and 8 coworkers (eg, technologists, secretaries), respectively, using 5-point scales and an unable-to-assess category. Physicians completed a self-assessment survey. Items addressed key competencies related to clinical competence, collaboration, professionalism, and communication.

Results.--Data from 101 pathologists and laboratory medicine physicians were analyzed. The mean number of respondents per physician was 7.6, 7.4, and 7.6 for peers, referring physicians, and coworkers, respectively. The reliability of the internal consistency, measured by Cronbach [alpha], was [greater than or equal to] .95 for the full scale of all instruments. Analysis indicated that the medical peer, referring physician, and coworker instruments achieved a generalizability coefficient of .78, .81, and .81, respectively. Factor analysis showed 4 factors on the peer questionnaire accounted for 68.8% of the total variance: reports and clinical competency, collaboration, educational leadership, and professional behavior. For the referring physician survey, 3 factors accounted for 66.9% of the variance: professionalism, reports, and clinical competency. Two factors on the coworker questionnaire accounted for 59.9% of the total variance: communication and professionalism.

Conclusions.--It is feasible to assess this group of physicians using multisource feedback with instruments that are reliable.

(Arch Pathol Lab Med. 2009;133:1301-1308)
Article Type: Report
Subject: Pathologists (Surveys)
Medical technologists (Surveys)
Organizational behavior (Management)
Work environment (Management)
Medical care (Quality management)
Medical care (Analysis)
Authors: Lockyer, Jocelyn M.
Violato, Claudio
Fidler, Herta
Alakija, Pauline
Pub Date: 08/01/2009
Publication: Name: Archives of Pathology & Laboratory Medicine Publisher: College of American Pathologists Audience: Academic; Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2009 College of American Pathologists ISSN: 1543-2165
Issue: Date: August, 2009 Source Volume: 133 Source Issue: 8
Topic: Event Code: 200 Management dynamics Canadian Subject Form: Organizational behaviour Computer Subject: Company business management
Geographic: Geographic Scope: United States Geographic Code: 1USA United States
Accession Number: 230247122
Full Text: In the United States and Canada, the Accreditation Council for Graduate Medical Education (ACGME) competencies (1) and the Royal College of Physicians and Surgeons of Canada CanMEDS roles (2) are established, key parameters for performance in practice. The ACGME competencies are patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and system-based care. (1) The CanMEDS roles include medical expert, scholar, communicator, collaborator, professional, manager, and health advocate. (2)

These competencies for practice have stimulated a new look within and across medicine for both practitioners and physicians in training, in laboratory medicine and in other disciplines of medicine. For example, the Education Committee of the College of American Pathologists defined pathology-specific competencies and used data from a survey of pathologists to create educational courses targeted to maintenance of each competence category. (3) Similarly, the Academy of Clinical Laboratory Physicians and Scientists proposed a curriculum to define goals and objectives for training, provide guidelines for instructional methods, and provide some examples of how outcomes can be assessed. (4) It suggested that assessment tools might include record reviews, checklists, global ratings, simulations, 360[degrees] global ratings, portfolios, standardized oral examinations, written examinations, and procedure/case logs. (4)

Both education and assessment tools have begun to be developed in response. For example, the Mayo Clinic in Rochester, Minnesota, created a basic-competency leadership and management program for its residents and fellows. (5) The Henry Ford Hospital in Detroit, Michigan, has developed tools to assess "customer" satisfaction in anatomic pathology, specifically surveying physician-users about services available. (6) Tools that efficiently assess physicians across a broad range of competencies are needed. The 360[degrees] evaluation is one of the tools that the Academy of Clinical Laboratory Physicians and Scientists has recommended to assess ACGME competencies. (4) This evaluation is particularly helpful for assessing competencies related to patient care, interpersonal and communication skills, professionalism, and systems-based practice. In a 360[degrees] global rating, questionnaires completed by multiple individuals in the training sphere of influence assess performance of the trainee. Supervisory faculty, medical technologists, clinicians, residents with whom the resident has consultative interactions, and patients in whose care the resident participates during the course of his or her work are all appropriate respondents.4 Although the term 360[degrees] evaluation is very descriptive, the more common and accurate term is multisource feedback (MSF). (7) Multisource feedback is preferred in medicine and other organizations as a 360[degrees] assessment implies assessment by subordinates (direct reports) as well as supervisors and is inconsistent with the language and direction that team-based collaborative care is taking.

Multisource feedback is a useful way to provide feedback about behaviors that can be addressed and changed. It is currently used as part of revalidation and quality improvement for practicing physicians with examples of MSF to be found for most specialties. (7) Multisource feedback can be designed to elicit feedback about all of the competencies, although, there are more objective tools for assessing patient care and medical knowledge. (7) If this type of tool is to be adopted by residency programs, it is helpful for faculty members to also have experience with it.

Studies of MSF show that reliable and valid instruments (questionnaires) can be developed. (7-12) It appears feasible to develop quality improvement programs in which most of the physicians in the discipline can be assessed by 8-10 coworkers, 8-10 peers, and 25 patients. (6-11) This number of raters produces acceptable reliability for both the overall instrument and the physician being assessed. (6-11) Furthermore, given that the intent of MSF is to guide professional development, studies have shown that participating physicians will use their feedback data to guide the changes they make. (7,8,12) Although studies could not be located in which MSF was used to assess pathologists and laboratory medicine physicians (P/LMPs) by searching MEDLINE, MSF has been used in another investigative medicine area, namely, radiology. (11)

The main purpose of the present study was to conduct an MSF study of P/LMPs using data provided by P/LMP peers, referring physicians, and coworkers and by self-assessment to determine the feasibility and reliability of an MSF system as well as to develop some initial evidence for validity. Our study had several questions:

1. What is the feasibility of an assessment system for P/LMPs that provides feedback from a P/LMP's peer physicians, referring physicians, coworkers, and self?

2. What questions about a P/LMP's practice can assessors answer?

3. What are the score profiles for each of the items (ie, mean and standard deviation) on the surveys?

4. Do the items on a survey group together into meaningful scales to guide performance improvement?

5. Are the instruments reliable for both the practice of P/LMPs and for the individual physician who is assessed?

MATERIALS AND METHODS

The College of Physicians and Surgeons of Alberta, Physician Achievement Review Program (CPSA-PAR), began developing MSF instruments in 1996 (www.par-program.org). (13) This program is mandatory and requires that every physician participate on a 5-year cycle. The original goal of the program was to provide feedback to physicians about 6 broad categories of performance: medical knowledge and skills, attitudes and behavior, professional responsibilities, practice improvement activities, administrative skills, and personal health. Instruments have been developed and tested for several disciplines. (13) As part of this work, a set of instruments was created and tested psychometrically for P/LMPs.

The instruments (questionnaires) were developed by a working group of P/LMPs. The working group comprised physicians from academic appointments; from tertiary, community, and regional settings; and from the disciplines of anatomic pathology, clinical pathology, general pathology, microbiology, clinical chemistry, hematopathology, neuropathology, pediatric pathology, and forensic pathology. The committee drew on previous instruments (13) and the CanMEDS competencies (2) to develop the items for each of the 4 questionnaires (P/LMP peers, referring physician, coworker, and self). The goal for each survey was to have a reasonable number of items to provide feedback about most aspects of practice across the 3 surveys. Similar to the decisions reached for the assessment of radiologists, but unlike other PAR instruments, there was not a patient questionnaire component. Having 2 instruments for physician peers (P/LMP peers and referring physician) recognized that the feedback provided about practice would be optimized by having 2 sources of input, each with a discrete set of questions. After the committee had developed the questionnaires, the questionnaires were mailed to every physician working in P/LMP in Alberta, Canada, for feedback about the items. Questionnaires were modified following that feedback.

The final instruments for P/LMP peers, referring physicians, and coworkers (eg, technologists) consisted of 39, 30, and 22 items, respectively, as shown on Tables 1 through 4. The self-assessment section used the items from the peer survey but were written in the first person. Raters were asked to use a 5-point rating scale (from 1 [strongly disagree] to 5 [strongly agree]). All questionnaires provided respondents with the option of being able to indicate they were "unable to assess" the physician on the item.

Each physician was responsible for completing a self-assessment and for identifying the 8 P/LMP peers, 8 referring physicians, and 8 coworkers who could answer the questions on the survey. Previous work had established that raters chosen by people being assessed do not provide significantly different evaluations than those selected by a third party. (14) Each P/LMP was asked to provide 8 assessors for each survey because previous studies (6,7,8,10-12) have shown that 8 to 12 surveys are likely to produce a generalizability coefficient ([Ep.sup.2]) [greater than or equal to] .70, suggesting the data provided to each physician is reliable (stable). (7,9,11)

Pivotal Research Inc (Edmonton, Alberta, Canada), a private company that handles the CPSA-PAR program, recruited the physicians under the direction of the CPSA. These physicians had to be licensed to practice and have been in practice in the province for at least 3 years. Pivotal Research provided the assessors with copies of the questionnaire, and responses went directly to Pivotal Research. Pivotal Research initiated reviews with all 120 P/LMPs who met the eligibility criteria. The study was initiated January 31, 2007, and concluded December 4, 2007, when the data collection for the study was closed. Participation in the PAR program is mandatory for all physicians in the province, although physicians can be exempt or their participation deferred because they are no longer practicing in the province, have retired, or have been in their current location for less than 3 years, or by providing evidence of extenuating personal circumstances. Pivotal Research provided a data set for the physicians but removed identities from the data. No identifying data (eg, gender, age, year or school of graduation from medical school, or location) were provided in the data set.

A number of statistical analyses were undertaken to address the research questions posed. Response rates were used to determine feasibility for each of the respondent groups (question 1). For each item on each survey, the percentage of the unable-to-assess answers, along with the mean and standard deviation, was computed to determine the viability of the items and the score profiles (questions 2 and 3, respectively). Items in which the unable-to-assess answer exceeded 15% on a survey might be in need of revision or deletion.

We used exploratory factor analysis to determine which items on each survey belonged together (ie, becoming a factor or scale; research question 4). This analysis allowed us to identify the factors and the number of factors for each instrument to describe the relative variance accounted for by each factor and its coherence. In this study, using individual-physician data as the unit of analysis for each survey, the items were intercorrelated using Pearson product moment correlations. The correlation matrix was then decomposed into principal components, and these were subsequently rotated to the normalized varimax criterion. Items were considered to be part of a factor if their primary loading was on that factor. The number of factors to be extracted was based on the Kaiser rule (ie, eigenvalues >1.0). In this analysis, we eliminated items with unable-to-respond rates [greater than or equal to] 50% because they represented items requiring considerable review.

The factors or scales established through exploratory factor analysis were used to establish the key domains (eg, clinical competence) for improvement, whereas the items within each factor provided more precise information about specific behaviors (eg, works at a reasonable pace, works to resolve conflict in the workplace, has effective verbal communication skills). Physician improvement could be guided by the scores on factors or items. This analysis made it possible to determine whether the instrument items were aligned into the appropriate constructs (factors) as intended.

Instrument reliability (stability) was assessed (research question 5). Internal consistency reliability was examined by calculating the Cronbach a coefficient for each of the rater groups and for each of the scales or factors for each rater group. This calculation provided an assessment of the overall internal consistency for each instrument as well as for each factor within the instruments. This analysis was followed by a generalizability analysis to determine the [Ep.sup.2] to ensure there were sufficient numbers of items and raters to provide stable data for each individual physician on each instrument. Normally, an [Ep.sup.2] [greater than or equal to] .70 suggests data are stable. (7,9,11) If the [Ep.sup.2] is low, it suggests that more raters or more items are required to enhance stability.

This study received approval from the University of Calgary, Conjoint Health Research Ethics Board, to undertake a psychometric analysis of the data collected.

RESULTS

We analyzed data for 101 of the 120 P/LMPs (84%). Of the physicians who participated, 63 were in anatomic pathology (of these 8 were also general pathologists), 23 were general pathologists, 6 were medical microbiologists, 3 were hematopathologists, 3 were infectious diseases specialists (of whom 2 were medical microbiologists), 2 were neuropathologists, and 1 had an internal medicine certificate. Of the 120 reviews initiated, 11 (9%) were deferred or exempt. Data from another 8 physicians were in process (ie, awaiting responses) and were not made available for analysis at the time data collection was closed. Data related to the physicians' sociodemographic background were not provided.

Of the 808 questionnaires available, there were 738 peer questionnaires (91%), 730 referring physician questionnaires (90%), and 742 coworker questionnaires (92%) based on 8 surveys per source per physician. Almost all of the physicians (100/101 or 99%) completed the self-assessment. Most of the physicians had data from all 4 sources (peers, referring physicians, coworkers, and self); 94 had all 4 sources, 5 had 3 sources, and 2 physicians had 1 source. The mean number of responses per participant was 7.6, 7.4, and 7.6 for peers, referring physicians, and coworkers, respectively.

Most items on the questionnaires could be answered by respondents. As presented in Tables 1, 3, and 4, the number of unable-to-assess items showed that 3 items (of 30; 10%) on the referring physician questionnaire and 4 items (of 22; 18%) on the coworker questionnaire had unable-to-assess rates [greater than or equal to] 15%. There were no items on the peer or self questionnaire with unable-to-assess rates [greater than or equal to] 15%. With the exception of 1 item on the self-assessment questionnaire, all items had means between 4 and 5 on the 5-point scale. The means on the self-assessment were lower than the means on the medical peer questionnaire.

The factor analysis showed that the data on the medical peer questionnaire decomposed into 4 factors that accounted for 68.8% of the total variance: reports and clinical competence, collaboration, educational leadership, and professional behavior (Table 2). For the referring physician survey, the items with 50% or more unable-to-assess items were eliminated, and a pairwise deletion was used, resulting in 3 factors that accounted for 66.9% of the variance: professionalism, reports, and clinical competency (Table 3). For the coworker questionnaire, the items with 50% or more unable-to-assess items were eliminated, and a pairwise deletion was used, resulting in 2 factors that accounted for 59.9% of the variance: communication and professionalism (Table 4).

Reliability analyses (Cronbach a reliability of internal consistency) indicated that all of the instruments' full scales had high internal consistency (Cronbach [alpha] > .95). The reliability for the factors (subscales) within each questionnaire had high internal consistency (Cronbach [alpha] > .87). The [Ep.sup.2] analysis showed that the medical peer, referring physicians, and coworker instruments resulted in an [Ep.sup.2] of .78, .81, and .80, respectively.

COMMENT

In this study, we developed and evaluated a set of MSF questionnaires to assess P/LMPs by peers, referring physicians, and coworkers to assess the feasibility and reliability of the instruments and to begin to develop evidence for validity. Physicians were assessed on a number of aspects of practice that the regulatory authority and the physicians themselves (through their participation on the committee and comments about the questionnaires) believed to be important. Although not designed to specifically assess ACGME (1) or CanMEDS (2) competencies, the items and the factors suggest that this instrument does assess some aspects of both sets of competencies. However, to develop a tool that actually assessed either set of competencies would require the addition of new items and retesting of the instrument and its factors.

We believe we met the goals of this assessment. This type of assessment is feasible in our setting as demonstrated by our very high response rates. Although this is partly explained by the fact that it is mandatory for physicians to participate and failure to participate can result in loss of license to practice, there is no legislation requiring their respondents (physicians or coworkers) to participate. These rates are consistent with the response rates for other groups of Canadian physicians (7,9,11) who have participated in MSF in conjunction with a regulatory authority.

The instruments are reliable at both an instrument and individual practitioner level in our setting. The reliability analysis (Cronbach [alpha]) suggests both the instrument and the scales are internally consistent. Furthermore, the [Ep.sup.2] data indicate that the data provided to each physician were also stable across raters. The instruments had sufficient numbers of items and assessors to provide reliable data. These findings are equivalent to, or higher than, those found in other studies. (7)

We have some initial evidence for the validity of the instruments, recognizing that establishing validity is a process and not a one-time study. Almost all of the questions could be answered by the responding physicians and coworkers. However, there were items on all questionnaires that many of the respondents were unable to assess; these items need to be examined. Some may be amenable to modification. Others may need to be deleted. On the coworker instrument, items that asked coworkers about how the physician worked with patients could not be answered. On the referring physician questionnaire, the items querying leadership and ethical behaviors need to be reconsidered. Peers had difficulty with items in which they did not routinely observe the physician (eg, discussion about alternate laboratory tests, undertaking technical procedures, managing conflicts of interest and committee work). Although many respondents could and did answer these questions, there is a tendency with MSF, particularly, from assessed physicians who are not satisfied with their results, to disregard the entire data set when they see items on surveys that cannot be answered.

The score profiles are negatively skewed. The range and the mean ratings were high, with most physicians receiving all of their ratings between 4 and 5. These profiles are similar to those of other groups. (7) Similarly, the self-ratings were lower than those provided by P/LMPs peers, a finding similar to those found in other studies of this nature. (7)

Our exploratory factor analyses found that items did group together into factors in ways that are consistent with the intent of the PAR program. The CPSA, as a regulatory authority, is concerned about professionalism, collaboration, communication, and clinical performance. As such, the factors we identified provide the general direction for physician improvement, whereas the individual items provide more specific feedback. By providing data by source, the P/LMPs can assess and compare the information provided by each of the groups. Each physician received descriptive data (means and standard deviations) on the scales and individual items for himself or herself as well as for the group as a whole.

Additional work will be required to examine the validity of the instruments. For example, it would be useful to determine whether physicians who are high performers on this assessment are high performers on other assessments that more objectively examine performance (eg, accuracy of laboratory reports). There are limitations to this study. This study focused on physicians in 1 province in Canada. All P/LMPs in the province were recruited because the program is mandatory. Data for 19 physicians were incomplete when the data collection was closed. However, in work of this nature, the pragmatics of providing feedback to physicians in a timely manner necessitated that comparator data be calculated so the physicians could receive their reports within a reasonable time frame. There is little reason to suspect that these physicians had different profiles than the 101 physicians whose data were available to us. We cannot be certain that P/LMPs in other parts of Canada or elsewhere in the world would have similar performance profiles. This study, like all but one MSF study, (14) permitted participating physicians to identify respondents. Although this may introduce bias, the CPSA wanted a system of administration that was feasible across all specialty groups. The results are dependent on the honesty of the professionals involved, and it is hoped that the assessors have a vested interest in the outcome. If future studies with these instruments, given the reality that P/LMPs are found in settings in which there is a hierarchy and known collaborations, it may be possible to assign raters for a further assessment of the validity of the instruments.

Multisource feedback is relatively new. With the advent of the ACGME (1) and CanMEDS (2) competencies, MSF has a certain appeal as a way to inform physicians about professionalism, collaboration, and communication, so that they can improve in directed ways. Multisource feedback appears to be a relatively inexpensive way of assessing these competencies and assessing the changes physicians make based on the feedback received. Although the present study did not examine the use physicians made of their MSF data, this would be another legitimate scholarly inquiry. A follow-up study to determine how the physicians used their data, the changes they made as a result of the feedback, and their perceptions of this type of assessment is certainly warranted and has been undertaken in other MSF work. (7,8,12) At an institutional or provincial level, aggregate data for this group of physicians could be used as part of a needs assessment to guide educational programs or institutional policy or procedures. Some discussions have taken place to see how these data can be used to improve care within health regions. Although not possible in our setting, correlating MSF data to other more objective data would further establish the evidence for the validity of this approach to assessment. Although this tool was not developed for residents, it is likely that many of the questions and the approach could be adapted for resident assessment.

We believe our MSF instruments for P/LMPs provide a viable method of assessing physicians comprehensively and providing guided feedback on a number of competencies and behaviors. The instruments were developed with a regulatory authority as a quality improvement program. Although the items focus on the needs of a regulatory authority, their breadth and scope may provide a base set of items on which to assess aspects of ACGME (1) and CanMEDS (2) competencies for other practicing physicians and residents.

The authors thank the College of Physicians and Surgeons of Alberta, namely, Trevor Theman, Registrar; John Swiniarski, Assistant Registrar; and Bryan Ward, Associate Registrar, for permission to conduct the assessment and Steve Dennis, President, Pivotal Research Inc, for the anonymous data set on which this study was conducted. The support of the working group is also appreciated: Pauline Alakija, Hallgrimur Benediktsson, Richard Berendt, Valerie Boras, Gwen Clark, Maire Duggan, Valerie Boras, Ray Lewkonia, Tony Morris, Lakshmi Puttagunta, Nargis Rayani, and George Wood.

References

(1.) Accreditation Council for Graduate Medical Education. ACGME competencies: July 1, 2007, requirements Web site. http://www.acgme.org/outcome/ comp/GeneralCompetenciesStandards21307.pdf. Accessed May 16, 2008.

(2.) Royal College of Physicians and Surgeons of Canada. The CanMEDS roles framework. http://www.rcpsc.medical.org/canmeds/index.php. Published 2005. Accessed May 16, 2008.

(3.) Hammond MEH, Filling CM, Neumann AR, Homburger HA. Addressingthe maintenance of certification challenge, Arch Pathol Lab Med. 2005;12 9:666-675.

(4.) Smith BR, Wells A, Alexander B, Bovill E, et al; for the Academy of Clinical Laboratory Physicians and Scientists. Curriculum content and evaluation of residency competency in clinical pathology (laboratory medicine): a proposal. Hum Pathol. 2006;37(8):934-968.

(5.) Hemmer PR, Karon BS, Hernandez JS, Cuthbert C, Fidler ME, Tazelaar HD. Leadership and management training for residents and fellows: a curriculum for future medical directors, Arch Pathol Lab Med. 2007;131(4):610-614.

(6.) Zarbo RJ. Determining customer satisfaction in anatomic pathology. Arch Pathol Lab Med. 2006;130(5):645-649.

(7.) Lockyer JM, Clyman SG. Multisource feedback(360-degreeevaluation). In: ES Holmboe, RE Hawkins, eds. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby Elsevier; 2008:75-85.

(8.) Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med. 2002;77(10)(suppl):S64-S66.

(9.) Violato C, Lockyer J. Selfand peer assessmentofpediatricians, psychiatrists and medicine specialists: implications for self-directed learning. Adv Health Sci Educ Theory Pract. 2006;11(3):235-244.

(10.) Archer J, Norcini J, Southgate L, Heard S, Davies H. mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? [published online ahead of print October 12, 2006]. Adv Health Sci Educ Theory Pract. 2006;13(2):181-192. doi: 10.1007/s10459-006-9033-3.

(11.) Lockyer JM, Violato C, Fidler HM. Assessment of radiology physicians by a regulatory authority [published online ahead of print March 28, 2008]. Radiology. 2008;247(3):771-778. doi: 10.1148/radiol.2473071431.

(12.) Fidler H, Lockyer J, Toews J, Violato C. Changing physicians' practices: the effect of individual feedback. Acad Med. 1999;74(6):702-714.

(13.) College of Physicians and Surgeons of Alberta. Physician Achievement Review program Web site. http://www.par-program.org/PAR-Inst.htm. Published 2005. Accessed May 16, 2008.

(14.) Ramsey PC, Wenrich MD, Carline JD, Inui TS, Larson EB, LeGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269(13):1655-1660.

Jocelyn M. Lockyer, PhD; Claudio Violato, PhD; Herta Fidler, MSc; Pauline Alakija, MD, FRCPC

Accepted for publication October 15, 2008.

From the Departments of Community Health Sciences (Drs Lockyer and Violato), Continuing Medical Education and Professional Development (Dr Lockyer and Ms Fidler), and Pathology (Dr Alakija), Faculty of Medicine, University of Calgary, Alberta, Canada.

The authors have no relevant financial interest in the products or companies described in this article.

Reprints: Jocelyn M. Lockyer, PhD, University of Calgary, Continuing Medical Education and Professional Development, 3330 Hospital Dr NW, Calgary, AB T2N 0L1, Canada (e-mail: lockyer@ucalgary.ca).
Table 1. Item Analysis of Pathologists and Laboratory Medicine
Physicians (P/LMPs) Peer and Self Questionnaires

                                           P/LMPs Peers

Items                           n     Mean     SD    UA, % (No.)

 1. Reports are clear           708   4.66   0.499    3.9 (29)
 2. Reports are concise         705   4.60   0.573    4.2 (31)
 3. Reports are correct         704   4.67   0.488    4.5 (33)
 4. Reports are timely          687   4.54   0.635    6.8 (50)
 5. Reports are complete        702   4.68   0.487    4.6 (34)

 6. Reports answer the          703   4.66   0.481    4.2 (31)
    intended questions

 7. Reports make
    appropriate management      666   4.60   0.543    9.6 (71)
    recommendations

 8. Reports discuss the
    limitations of the study    655   4.56   0.566   11.1 (82)
    reported

 9. Reports offer a
    differential diagnosis,     693   4.65   0.490    6.1 (45)
    where  appropriate

10. Physician applies           670   4.70   0.470    9.2 (68)
    safe laboratory practice

11. Physician discusses
    alternatives to tests       619   4.56   0.540   16.1 (119)
    that may be of limited
    clinical usefulness

12. Physician consults
    with laboratory             722   4.70   0.480    2.2 (16)
    physicians
    appropriately

13. Physician works
    within the limits of        727   4.68   0.474    1.5 (11)
    his/her  expertise

14. Physician is able to        730   4.69   0.487    1.1 (8)
    analyze complex problems

15. Physician is                717   4.64   0.550    2.8 (21)
    reasonably available for
    consultation

16. Physician
    participates in             691   4.64   0.528    6.4 (47)
    professional development
    activities

17. Physician contributes
    to the education of         697   4.48   0.630    5.6 (41)
    laboratory physician
    colleagues

18. Physician contributes
    to the education of         652   4.49   0.631   11.5 (85)
    clinical colleagues

19. Physician makes             719   4.61   0.500    2.6 (19)
    relevant clinical
    correlations

20. Physician contributes       667   4.57   0.547    9.6 (71)
    to quality improvement

    21. Physician uses
    resources in an efficient   682   4.54   0.550    7.5 (55)
    and  appropriate way

22. Physician shares
    workload responsibly with   674   4.55   0.580    8.5 (63)
    colleagues

23. Physician                   723   4.60   0.557    1.9 (14)
    collaborates with
    colleagues

24. Physician performs
    technical procedures        527   4.63   0.498   19.4 (143)
    competently

25. Physician maintains a
    healthy balance between     595   4.49   0.615    9.2 (68)
    professional and personal
    responsibilities

26. Physician
    demonstrates appropriate    669   4.47   0.608   14.5 (107)
    behavior in stressful
    situations

27. Physician works well
    with residents and other    631   4.60   0.531   10.8 (80)
    trainees

28. Physician works well
    with nurses,                656   4.60   0.538    2.3 (17)
    technologists,
    transcriptionists, and
    other support staff

29. Physician accepts
    responsibility for          721   4.67   0.489    2.3 (17)
    his/her professional
    actions

30. Physician accepts           694   4.52   0.588    6.0 (44)
    feedback constructively

31. Physician exhibits          721   4.66   0.495    2.2 (16)
    ethical behavior toward
    peer

32. Physician exhibits
    professional behavior       722   4.66   0.515    1.9 (14)
    toward peer

33. Physician
    demonstrates effective      682   4.48   0.628    7.3 (54)
    leadership where
    appropriate

34. Physician manages
    conflicts of interest       597   4.48   0.586   18.7 (138)
    appropriately

35. Physician contributes       670   4.54   0.600    9.2 (68)
    to departmental meetings

36. Physician
    participates in facility    620   4.45   0.665   16.0 (118)
    or regional  committees

37. Physician prioritizes
    professional duties
    effectively and             631   4.51   0.571   14.5 (107)
    appropriately when faced
    with multiple  patients
    and work-related problems

38. Physician gives             688   4.48   0.558    6.6 (49)
    constructive feedback

39. Physician works
    effectively with other
    professionals to prevent    667   4.41   0.645    9.6 (71)
    and/or resolve conflict

                                           P/LMPs Self

Items                           n     Mean     SD    UA, % (No.)

 1. Reports are clear            99   4.53   0.595     1.0 (1)
 2. Reports are concise          99   4.51   0.612     1.0 (1)
 3. Reports are correct          99   4.51   0.542     1.0 (1)
 4. Reports are timely           99   4.19   0.752     1.0 (1)
 5. Reports are complete         99   4.48   0.560     1.0 (1)

 6. Reports answer the           99   4.51   0.560     1.0 (1)
    intended questions

 7. Reports make
    appropriate management       97   4.23   0.621     3.0 (3)
    recommendations

 8. Reports discuss the
    limitations of the study     96   4.33   0.627     4.0 (4)
    reported

 9. Reports offer a
    differential diagnosis,      98   4.47   0.522     2.0 (2)
    where  appropriate

10. Physician applies           100   4.50   0.541     0
    safe laboratory practice

11. Physician discusses
    alternatives to tests        91   4.24   0.638     9.0 (9)
    that may be of limited
    clinical usefulness

12. Physician consults
    with laboratory             100   4.52   0.559     0
    physicians
    appropriately

13. Physician works
    within the limits of        100   4.52   0.522     0
    his/her  expertise

14. Physician is able to        100   4.45   0.557     0
    analyze complex problems

15. Physician is                100   4.44   0.715     0
    reasonably available for
    consultation

16. Physician
    participates in              99   4.41   0.589     1.0 (1)
    professional development
    activities

17. Physician contributes
    to the education of         100   4.22   0.705     0
    laboratory physician
    colleagues

18. Physician contributes
    to the education of          99   4.19   0.665     1.0 (1)
    clinical colleagues

19. Physician makes             100   4.39   0.530     0
    relevant clinical
    correlations

20. Physician contributes       100   4.28   0.637     0
    to quality improvement

    21. Physician uses
    resources in an efficient    99   4.36   0.579     0
    and  appropriate way

22. Physician shares
    workload responsibly with    97   4.41   0.554     2.0 (2)
    colleagues

23. Physician                    98   4.46   0.540     1.0 (1)
    collaborates with
    colleagues

24. Physician performs
    technical procedures         93   4.35   0.583     7.0 (7)
    competently

25. Physician maintains a
    healthy balance between     100   3.86   0.888     0
    professional and personal
    responsibilities

26. Physician
    demonstrates appropriate    100   4.27   0.548     0
    behavior in stressful
    situations

27. Physician works well
    with residents and other     94   4.36   0.602     6.0 (6)
    trainees

28. Physician works well
    with nurses,                100   4.51   0.522     0
    technologists,
    transcriptionists, and
    other support staff

29. Physician accepts
    responsibility for          100   4.59   0.514     0
    his/her professional
    actions

30. Physician accepts           100   4.34   0.555     0
    feedback constructively

31. Physician exhibits          100   4.54   0.540     0
    ethical behavior toward
    peer

32. Physician exhibits
    professional behavior       100   4.52   0.522     0
    toward peer

33. Physician
    demonstrates effective      100   4.24   0.638     0
    leadership where
    appropriate

34. Physician manages
    conflicts of interest        96   4.24   0.557     4.0 (4)
    appropriately

35. Physician contributes       100   4.19   0.706     0
    to departmental meetings

36. Physician
    participates in facility    100   4.04   0.803     0
    or regional  committees

37. Physician prioritizes
    professional duties
    effectively and             100   4.29   0.608     0
    appropriately when faced
    with multiple  patients
    and work-related problems

38. Physician gives              99   4.15   0.560     1.0 (1)
    constructive feedback

39. Physician works
    effectively with other
    professionals to prevent     99   4.22   0.599     1.0 (1)
    and/or resolve conflict

Abbreviation: UA, unable to assess.

Table 2. Peer Questionnaire Rotated Component Matrix

                                 Reports and
                                  Clinical
Items                            Competence    Collaboration

 1. Reports are clear               0.727
 2. Reports are concise             0.658
 3. Reports are correct             0.803
 4. Reports are timely              0.512
 5. Reports are complete            0.786

 6. Reports answer the
    intended questions              0.726

 7. Reports make
    appropriate management          0.644
    recommendations

 8. Reports discuss the
    limitations of the study
    reported                        0.675

 9. Reports offer a
    differential diagnosis,
    where appropriate               0.797

10. Physician applies
    safe laboratory practice        0.574

11. Physician discusses
    alternatives to tests
    that may be of limited
    clinical usefulness             0.560

12. Physician consults
    with laboratory
    physicians appropriately                       0.588

13. Physician works
    within the limits of
    his/her expertise               0.607

14. Physician is able to
    analyze complex problems        0.639

15. Physician is
    reasonably available for
    consultation                                   0.662

16. Physician
    participates in
    professional development
    activities                                     0.560

17. Physician contributes
    to the education of
    laboratory physician
    colleagues

18. Physician contributes
    to the education of
    clinical colleagues

19. Physician makes
    relevant clinical
    correlations                    0.565

20. Physician contributes
    to quality improvement

21. Physician uses
    resources in an efficient
    and appropriate way             0.440

22. Physician shares
    workload responsibly with
    colleagues                                     0.715

23. Physician
    collaborates with
    colleagues                                     0.683

25. Physician maintains a
    healthy balance between
    professional and
    personal responsibilities

26. Physician
    demonstrates appropriate
    behavior in stressful
    situations

27. Physician works well
    with residents and other
    trainees                                       0.526

28. Physician works well
    with nurses,
    technologists,
    transcriptionists, and
    other support staff                            0.588

29. Physician accepts
    responsibility for
    his/her professional
    actions                                        0.664

30. Physician accepts
    feedback constructively                        0.663

31. Physician exhibits
    ethical behavior toward
    peer                                           0.657

32. Physician exhibits
    professional behavior
    toward peer                                    0.663

33. Physician
    demonstrates effective
    leadership where
    appropriate

34. Physician manages
    conflicts of interest
    appropriately

35. Physician contributes
    to departmental meetings

36. Physician
    participates in facility
    or regional committees

37. Physician prioritizes
    professional duties
    effectively and
    appropriately when faced
    with multiple patients
    and work-related problems

38. Physician gives
    constructive feedback

39. Physician works
    effectively with other
    professionals to prevent
    and/or resolve conflict

Variance accounted for
  (overall 68.82), %               57.40           4.87

Cronbach [alpha]
  (overall = .98)                    .96            .94

                                 Educational   Professional
Items                            Leadership      Behavior

 1. Reports are clear
 2. Reports are concise
 3. Reports are correct
 4. Reports are timely
 5. Reports are complete

 6. Reports answer the
    intended questions

 7. Reports make
    appropriate management
    recommendations

 8. Reports discuss the
    limitations of the study
    reported

 9. Reports offer a
    differential diagnosis,
    where appropriate

10. Physician applies
    safe laboratory practice

11. Physician discusses
    alternatives to tests
    that may be of limited
    clinical usefulness

12. Physician consults
    with laboratory
    physicians appropriately

13. Physician works
    within the limits of
    his/her expertise

14. Physician is able to
    analyze complex problems

15. Physician is
    reasonably available for
    consultation

16. Physician
    participates in
    professional development
    activities

17. Physician contributes
    to the education of
    laboratory physician
    colleagues                       0.724

18. Physician contributes
    to the education of
    clinical colleagues              0.745

19. Physician makes
    relevant clinical
    correlations

20. Physician contributes
    to quality improvement           0.587

21. Physician uses
    resources in an efficient
    and appropriate way

22. Physician shares
    workload responsibly with
    colleagues

23. Physician
    collaborates with
    colleagues

25. Physician maintains a
    healthy balance between
    professional and
    personal responsibilities                      0.626

26. Physician
    demonstrates appropriate
    behavior in stressful
    situations                                     0.538

27. Physician works well
    with residents and other
    trainees

28. Physician works well
    with nurses,
    technologists,
    transcriptionists, and
    other support staff

29. Physician accepts
    responsibility for
    his/her professional
    actions

30. Physician accepts
    feedback constructively

31. Physician exhibits
    ethical behavior toward
    peer

32. Physician exhibits
    professional behavior
    toward peer

33. Physician
    demonstrates effective
    leadership where
    appropriate                                    0.532

34. Physician manages
    conflicts of interest
    appropriately                                  0.485

35. Physician contributes
    to departmental meetings         0.589

36. Physician
    participates in facility
    or regional committees           0.658

37. Physician prioritizes
    professional duties
    effectively and
    appropriately when faced                       0.568
    with multiple patients
    and work-related problems

38. Physician gives
    constructive feedback                          0.595

39. Physician works
    effectively with other
    professionals to prevent
    and/or resolve conflict                        0.590

Variance accounted for
  (overall 68.82), %                 3.86          2.69

Cronbach [alpha]
  (overall = .98)                     .87           .91

Table 3. Referring Physician Descriptive Statistics and Rotated
Component Matrix

                                                              UA,
Items                                  n    Mean    SD     % (No.)

 1. Reports are clear                 726   4.71   0.500     0.4 (3)

 2. Reports are concise               727   4.66   0.540     0.4 (3)

 3. Reports are correct               720   4.72   0.507     1.1 (8)

 4. Reports are clinically
    relevant                          726   4.72   0.488     0.4 (3)

 5. Reports are complete              727   4.70   0.524     0.4 (3)

 6. Reports are timely                725   4.55   0.681     0.5 (4)

 7. Reports take into
    account the clinical
    information I provide             715   4.69   0.522    1.8 (13)

 8. Reports make appropriate
    management  recommendations       654   4.49   0.682   10.0 (73)

 9. Reports suggest
    appropriate consultation/
    additional testing                692   4.59   0.601    5.1 (37)

10. Reports discuss the
    limitations of the report/
    investigation                     712   4.62   0.575    2.1 (15)

11. Reports offer a
    differential diagnosis,
    where  appropriate                707   4.63   0.559    3.0 (22)

12. The physician performs
    and communicates  urgent
    consultations promptly            700   4.74   0.504    4.1 (30)

13. The physician suggests
    appropriate laboratory
    tests                             656   4.57   0.602   10.1 (74)

14. The physician discusses
    alternatives to test
    requests that may be of
    limited clinical usefulness       609   4.55   0.647   16.2 (118)

15. The physician consults
    physician colleagues
    appropriately                     633   4.63   0.545   13.3 (97)

16. The physician works
    within the limits of
    his/her expertise                 681   4.66   0.510    6.7 (49)

17. The physician is able
    to analyze complex
    problems                          659   4.66   0.530    9.6 (70)

18. The physician is
    reasonably available for
    consultation                      710   4.71   0.502    2.6 (19)

19. The physician
    demonstrates knowledge of         704   4.68   0.492    3.4 (25)
    current medical practice

20. The physician
    participates effectively in
    educational rounds (eg,
    CPD, M&M rounds)                  496   4.65   0.541   32.1 (234)

21. The physician performs
    technical procedures
    competently (a)                   316   4.72   0.504   56.7 (414)

22. The physician accepts
    responsibility for his/her
    professional actions              641   4.69   0.487   12.2 (89)

23. The physician accepts
    feedback constructively           585   4.59   0.557   19.7 (144)

24. The physician exhibits
    professional behavior
    toward peer                       703   4.75   0.454    3.6 (26)

25. The physician exhibits
    ethical behavior toward
    peer                              667   4.77   0.431    8.4 (61)

26. The physician exhibits
    professional and ethical
    behavior (a) toward
    patients and their
    families                          320   4.73   0.463   56.0 (409)

27. The physician
    demonstrates effective
    leadership where
    appropriate                       510   4.63   0.552   30.0 (219)

28. The physician manages
    conflicts of interest
    appropriately (a)                 327   4.62   0.579   55.1 (402)

29. The physician
    collaborates with
    colleagues                        675   4.69   0.510    7.5 (55)

30. The physician seeks
    additional information
    where appropriate                 677   4.71   0.487    7.3 (53)

Cronbach [alpha] (.98
  overall)
Variance accounted for
  (overall 66.91), %

Items                                 Professionalism   Reports

 1. Reports are clear                                    0.825

 2. Reports are concise                                  0.762

 3. Reports are correct                                  0.787

 4. Reports are clinically
    relevant                                             0.770

 5. Reports are complete                                 0.789

 6. Reports are timely                                   0.511

 7. Reports take into
    account the clinical
    information I provide                                0.582

 8. Reports make appropriate
    management  recommendations

 9. Reports suggest
    appropriate consultation/
    additional testing

10. Reports discuss the
    limitations of the report/
    investigation

11. Reports offer a
    differential diagnosis,
    where  appropriate                                   0.561

12. The physician performs
    and communicates  urgent
    consultations promptly

13. The physician suggests
    appropriate laboratory
    tests

14. The physician discusses
    alternatives to test
    requests that may be of
    limited clinical usefulness

15. The physician consults
    physician colleagues
    appropriately                          0.574

16. The physician works
    within the limits of
    his/her expertise                      0.590

17. The physician is able
    to analyze complex
    problems                                             0.507

18. The physician is
    reasonably available for
    consultation                           0.600

19. The physician
    demonstrates knowledge of              0.565
    current medical practice

20. The physician
    participates effectively in
    educational rounds (eg,
    CPD, M&M rounds)                       0.559

21. The physician performs
    technical procedures
    competently (a)

22. The physician accepts
    responsibility for his/her
    professional actions                   0.692

23. The physician accepts
    feedback constructively                0.714

24. The physician exhibits
    professional behavior
    toward peer                            0.805

25. The physician exhibits
    ethical behavior toward
    peer                                   0.780

26. The physician exhibits
    professional and ethical
    behaviora toward patients
    and their families

27. The physician
    demonstrates effective
    leadership where
    appropriate                            0.662

28. The physician manages
    conflicts of interest
    appropriately (a)

29. The physician
    collaborates with
    colleagues                             0.669

30. The physician seeks
    additional information
    where appropriate                      0.591

Cronbach [alpha] (.98
  overall)                                   .94          .94
Variance accounted for
  (overall 66.91), %                       57.48         5.66

                                       Clinical
Items                                 competence

 1. Reports are clear

 2. Reports are concise

 3. Reports are correct

 4. Reports are clinically
    relevant

 5. Reports are complete

 6. Reports are timely

 7. Reports take into
    account the clinical
    information I provide

 8. Reports make appropriate
    management  recommendations         0.692

 9. Reports suggest
    appropriate consultation/
    additional testing                  0.696

10. Reports discuss the
    limitations of the report/
    investigation                       0.573

11. Reports offer a
    differential diagnosis,
    where  appropriate

12. The physician performs
    and communicates  urgent
    consultations promptly              0.513

13. The physician suggests
    appropriate laboratory
    tests                               0.763

14. The physician discusses
    alternatives to test
    requests that may be of
    limited clinical usefulness         0.716

15. The physician consults
    physician colleagues
    appropriately

16. The physician works
    within the limits of
    his/her expertise

17. The physician is able
    to analyze complex
    problems

18. The physician is
    reasonably available for
    consultation

19. The physician
    demonstrates knowledge of
    current medical practice

20. The physician
    participates effectively in
    educational rounds (eg,
    CPD, M&M rounds)

21. The physician performs
    technical procedures
    competently (a)

22. The physician accepts
    responsibility for his/her
    professional actions

23. The physician accepts
    feedback constructively

24. The physician exhibits
    professional behavior
    toward peer

25. The physician exhibits
    ethical behavior toward
    peer

26. The physician exhibits
    professional and ethical
    behaviora toward patients
    and their families

27. The physician
    demonstrates effective
    leadership where
    appropriate

28. The physician manages
    conflicts of interest
    appropriately (a)

29. The physician
    collaborates with
    colleagues

30. The physician seeks
    additional information
    where appropriate

Cronbach [alpha] (.98
  overall)                                .89
Variance accounted for
  (overall 66.91), %                     3.77

Abbreviations: UA, unable to assess; CPD, continuing professional
development; M&M, morbidity and mortality

(a) Item was not included in factor analysis because of low response
rate.

Table 4. Coworker Descriptive Statistics and Rotated Component Matrix

Item                               n    Mean     SD    UA, % (No.)

 1. Responds in a timely
    fashion                       738   4.36   0.758    0.5 (4)

 2. Listens effectively           740   4.38   0.689    0.3 (2)

 3. Treats me with respect        740   4.61   0.579    0.1 (1)

 4. Contributes to the
    teaching of coworkers         672   4.52   0.624    9.0 (67)

 5. Respects the training
    and knowledge of coworkers    709   4.50   0.600    4.3 (32)

 6. Demonstrates appropriate
    concern for coworker safety   627   4.35   0.631   15.2 (113)

 7. Demonstrates appropriate
    behavior in stressful
    situations                    683   4.22   0.776    7.4 (55)

 8. Works to resolve and/or
    prevent conflict in the
    workplace                     627   4.06   0.835   15.1 (112)

 9. Responds professionally
    and skillfully to urgent
    situations                    712   4.46   0.653    3.9 (29)

10. Demonstrates respect
    for coworkers regardless of
    sex, ethnicity, or
    disability                    731   4.53   0.641    1.5 (11)

11. Has effective verbal
    communication skills          740   4.36   0.713    0.3 (2)

12. Has effective written
    communication skills          678   4.37   0.643    8.6 (64)

13. Gives constructive
    feedback                      702   4.25   0.709    5.3 (39)

14. Is willing to take
    responsibility for errors
    in the laboratory             604   4.30   0.711   18.5 (137)

15. Is organized                  702   4.21   0.797    5.4 (40)

16. Is knowledgeable              732   4.72   0.478    1.2 (9)

17. Is available for
    questions                     738   4.51   0.619    0.4 (3)

18. Introduces him/herself
    appropriately to patients
    and/or family members (a)     177   4.56   0.610   76.0 (564)

19. Gives patients enough
    information to understand
    what will occur during the
    procedure (a)                 97    4.58   0.734   86.8 (644)

20. Respects patient
    confidentiality               605   4.67   0.497   18.5 (137)

21. Answers patient
    questions appropriately (a)   133   4.58   0.642   81.9 (608)

22. Demonstrates respect
    for patients regardless of
    sex, ethnicity, or
    disability (a)                310   4.67   0.541   58.2 (432)

Cronbach [alpha] (.95
  overall)

Variance accounted for
  (overall 59.93), %

Item                              Communication   Professionalism

 1. Responds in a timely
    fashion                                            0.569

 2. Listens effectively               0.711

 3. Treats me with respect                             0.587

 4. Contributes to the
    teaching of coworkers                              0.610

 5. Respects the training
    and knowledge of coworkers                         0.666

 6. Demonstrates appropriate
    concern for coworker safety                        0.608

 7. Demonstrates appropriate
    behavior in stressful
    situations                        0.648

 8. Works to resolve and/or
    prevent conflict in the
    workplace                         0.740

 9. Responds professionally
    and skillfully to urgent
    situations                        0.567

10. Demonstrates respect
    for coworkers regardless of
    sex, ethnicity, or
    disability                        0.552

11. Has effective verbal
    communication skills              0.828

12. Has effective written
    communication skills              0.765

13. Gives constructive
    feedback                          0.669

14. Is willing to take
    responsibility for errors
    in the laboratory                 0.619

15. Is organized                      0.649

16. Is knowledgeable                                   0.726

17. Is available for
    questions                                          0.708

18. Introduces him/herself
    appropriately to patients
    and/or family members (a)

19. Gives patients enough
    information to understand
    what will occur during the
    procedure (a)

20. Respects patient
    confidentiality                                    0.709

21. Answers patient
    questions appropriately (a)

22. Demonstrates respect
    for patients regardless of
    sex, ethnicity, or
    disability (a)

Cronbach [alpha] (.95
  overall)                            .92               .87

Variance accounted for
  (overall 59.93), %                54.18              5.76

Abbreviation: UA, unable to assess.

(a) Item was not included in factor analysis because of low response
rate.
Gale Copyright: Copyright 2009 Gale, Cengage Learning. All rights reserved.