Document Detail

Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey.
Jump to Full Text
MedLine Citation:
PMID:  23268489     Owner:  NLM     Status:  MEDLINE    
CONTEXT: Failure to notify patients of test results is common even when electronic health records (EHRs) are used to report results to practitioners. We sought to understand the broad range of social and technical factors that affect test result management in an integrated EHR-based health system.
METHODS: Between June and November 2010, we conducted a cross-sectional, web-based survey of all primary care practitioners (PCPs) within the Department of Veterans Affairs nationwide. Survey development was guided by a socio-technical model describing multiple inter-related dimensions of EHR use.
FINDINGS: Of 5001 PCPs invited, 2590 (51.8%) responded. 55.5% believed that the EHRs did not have convenient features for notifying patients of test results. Over a third (37.9%) reported having staff support needed for notifying patients of test results. Many relied on the patient's next visit to notify them for normal (46.1%) and abnormal results (20.1%). Only 45.7% reported receiving adequate training on using the EHR notification system and 35.1% reported having an assigned contact for technical assistance with the EHR; most received help from colleagues (60.4%). A majority (85.6%) stayed after hours or came in on weekends to address notifications; less than a third reported receiving protected time (30.1%). PCPs strongly endorsed several new features to improve test result management, including better tracking and visualization of result notifications.
CONCLUSIONS: Despite an advanced EHR, both social and technical challenges exist in ensuring notification of test results to practitioners and patients. Current EHR technology requires significant improvement in order to avoid similar challenges elsewhere.
Hardeep Singh; Christiane Spitzmueller; Nancy J Petersen; Mona K Sawhney; Michael W Smith; Daniel R Murphy; Donna Espadas; Archana Laxmisan; Dean F Sittig
Related Documents :
3618549 - Diagnosis of infective and inflammatory disorders by flow cytometric analysis of blood ...
23341179 - Bleeding risk assessment using whole blood impedance aggregometry and rotational thromb...
22932179 - Relationship between 6-minute walk test and pulmonary function test in stable chronic o...
8925419 - Recent advances in the laboratory diagnosis of equine parasitic diseases.
23761199 - Usefulness of pcr-hrma in identification of non-fermentative gram-negative rods recover...
23687549 - Rapid (13)c urea breath test to identify helicobacter pylori infection in emergency dep...
16624729 - An investigation into the role of chlamydophila spp. in bovine upper respiratory tract ...
19698289 - Assessment of semen quality, sperm cryopreservation and heterologous ivf in the critica...
23574879 - Patterns of executive dysfunction in amnestic mild cognitive impairment.
Publication Detail:
Type:  Journal Article; Research Support, N.I.H., Extramural; Research Support, U.S. Gov't, Non-P.H.S.     Date:  2012-12-25
Journal Detail:
Title:  Journal of the American Medical Informatics Association : JAMIA     Volume:  20     ISSN:  1527-974X     ISO Abbreviation:  J Am Med Inform Assoc     Publication Date:    2013 Jul-Aug
Date Detail:
Created Date:  2013-06-12     Completed Date:  2013-09-03     Revised Date:  2014-03-27    
Medline Journal Info:
Nlm Unique ID:  9430800     Medline TA:  J Am Med Inform Assoc     Country:  United States    
Other Details:
Languages:  eng     Pagination:  727-35     Citation Subset:  IM    
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms
Cross-Sectional Studies
Diagnostic Tests, Routine*
Electronic Health Records*
Medical Records Systems, Computerized
Physicians, Primary Care*
Primary Health Care
United States
Grant Support

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): J Am Med Inform Assoc
Journal ID (iso-abbrev): J Am Med Inform Assoc
Journal ID (hwp): amiajnl
Journal ID (publisher-id): jamia
ISSN: 1067-5027
ISSN: 1527-974X
Publisher: BMJ Publishing Group, BMA House, Tavistock Square, London, WC1H 9JR
Article Information
Download PDF
Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to
Received Day: 13 Month: 8 Year: 2012
Revision Received Day: 19 Month: 11 Year: 2012
Accepted Day: 20 Month: 11 Year: 2012
Print publication date: Month: 7 Year: 2013
Electronic publication date: Day: 25 Month: 12 Year: 2012
pmc-release publication date: Day: 25 Month: 12 Year: 2012
Volume: 20 Issue: 4
First Page: 727 Last Page: 735
PubMed Id: 23268489
ID: 3721157
Publisher Id: amiajnl-2012-001267
DOI: 10.1136/amiajnl-2012-001267

Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey
Hardeep Singh1
Christiane Spitzmueller2
Nancy J Petersen1
Mona K Sawhney1
Michael W Smith1
Daniel R Murphy1
Donna Espadas1
Archana Laxmisan1
Dean F Sittig3
1Department of Medicine, Houston VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center and Section of Health Services Research, Baylor College of Medicine, Houston, Texas, USA
2Department of Psychology, University of Houston, Houston, Texas, USA
3University of Texas School of Biomedical Informatics and UT-Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas, USA
Correspondence to Dr Hardeep Singh, Department of Medicine, Houston VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center (152), Section of Health Services Research, Baylor College of Medicine, 2002 Holcombe Blvd, Houston, TX 77030, USA;


Failure to follow up on abnormal test results (‘missed test results’) is a global patient safety concern.1–5 Electronic health records (EHRs) are increasingly used to notify practitioners of abnormal test results.6–8 EHR-based test result notification systems can reduce, if not eliminate, many of the communication problems inherent with paper-based records.9–12 However, ensuring that test results receive appropriate follow-up remains challenging even with electronic transmission.5, 13, 14 Previous studies of practitioner responses to abnormal test result notifications through EHRs found that 7–8% of abnormal test alerts had no evidence of timely follow-up, even when practitioners had electronically acknowledged result receipt.10, 11, 14

Although the proposed Stage 2 meaningful use regulations15 to be implemented in 2014 in the USA include structured laboratory result reporting functionality, little is known about how to optimize EHR-based test result management. Currently, several major EHR vendors employ a reporting functionality whereby providers receive notification of results in their inboxes (similar to email). This functionality is being increasingly adopted across EHRs and is likely to have a great impact on future test result management practices.

In complex healthcare settings that use EHRs, test result management encompasses many contextual factors such as clinical workflow, user behaviors, and organizational policies and procedures.16, 17 Electronically transmitted test results may be missed or overlooked for a variety of reasons related to EHR design and use.18, 19 For instance, while most EHRs deliver abnormal results to practitioners through an inbox, the inbox often also holds less important data (‘noise’) that could dilute important or urgent information (‘signals’).19 Other factors contributing to missed test results include policies and procedures related to the use of EHRs, such as unclear responsibility for follow-up of abnormal results when multiple providers are involved in patient care.10 Thus, contextual factors20 that affect test results management derive from both technical and social dimensions of the EHR-enabled health care system.

To better understand contextual factors that affect practitioners’ test results management practices within the setting of a comprehensive EHR, we designed and administered a nationwide survey to primary care practitioners (PCPs) practicing in all facilities of the Department of Veterans Affairs (VA). Our objective was to use survey data on PCPs’ experiences, practices, and preferences to create a body of knowledge upon which to base future improvements in EHR-based test result reporting systems.


Between June 10, 2010 and November 5, 2010, we conducted a cross-sectional, web-based survey of PCPs in VA settings nationwide. The local institutional review board approved the study. Within the VA system, most patients are assigned to staff PCPs, who serve as a coordinating hub for most care and thus depend heavily on the clinical information sent to their EHR inboxes. A smaller number of patients are assigned to trainees or subspecialists who see patients on a part-time basis, usually one half-day per week. Thus, these practitioners have a smaller panel (the total number of patients a PCP is responsible for) due to their other competing responsibilities. Our previous experience suggested that staff PCPs’ experiences, practices, and preferences related to test result management might be different from those of trainees and subspecialists who serve as PCPs.16, 18 Thus, we used a large administrative VA database, the Veterans Health Administration Support Service Center Clinical Care: Primary Care Management Module, to identify all PCPs with a minimum primary care patient panel size of 250 (N=5001). This strategy allowed us to exclude trainees and subspecialists because they are generally expected to have small panel sizes.

Survey development

Survey development was guided by literature review1, 6, 21–34 and by a conceptual model describing multiple inter-related dimensions of EHR use (see ‘Survey content areas’ section below; also see table 1 which lists model dimensions).35 A psychometrician (CS) guided the survey development process, which included item writing and refinement, soliciting input from subject matter experts in EHR use, and iterative content review. After refining all survey items, we pilot tested the survey with 10 PCPs for readability, clarity, and ease of completion in a web-based format. With the exception of demographic items and 10 open-ended items, items were rated on a 5-point Likert-type scale with response options ranging from ‘strongly disagree’ to ‘strongly agree.’ Survey completion time was approximately 20–25 min.

Survey content areas
Hardware and software, content, and user interface

The VA uses the Computerized Patient Record System (CPRS) as its EHR in all its facilities. CPRS uses the ‘View Alert’ notification system to communicate test results (as well as other important clinical information) to practitioners through an inbox. The View Alert system also displays notifications as alerts with various priorities. Practitioners see their patients’ alerts each time they log in to the system or switch between patient records (figure 1). Alerts remain within the inbox until read by the practitioner, but they may be removed automatically if unopened after a certain time period (eg, 14 or 30 days). Although this functionality is used across the VA, individual facilities have discretion over which types of alerts practitioners must receive (eg, they can allow flexibility for practitioners to turn off certain notifications, such as normal test results). Our survey items assessed PCPs’ perceptions of the View Alert notification system, including their views on the content and quantity of alerts received, perceived ease of use, and use of the EHR's features for alert processing and follow-up.


Our survey assessed several characteristics of the respondents including age, gender, race, job classification (ie, academic physician, non-academic physician, nurse practitioner, physician assistant), years employed at the VA, native language (English or other), perceived adequacy of training related to the EHR notification system, and prior use of an EHR other than CPRS.

Workflow and communication

Workflow items assessed perceptions of alert burden, processes related to notifying patients of their test results, and practices used to support alert management.

Organizational features

These items assessed institutional cultural norms and expectancies regarding management of alerts, technical support for alert notifications, organizational support to facilitate patient notification, and the amount of protected time (ie, specifically designated and compensated time) clinicians were given by their organizations to manage alerts. Protected time was measured in hours per week; all other response choices and coding were the same as for other item categories on the 5-point Likert-type scale.

External rules and regulations

We assessed whether practitioners were aware of and adhered to a national VA policy36 released in 2009 requiring notification of patients within 14 days for both normal and abnormal test results.

System measurement and monitoring

We assessed perceptions of practices related to measuring and monitoring alert follow-up at the system level.

New features and functions to improve EHR-based notification

We inquired whether PCPs would endorse several potential strategies to improve EHR-based notification. Strategies were selected based on thematic areas of improvement identified in previous work8, 10, 11, 16, 18, 37 and addressed both technical and social dimensions.

Survey administration

We solicited support from the section chiefs of primary care at 142 VA facilities nationwide by asking them to email all PCPs at their respective sites to orient them to the project. We subsequently invited all participants by sending a personalized email from the principal investigator (HS) that described the study and provided a link to the web-based survey instrument. To increase response rates, invitation emails and subsequent reminders were followed by telephone attempts to reach non-respondents. In keeping with VA policies, we did not use monetary or other incentives for participation.

Data analysis

Data were downloaded from the internet survey administration service and were analyzed using SPSS statistical software. We generated descriptive statistics to summarize the characteristics of respondents and to aggregate responses to other survey items using descriptive statistics. Likert-type item responses were collapsed into three categories of ‘agree or strongly agree,’ ‘disagree or strongly disagree,’ and ‘neither agree nor disagree.’


Of 5001 PCPs invited, 2590 (51.8%) responded. Figure 2 shows the geographic distribution of the respondents by Veterans Integrated Service Network (regional divisions of VA). Table 2 shows the characteristics of the respondents. Characteristics of non-respondents were not available for comparison with the respondent group. The vast majority of respondents had considerable experience with the VA EHR, having worked within the VA for 2 or more years. Less than half (45.7%) reported having received sufficient training on the View Alert system, and only a minority (13.7%) reported any refresher training (data not shown in the table). Nevertheless, the majority believed they had the knowledge (74.4%) and proficiency (81.8%) necessary to use the View Alert system (data not shown in the table). Nearly half (46.6%) also had prior experience using a non-VA EHR. Of these, only 19% of providers thought that the non-VA EHRs they used were better than VA's CPRS; 55% indicated that the non-VA EHR they used was overall inferior to CPRS, and 26% perceived it was about the same. No particular non-VA EHR was consistently identified as better or worse than CPRS.

Table 3 shows the distribution of responses of selected items related to use and perceptions of the EHR technology (hardware/software, clinical content, and human–computer interface). Although most PCPs found the EHR alert notification system easy to use, more than half believed that the EHR software did not have convenient features for notifying patients of test results. Less than two-thirds of practitioners used a basic sorting feature for alerts, and just over a third used an enhanced functionality to process alerts. Both of these features are intended to promote efficiency when managing alerts. In addition, a significant proportion of PCPs had not altered their alert notification filters to help customize the amount and types of alerts they received. Thus, they received only notifications determined important (and set as default) by the facility (43.0%; data not shown).

Table 4 shows the distribution of responses for survey items related to workflow and communication. A majority of practitioners stayed after hours or came in on weekends to address alerts; only about a third had remote access to the EHR to manage alerts after hours. More than half of PCPs electronically assigned a surrogate or covering practitioner to handle their alerts when out of the office, a process we defined as ‘electronic hand-off.’ Just over a third reported having the help needed for notifying patients of test results. Patient notification varied between normal and abnormal results. Almost half of PCPs reported that they did not immediately notify patients of normal test results and relied on the patient's next visit to notify them, whereas about one-fifth relied on the next visit to notify patients of abnormal results.

Table 5 summarizes responses to items that assessed people and organizational features affecting alert management. Only about a third of respondents reported having or accessing an assigned technical contact to help with alerts; most admitted receiving help with alert management from colleagues. About a third of respondents reported receiving protected time for alert management; of these, most respondents reported it to be at least 4 hours per week (69.8%).

About half of respondents (54.2%) were aware of the 2009 Veterans Health Administration (VHA) policy (ie, external rules and regulations) regarding notification of patients regarding test results. However, among those aware of the policy only 20% reported that they changed their result notification practices accordingly (data not shown in tables). Almost a third (31.9%) reported that their supervisors monitored how they managed notifications, but fewer (23.9%) received feedback on their follow-up practices (ie, system measurement and monitoring).

Table 6 summarizes PCPs’ assessments of potential new features and functions to improve EHR-based notification. PCPs strongly endorsed four functionalities to reduce loss of alert information in the EHR (preventing automated deletion of alerts, being able to retrieve deleted alerts, having a back button, and being able to access alerts in the inbox for at least 30 days). To improve alert management options, most agreed with the need for a feature to remind them at a later date of the necessity to take a follow-up action. In addition, most PCPs endorsed the need for a separate messaging system within the EHR to allow providers to communicate, rather than the traditional method of including these human-generated messages along with the EHR-generated notification system that was used for result notification. About two-thirds of respondents endorsed one or more new visualization techniques, including a separate window for high-priority alerts or a method to filter or color-code alerts based on type. Most PCPs supported strategies to improve safety related to hand-offs, including being able to assign responsibility of test result follow-up and to display who is responsible for follow-up. About half were in favor of receiving feedback about their alert management performance.


Test result reporting has substantial patient safety implications and is now being considered in the meaningful use criteria for EHRs.38 However, our data suggest that despite the use of an advanced EHR system, both social and technical challenges exist in ensuring the reliability of test result notification to practitioners and patients within one of the world's largest healthcare systems. Other healthcare systems, most of which have a shorter history of EHR use, are likely to face similar challenges as they begin adopting EHRs. Several technical as well as social (personal, workflow, and organizational) factors need to be addressed in order for EHR-based test reporting functionality to be successful. PCPs in our sample endorsed several new features and functions to reduce loss of information in the EHR and to improve visualization of alerts that communicate test results.

We found that most providers do not routinely notify patients of normal test results, and a substantial proportion use the next scheduled visit to notify patients of both normal and abnormal results. A 1996 study conducted outside the VA reached similar conclusions, suggesting that this pattern has changed little in the last two decades.30 Recent initiatives within the VA are developing additional guidance to address test result management, which might change some of the patterns we found.39 As healthcare quality improvement efforts increasingly emphasize patient engagement, alternative methods are being explored for patients to receive test results. For instance, results are increasingly accessible to patients through online portals,40 although adoption has been slower than expected.41, 42 Furthermore, the Department of Health and Human Services recently proposed a rule43 allowing patients to access test results directly from the laboratory upon patient request (ie, bypassing the provider). However, providers’ interpretation of test results within the context of the patient's other clinical conditions remains essential.44 Because most PCPs receive hundreds of test results a week,31 one area for potential improvement is PCPs’ resources to facilitate patient notification. Just over a third of the PCPs reported having administrative support for patient notification, presenting an opportunity to leverage the current shift towards team-based models of care (eg, medical home teams45). For example, with appropriate task delegation and clarity of roles, other members of the team could be responsible for tracking results and notifying patients.45 Future EHR development should focus on innovative methods to facilitate patient notification of test results.

Our findings underscore the additional time burden induced by EHR-based alert systems, with the majority of respondents staying after hours, coming in on weekends, or using remote access from home to deal with notifications. Unfortunately, there is little or no reimbursement for non-face-to-face time related to documentation, follow-up, and patient notification of test results. Although the interventions proposed in the survey focused mostly on technological features and workflow, several organizational and policy interventions are also needed to improve the outcomes of EHR-based test results reporting.

Based on our data, we recommend several strategies for improving test results reporting though EHRs. First, being able to easily retrieve and readily access critical test result data is an absolute requirement for safe and efficient care, and the ability to do so (eg, to retrieve previously viewed alerts) was endorsed by a large majority of respondents. For instance, providers might accidently delete a result notification or need to return to an alert they processed some time ago, and these workflow requirements should inform EHR design. Second, to effectively address alert-related information overload,19 providers need better methods to display, sort, and visualize test result information46 according to patient, date, type, priority, and origin of alerts. Usability testing is essential to evaluate EHR support features for planning and prioritizing among high volumes of alerts. Third, most providers agreed with the need for strategies to improve electronic hand-offs in the EHR, including assignment of responsibility. There is surprisingly little knowledge to guide effective policy setting or practice management in this area. While health information technology can prevent communication breakdowns,47–50 vulnerabilities related to teamwork and care coordination51, 52 have received less attention, and novel methods to reduce ambiguities related to these hand-offs are needed. Lastly, EHRs need to be able to support the ‘prospective’ memory53 of providers (memory of an intent to perform a future task) by facilitating alert tracking and self-reminders for future tasks. Some of these features should be deemed as universal requirements in EHRs.

Most of our results are generalizable outside the VA even though there are differences in EHR features, usage, organizational, workflow, and policy-related factors. This is due to the fact that the VA has been a leader in the successful integration and use of comprehensive EHRs, and many of our findings are valuable for other healthcare institutions that are moving to more integrated, technology-enabled care environments as required to meet the new patient management challenges prescribed by accountable care organizations.54 Although the alert notification system was specific to the VA's EHR, most major EHRs use similar methods to notify providers of important clinical findings, and others will follow in compliance with meaningful use requirements. Our study had a relatively low response rate. However, this is comparable to or higher than the response rates of many other physician surveys,55–58 which traditionally are not very high.59, 60 In fact, the large number of completed surveys represents one of the largest total responses regarding EHR characteristics ever reported.61, 62 Studies that report higher response rates for physician surveys often utilize monetary incentives for participants (which may lead to a different bias in the results), a practice not permitted for data collection within the VA. Lastly, we did not have access to any demographic or practice-based characteristics of non-respondent physicians for comparison.

Overall, our survey data suggest that current capabilities for test result management are limited even within a well-established, mature EHR. A comprehensive socio-technical approach is needed to optimize EHR-based test result management. Key components of this socio-technical approach will include: (1) design, development, and testing of EHR features and functions to support physicians’ test result management workflows and inclusion of these new features in all EHRs; (2) new policies and procedures at the local institutional and national level regarding appropriate methods and timeliness of patient notification; and (3) commitments from organizational leadership and payers to acknowledge and support the additional, non-face-to-face work required to provide care for patients in an EHR-enabled healthcare system.


Funding: The research reported here was supported by the Department of Veterans Affairs National Center for Patient Safety and in part by the Houston VA HSR&D Center of Excellence (HFP90-020), the National Library of Medicine (R01 LM006942), and the Office of the National Coordinator for Health Information Technology (ONC #10510592).

Competing interests: None.

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or other funding agencies. The corresponding author had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Ethics approval: Baylor College of Medicine Institutional Review Board approved this study.

Provenance and peer review: Not commissioned; externally peer reviewed.

We thank Annie Bradford, PhD for assistance with medical editing.

1. Wahls TL,Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam PractYear: 2007;8:3217519017
2. Schiff GD,Hasan O,Kim S,et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern MedYear: 2009;169:1881–719901140
3. Lorincz CY,Drazen E,Sokol PE,et al. Research in Ambulatory Patient Safety 2000–2010: a 10-year review. Am Med AssocYear: 2011 Available at: (accessed 3 Dec 2012).
4. Wynia MK,Classen DC. Improving ambulatory patient safety. JAMAYear: 2011;306:22 2504–5
5. Callen JL,Westbrook JI,Georgiou A,et al. Failure to follow-up test results for ambulatory patients: a systematic review. J Gen Intern MedYear: 2011
6. Murff HJ,Gandhi TK,Karson AK,et al. Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems. Int J Med InformYear: 2003;71:137–4914519406
7. Poon EG,Wang SJ,Gandhi TK,et al. Design and implementation of a comprehensive outpatient Results Manager. J Biomed InformYear: 2003;36:80–9114552849
8. Singh H,Vij MS. Eight recommendations for policies for communicating abnormal test results. Jt Comm J Qual Patient SafYear: 2010;36:226–3220480756
9. Tenner CT,Shapiro NM,Wikler A. Improving health care provider notification in an academic setting: a cascading system of alerts. Arch Intern MedYear: 2010;170:39220177047
10. Singh H,Thomas EJ,Mani S,et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?Arch Intern MedYear: 2009;169:1578–8619786677
11. Singh H,Thomas EJ,Sittig DF,et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain?Am J MedYear: 2010;123:238–4420193832
12. Singh H,Arora H,Vij M,et al. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform AssocYear: 2007;14:459–6617460135
13. Casalino LP,Dunham D,Chin MH,et al. Frequency of Failure to Inform Patients of Clinically Significant Outpatient Test Results. Arch Intern MedYear: 2009;169:1123–919546413
14. Elder NC,McEwen TR,Flach J,et al. The management of test results in primary care: does an electronic medical record make a difference?Fam MedYear: 2010;42:327–3320455108
15. Health Information Technology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information TechnologyOffice of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services. Federal RegisterYear: 2012 ( cited 2012 July 25. ); 77:13845–03–07/pdf/2012–4430.pdf (accessed 3 Dec 2012).
16. Hysong SJ,Sawhney MK,Wilson L,et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis MakYear: 2011;11:2221486478
17. Final Report: Summary review: Evaluation of Veterans Health Administration procedures for communicating abnormal test results. Report number 01-01965-24. 11-25-2002. Ref Type: Report.
18. Hysong SJ,Sawhney MK,Wilson L,et al. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform AssocYear: 2010;17:71–720064805
19. Singh H,Spitzmueller C,Petersen NJ,et al. Information overload and missed test results in electronic health record-based settings. Arch Intern Med. In press.
20. Taylor SL,Dy S,Foy R,et al. What context features might be important determinants of the effectiveness of patient safety practice interventions?BMJ Qual SafYear: 2011;20:611–17
21. Yano EM,Simon B,Canelo I,et al. 1999 VHA survey of primary care practices. VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. 00-MC12. 2000. VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. Ref Type: Report.
22. Forward momentum: Hospital use of information technology http://www.aha org/content/00-10/FINALNonEmbITSurvey105 pdf. (Year: 2005 (accessed 3 Dec 2012).
23. Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput InteractYear: 1995;7:57–78
24. Clason DL,Dormody TJ. Analyzing data measured by individual Likert-type items. J Agricultural EducYear: 2000;35:31–6
25. Lyons SS,Tripp-Reimer T,Sorofman BA,et al. VA QUERI informatics paper: information technology for clinical guideline implementation: perceptions of multidisciplinary stakeholders. J Am Med Inform AssocYear: 2005;12:64–7115492035
26. Campbell EG,Regan S,Gruen RL,et al. Professionalism in medicine: results of a national survey of physicians. Ann Intern MedYear: 2007;147:795–80218056665
27. Jha AK,Ferris TG,Donelan K,et al. How common are electronic health records in the United States? A summary of the evidence. Health Aff (Millwood)Year: 2006;25:w496–50717035341
28. Cutler DM,Feldman NE,Horwitz JR. U.S. adoption of computerized physician order entry systems. Health Aff (Millwood)Year: 2005;24:1654–6316284040
29. Ash JS,Gorman PN,Seshadri V,et al. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform AssocYear: 2004;11:95–914633935
30. Boohaker E,Ward RE,Uman JE,et al. Patient notification and follow-up of abnormal test results. A physician survey. Arch Intern MedYear: 1996;156:327–318572844
31. Poon E,Gandhi T,Sequist T,et al. "I wish I had seen this test result earlier!": dissatisfaction with test result management systems in primary care. Arch Intern MedYear: 2004;164:2223–815534158
32. Hickner J,Graham DG,Elder NC,et al. Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health CareYear: 2008;17:194–20018519626
33. Elder NC,Dovey SM. Classification of medical errors and preventable adverse events in primary care: a synthesis of the literature. J Fam PractYear: 2002;51:927–3212485545
34. Elder NC,Pallerla H,Regan S. What do family physicians consider an error? A comparison of definitions and physician perception. BMC Fam PractYear: 2006;7:7317156447
35. Sittig DF,Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health CareYear: 2010;19(Suppl 3):i68–7420959322
36. VHA Directive 2009–019: Ordering and Reporting Test Results. Veterans Health Administration. Year: 2009 (accessed 3 Dec 2012).
37. Singh H,Wilson L,Reis B,et al. Ten strategies to improve management of abnormal test result alerts in the electronic health record. J Patient SafYear: 2010;6:1212320563228
38. Blumenthal D,Tavenner M. The "Meaningful Use" Regulation for Electronic Health Records. N Engl J MedYear: 2010;363:501–420647183
39. Sittig DF,Singh H. Improving test result follow-up through electronic health records requires more than just an alert. J Gen Intern MedYear: 2012;27:12353722790618
40. Fraser HS,Allen C,Bailey C,et al. Information systems for patient follow-up and chronic management of HIV and tuberculosis: a life-saving technology in resource-poor areas. J Med Internet ResYear: 2007;9:e2917951213
41. Archer N,Fevrier-Thomas U,Lokker C,et al. Personal health records: a scoping review. J Am Med Inform AssocYear: 2011;18:515–2221672914
42. Sittig DF. Personal health records on the internet: a snapshot of the pioneers at the end of the 20th Century. Int J Med InformYear: 2002;65:1–611904243
43. CLIA Program and HIPAA Privacy RulePatients’ Access to Test Reports (42 CFR 493, 45 CFR 164). Fed RegistYear: 2011;76
44. Davis Giardina T,Singh H. Should patients get direct access to their laboratory test results? An answer with many questions. JAMAYear: 2011;306:2502–322122864
45. Singh H,Graber M. Reducing diagnostic error through medical home-based primary care reform. JAMAYear: 2010;304:463–420664048
46. Woods DD,Patterson E,Roth EM. Can we ever escape from data overload? A cognitive systems diagnosis. Cogn Technol WorkYear: 2002;4:22–36
47. Brantley SD,Brantley RD. Reporting significant unexpected findings: the emergence of information technology solutions. J Am Coll RadiolYear: 2005;2:304–717411822
48. Kuperman GJ,Teich JM,Tanasijevic MJ,et al. Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform AssocYear: 1999;6:512–2210579608
49. Petersen LA,Orav EJ,Teich JM,et al. Using a computerized sign-out program to improve continuity of inpatient care and prevent adverse events. Jt Comm J Qual ImprovYear: 1998;24:77–879547682
50. Singh H,Naik A,Rao R,et al. Reducing diagnostic errors through effective communication: harnessing the power of information technology. J Gen Int MedYear: 2008;23:489–94
51. Woolf SH,Kuzel AJ,Dovey SM Jr,et al. A string of mistakes: the importance of cascade analysis in describing, counting, and preventing medical errors. Ann Fam MedYear: 2004;2:317–2615335130
52. Gandhi TK,Sittig DF,Franklin M,et al. Communication breakdown in the outpatient referral process. J Gen Intern MedYear: 2000;15:626–3111029676
53. Dismukes K. Remembrance of things future: prospective memory in laboratory, workplace, and everyday settings. Hum Factors Ergon SocYear: 2010;6:79–122
54. Fisher ES,McClellan MB,Safran DG. Building the path to accountable care. N Engl J MedYear: 2011;365:2445–722204720
55. Kellerman SE,Herold J. Physician response to surveys. A review of the literature. Am J Prev MedYear: 2001;20:61–711137777
56. Singh H,Thomas EJ,Wilson L,et al. Errors of diagnosis in pediatric practice: a multisite survey. PediatricsYear: 2010;126:70–920566604
57. Asch DA,Jedrziewski MK,Christakis NA. Response rates to mail surveys published in medical journals. J Clin EpidemiolYear: 1997;50:1129–369368521
58. Cummings SM,Savitz LA,Konrad TR. Reported response rates to mailed physician questionnaires. Health Serv ResYear: 2001;35:1347–5511221823
59. Halpern SD,Kohn R,Dornbrand-Lo A,et al. Lottery-based versus fixed incentives to increase clinicians’ response to surveys. Health Serv ResYear: 2011;46:1663–7421492159
60. Nicholls K,Chapman K,Shaw T,et al. Enhancing response rates in physician surveys: the limited utility of electronic options. Health Serv ResYear: 2011;46:1675–8221492157
61. Poon EG,Wright A,Simon SR,et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med CareYear: 2010;48:203–920125047
62. Simon SR,Soran CS,Kaushal R,et al. Physicians’ use of key functions in electronic health records from 2005 to 2007: a statewide survey. J Am Med Inform AssocYear: 2009;16:465–7019390104


[Figure ID: AMIAJNL2012001267F1]
Figure 1 

Alert notification window in the VA Computerized Patient Record System (CPRS).

[Figure ID: AMIAJNL2012001267F2]
Figure 2 

Distribution of respondents within 21 Veterans Affairs Networks across the USA (Veterans Integrated Service Networks or VISNs).

[TableWrap ID: AMIAJNL2012001267TB1] Table 1 

Eight interactive socio-technical health information technology (HIT) dimensions addressed for missed test results

Dimension Definition
Hardware and software Equipment and software required to run the healthcare applications
Clinical content Data, information, and knowledge entered, displayed, or transmitted in EHRs
Human computer interface Aspects of the EHR system that users interact with (eg, see, touch, or hear)
People Humans involved in the design, development, implementation, and use of HIT
Workflow and communication Work processes needed to ensure that each patient receives the care they need at the time they need it
Organizational policies and procedures Internal culture, structures, policies, and procedures that affect all aspects of HIT management and healthcare
External rules, regulations, and pressures External forces that facilitate or place constraints on the design, development, implementation, use, and evaluation of HIT in the clinical setting
System measurement and monitoring Measurement of system availability, use, effectiveness, and unintended consequences of system use

EHR, electronic health record.

[TableWrap ID: AMIAJNL2012001267TB2] Table 2 

Characteristics of survey respondents (n=2590)

Characteristic n (%)
 20–39 338 (13.1)
 40–49 685 (26.4)
 50–59 961 (37.1)
 60 and over 402 (15.5)
 Missing 204 (7.9)
 Male 1080 (41.7)
 Female 1343 (51.9)
 Missing 167 (6.4)
 White 1630 (62.9)
 Black 118 (4.6)
 Asian 431 (16.6)
 Other 188 (7.3)
 Missing 223 (8.6)
Job classification
 Physician, academic 438 (16.9)
 Physician, non-academic 1228 (47.4)
 Nurse practitioner 561 (21.7)
 Physician assistant 204 (7.9)
 Missing 159 (6.1)
Years at VA
 <2 437 (16.9)
 2–10 1219 (47.1)
 11–20 589 (22.7)
 >20 201 (7.8)
 Missing 144 (5.6)
Native language
 English 1911 (73.8)
 Other 498 (19.2)
 Missing 181 (7.0)

VA, Department of Veterans Affairs.

[TableWrap ID: AMIAJNL2012001267TB3] Table 3 

Technology-related items associated with EHR-based test result management

Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing
Item n (%) n (%) n (%) n (%)
Hardware and software
 CPRS has convenient features for notifying patients of test results 628 (24.2) 1437 (55.5) 421 (16.3) 104 (4.0)
 Has a convenient way of generating letters in CPRS for patient notification 1091 (42.1) 1042 (40.2) 347 (13.4) 110 (4.2)
Has an automated voice messaging system that can notify patients of test results 38 (1.5) 2337 (90.2) 113 (4.4) 102 (3.9)
 Uses My HealtheVet* to notify patients of test results 45 (1.7) 2308 (89.1) 134 (5.2) 103 (4.0)
Clinical content
 Acknowledges (ie, clicks on) all high priority alert notifications 2365 (91.3) 80 (3.1) 112 (4.3) 33 (1.3)
 Follows up on all high priority alert notifications received 2372 (91.6) 77 (3.0) 97 (3.7) 44 (1.7)
 Acknowledges all alert notifications received regardless of priority 2169 (83.7) 228 (8.8) 146 (5.6) 47 (1.8)
Human–computer interface
 Finds the alert notification system in CPRS easy to use 1842 (71.1) 326 (12.6) 393 (15.2) 29 (1.1)
 Follows up on all alert notifications received 1968 (76.0) 311 (12.0) 258 (10.0) 53 (2.0)
 Alert notification system in CPRS makes it possible for providers to miss test results 1440 (55.6) 555 (21.4) 506 (19.5) 89 (3.4)
 Sorts alert notifications when necessary according to urgency, patient name, location, or alert date/time 1625 (62.7) 562 (21.7) 381 (14.7) 22 (0.8)
 Uses the Process All function of the system when necessary 981 (37.9) 1219 (47.1) 367 (14.2) 23 (0.9)

Italicized items are discussed in the text.

*My HealtheVet is a free, online personal health record for VA patients.

†CPRS (Computerized Patient Record System) is the Department of Veterans Affairs electronic health record software.

‡The Process All feature allows clinicians to efficiently process alerts one after another without returning to the View Alert window.

EHR, electronic health record; VA, Department of Veterans Affairs.

[TableWrap ID: AMIAJNL2012001267TB4] Table 4 

Workflow and communication items associated with EHR-based test result management

Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing
Item n (%) n (%) n (%) n (%)
Workflow and communication
Alert burden
  The number of alerts exceeds what they can effectively manage 1803 (69.6) 321 (12.4) 432 (16.7) 34 (1.3)
  Receives too many alerts to easily focus on most important ones 2078 (80.2) 215 (8.3) 244 (9.4) 53 (2.0)
  In the past year, missed test results that led to delayed patient care 772 (29.8) 1212 (46.8) 537 (20.7) 69 (2.7)
  Uses remote access after hours or on weekends to manage alerts 893 (34.5) 1327 (51.2) 320 (12.4) 50 (1.9)
  Has remote access to CPRS 953 (36.8) 1599 (61.7) 38 (1.5)
  Stays after hours or comes in on weekends to manage alerts 2218 (85.6) 219 (8.5) 105 (4.1) 48 (1.9)
  Wishes the system provided them with more alerts 151 (5.8) 2190 (84.6) 208 (8.0) 41 (1.6)
  Receives too many alert notifications per day 2251 (86.9) 164 (6.3) 166 (6.4) 9 (0.3)
  Uses additional paper-based methods to help follow test results 1451 (56.0) 1103 (42.6)
  Gets too many FYI only alert notifications that require a signature even though no action on their part is required 2071 (80.0) 250 (9.7) 258 (10.0) 11 (0.4)
  Often receives alert notifications where they are unsure as to why they were sent to them 1601 (61.8) 539 (20.8) 432 (16.7) 18 (0.7)
Patient notification
  Consistently notifies patients of abnormal test results 2146 (82.9) 151 (5.8) 185 (7.1) 108 (4.2)
  Has the help needed for notifying patients of test results 981 (37.9) 1180 (45.6) 332 (12.8) 97 (3.7)
  Consistently notifies patients of normal test results 1167 (45.1) 902 (34.8) 415 (16.0) 106 (4.1)
  Relies on a patient's next visit to notify them of their abnormal test results 521 (20.1) 1556 (60.1) 407 (15.7) 106 (4.1)
  Relies on a patient's next visit to notify them of their normal test results 1193 (46.1) 858 (33.1) 431 (16.6) 108 (4.2)
Use of surrogates
  Alert notifications related to surrogates create new safety concerns 1339 (51.7) 482 (18.6) 730 (28.2) 39 (1.5)
  Has support staff to assist with management of test result alert notifications 873 (33.7) 1304 (50.3) 311 (12.0) 102 (3.9)
  Assigns a surrogate to take care of alert notifications when out of the office 1525 (58.9) 788 (30.4) 239 (9.2) 38 (1.5)

Italicized items are discussed in the text.

CPRS, Computerized Patient Record System; EHR, electronic health record; FYI, for your information.

[TableWrap ID: AMIAJNL2012001267TB5] Table 5 

Organizational and people items associated with EHR-based test result management

Agree or strongly agree
Disagree or strongly disagree Neither agree nor disagree
Item n (%) n (%) n (%) n (%)
Organizational policies and procedures
Technical support
  Clinic has an assigned technical contact that can be accessed for help with alert notifications 908 (35.1) 947 (36.6) 651 (25.1) 84 (3.2)
  IT help person provides useful resources 621 (24.0) 882 (34.1) 988 (38.1) 99 (3.8)
  With questions about or problems with alert notifications, gets the help needed from IT 887 (34.2) 809 (31.2) 807 (31.2) 87 (3.4)
  With questions about or problems with alert notifications, gets the help needed from colleagues 1564 (60.4) 348 (13.4) 587 (22.7) 91 (3.5)
Protected time
  Has protected clinical time slots to manage alert notifications* 779 (30.1) 1773 (68.5) 38 (1.5)
Supportive norms
  Colleagues believe the alert notifications in CPRS help them get their job done effectively 640 (24.7) 849 (32.8) 1041 (40.2) 60 (2.3)
  Supervisor believes alert notifications in CPRS are an essential component of effective primary care 1403 (54.2) 143 (5.5) 981 (37.9) 63 (2.4)
  Senior management has emphasized the importance of the use of alert notifications in high quality care 1359 (52.5) 189 (7.3) 981 (37.9) 61 (2.4)
Performance expectancy
  Using alert notifications in CPRS enhances providers’ ability to provide safe patient care 2098 (81.0) 167 (6.4) 275 (10.6) 50 (1.9)
  Using alert notifications in CPRS enhances providers’ effectiveness on the job 1803 (69.6) 311 (12.0) 423 (16.3) 53 (2.0)
  Using alert notifications in CPRS increases providers’ productivity 952 (36.8) 858 (33.1) 723 (27.9) 57 (2.2)
  Using alert notifications in CPRS allows providers to meet performance standards 1312 (50.7) 542 (20.9) 679 (26.2) 57 (2.2)

Italicized items are discussed in the text.

*Of the 779 providers who reported receiving protected time, 224 (28.8%) reported <4 h/week, 544 (69.8%) reported ≥4 h/week, and 11 (1.4%) did not report the number of hours of protected time.

CPRS, Computerized Patient Record System; EHR, electronic health record.

[TableWrap ID: AMIAJNL2012001267TB6] Table 6 

New features and functions to improve EHR-based notification

Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing
Item n (%) n (%) n (%) n (%)
Hardware and software
Improving communication and alert management options
  I would like to be able to set reminders for myself for future actions 2160 (83.4) 64 (2.5) 213 (8.2) 153 (5.9)
  I would like to have a messaging system within CPRS that would allow providers to communicate with one another - this would be outside the View Alert system 1826 (70.5) 174 (6.7) 444 (17.1) 146 (5.6)
Human–computer interface
Improving alert visualization
  I would like to receive high priority test result notifications in one window, and all other alert notifications in another window 1611 (62.2) 474 (18.3) 370 (14.3) 135 (5.2)
  I would like an option to display only certain alert notifications at a time (ie, filter to display only surrogate, inpatient, or high priority alerts) 1722 (66.5) 321 (12.4) 398 (15.4) 149 (5.8)
  I would like to have my alert notifications color-coded according to type (eg, surrogate, inpatient, or high priority alerts) 1720 (66.4) 316 (12.2) 414 (16.0) 140 (5.4)
Better processing and tracking of alerts
  I would like to be able to retrieve my deleted alert notifications 2036 (78.6) 193 (7.5) 228 (8.8) 133 (5.1)
  High priority alert notifications should not disappear until I actively delete them after taking follow-up action 1913 (73.9) 266 (10.3) 273 (10.5) 138 (5.3)
  There should be a mechanism in the alert notification system in CPRS to display the name of the person responsible for following up on the test result alert 1596 (61.6) 292 (11.3) 557 (21.5) 145 (5.6)
  I would like to have a ‘back button’ in CPRS to retrieve the prior window 2173 (83.9) 67 (2.6) 212 (8.2) 138 (5.3)
  All unacknowledged alert notifications should stay in the alert notification window for at least 30 days 1689 (65.2) 318 (12.3) 445 (17.2) 138 (5.3)
Workflow and communication
Improving the surrogate process
  As a surrogate, I should only receive high priority alert notifications for patients assigned to the provider who is out of the office 1659 (64.1) 432 (16.7) 360 (13.9) 139 (5.4)
  Currently it is not possible for a provider to directly assign, without IT assistance, more than one surrogate in CPRS. This capability should be introduced 1555 (60.0) 264 (10.2) 618 (23.9) 153 (5.9)
System measurement and monitoring
Improving feedback
  I would like to receive feedback about my performance related to follow-up of high priority alert notifications 1264 (48.8) 479 (18.5) 708 (27.3) 139 (5.4)

CPRS, Computerized Patient Record System; EHR, electronic health record.

Article Categories:
  • 1506
Article Categories:
  • Focus on Human Factors and System Utilization
Series: Research and applications.

Keywords: missed test results, medical errors, diagnostic errors, lack of follow-up, health information technology, electronic health records.

Previous Document:  A sequence labeling approach to link medications and their attributes in clinical notes and clinical...
Next Document:  Prevalence of and risk factors for autopsy-determined atherosclerosis among US service members, 2001...