By Jamie Kelly

WE HUMANS HAVE AN INNATE NEED TO compare. It is a core element of our decision-making processes. We select our homes, cars, computers, and even mates using a slew of criteria against which we can hold up any two options and quickly proclaim a winner. This craving to contrast is great for Consumer Reports’ sales, but is not so easily satisfied when it comes to selecting health care information technology.

It is important, when we compare tools, that we take into account how they are being used. Direct product comparison, by way of sheer numerical data, is not always possible, particularly in relation to the prevention of medication administration errors. Instead, hospitals seeking patient safety technology, such as bar code point-of-care (BPOC) systems, need an understanding of product nuances that can be derived only through indepth data analysis. Such data analysis can allow for internal process improvement and allow hospitals to evaluate error-prevention rates in a more meaningful way.

Bar Code Point-of-Care Systems
More than 400 hospitals have implemented BPOC systems to automatically verify the “five rights” of medication administration and to reduce medication administration errors. Receiving patient-specific data from various clinical information systems, BPOC systems employ bar-code scanning devices to automate the identification of patient, caregiver, and medication. The clinician scans his or her bar-coded ID to log onto the system, and then scans the patient’s identification band to access the patient’s profile. Finally, the medication’s bar-coded label is scanned, allowing the clinician to intercept any errors before administration. Medication administration events are automatically documented in the patient’s medication administration record (MAR). BPOC systems also record medication administration data that can be used to detect system errors in the medication-use process, identify user non-compliance, and gather data to fuel on-going optimization of the systems.

Challenges in Comparing Error-Prevention Rates
As creatures of comparison, we love a tidy percentage that sums up an entire system’s value in three digits or less, and IT vendors produce percentages aplenty to feed our need. But what lies behind those digits?

Hospitals with BPOC systems have learned that the most common warning presented to a nurse is “late dose.” Hospitals usually define “late” as within an hour or two of the time the dose was due. Although some medications are more time sensitive, such as pre-operative antibiotics, most medications do not have a significantly diminished effect if given as much as two hours late. With this understanding, hospitals may elect to exclude late-dose warnings from their error prevention rates to get a more meaningful picture of the system’s impact on patient safety.

In 2002, the University of Wisconsin Hospitals and Clinics cited an 87% reduction in medication errors—including late-dose warnings— after implementing a BPOC system. A 2004 observational study at Lancaster General Hospital found a medication administration error reduction rate of 54%, excluding late-dose warnings. Because late doses can account for more than half of all warnings, this data alone does not indicate which hospital had better results from their BPOC system.

New Error Reduction Study
While it may be difficult to draw conclusions from the results of the two studies mentioned above, there is enormous value in analyzing system data. Recently, Bridge Medical Inc., a vendor of BPOC systems, set out to test their MedPoint system’s effectiveness in helping the nurses at six hospitals avoid serious medication administration errors. Each hospital contributed six continuous months of MedPoint data from one medical/surgical unit, covering a total of 446,195 administrations. An analyst evaluated all system warnings presented to the nurses during the study period, as well as the actions taken by those nurses as a result of the warnings.

When faced with a system warning during a medication administration, a nurse may elect to either ignore a warning and proceed with the administration, or heed the warning and cancel the administration. When a nurse cancels the administration, the system flags a potential save. In the case of “wrong route” and “wrong dose” warnings, the system allows the nurse to correct the route or dose prior to administration, and both actions could represent saves. When system data showed that a nurse reacted to a warning by aborting or altering the administration, the analyst considered it a preventive action. The number of these actions as a percentage of total warnings—the preventive action rate (PAR) for a given warning—yields the main construct of this study. Therefore, the PAR reflects the consistency with which the nurse, using clinical judgment, alters an administration based on information provided by the BPOC system.

Warnings that Matter Most
The MedPoint system issues more than 30 different warnings. Some warnings are intended to improve nursing process and documentation, while others are meant to intercept clinically significant errors that might harm the patient. The Bridge study focused in on five clinically significant warnings: wrong medication, wrong dose, wrong time (excluding late-dose warnings), wrong patient, and wrong route. The data analysis showed that, during the six-month study period, MedPoint assisted in a total of 16,633 saves related to “five rights” violations—on average 15.4 saves per day across all six sites.

The study also found that the PAR for the clinically significant warnings more than doubled the PAR for all warnings, 52% and 24%, respectively. This is not so surprising when you consider that late-dose warnings accounted for 53% of all warnings, and since there is no preventive action for a late dose, the warning has a substantial impact on the overall PAR.

Of the clinically significant warnings, the nurses heeded wrong-patient, wrong-route, and wrong-dose warnings 100% of the time. The study data identified 243 instances in which the system stopped nurses after scanning the wrong patient. However, not all warnings have this consistent effect on nurse behavior.

Putting PAR to Work for System Optimization
Because we want tools to work reliably and as intended, we would expect a quality BPOC system to yield consistently high PAR rates for all warnings. The study found that site-specific usage can significantly lower PAR data and present hospitals with an opportunity to alter practices, and thereby reduce ineffective warnings. For example, the PAR for wrong-time warnings varied from site to site, but averaged only 31%, suggesting that nurses perceived little risk and ignored the warning two out of three times. The high volume and low PAR for wrong-time warnings at one site could be attributed to a narrow dose-due window. With this information, the site may decide to widen the window to decrease low-impact warnings. Across the six study sites, there were numerous opportunities for pharmacy to optimize system set-up, alter the formulary, or change current practice to increase PAR. For example, to reduce wrong-dose warnings, pharmacists can look for drugs which, when entered in the pharmacy information system, have dose units (i.e., mg, mEq, mL, etc.) that conflict with the dose units set up in the MedPoint formulary. These discrepancies result in a wrong-dose warning that the nurse will likely chose to bypass. Pharmacy may also keep an eye on latedose warnings given in instances when a nurse may have documented that a medication was not available at the time of scheduled administration. These events provide impetus to investigate cart-fill processes and the time it takes to deliver medications to the units, in an attempt to find possible process improvements.

Know Thy Data
Low PAR can be a reflection of creative system usage. For wrong-medication warnings, PAR varied from 62% in one facility to 19% in another. How is it possible that nurses are ignoring four out of five warnings at the latter site? Investigation revealed that this facility has a high rate of “no order in the system” warnings, which notify nurses when they scan a medication for which there is no order in the pharmacy information system. The nurses at this hospital use MedPoint to document all IV start times, accuchecks, dressing changes, and saline flushes. These actions do not follow physician orders, and MedPoint, which fundamentally assumes that all medication orders come from the pharmacy information system as a matter of safe practice, recorded these warnings as potential errors. In fact, in March 2005, the hospital had 4,640 “no order in system” warnings related to these three practices, but the hospital confirmed it averages approximately 350 true “no order in system” warnings per month.

Therefore, the hospital with a 62% PAR is not necessarily averting wrong medication errors more effectively than the hospital with a 19% PAR. The BPOC system will perform as designed, but the user may stretch its intended application, thereby affecting the system data. Fortunately, hospitals can adjust their reporting of saves to isolate only clinically relevant PAR.

The Value of Data
Quick numeric comparisons, while meeting our innate need for contrast, do not necessarily provide instructive data. As shown through Bridge Medical’s study, in-depth data analysis takes us beyond the simple tallying of error preventions and allows us to actually analyze PAR and apply data to internal process improvements.
Jamie Kelly is the principal of Entropy Research, a consulting firm based in San Diego. With 10 years of health care IT marketing expertise, Kelly works with her clients, including Bridge Medical, Inc., to improve patient safety.