When the numerator is zero: another lesson on risk.
Stereoencephalotomy (Complications and side effects)
|Author:||Ho, Adrienne K.|
|Publication:||Name: The American Biology Teacher Publisher: National Association of Biology Teachers Audience: Academic; Professional Format: Magazine/Journal Subject: Biological sciences; Education Copyright: COPYRIGHT 2009 National Association of Biology Teachers ISSN: 0002-7685|
|Issue:||Date: Nov-Dec, 2009 Source Volume: 71 Source Issue: 9|
|Geographic:||Geographic Scope: United States Geographic Code: 1USA United States|
A biology teacher has developed a rare and troubling neurologic disorder. The signs and symptoms are well controlled with medications but the side effects are unpleasant. He hears of an experimental treatment that has produced impressive results. The treatment involves surgically ablating selective parts of the brain at close proximity to the brainstem. There is a risk that, during the procedure, vital parts of the brain could be inadvertently damaged and the patient could become paralyzed.
Our teacher decides to see the neurosurgeon, a pioneer in this experimental procedure, in a big medical center. The good doctor enthusiastically tells him that the procedure may bring about a cure. The following conversation ensues:
Teacher: What is the risk of paralysis?
Surgeon: I can't really give you a figure. All I cart say is, so far, 900 such procedures have been performed around the world, and no patient has suffered paralysis. Yours truly has personally performed 200, and so far, no problem. Touch wood.
The good teacher is too polite to say how he really feels. While he likes the prospect of a cure without medications, he is afraid of the possibility of paralysis. He recalls a recent article (Dougherty et al., 2009) on absolute and relative risks in The American Biology Teacher, his favorite journal. After reviewing the excellent article, the teacher has the following question: If zero paralysis has occurred after 900 cases, should the risk be calculated by dividing the number of paralysis cases (zero) by the number of cases performed (900) (Dougherty et al., 2009)? That would give a risk of zero! Simple logic suggests that it cannot be zero. When doctors started doing this procedure, could they have declared the risk to be zero after, say, 10 uncomplicated cases? By the same token, just because 100 bombs have been diffused without one going off prematurely does not mean that bomb disposal is always safe work and all that cumbersome protective gear can be discarded.
Our biology teacher wants to know what the risk really is before he makes up his mind about surgery. He has two choices. First, he can wait until more cases have been recorded from around the world. The problem is that his disease is rare, and he may have to wait for years before enough data can be collected. That seems unacceptable. Second, he can try to figure out the risk based on the 900 uncomplicated cases. He starts with a literature search and uncovers three papers dealing with risk estimation when the numerator is zero (Hanley et al., 1983; Eypasch et al., 1995; Ho et al., 2000). The mathematics involved is somewhat complex. Always on the lookout for interesting topics to teach, our good teacher is determined to introduce this concept of risk assessment to his students, but is wary that many of them may not have enough mathematical aptitude (Dougherty et al., 2009) to understand the analysis. Undaunted, he takes out his pen and paper and starts writing.
* Problem Solving
Let the true risk of paralysis with each operation be r. (Of course, r is the million dollar unknown he wants to figure out.) The probability of no paralysis of each operation must be 1 - r. What is the probability of two operations without paralysis? Well, the chance has to be [(1 - r).sup.2]. The chance of no paralysis after n procedures is [(1 - r).sup.n]. Since 900 uncomplicated cases have been performed worldwide, n = 900. Using EXCEL, the teacher plots the chance of no paralysis after 900 operations, [(1 - r).sup.n], on the vertical axis against the risk of paralysis r on the horizontal axis, for r ranging from, say, 0.0002 to 0.005 (Figure 1). Again, he does not know where r lies on the horizontal axis. For all he knows, the true value of r may even lie somewhere outside of this arbitrary range he has chosen. However, he has to start somewhere.
Let's say for the moment that the true value of r, the million dollar unknown, is very roughly 0.0008 (point A in the graph), meaning that there should be roughly eight paralysis cases, on average, for every 10,000 surgeries. (The number 8 is the product of 0.0008 x 10,000.) This translates to 0.8 case for every 1,000 surgeries, or 0.8/1000 x 900 = 0.72 paralysis for every 900 cases.
The graph says that at r = 0.0008, there is a roughly 50% chance of no paralysis after 900 cases. In other words, the chance of a clean record after 900 cases is, well, about the same as flipping a coin. One could just as likely have at least one case of paralysis after 900 cases and, suddenly, the reassurance given by the doctor, based on no paralysis after 900 cases, is not so reassuring. Remember that we are not talking about loosing a few strands of hair here; we are talking about a serious complication.
Let's say for the moment that the true value of r is very roughly 0.0033 (point B in the graph), meaning that, for every 10,000 surgeries, there should be roughly 0.0033 x 10,000, or 33 paralysis cases, on average. This translates to about 3.3 cases for every 1,000 surgeries, or 3.3/1000 x 900=3 paralysis cases for every 900 surgeries. According to the graph, the chance of zero complications after 900 surgeries is 0.05, or 5%. In spite of this low probability, that is exactly what has happened--no paralysis after 900 cases. It is therefore unlikely that the 0/900 record has merely been due to incredibly good luck. In somewhat convoluted words, there is a 95% chance that it is not luck that no paralysis has occurred after 900 surgeries. The good teacher likes the implications of being able to say with 95% confidence that the true risk of paralysis is no more than 0.0033.
[FIGURE 1 OMITTED]
It turns out that Rumke (1975) had determined the rule of thumb for determining the 95% confidence limit of the risk of an adverse event when the numerator for risk calculation is zero. It is 3/n, where n is the number of cases performed. In our case, after 900 uncomplicated cases, one can say with 95% confidence that the risk of paralysis should be no more than 3/900, or, 0.0033.
The next question our teacher needs to ask is: What is his chance of paralysis if he were to undergo surgery? In other words, what is the chance that the 901st case in the world (his case) to be performed will result in paralysis? The answer has already been determined above. He can be 95% confident that the chance of paralysis with his surgery is no more than 0.0033, or 1/300. He would like to have the risk at less than 0.0008, or 1/1250, but he is only 50% confident that this more attractive "worst case scenario" risk estimate is accurate. Notice that the right vertical axis in the graph (Figure 1) is the confidence limit, and is the percentage form of 1 minus the corresponding values on the left vertical axis. Reading off the graph, our teacher can also say with 99% confidence that the chance that his surgery will be complicated by paralysis is at worst 0.005, or 1/200. One may notice by now that if our teacher should demand that the accuracy of the estimated risk be stated with a very high degree of confidence, then that estimated risk must necessarily become very conservative. In other words, the estimated risk would be relatively high and might indeed be an over-estimation. Such is the price to pay for demanding a high level of confidence when estimating the risk of a complication. The confidence limit one picks will depend in part on how badly one wants the treatment relative to how serious the undesirable complication in question is. If the potential complication is quite severe and could potentially overshadow the benefits, one might wish to gauge the risk by choosing a high confidence level such as 95%, or even 99%. If the potential complication is not that serious, especially when considered in light of the potential gain from treatment, then one might not be particularly worried and accept a risk estimate based on a more pragmatic confidence level. Notice that it is unreasonable to demand a risk estimate with absolute, i.e., 100%, certainty. No doctor can promise with certainty that no complication will ever occur. With that in mind, our beloved teacher can make a more informed decision as to whether he wishes to go ahead with the surgery.
Scientists, engineers, and doctors are constantly developing new technologies, techniques, and therapeutics to improve health care. The Food and Drug Administration is extremely safety conscious on the one hand. On the other hand, it has the mandate of approving new therapeutics in a timely fashion so the public can benefit. In spite of rigorous and stringent testing and lengthy trials, highly beneficial therapeutics are sometimes approved before all plausible adverse effects have necessarily emerged. Estimating the risk of a rare, but plausible, serious complication when none has yet occurred is therefore an important concept.
The excitement of new discoveries often fuels exuberance that sometimes leads to loss of objectivity. For example, when minimally-invasive removal of the diseased gallbladder was developed a couple of decades ago, one of the theoretical concerns was injury to the common bile duct by laparoscopic instruments. After several small series of laparoscopic cholecystectomies without common bile duct injury, surgeons could hardly wait to declare that the new technique had led to no such injury (Eypasch et al., 1995). Now our biology teacher (and his class) knows that does not mean zero risk. These days, laparoscopic removal of many kinds of diseased organs, including the gallbladder, is routine. However, as more data have accumulated, it has turned out that common bile duct injury does occur and surgeons ignore this risk at their own, and their patients', peril.
1. Pretend that one of the students has to make the same decision based on the same information used in this article.
2. Go through the logic as outlined above.
3. After assuming no complication after 900 cases, and going through the reasoning, ask the students to plot a graph with multiple curves representing n = 600, 900, 1200 cases. This would be a good exercise to also explore the powerful graph-plotting functions of EXCEL.
4. Using the graph, determine the risk with 95% confidence for the above values of n, and verify the 95% confidence level for risk estimation is 3/n.
5. Plot more graphs for n equaling small numbers, e.g., 25, 30, 90, and see if the rule of thumb of risk estimation with 95% confidence equaling 3/n still holds at smaller values of n. The class will discover that when n is 30, the rule of thumb 3/n for 95% confidence level has an error of about 5%. When n becomes smaller, the rule of thumb produces increasingly erroneous results. Those interested in estimating risks of rare events for large and small values of n should refer to Newcombe et al. (2000).
6. Ask the students to make the same argument for 90% confidence and 99% confidence and come up with the respective rule of thumb (answer: 2.3/n and 4.6/n, respectively). To get to the 99% confidence level, teach the students how to adjust the scale of the graph in EXCEL by double-clicking on the vertical axis and defining the minimum and maximum at 0 and 0.01. This would allow easier determination of the corresponding r values on the x-axis.
7. If the mathematics proficiency of the students allows, review the solutions offered in Hanley et al. (1983) and Ho et al. (2000). The formula for the rule of thumb is -ln[1-CL/100]/n, where In is the natural logarithm and CL is the confidence level such as 90%, 95%, 99% (Table 1).
8. The minimum value for n has already been determined for the 95% confidence level. To determine the minimum value for n at which the rule of thumb for 90% and 99% confidence level produces a [less than or equal to] 5% error, the class can use various values of n surrounding 23 for the 90%, and 46 for the 99% confidence levels, respectively. The results are summarized in Table 1.
9. The more severe a complication, the more cautious one should become. At this point, discuss when one might use the risk estimate with 90%, 95%, or 99% confidence.
A zero numerator does not necessarily mean zero risk. It is possible to estimate the risk of a rare plausible complication even when it has not occurred, and when only the denominator is known. This is done through evaluating the likelihood of n successful trials for various confidence levels. The concept can be easily taught to nonmathematics students using a simple graph.
Dougherty, M.J. & Mclnerney, ,I.D. (2009). What's the denominator? A lesson on risk. The American Biology Teacher, 71(1), 31-33.
Eypasch, E., Lefering, R., Kum, C.K. & Troidl, H. (1995). Probability of adverse events that have not yet occurred: a statistical reminder. British Medical Journal, 322 (7005), 619-620.
Hanley, J.A. & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? Interpreting zero numerators. Journal of the American Medical Association, 249(13), 1743-1745.
Ho, A.MH., Chunk, D.C. & Joynt, G.M. (2000). Neuraxial blockade and hematoma in cardiac surgery. Estimating the risk of a rare adverse event that has not (yet) occurred. Chest, 117(2), 551-555.
Newcombe R.G. & Altman, D.G. (2000). Proportions and their differences. In: D. G. Altman, D. Machin, T.N. Bryant & MJ. Gardner (Eds.), Statistics with Confidence, 2nd Ed. Bristol, England: BMJ.
Rumke, C.L. (1975). Implications of the statement: No side effects were observed. New England Journal of Medicine, 292(7), 372-373.
ADRIENNE K. HO is a second-year student at Li Po Chun United World College, Shalin, NT, Hong Kong; e-mail: Adi.K.M.Ho@gmail.com.
Table 1. The rules of thumb for estimating the risk (r) of a complication when no complication has occurred after n trials, at various confidence limits; and the minimum number of trials (n) without a complication to allow the use of the rules of thumb for estimating r with error. Confidence limits (%) 90 95 99 Estimated risk (r) 2.3/n 3/n 4.6/n Minimum number of trials (n) to produce a <5% 23 30 46 error compared to graphs or calculations
|Gale Copyright:||Copyright 2009 Gale, Cengage Learning. All rights reserved.|