Subject: When is Evidence Based Medicine not Science Based Medicine?
Background: Carbon Monoxide Poisoning. There are many cases of CO Poisoning throughout the world every year. CO produced via indoor-heating appliances, car exhaust, or any number of other sources can be inhaled into the body where it competes with Oxygen for space on the O2 carrying molecule, Hemoglobin. The effects of CO poisoning effectively mirror those of asphyxia, since the binding affinity of CO with hemoglobin is much, much stronger than that of O2. Essentially the difference between O2 and CO's "stickiness" to hemoglobin is roughly that between 2-sided sticky tape and KrazyGlue (tm). CO also binds very strongly to myoglobin, which is present in muscles (including heart muscle) and helps to maintain sufficient oxygen for metabolic processes. Wikipedia has a nice summary of this relatively common form of poisoning. If you look at the second paragraph in the "treatment" section, you see the following:
A significant controversy in the medical literature is whether or not
hyperbaric oxygen actually offers any extra benefits over normal high flow
oxygen in terms of increased survival or improved long term outcomes. There have been clinical trials[24][30][31][32][33][34]
in which the two treatment options have been compared; of the six performed, four found hyperbaric oxygen improved outcome and two found no benefit for hyperbaric oxygen. Some of these trials have been criticized for apparent flaws in their implementation.[35][36][37]
A recent robust review of all the literature on carbon monoxide treatment
concluded that the role of hyperbaric oxygen is unclear and the available
evidence neither confirms nor denies a clinically meaningful benefit. The
authors suggested a large, well designed, externally audited, multicentre trial to compare normal oxygen with hyperbaric oxygen.[2]
The authors of this review of the literature (published in Toxicological Reviews and reference [2] above) then contacted the well-known and respected Cochrane Collaboration and suggested that they provide a "review" of the use of Hyperbaric Oxygen (HBO) in the treatment of CO Poisoning. The Cochrane Review that was subsequently published was essentially the same as the Toxicological Reviews paper, and had the same assessment and recommendations, namely that despite several clinical trials, there was insufficient evidence to support the use of HBO in CO Poisoning, and that a multicenter trial should be undertaken to gather enough evidence to put this question to rest once and for all.
Let's take a step back and start by analyzing from scientific plausibility and basic-science research. This is a critical step that distinguishes much of Evidence Based Medicine from Science Based Medicine. Having a therapy grounded in sound basic scientific plausibility should be the first step for gaining acceptance, and ideally should be a requirement before even starting to investigate the effects of any given therapy (can you hear me, homeopathy "researchers"?). Here the science is at least plausible. Binding affinities are dependent in part on the partial pressure of a gas in solution, and therefore having a higher partial pressure of oxygen in the blood would lead to a displacement of CO from the O2 binding sites in hemoglobin and myoglobin. Other direct effects of CO poisoning (such as mitochondrial interference) may not be as affected, but at least there is plausibility enough to suggest that HBO would provide benefit.
In the design of clinical trials, some are more rigorously designed and carried out than others. The goal of designing and carrying out a trial is to attempt to remove any potential sources of bias so that only one variable can be analyzed, in this case HBO versus regular Oxygen therapy. The study needs to have a sufficiently large number of patients to be able to detect an effect, and the hypothesis should be clearly identified prior to the start of the study. This last point is critical to avoid "cherry-picking" of data. If you get handed the raw data for a particular study and go through with a statistical analysis, it is likely that you can find something to support a particular ideology (such as has been the case for many Supplements, Complementary and Alternative Medicine studies, as well as studies on effectiveness of prayer). By defining at the beginning what it is you intend to measure, you avoid falling into the trap of "anomaly-hunting."
It turns out that a very well-designed study comparing normal O2 with HBO was published in the New England Journal of Medicine (NEJM), a highly-respected medical journal, largely because of stringent peer-review process. This study started way back in 1992, because even though CO poisoning is relatively common, it took some time to gather the number of cases needed for statistical significance. The study had all of the buzzwords you want to hear from a "definitive" study, with maybe one exception. It was Prospective, Randomized, Controlled, TRIPLE blinded (not just double-blinded, meaning that the patients, doctors, AND statisticians were blinded as to who was receiving which therapy). The only thing it did not have going for it was that it was a single center study, not a multi-center study. The results of the study were quite significant, as it showed strong effects of HBO in treating CO Poisoning.
Now to return to the question of when is EBM not SBM. Well, my pithy answer is: when there is an axe to grind.
It seems that Dr. Juurlink, who was the lead author on the Cochrane Review, and the last author (usually this means the lead Investigator) in the Tox Review paper, doesn't like HBO. Or maybe he doesn't like Dr. Weaver, the author of the NEJM paper. It seems that he assumed that, due to incorrect assumptions about methodology, he did not think that the NEJM paper constituted a "definitive" study. There is a very telling exchange between the two, as well as another ER doc and friend of mine, on the discussions/feedback page for the Cochrane Review article. Read the exchange from bottom to top, as it is posted in reverse chronological order, with oldest post at the bottom. I can summarize it for you here as well:
July 2006, by Dr. Weaver: The Review makes incorrect statements of fact andFrom the Cochrane Review summary:
draws conclusions that are invalid. Our study was extensively peer-reviewed and no-one else found the objections raised in the Review.
Sept 2006, by Dr. Juurlink: We corrected one of the incorrect
statements of fact. We continue to speculate that you changed your mind about outcome measures midway through the study.
Sept 2006, by Dr. Weaver: Your objections are not supported by the evidence and betray bias against the study outcome. I don't think I can convince you that you are wearing blinders, so bye-bye!
Oct 2006, by Dr. Juurlink: Good riddance, ya loony! And what I said before,
too!
Oct 2006, by Dr. Logue: The Cochrane Review failed to follow the procedures
established by the Cochrane collaboration.
Oct 2006, by Dr. Juurlink: Be Gone, you Emergency Medicine-shill!
Nov 2006, by Dr. Logue: Really, I'm just trying to figure out what to do in the patient's best interest, and here you are accusing people of fabricating results without taking the trouble to do some simple fact-checking.
[silence]
However, the review of published trials found conflicting, potentiallyThis is rather strong language. The NEJM study is not weak evidence, is not biased insofar as such things can be quantified and determined, and the lead author for the review failed to follow established Cochrane Collaboration procedures. These types of incidents need to be fully and openly discussed in the interest of truly pursuing scientific evidence to support treatments that will improve patient outcomes. And although both the NEJM and the Cochrane Collaboration enjoy a significant amount of prestige, and although published articles from each source carry significant persuasive authority, they are not immune to human nature. The science should be discussed on its own merit. I'd like to finish with a quote from Dr. Logue's response:
biased, and generally weak evidence regarding the efficacy of hyperbaric oxygen for the prevention of neurological symptoms
...I doubt that a separate panel of experts would make similar accusations of academic dishonesty without thoroughly verifying the facts first. In the world of evidence based medicine the end does not justify the means.
1 comment:
First of all, it's hard to critique much not knowing the intended audience (medical?) and format (blog?). Anyway, a few comments:
-- Good topic: controversial, but equipoise is possible.
-- You, however, seem to take a position favoring HBOT. If you want to argue for HBOT, then make that the central thrust of the article. However, if you are intending to highlight the ways in which an agenda can bias science, the article is quite unbalanced. Much of the HBOT literature comes from "true believers" who are greatly invested in demonstrating that HBOT works; I've met many of them and they can be messianic in their fervor. I think much of the skepticism of HBOT in the general medical community is in reaction to the evangelical manner of the Hyperbaric folks. There are certainly agendas and axes on both sides, and a more balanced article might better make your point, without calling into question your own agenda.
-- If intended for a medical audience, you can skim the summary of what CO poisoning is. If intended for a general audience, the literature component is a bit dense and might better be summarized rather than quoted so extensively.
-- An interesting point that occurred to me, is that if this NEJM study was so powerful, why was it not persuasive? (I didn't read it.) It may be a digression, but it might tie back into the central theme: "What does it take for evidence to persuade?"
-- Too long and wordy. Tighten it up a bit.
-- Biggest potential improvement. You're very formal in much (but not all of the piece), and it comes off as dry, as if for a medical journal. Give it a "hook." The introductory paragraph should use some colorful language, pose the lemma in a vivid and interesting way and draw the reader in. Read an old Stephen Jay Gould article for an example -- he was a master at that. Then, at the end, "wrap it up" with a summary that ties back to the intro paragraph. Again, colorful language better satisfies readers.
-- The "subject" is clear statement of your lemma, but not a great title. Make it provocative. For example, the pithy "What's the difference between Evidence and Science?" has more pop. Or go oblique: "On Axes and Evidence." Something that will intrigue the reader.
One of the fun things about my blog is that I actually have a lot of data on what gets read and what gets linked or quoted. I have noticed that there is a very clear correlation between the articles where I took the time to come up with a "sound bite" that is concise and memorable, and the traffic that various posts generate.
Have fun with it. I look forward to reading the next draft.
Post a Comment