Evidence Based Medicine-The Dark Side Part two

The core of EBM is the randomized Controlled Trial (RCT) and the systematic review ( meta-analysis) of multiple RCTs.

Two recent and widely quoted meta-analyses (MAs) of mammography effectiveness reached different conclusions working with the same original data base but eliminating different studies from the universe of analysis. Olsen and Gotzche from Denmark (Screening for breast cancer with mammography.Cochrane database Syst.Rev.2001;CD001877) eliminated all but two studies in their MA and found that the all cause mortality was no different in the screened group. The USPSTF analysis was more inclusive in their pooled analysis and found a 16% reduction in breast cancer mortality. What do you do when the meta-analyses conflict?

Steve Goodman (Ann Int Med 3 sept 2002 volume 137 issue 5 pages 363-365) explains that a MA itself is basically an observational design using published studies as the subjects. What subjects you keep and which you eliminate from the analysis obviously can turn the results of the analysis around. He says " This controversy shows that the justification of why studies are included or excluded ...can rest on competing claims of methodologic authority" These claims, he says, " look little different from the traditional claims of medical authority that proponents of evidence-based medicine have criticized"
In the RCT the process of subject elimination is more transparent and once the study is underway there are safeguards against eliminating subject whose upcomes go against the researchers desired outcome.MAs are transparent to the extent that the reader is informed of what studies are eliminated but there is nothing to prevent the researcher from doing pre study simulations to see what eliminations lead to which results and proceeding accordingly. Out right fraud could occur but more likely the bias of the researchers would be the culprit. The point is the MA could be rigged.Entities (HMOS, Governments etc) who incur costs with a given medical intervention would welcome MAs that show no effectiveness. Follow the money.

• I think the main point here is really a MA should be considered context dependent not universally true. The context is which studies are included and ignored and what is the outcome statistic used. In the two studies quoted above, different data sets were selected and difference outcome measures used. So " Is mammography effective"?, well,it depends.

Evidence Based Medicine- The Dark Side Part One

The most recent definition of Evidence based medicine (EBM) preempts criticism in its inclusiveness. " The integration of the best evidence with clinical expertise and patient values" Previous criticism of EBM claimed there was neglect of clinical expertise and the patient's values and desires.
Who could object to the inclusive definition." It is mom and apple pie and more pie. Surely there could be no dark side to that.

However, when most physicians think of EBM,their thoughts focus on randomized control trials (RCT) and meta-analysis rather than the all embracing arguably vacuous definition. This is because RCTs and meta-analysis are really what EBM is all about.
A darker side could emerge if in the zeal to champion the RCTs the other factors one needs to consider are neglected. Those factors are, of course, prior evidence and biological plausibility.Dr. Steven Goodman from Johns Hopkins has cogently argued for this as have Sehon and Stanley in a wide ranging and very interesting philosophical analysis of the EMB issue.
Another author, Miles Little,(ANZ Journal of Surgery.Vol.73,issue 4, pg 177, April 2003) speaks of EBM's "cult status" and lists 8 areas of concern including its unwitting paternalism,its unstable truths and its reductionism.
Bandolier ( a journal of evidence based medicine) admits that references to EMB can serve as a talisman. To say that the author's position is evidence based is supposed to end the discussion much as saying "its God's Will" might have served in an earlier era.

In anyone thinks there will always be an arrow from RCT to clinical algorithm for all or most clinical problems here are few of reasons why that will not be true. There is not enough time or money or interested funding groups to do all the RCTs needed for all the issues. RCTs may be contradictory (i.e. ALLHAT and the 2Nd Austalian National blood pressure study). RCTs become outdated as technical advances pile up faster that RCT can catch up.

If there are problems and limitations with RCTs, there are meta problems and issues with meta-analysis, an issue to consider later.

McMaster Clinical epidemiology Group: Evidence based medicine is best spoon fed

In an April 2000 BMJ editorial(BMJ 2000:320. 954-955 (8 April)) Jaeschke,Cook and Haynes-from the McMaster Clinical epidemiology Group- state that their residents as well as British GPs prefer "preappraised " evidence, rather than doing the analysis themselves. They have concluded that not all trainees are interested in attaining advance EBM analytic skills. The interesting thing here is how the authors reached that conclusion. It was not through a randomized controlled trial, nor was it through a systematic observational study but as they admit "after a decade of unsystematic observation".This is the type of clinical evidence that they -rightly enough-have placed on the bottom tier of the clinical evidence hierarchy. Is this not the same type of evidence that they denigrate in their advocacy of the RCTs. How does their pronoucement differ from the dogma that old clinical profs passed down to their proteges? And yet it is on that basis that they make their recommendation of best ensuring evidence based care by supplying preappraised evidence based summaries. Does this sound like "Don't worry we will analyze the data and tell you what to do"? Is there a difference between the dogma we learned from old clinicians based on their experience and the dogma from the EBM analytic experts? In either case it seems the trainees of days gone by and the trainees of today want to be told what to do. What do the trainees do when two meta analyses on the same issue differ?

The random and the deterministic in medicine

Forty years ago pathophysiological concerns were stressed in medical education and statistics and epidemiology were little more than John Snow and the Broad St. Pump and the mean and standard deviations and t tests.The landscape has changed. Logistic regressions, Markow simulations, and even more obscure statistical tools populate the methods sections of medical articles.There is much "black box" output.
There is a theme in the history of medicine of an affection for determinisitic thinking and an antipathy for statistics. Statistics, after all, to a large degree began in the study of gambling. Coin flips, and rolls of dice were not the stuff that physicians were concerned with. When did the pendulum swing so far to the data side ? When did concerns with aggregate data begin to push out thoughts about what exactly is going on?.
The following is an example how how what at first seemed to be a random occurence was determined to a large degree by a SNP or single nucleotide polymorphism. The small percent of children treated with thioprine type drugs who developed a serious leukopenia did appear random until it was learned than about 1/300 Caucasians are deficient in TPMT (Thiopurine methyl transferase) which in turn was caused by a single misplaced nucelotide. Now testing for that enzyme deficiency or for the genetic variation allows for downward adjustment of the chemotherapy dose for those patients with TPMT deficiency and the previously random serious marrow damage is avoided.
Of course, every situation with bad clinical outcomes that can be described in aggregate numbers is not that simple. But, at least some of the time what appears at first to be random is shown to be determined by a describable -and often controlable -mechanism. At least some of the time and and at some level random is simply ignorance.
div>

subject matter experts and methodologists

The American College of Chest Physician's most recent treatise on venous thrombosis (The seventh ACCP Conference on Antithrombotic and Thrombolytic therapy)mentions in its methods section that two kinds of experts are involved in their very labor intensive review.There are subject matter experts -these presumably include physicians who actually treat DVTs- and also individuals who are well versed in statistics and the methodology of various aspects of clinical epidemiology. I believe this is a good thing. Less of a good thing are a number of articles I have seen over the past few years where all of the authors seem to be or at least tacitly claim to be in the latter category. Typical is a review article (a "systematic review") by generalists of a topic traditionally in the purview of a specialist, for example Parkinson Disease. As likely as not the multiple authors are members of a general internal medicine department and several if not all have appended to the defining initials after their names the letters"MPH". Their methodological reviews may be good or not but what is often missing is the insight of a subject matter expert.For example, someone who has actually treated many patients with the disease of interest. Such a person can put the data analysis in some type of real life clinical perspective.
Thirty years ago journal editorialists needed only to be the subject expert.Now they also either need to be experts in or need the help of experts in statistics and epidemiology to put issues into a contextual mix blending an analysis of the data currently being published with the prior evidence,the biological plausibility concerns and the clinical relevance.
I believe that review articles should routinely include among the authors someone who is in fact a subject matter expert even if others do the methodological heavy lifting. There is plenty of room for both types of experts in the advancing of medical knowledge and both types are needed.Evidence based medicine is currently defined as the integration of the best evidence with clinical expertise and patient values. Liklihood ratios and "NNTs" are great-atleast some find them useful- but we also may benefit from the insight from someone who knows the trees as well as those who describe the forrest in aggregate data terms.

There is a fundamental conflict regarding how to determine the effectiveness of cancer screening

Current preventive medicine (apart from immunizations) focuses on cancer screening and coronary artery disease prevention. Cervical cancer screening has moved past the stage of debate. Screening for breast cancer, colon cancer, prostate cancer and lung cancer have not, although different aspects of each are at issue.

In the debate about the value of lung cancer screening a foundational conflict about how decisions are made regarding the effectiveness of screening becomes evident. Specifically, the question becomes which statistical method should be determinative in analyzing the data from randomized population screening trials. Those who believe the final answer lies in disease specific mortality have decided that lung cancer screening is of no value (and this view is the more commonly held one). Dissenters believe that in this type of trial-as opposed to a treatment trial-cure rate or five year survival is the analytic technique that uncovers the truth. A prolific advocate of this dissenting view is Dr. Gary M. Strauss. According to the generally accepted paradigm a reduction in the cause specific mortality in a randomized trial is accepted as the definitive measure of effectiveness. Strauss and others question the assumptions underlying the paradigm.
It gets more complicated. Dr. William C. Black in the Feb. 6, 2002 issue of the Journal of the NCI argues that all cause mortality may be less affected by bias than disease specific mortality. Once the data are collected, and in these screening trials it may take years, the problem remains that there are widely different views as to how to analyze the outcome (i.e what measure of effectiveness do you use) and the method you choose may determine the answer and therefore policy decisions and advice to patients based on the study.
Most of the controversy over lung cancer screening involved trials using chest xrays. As thoracic imaging evolves, similar rhetorical exchanges will likely take place following publications of screening projects with various generations of CT scans.

Does this mean that our ability to use the tools of epistemology lags behind our technological advances?

Relative risks less than two

A recent article in Annals of Internal Medicine claimed that Vit E increased the risk of death. The relative risk (RR) was 1.01. If a relative risk of 2 increases the risk of the outcome of interest two fold,then a relative risk of 1.01 increases the risk by 1/100. This is clearly a difference too small to really measure and too small to care about.
Relative risks of less than two are frequently published in medical journals and often represent more noise than signal but when repeated in the lay press alarms some,confuses others and generally give even more reason for folks to think that the medical profession is not too sure about much of anything.Of course, a relative risk even as "big" as two does not guarantee a causal relationship, but courts are beginning to accept a relative risk of 2 or more as fulfilling the "more likely than not" standard of proof for some types of civil litigation. Let's not forget the role of prior evidence and biological plausibility in considering what significance should be given to " statistically significant" RRs.