Evidence Based Medicine-The Dark Side Part two
The core of EBM is the randomized Controlled Trial (RCT) and the systematic review ( meta-analysis) of multiple RCTs.
Two recent and widely quoted meta-analyses (MAs) of mammography effectiveness reached different conclusions working with the same original data base but eliminating different studies from the universe of analysis. Olsen and Gotzche from Denmark (Screening for breast cancer with mammography.Cochrane database Syst.Rev.2001;CD001877) eliminated all but two studies in their MA and found that the all cause mortality was no different in the screened group. The USPSTF analysis was more inclusive in their pooled analysis and found a 16% reduction in breast cancer mortality. What do you do when the meta-analyses conflict?Steve Goodman (Ann Int Med 3 sept 2002 volume 137 issue 5 pages 363-365) explains that a MA itself is basically an observational design using published studies as the subjects. What subjects you keep and which you eliminate from the analysis obviously can turn the results of the analysis around. He says " This controversy shows that the justification of why studies are included or excluded ...can rest on competing claims of methodologic authority" These claims, he says, " look little different from the traditional claims of medical authority that proponents of evidence-based medicine have criticized"
In the RCT the process of subject elimination is more transparent and once the study is underway there are safeguards against eliminating subject whose upcomes go against the researchers desired outcome.MAs are transparent to the extent that the reader is informed of what studies are eliminated but there is nothing to prevent the researcher from doing pre study simulations to see what eliminations lead to which results and proceeding accordingly. Out right fraud could occur but more likely the bias of the researchers would be the culprit. The point is the MA could be rigged.Entities (HMOS, Governments etc) who incur costs with a given medical intervention would welcome MAs that show no effectiveness. Follow the money.
- I think the main point here is really a MA should be considered context dependent not universally true. The context is which studies are included and ignored and what is the outcome statistic used. In the two studies quoted above, different data sets were selected and difference outcome measures used. So " Is mammography effective"?, well,it depends.