Likelihood ratios and Evidence Based Medicine, what does the evidence show?

Part of the teachings of EMB is the use of the likelihood ratio. It is defined as the likelihood that a given test result would be expected in a patient with the disease of interest compared to the likelihood that the same result would be expected in someone without that disease. To use one , you have to "estimate" the pretest probability of a disease in a given setting. It has always appeared to me that this means- much of the time- you simply make up a number.If that is true, a group of doctors given the same clinical scenario would make up different numbers. Now there is evidence that this is exactly what happens. Drs. Phelps and Levitt (Acad. Emerg. Med,2004 June:11 (6) 691-694) gave clinical vignettes to a group of IM and ER trainees and attendings and asked them to estimate the likelihood of several illnesses. The numbers varied widely.The smallest difference in the pretest probability was 70%. If you think about "making up" a number as opposed to "estimating" a number, the validity of the whole process is perceived differently. Here, at least, the "evidence" that a particular element of Evidence Based Medicine is valid in a simulated real world setting is lacking and in fact suggests that it is not. The Ontario EMB experts in their book "Evidence Based Medicine" indicate there are several ways to estimate the pretest probability besides the physician using her experience including national and regional data bases. But in the real world it seems that the most common way of estimating pretest probability is to use one's own experience because such data is either non existent or you cannot access it. In this regard perhaps older, more experienced doctors would have the edge.

EMB dogma states that the likelihood ratio is the best way to judge how much a test result help to make the diagnosis. In theory this is clearly the case but the scheme does not work very well if the pretest probability value is is so nebulous.