O’Shaugnessy’s to RES, MD: The new O’Shaughnessy’s has a great piece by Jack McCue, MD, describing the establishment’s preoccupation with the alleged “quality of the evidence.” We need to challenge the Cult of the RCT and the dismissal of “mere anecdotal evidence,” especially when the latter is coming from hundreds of thousands.
RES to O’S: Peter Simon (now retired from RI Dept of Health) reminded me that a professor of ours in the dept of epidemiology at Johns Hopkins wrote an important paper making that point and saying a good cohort study could do the job and RCTs were overly endorsed and supported by Big Pharma. Check George Comstock, Journal of Epidemiology about 30 years ago.
Joe D. Goldstrich, MD, found these two papers by Comstock in Epidemiology:
Neither one seems to fit the paper as recalled by Dr. Simon, so we’ll keep looking. Comstock was obviously a brilliant, humane doctor. Here are some riffs from his 2003 interview in Epidemilology:
DS: When you first got started, what was epidemiology like? What did an epidemiologist do? How was it different from the way we practice it today?
GC: Statistics plays a more prominent role today. Epidemiology back then was based more on logic and whether the results made sense biologically. My early papers don’t have a P value or significance test. The “biostatisti- cians” were more like today’s project directors. They knew how to carry out studies and how to avoid biases.
Technology has improved. We are able to measure more things in different biologic samples. But I don’t think the basic principles of epidemiology have changed at all. It’s just that we now rely more on statistical significance and less on what makes sense.
DS: Do you think that is a change for the worse?
GC: It’s a mixture. One positive advance is our ability to handle multiple variables in our statistical analyses. The best we could do in our early studies was stratification or analysis of variance with a limited number of variables at a time.
But I believe we pay too much attention today to the statistical significance and not enough to replication. Even in a huge study with highly significant P values, the results could still be due to chance. The tendency is to rely on results from one big study or from pooling studies. I think this is a mistake. I would put much more faith in results from 3 relatively small studies showing the same results in different populations and done in different ways, regardless of the significance level, than I would in any one study with a very small P value.
I think a good example of how a big study can lead you astray is the University Group Diabetes Project. This was a well-done study that showed an increased risk for cardiovascular death associated with one of the oral antidiabetic agents. If there hadn’t already been other studies underway, it is not likely that anyone would have initiated another study to confirm this finding, and we might not be using oral treat- ments for diabetes today. However, the other studies didn’t confirm the UGDP’s findings.
DS: Over the years there has been debate about the relationship between epidemiology and public health practice and about the failings of the discipline. Has epidemiology lost its roots in public health?
GC: I’m glad you brought that up. I think that we have moved too far from public health. Not many on the Hopkins faculty, for example, have had contact with a public health department. I think that is too bad, because epidemiology has to be the basic science of public health. Public health is both the study of populations and health services.
I think most of the criticisms epidemiology has faced are not due to the discipline per se. One factor is that there is insufficient acknowledgment that we are almost invariably looking at associations and not causes. In particular, the media want to treat every study as if it were the final answer. And we as investigators get trapped into that because of the questions reporters ask us and what gets reported. The media point us in that direction and we follow along far too readily. Even if we had said, “We really need to replicate this, we really need to look back to see what other people know,” that doesn’t tend to get reported. It is no great surprise that various critics jump on us. This is what they see of epidemiology.
We also are not humble enough in our written work. We often state that more research is needed without specify- ing that it needs to be replicated in a slightly different way or allowing that the conclusions could be wrong.
About George W. Comstock, MD (from the Johns Hopkin School of Public Health site):
Horace Mann’s admonition to “Be ashamed to die until you have won some victory for humanity” is engraved on George Comstock’s keychain.
As a commissioned officer in the U.S. Public Health Service for 21 years, he served first with the U. S. Coast Guard on shore and at sea. Later, he was with the Research Unit of the Tuberculosis Control Division with major responsibility for the controlled trials of BCG vaccination in Georgia, Alabama, and Puerto Rico, and of preventive treatment of latent tuberculosis infection in Alaska. Results of these studies had major influences on tuberculosis control in the U. S.
For five decades as director of the Johns Hopkins Training Center for Public Health Research in the Washington County Health Department, he supervised the community-wide collection of health-related data and serum specimens that have proved invaluable for research on the identification of risk factors and for long-term longitudinal studies of chronic disease. He also served as editor-in-chief of the American Journal of Epidemiology for nine years.
