About a year ago I criticized a “study” from Mississippi State University alleging that implementation of a smoking ban in Starkville, MS caused a dramatic decline in heart attacks. The problem was that the study was in fact just a press release and the data from control groups hadn’t even been collected yet. Without control groups there was no justification for asserting that the decline in heart attacks was caused by the smoking ban rather than other factors. I challenged the authors, Dr. Robert McMillen and Dr. Robert Collins, on this point and they declined to respond, saying only that they would continue their research when more data arrived.
Now the data is in and they have released another study confirming their hypothesis. To their credit they have control groups this time. What they don’t have are tests of statistical significance. I’ll hand it over to Michael Siegel:
This study violates the most fundamental principal of epidemiology and biostatistics: you must evaluate any scientific hypothesis to see whether the results could be explained by chance. In other words, you must determine whether your results are statistically significant.
A persistent flaw in studies of this type is reliance on small population sizes to extrapolate an exaggerated effect. This one appears no different. Siegel again:
I did my own calculations based on the results reported in the study and based on a conservative estimate which maximizes the likelihood of finding a statistically significant difference, I found that the difference between the two rates of decline was not even close to being statistically significant. […]
For these findings, which are exquisitely sensitive to a simple shift in one heart attack here and one heart attack there, one must not put any confidence in their statistical meaning. Clearly, the role that these are just chance differences cannot be ruled out given the small sample size. Nevertheless, the study goes as far as telling us the exact cost savings from the heart attacks averted due to the smoking ban.
McMillen and Collins are getting closer to doing real science. Close, but still no cigar!