What really happened in Starkville, revisited

About a year ago I criticized a “study” from Mississippi State University alleging that implementation of a smoking ban in Starkville, MS caused a dramatic decline in heart attacks. The problem was that the study was in fact just a press release and the data from control groups hadn’t even been collected yet. Without control groups there was no justification for asserting that the decline in heart attacks was caused by the smoking ban rather than other factors. I challenged the authors, Dr. Robert McMillen and Dr. Robert Collins, on this point and they declined to respond, saying only that they would continue their research when more data arrived.

Now the data is in and they have released another study confirming their hypothesis. To their credit they have control groups this time. What they don’t have are tests of statistical significance. I’ll hand it over to Michael Siegel:

This study violates the most fundamental principal of epidemiology and biostatistics: you must evaluate any scientific hypothesis to see whether the results could be explained by chance. In other words, you must determine whether your results are statistically significant.

A persistent flaw in studies of this type is reliance on small population sizes to extrapolate an exaggerated effect. This one appears no different. Siegel again:

I did my own calculations based on the results reported in the study and based on a conservative estimate which maximizes the likelihood of finding a statistically significant difference, I found that the difference between the two rates of decline was not even close to being statistically significant. […]

For these findings, which are exquisitely sensitive to a simple shift in one heart attack here and one heart attack there, one must not put any confidence in their statistical meaning. Clearly, the role that these are just chance differences cannot be ruled out given the small sample size. Nevertheless, the study goes as far as telling us the exact cost savings from the heart attacks averted due to the smoking ban.

McMillen and Collins are getting closer to doing real science. Close, but still no cigar!


What really happened in Starkville?

Last week I briefly commented on a study of heart attack (AMI) rates in Starkville, MS that found a 27% reduction in the AMI rate in the three years following the imposition of a smoking ban. Though I was initially dismissive, I decided to email the authors, Dr. Robert McMillen and Dr. Robert Collins, to ask for a copy of the study and to see the data. I expected to find the usual flaws, but what I found was even worse: In truth there is no completed study, the relevant data haven’t even been collected yet, and there are no grounds for concluding that the decline in heart attacks had anything to do with Starkville’s smoking ban.

Journalists covering the story reported that the results came from a study that had just been released. This is understandable, since that is what Mississippi State’s* press release claims:

A Mississippi State study released Monday [Nov. 9] shows a 27 percent decrease in heart attacks among Starkville residents since the city passed a smoking ban in 2006.

The “study” was actually just a two page preliminary report [PDF] that has not been published in a journal or subject to peer review. Nonetheless, in the press release and the report the authors express certainty that the decline is attributable to the smoking ban:

Without spending a dime, the community of Starkville has decreased heart attacks by more than 25% […]

Current research shows that smoking bans in Starkville have contributed to a 27% reduction in the heart attacks of local residents […]

There are two flaws that plague studies of heart attack reductions in small cities like Starkville. One is the size of the population, which makes wild fluctuations in the data possible. Starkville’s resident population is just a little over 20,000, so there’s nothing to be done about that. The other flaw is the lack of control groups. AMI rates could be declining for reasons unrelated to smoking bans, so it’s necessary to compare communities with smoking bans to similar communities without them.

Admirably, McMillen and Collins intend to collect data from other Mississippi cities to act as control groups. However they don’t have this data yet. When I wrote to them asking for a copy of the study, all I got was this email from Dr. McMillen:

The results that we released in our 2 page report are preliminary results form a larger study. The study examine AMI admissions from several communities that have passed smoke-free ordinances, as well as communities that have not. The latter will serve as controls for the purpose of comparison. The hospital in Starkville was the first to provide us with AMI admission data. When we have received data from the other communities in our study we will prepare a manuscript that compares AMI admissions before and after the smoke-free laws were enacted in communities with and without smoke-free laws.

Without the data from control groups there is no evidence that the decline was caused by the smoking ban. There’s absolutely no way to know whether the ban deserves complete, partial, or zero credit for the decline, and any statement to the contrary is pure speculation.

Michael Siegel has looked into Mississippi’s heart attack data and found that the AMI mortality rate has been trending downward throughout the state:

[…] from 2000-2002 to 2003-2005, there was a 19.3% decline in heart attack deaths in Mississippi. And from the period 2000-2002 to 2006-2008, there was a 29.2% decline in heart attack deaths in Mississippi.

From 2003 to 2008, heart attack deaths in Mississippi fell by 19.1%. From 2002 to 2008, they fell by 29.4%. And from 2000 to 2008, they fell by more than one-third: by 33.9%.

This demonstrates that heart attacks were declining substantially in the state of Mississippi anyway, even in the absence of a smoking ban. It suggests that if Starkville had not enacted a smoking ban, the study still would have found a very large decline in heart attacks from 2003-2005 to 2006-2008.

On Friday I wrote to McMillen and Collins to ask on what basis they reached their conclusion without having received the data from their control groups:

In the two page report you imply rather strongly that the decline in Starkville’s AMI rate is due to the smoking ban (“Without spending a dime, the community of Starkville has decreased heart attacks by more than 25%…”) (“Current research shows that smoking bans in Starkville have contributed to a 27% reduction in the heart attacks of local residents…”). I am curious to know how you have reached this conclusion without having received the data from control groups, as this seems to go far beyond what one could determine from your preliminary results.

I’ve yet to receive any response from them.

In two of my emails I requested to see the data from Starkville broken down year-by-year. If we are to believe the studies from Pueblo and Helena, the bulk of the decline should have occurred immediately after the ban took effect. It would be useful to know if the same pattern occurred in Starkville. They have yet to release this data.

Collins and McMillen have given their statements a veneer of scientific objectivity without providing the transparency needed to substantiate their claims. This hasn’t stopped them from using their results to push for a statewide smoking ban:

Data from the Mississippi State Department of Health and national figures indicates the state would save an estimated $125 million annually in health care expenses if a legislatively mandated smoking ban was enacted, [Collins] added.

“I plead with the Mississippi Legislature to ban smoking in public places,” Collins said. “Our data reflects the findings of every other community that has looked at what happened when smoking is banned in public venues.”

This is yet another example of what Siegel aptly calls “science by press release.” McMillen and Collins have no basis on which to rest their claims, have not published them in any journal, and haven’t even collected all of their data. Yet because they were critical of tobacco a willing press disseminated their conclusions far and wide without asking difficult questions or consulting critics.

When this study is completed it will be interesting to see what if finds. At that time its results could possibly be informative in the debate over a statewide smoking ban. But to go public with results now, before the data is even collected, is a corruption of science for the pursuit of political ends and unfortunately all too typical for today’s anti-smoking movement.