What really happened in Starkville, revisited

About a year ago I criticized a “study” from Mississippi State University alleging that implementation of a smoking ban in Starkville, MS caused a dramatic decline in heart attacks. The problem was that the study was in fact just a press release and the data from control groups hadn’t even been collected yet. Without control groups there was no justification for asserting that the decline in heart attacks was caused by the smoking ban rather than other factors. I challenged the authors, Dr. Robert McMillen and Dr. Robert Collins, on this point and they declined to respond, saying only that they would continue their research when more data arrived.

Now the data is in and they have released another study confirming their hypothesis. To their credit they have control groups this time. What they don’t have are tests of statistical significance. I’ll hand it over to Michael Siegel:

This study violates the most fundamental principal of epidemiology and biostatistics: you must evaluate any scientific hypothesis to see whether the results could be explained by chance. In other words, you must determine whether your results are statistically significant.

A persistent flaw in studies of this type is reliance on small population sizes to extrapolate an exaggerated effect. This one appears no different. Siegel again:

I did my own calculations based on the results reported in the study and based on a conservative estimate which maximizes the likelihood of finding a statistically significant difference, I found that the difference between the two rates of decline was not even close to being statistically significant. […]

For these findings, which are exquisitely sensitive to a simple shift in one heart attack here and one heart attack there, one must not put any confidence in their statistical meaning. Clearly, the role that these are just chance differences cannot be ruled out given the small sample size. Nevertheless, the study goes as far as telling us the exact cost savings from the heart attacks averted due to the smoking ban.

McMillen and Collins are getting closer to doing real science. Close, but still no cigar!

Share

What really happened in Starkville?

Last week I briefly commented on a study of heart attack (AMI) rates in Starkville, MS that found a 27% reduction in the AMI rate in the three years following the imposition of a smoking ban. Though I was initially dismissive, I decided to email the authors, Dr. Robert McMillen and Dr. Robert Collins, to ask for a copy of the study and to see the data. I expected to find the usual flaws, but what I found was even worse: In truth there is no completed study, the relevant data haven’t even been collected yet, and there are no grounds for concluding that the decline in heart attacks had anything to do with Starkville’s smoking ban.

Journalists covering the story reported that the results came from a study that had just been released. This is understandable, since that is what Mississippi State’s* press release claims:

A Mississippi State study released Monday [Nov. 9] shows a 27 percent decrease in heart attacks among Starkville residents since the city passed a smoking ban in 2006.

The “study” was actually just a two page preliminary report [PDF] that has not been published in a journal or subject to peer review. Nonetheless, in the press release and the report the authors express certainty that the decline is attributable to the smoking ban:

Without spending a dime, the community of Starkville has decreased heart attacks by more than 25% […]

Current research shows that smoking bans in Starkville have contributed to a 27% reduction in the heart attacks of local residents […]

There are two flaws that plague studies of heart attack reductions in small cities like Starkville. One is the size of the population, which makes wild fluctuations in the data possible. Starkville’s resident population is just a little over 20,000, so there’s nothing to be done about that. The other flaw is the lack of control groups. AMI rates could be declining for reasons unrelated to smoking bans, so it’s necessary to compare communities with smoking bans to similar communities without them.

Admirably, McMillen and Collins intend to collect data from other Mississippi cities to act as control groups. However they don’t have this data yet. When I wrote to them asking for a copy of the study, all I got was this email from Dr. McMillen:

The results that we released in our 2 page report are preliminary results form a larger study. The study examine AMI admissions from several communities that have passed smoke-free ordinances, as well as communities that have not. The latter will serve as controls for the purpose of comparison. The hospital in Starkville was the first to provide us with AMI admission data. When we have received data from the other communities in our study we will prepare a manuscript that compares AMI admissions before and after the smoke-free laws were enacted in communities with and without smoke-free laws.

Without the data from control groups there is no evidence that the decline was caused by the smoking ban. There’s absolutely no way to know whether the ban deserves complete, partial, or zero credit for the decline, and any statement to the contrary is pure speculation.

Michael Siegel has looked into Mississippi’s heart attack data and found that the AMI mortality rate has been trending downward throughout the state:

[…] from 2000-2002 to 2003-2005, there was a 19.3% decline in heart attack deaths in Mississippi. And from the period 2000-2002 to 2006-2008, there was a 29.2% decline in heart attack deaths in Mississippi.

From 2003 to 2008, heart attack deaths in Mississippi fell by 19.1%. From 2002 to 2008, they fell by 29.4%. And from 2000 to 2008, they fell by more than one-third: by 33.9%.

This demonstrates that heart attacks were declining substantially in the state of Mississippi anyway, even in the absence of a smoking ban. It suggests that if Starkville had not enacted a smoking ban, the study still would have found a very large decline in heart attacks from 2003-2005 to 2006-2008.

On Friday I wrote to McMillen and Collins to ask on what basis they reached their conclusion without having received the data from their control groups:

In the two page report you imply rather strongly that the decline in Starkville’s AMI rate is due to the smoking ban (“Without spending a dime, the community of Starkville has decreased heart attacks by more than 25%…”) (“Current research shows that smoking bans in Starkville have contributed to a 27% reduction in the heart attacks of local residents…”). I am curious to know how you have reached this conclusion without having received the data from control groups, as this seems to go far beyond what one could determine from your preliminary results.

I’ve yet to receive any response from them.

In two of my emails I requested to see the data from Starkville broken down year-by-year. If we are to believe the studies from Pueblo and Helena, the bulk of the decline should have occurred immediately after the ban took effect. It would be useful to know if the same pattern occurred in Starkville. They have yet to release this data.

Collins and McMillen have given their statements a veneer of scientific objectivity without providing the transparency needed to substantiate their claims. This hasn’t stopped them from using their results to push for a statewide smoking ban:

Data from the Mississippi State Department of Health and national figures indicates the state would save an estimated $125 million annually in health care expenses if a legislatively mandated smoking ban was enacted, [Collins] added.

“I plead with the Mississippi Legislature to ban smoking in public places,” Collins said. “Our data reflects the findings of every other community that has looked at what happened when smoking is banned in public venues.”

This is yet another example of what Siegel aptly calls “science by press release.” McMillen and Collins have no basis on which to rest their claims, have not published them in any journal, and haven’t even collected all of their data. Yet because they were critical of tobacco a willing press disseminated their conclusions far and wide without asking difficult questions or consulting critics.

When this study is completed it will be interesting to see what if finds. At that time its results could possibly be informative in the debate over a statewide smoking ban. But to go public with results now, before the data is even collected, is a corruption of science for the pursuit of political ends and unfortunately all too typical for today’s anti-smoking movement.

*Corrected.

Share

A Starkville miracle

Speaking of heart attack miracles, a new one has been discovered to justify a possible statewide smoking ban in Mississippi:

A ban on indoor smoking has paid big dividends for the health of Starkville residents.

In the three years after the ban was enacted in 2006, Starkville had 27 percent fewer heart attacks than in the three years preceding the ban, based on the results of a Mississippi State University Social Science Research Study released Monday.

Of course none of the reporters covering the story bothered to include of a word of skepticism, and dramatic results like this don’t appear in large samples in the UK, Australia, and US states. But hey, Starkville!

More on the problems with small-town heart attack studies here and here.

Share

Motorhome Diaries update

The guys have published the full account of their detainment and it reflects some pretty appalling treatment from the Jones County police, including being pepper sprayed, having their vehicle searched without consent, and being locked up without being read their rights, being informed of the charges against them, or allowed to contact help. Read it all here.

Share

MHD crew arrested on the road

One of the surprise treats from my weekend family trip to Austin was crossing paths with my friends Jason and Pete from Motorhome Diaries, along with their new companion Adam. Together we had a great time crashing a Southwestern University frat party on Friday. (Personal discovery: I have a lot more fun at frat parties now than I did when I was actually in college.)

But it was only a matter of time before the RV of freedom hit a speed bump. All three have been arrested following a traffic stop that they documented with cameras and video cameras. The reasons for arrest in Jones County, Mississippi:

Adam Mueller – Disorderly Conduct and Disobeying an Officer
Pete Eyre – Possession of a Beer in a Dry County
Jason Talley – Disorderly Conduct, Disobeying, and Resisting Arrest

Obviously we don’t have the full story yet, but knowing these guys I’m going to guess that “disorderly conduct” is code for asserting their constitutional rights and not duly respecting the officers’ authoriteh. Pete was probably the calmest of them all, making possession of beer the only thing police could think to charge him with. (And speaking of which, it’s 2009! WTF is wrong with a county in which adults can get arrested for possession of beer?)

The guys are expected to be arraigned tomorrow. Hopefully they’ll get out soon and be able to report all the details. In the meantime, you might want to give them a little support at their PayPal link.

Previously: Pete videotaped an Arlington, VA officer illegally parking and was then trailed by that officer as he attempted to walk home.

Update: More details in the morning once the guys have their computers back. In the meantime, from @jdtalley’s Twitter feed: “The #MHD crew has been released from Jones County jail. I was pepper sprayed and choked for refusing to give ID.”

Share