Lazy reporting and the Pueblo ban study

The Centers for Disease Control have issued a new report about the impact of the smoking ban in Pueblo, Colorado. The study has the media breathlessly repeating claims that the ban dramatically saves lives. “A smoking ban caused heart attacks to drop by more than 40 percent in one U.S. city and the decrease lasted three years, federal health experts reported Wednesday,” writes Reuters reporter Maggie Fox, who doesn’t bother quoting any dissenting sources. Mary Engle at the LA Times health blog says uncritically that whatever the mechanism behind the fall in heart attacks, “Pueblo’s smoking ban can take the credit.” Bill Scanlon at the Rocky Mountain News throws science to the wind and extrapolates that Colorado will see a statewide “sharp decline” in heart attacks in 2009 — more than two years after its ban went into effect.

I realize times are tough in newsrooms, but there’s no excuse for such biased, lazy reporting. Journalists should treat the claims of ideologically driven anti-smoking groups with just as much skepticism as they would junk science coming from big tobacco companies.

Since the CDC’s report is going to be cited constantly by smoking ban advocates it’s worth taking a look at its methodology and limitations. Fortunately it’s straightforward enough that any moderately intelligent person can understand it. The following is my layman’s reading of the results, with the caveat that I’m approaching this without formal training. Nonetheless, it’s clear that one shouldn’t take this study’s conclusions at face value. Its use by anti-smoking groups, researchers, and the press to promote smoking bans is a case study in the abuse of science for political ends.

The study examines rates of heart attacks (acute myocardial infarctions, “AMI” for short) in three areas: the city limits of Pueblo, which implemented a smoking ban in bars and restaurants, and the unincorporated parts of Pueblo County and all of El Paso County, which did not have smoking bans. The researchers compare the rates in the 18 months prior to the ban’s implementation on July 1, 2003 (pre-implementation), and the two 18 month periods in which the ban has been in effect (Phases I and II). The researchers estimate the impact of the smoking ban by comparing the rate of AMI in Pueblo to the control areas. On the surface, the results are striking:

The rate of AMI hospitalizations for city residents decreased 27%, from 257 per 100,000 person-years during the 18 months before the ordinance’s implementation to 187 during the 18 months after it (the Phase I post-implementation period).* This report extends that analysis for an additional 18 months through June 30, 2006 (the Phase II post-implementation period). The rate of AMI hospitalizations among city residents continued to decrease to 152 per 100,000 person-years, a decline of 19% and 41% from the Phase I post-implementation and pre-implementation period, respectively. No significant changes were observed in two comparison areas.

That’s a huge impact. If smoking bans really do reduce heart attacks by 41% in just three years, even ardent libertarians might be tempted to support them. But before we do that, let’s look at two obvious questions. The first is whether this reduction occurs because non-smokers are no longer exposed to secondhand smoke or because smokers light up less frequently or even quit entirely due to the inconvenience. The second is whether the effect is truly that large or if it’s exaggerated by problems with the sample.

Let’s start with the question of who benefits from the ban. The authors of the report write:

These findings suggest that smoke-free policies can result in reductions in AMI hospitalizations that are sustained over a 3-year period and that these policies are important in preventing morbidity and mortality associated with heart disease. This effect likely is mediated through reduced SHS exposure among nonsmokers and reduced smoking, with the former making the larger contribution.

The authors do not have a good basis for claiming that non-smokers are the primary beneficiaries. Their data do not track the smoking status of AMI patients. Nor do they rigorously control for secondhand smoke exposure; city residents could be working or socializing in smoking bars outside the city limits and vice-versa, or residents within the ban area could be smoking more in their homes. At best, the distribution of reduced heart attacks is a matter of speculation based on other studies. You don’t have to take my word on this. The CDC’s own editorial accompanying the report states:

Smoke-free policies have been found to prompt some smokers to quit smoking (1); because active smoking is a major risk factor for heart disease and AMI, this effect also would be expected to reduce heart disease and AMI rates at a population level… [Because] no data were available on whether study subjects were nonsmokers or smokers, determining what portion of the observed decrease in hospitalizations was attributable to reduced SHS exposure among nonsmokers and what portion was attributable to increased quitting among smokers was not possible.

This is the least of the study’s problems. Let’s talk now about its samples. The trend within the city of Pueblo is based on just three study periods.* To know if the declines in the two post-ban periods are meaningful we have to know if the AMI rate in the pre-implementation period is normal. If those 18 months just happened to have an abnormally high number of heart attacks due to chance variation, then the decline in the two following periods might be a figment of the data. To conclude that the decline is real the authors would have to look at multiple periods before the ban. At the very least they would have to go back beyond 18 months. Failing that, they should examine AMI rates in similarly sized towns to see how much random variance to reasonably expect. They don’t gather any of this data.

I don’t have that data either, but I do have the study’s own comparison of rates within the three geographic areas. Here’s the handy graph presenting them:

Pueblo smoking ban study

Notice anything? The first bar, representing the city of Pueblo’s pre-ban rate of AMI, is freakishly higher than all the others. Maybe that was normal for Pueblo. Maybe that time period was a particularly bad year for heart attacks. Based on the data in this study, we simply don’t know.

Look also at the changes between Phases I and II. Within the city the number of heart attacks falls from 291 to 237, a change of 18.6% that the study authors credit to the smoking ban. Within the county, heart attacks rise from 76 to 92, a change of 21%. If random variance in a control group is larger than variance in the sample, that’s a problem for the study. It suggests that the large effect touted by the researchers could plausibly be due at least in part to random variation. That’s why samples based on short time frames and small populations need to be taken with very big grains of salt. (The county sample is smaller than that of the city, but this much random variance should still be of concern.)

The only reason the authors can claim the Phase II decline is significant is because the pre-implementation rate of AMI in Pueblo is so high. But again, we have no idea if that rate is an outlier or not. Nor do we have a good explanation for why the initial decline continued. The CDC’s editorial states:

The continued decrease in AMI hospitalizations observed in this study might be a result of a combination of 1) the immediate reduction in SHS exposure among nonsmokers that occurred when the city of Pueblo smoke-free ordinance was implemented, 2) further reductions in this exposure that occurred because of increased compliance with the ordinance and increased adoption of smoke-free home rules over time, and 3) increased quitting among smokers as a result of the ordinance and associated changes in social norms.

That’s true, it might. But the study provides no evidence that non-compliance was ever a problem or that patterns of smoking at home changed. These are purely ad hoc explanations. The decline might just be random.

Notice too that the rate of heart attacks in the city is higher than the rate in the control groups even three years after the ban has been in effect. Does this mean that if Pueblo County and El Paso County implemented smoking bans (which, thanks to the state legislature, they have), then 3 years later their AMI rates would also fall by 41%? Is there something about the residents in the city of Pueblo that, all else equal, makes their hearts seize up at nearly twice the rate as residents of nearby areas? It seems more likely that those first 18 months in Pueblo just happened to be an outlier period.

To test the accuracy of the control group we can also look at state data as a whole, as Michael Siegel did in his similar critique of the original Pueblo study. The authors claim that their data show no existing trend toward declining heart attacks, but the much larger sample of statewide data suggests that a downward trend does exist. If their small control groups don’t pick up the statewide trend that we suspect is happening, then they need to consider the possibility that their control groups are too small. Similarly, they have to consider that part of the decline in Pueblo might be explained by existing trends.

We can also ask why we hear so much about health miracles in Pueblo, CO, Helena, MT and other small towns, but not in California, New York, Washington, or other large populations that have lived under comprehensive smoking bans for many years. If we could really expect smoking bans to reduce heart attacks by 41%, wouldn’t we have noticed the effect on millions of people by now?

None of this means that smoking bans don’t reduce the rate of heart attacks. Given what we know about tobacco and smoking behavior, it would be surprising if they didn’t. But, as they say in the diet plan commercials, Pueblo’s results may not be typical. This study does not show conclusively that Pueblo’s smoking ban is responsible for the drastic reductions, that the primary reduction in heart attacks accrued to non-smokers, or that other jurisdictions should expect similar declines. It is at best suggestive. Further studies based on larger populations are necessary to accurately gauge the effects and it appears so far that they are not nearly as large as implied here.

The study itself is not terrible. It’s not particularly good either. The real problem is its abuse by lazy reporters and ideologically driven anti-smoking organizations, which the researchers seem all too happy to abet. Given that the study’s conclusions will inevitably be used to restrict smokers’ and business owners’ rights, all of these groups have an obligation to act more responsibly.

Update 1/5/08: Michael Siegel is back from vacation and has posted his own critique of the study. We cover much of the same ground, but there’s one key finding I missed:

…this is exactly what this study does: it knowingly uses a comparison county in which it has been documented that the smoking prevalence over the study period has increased from 17.4% to 22.3%.

The study doesn’t try to hide this fact. It openly acknowledges that the reported smoking prevalence in El Paso County (the comparison group) increased from 17.4% in 2002-2003 to 22.3% in 2004-2005.

Given this finding, El Paso County simply cannot be used as a comparison population. You can’t take a population in which you know that smoking prevalence increased substantially and “pretend” that it represents a reasonable area in which to evaluate the baseline secular trends in heart attack admission rates that would have occurred in the smoking ban city in the absence of the smoking ban.

Siegel also notes that heart attack rates in Colorado and nationally dropped significantly during the study period, further casting doubt on the quality of the control groups. Whole thing here.

*Edited 1/4/09


4 thoughts on “Lazy reporting and the Pueblo ban study”

  1. I remember reading an article about 3rd hand smoke a couple years ago- I thought the issue died.

    I love the dramatic tone, though- ‘Third-hand smoke is what one smells when a smoker gets in an elevator after going outside for a cigarette… “Your nose isn’t lying,” he said. “The stuff is so toxic that your brain is telling you: ’Get away.’”

  2. A huge problem with this study is that the area studied is a relatively small one. It is very cherry-picked, which is all too typical of this type of study. When larger populations at the state level are studied there is no significant difference in heart attacks before and after smoking bans.

Comments are closed.