There is (and has been) a battle raging in New York State over the sale of wine in grocery stores. In his Table Hopping blog, Times Union writer Steve Barnes linked to an article that supposedly espouses some positive effects of selling wine in grocery stores. The study in question was conducted by the American Association of Wine Economists, so there could be a possibility of bias. Just maybe. In today’s post, we’re going to go through a little crash course (no pun intended) in research methodology to show why you should be wary of this study’s conclusions.
So first off, the goal of any research is to uncover how certain aspects of world work. What are the relationships between certain phenomena? In the case of this study, titled “Regulating the Availability of Beer, Wine, and Spirits in Grocery Stores: Beverage-Specific Effects on Prices, Consumption and Traffic Fatalities” it is looking at the relationship between the availability of alcoholic beverages and their prices, consumption and traffic fatalities. The abstract makes clear that the researchers (from Cornell nonetheless) believe their study demonstrates that higher shares of wine availability are correlated with lower traffic fatalities. As Steve Barnes’ post indicates, the implied effect of having wine in grocery stores means less traffic deaths. So imagine you’re a state considering whether or not to sell wine in grocery stores. You hear about this study, or maybe even read its abstract, and you see that the availability of wine in grocery stores may not increase social problems, but ACTUALLY DECREASE them? Selling wine in grocery stores is associated with lower traffic fatalities? Well sign us up!
It’s easy to read a lot of headlines about studies and be mislead. Most news outlets don’t actually conduct any due diligence on the studies they cite and report on. Usually they just read the abstract or conclusion section and post that as if it’s the gospel. But the key to any study is not its results, but how it got its results. And so here’s where we take a look at how the American Association of Wine Economists came to conclude that if you sell wine in grocery stores, you can decrease the amount of traffic fatalities.
Since I don’t want this post to be incredibly long, I’ll focus solely on the study’s traffic related conclusion. I think the best way to go about looking at this study’s methods is to brainstorm about how you might go about studying the relationship between wine availability in grocery stores and traffic fatalities. Here we break down our independent and dependent variables. Independent variables are the phenomena that have an effect on our dependent or outcome variables. We can manipulate the independent variables to see how they would change the outcome. Our variables depend on our research question.
Research Question: What are the effects of wine availability in grocery stores (an alcohol policy) on traffic fatalities (a social problem)?
Independent Variable: To find our independent variable, we need to find some way to measure wine availability in grocery stores. Now, these policies are state determined and since the policy in question for New York State is at the state level, we should use state-level data. So our state level variable would be whether a state has wine available in grocery stores or not. It’s a binary variable, yes or no. My beef with the AAWE study is not with its independent variables. They use state level data on wine availability.
Dependent Variable: Now here is where it gets tricky. We want to measure traffic fatalities by state, since we’re using state level independent variables. But we simply don’t want to use any traffic fatalities, but rather, ALCOHOL RELATED traffic fatalities. Why? Because it doesn’t make sense to think that the availability of wine in grocery stores would have any effect on non-alcohol traffic deaths right? People don’t drive to the grocery store, buy wine and then crash into other people and kill them because they bought a bottle of wine. They would crash into other people and kill them perhaps if they had been drinking the wine, therefore making it an alcohol-related death. So are dependent variable would be a measure of alcohol related traffic fatalities by state. This data is available through NHTSA. In fact, the AAWE study DOES get its data from the NHTSA but they don’t use alcohol-related fatality data in their study. That’s the first reason why their study isn’t convincing. Who cares about the effect of wine in grocery stores on all traffic deaths? It doesn’t make any sense to think that there would/should be an effect.
BUT the study says there is an effect? So doesn’t that mean something? If the sales of wine in grocery stores has an effect on all traffic fatalities, then isn’t that still a good thing? Well, it would be, but only if we could say that the wine availability was the reason traffic fatalities decreased. So what we’re talking about here, are control variables.
What are control variables? In the criminal justice field, we use the following example to explain them. Homicides go up as ice cream sales go up. Does that mean ice cream sales cause murders? Of course not. It’s simply that ice cream sales go up in the summer and there are more homicides in the summer. So the original relationship was said to be spurious, as in there was something else accounting for the relationship, a third variable. Or maybe more. The purpose of a control variable is to put these “third variables” into the model that might have an effect on our dependent variable and control for their effect. If we control for all other possible influences on the dependent variable, then we can say our independent variable did have an effect on the outcome.
The AAWE study does insert some control variables into the study, but not the variables necessary to convince me that they’re really isolated the effect of wine availability. For example, what could have an effect on traffic fatalities? Well, I can think of a few things. The number of cars on the road, the number of young drivers, the number of older drivers, weather conditions, road quality, road lighting, speed limits, you get the picture. Did the AAWE study control of any of this? The closest they come is a variable measuring how many miles a vehicle has traveled per driver. That gets us an imperfect (to say the least) measure of vehicle use but not much else. So in that respect, the AAWE fails to control for spuriousness as there are MANY other explanations for low traffic fatality rates.
So, in conclusion, what did we learn? The AAWE study does a poor job of measuring and operationalizing its variables. It doesn’t control for enough other factors so we can’t be sure that they are really isolating the effect of wine availability in grocery stores. Finally, even if they did isolate an effect, their conclusions are hard to believe because they didn’t measure the social problem of real concern, which is alcohol related traffic fatalities. One might argue that alcohol related fatalities are included in overall fatalities, which is true, but measuring the total fatalities includes too many other types of fatalities to make any meaningful contribution. It would be like analyzing the effect of gun laws on all types of crime. Sure, there would be measurements of gun crimes aggregated into the dependent variable, but it doesn’t make sense because it would include non-gun related crimes such as money laundering, drunk driving, child pornography possession, crimes that had nothing to do with guns. There’s too much other stuff clogging up the data.
My opinion on wine in grocery stores remains the same. Allow wine to be sold in grocery stores. However, the AAWE study did nothing to influence my opinion and frankly, anyone advocating for wine in grocery stores that uses this study in their argument, is doing a disservice to their cause. If you want to read the study, you can find it in PDF format here.