In preparation for the BA’s annual mid-year release, I’ve been evaluating various data sources to fill in gaps in our survey collection. Because we typically receive fewer responses to our mid-year survey, it becomes more important to extrapolate from other data sources to get an accurate sense of the overall market.

State sources are the gold standard, but they have two problems: they are typically months behind, and they aren’t available for most states. Because of this, I’ve been evaluating when and where scan data might be used to estimate trends in the total market.

Both Julia and I have written recently about some of the ways that scan data may not capture overall small and independent brewer market trends, but just because a tool isn’t perfect doesn’t mean it isn’t useful. So, if we line up scan data against more verifiable data streams, how does it perform? NOTE: This piece was written using IRI Group scan data. Other data streams, like Nielsen’s all outlet data (which I don’t have access to) may produce different results.

The short version:

1. In aggregate, scan data do a very good job of measuring the overall growth trend of the set of breweries included
2. Scan data are not as strong an indicator of the growth of individual brewers
3. The breweries that scan data do not cover have a very different growth trend
4. Overall craft is growing faster than scan data and the on-premise is probably doing better than we think it is

Put differently, scan data appear to be very strong in measuring the collective set of brewers selling in off-premise. But the set of brewers not included appear to be different enough from the breweries in scan data that it be considered indicative of anything other than what it is–a picture of off-premise. In stats terms, this means that scan data may be a fairly unbiased estimator of the off-premise growth parameter, but the selection bias of the breweries in scan data creates a biased estimator of the overall craft growth parameter.

How did I measure this? I used two data sources: Our own mid-year survey and Oregon state data. Since we’re still finalizing mid-year survey results I won’t go too deep into that analysis other than to say that there was a very strong correlation between the growth rates brewers reported to us and those found in scan data. The one issue with overall trends was that scan data isn’t weighted properly if you’re trying to measure the overall market–it tends to overweight larger brewers and underweight smaller brewers. This makes sense since draught as a percentage of production tends to decrease as brewers grow.

The Oregon data provide an opportunity for a more transparent analysis. Right now, Oregon provides four months of 2016 data for sales of Oregon breweries in Oregon. I simply lined up the 2015 and 2016 numbers with IRI Group scan data on those companies sales in Oregon. In all 47 Oregon craft breweries had data in both the April report and had scan data available (scan data was YTD through 5/1/16. A few breweries had scan data but weren’t in April and many were in April but didn’t have scan). Here’s a summary of their growth:

• IRI data = 10.4%
• Oregon state data = 9.9%

That’s pretty darn good! The correlation between individual brewer growth rates in the two data streams was only r = .26 (which suggests a weak relationship, but still a lot of variation), but when aggregated (and so controlling for different business models, etc.) the scan data lines up very well with the state data.

The problem is that only 72 percent of Oregon craft brewer volume comes from breweries that show up in scans (methodological note: Oregon is a bit of an outlier, so that’s probably lower than in other states, more on this later). That’s not to say that scans include 72 percent of craft brewer volume–just that 72 percent of craft brewer volume comes from breweries you can find in scans.

That wouldn’t be an issue if the other 29 percent of volume came from breweries that look like the breweries that are included. But, they don’t look the same. They are smaller, clearly more on-premise (or own-premise) focused, and most importantly, growing much faster. The 28 percent that are not in scans are growing their Oregon sales by 18.3 percent. That means that overall growth for Oregon craft brewer sales in Oregon is 12.2 percent, nearly  two points higher than what scan data finds.

Equally important to the difference in growth rates is the difference in incremental volume. The Oregon brewers included in scan added just under 11,000 barrels in Oregon sales volume through April. The brewers not in scan added over 7,000. In all around 40 percent of the barrel growth comes from brewers not in scan data. These are the type of barrels the TTB keeps adding in its revisions.

So what does this all mean?

Taken as a whole, scan data appears to do a good job of measuring the aggregated experience of the top 25 percent of craft brewers (who represent much more in volume terms). However, there is strong evidence that the tail (and consequently the craft category as a whole) continues to be healthier than scan data has suggested. So a slowdown for regionals may be real, but craft’s overall slowdown is not as sharp.

Let me caveat that Oregon is a unique state, and so you might not find the same gap in other states. I’m more confident that the trends I’m describing are real since similar things showed up in our mid-year survey data. I’ll plan to break these trends down more once we’ve finished the mid-year numbers. Stay tuned for that release later this month!