DtC Report methodology still obscure: lack of disclosure makes current analysis untrustworthy & irrelevant for business decisions

Shipcompliant blasted out a mass email Feb. 13 that seemed to promise some degree of transparency for the data manipulation process: “The Highlights and Methodology of the 2019 Direct-to-Consumer Wine Shipping Report.”

 

Sadly, that email contained no actual methodology beyond that of its original announcement. As a result, this offered no more assurance on the relevance or  trustworthiness of the data than it did on  January 24, when Wine Industry Insight took a look at the Wines Vines Analytics data manipulation method and found that it fell far short of the the sort of transparent standards and disclosures needed to inspire confidence and relevance for business decisions.

 

There is no question that Shipcompliant’s basic data is accurate. However, the obscure process in trying to take that data and create an industry-wide picture cannot be trusted on its face and is irrelevant to wineries and other direct shippers because of an appalling lack of transparency and disclosure.

 

One major failing of the report is its irrelevance for business decision making. In addition to the shortcomings discussed below, the report offers no guidance on where the source of the millions of transactions analyzed:

 

 

• Do those transactions come mainly from very large players? In that case, the data could be irrelevant to small and medium-sized wineries.
• What kinds of shippers provided the data? Wineries? Multi-winery conglomerates? Online flash sales operations? Winery clubs? Other sorts of wine clubs such as those run by publications and retailers?

While there is no question that the Sovos data is valuable, in short, the report fails completely fails flat as a tool for assessing the health and progress of  DtC and as a business decision tool because of its lack of disclosure.

 

 

If most of the study’s transactions are based on huge players like Naked Wines, Lot18, or Constellation, then it lacks relevance and provides no guidance for a the vast majority of U.S. wineries, retailers, and wine clubs.

Wine Industry Deserves better

The wine business of a multi-billion dollar industry and needs statistics and data analysis that rises to a truly national stature. As one prominent wine industry player said of this study: “While I know what they are claiming, without support and disclosure, I don’t know how to interpret the information or how to make it actionable.”

 

 

This pdf — “Understanding IRI Household-Based and Store-Based Scanner Data” — offers one example of the sorts of  details and transparency that make the outcome of a data analysis trustworthy.

A closer look at the email blast

Here is a direct and complete quote from the Feb. 13 email blast (in italics) along with annotations.

How is the report created?

Wines Vines Analytics created a custom, proprietary algorithm

 

• Algorithms are frequently biased and incorrect. No one expects Wine Vines Analytics to reveal the trade secrets in their algorithm. but to be worth of trust, transparency demands that it reveal what (not how) it accomplishes its mission. I have created working algorithms and know it is not hard to explain the what without revealing the how.
  

 

• Significantly, both IRI and Nielsen are specific about where they get their data and where (and why) there may be gaps. Both have proprietary algorithms that help estimate the gaps where they have data missing. Both companies are up front about the gaps.
  

that uses its database of U.S. wineries.

 

• That database is for sale from multiple sources and is updated monthly by the TTB. This tells us nothing about why it is relevant to this study.

Their database comprises 9,997 wineries, some holding multiple licenses.

 

• There are 11,496 wineries in the United States. Simply revealing the number of wineries in the database is irrelevant. Further, the prominent mention of that number leaves the incorrect misperception that they have analyzed data from all of them.
  

 

• Wineries are not the only DtC sellers. Does this report cover only wineries or did it include non-winery, direct-sales companies such as flash sales organizations and non-winery wine clubs. This need for segmentation applies to all further annotations, below, where wineries are referenced.

 

• Information that would be relevant and useful would include:
  • How many wineries are represented in the data.
  • How many cases of wine  did each winery process through Shipcompliant?
  • What how many cases were shipped to which state?

This algorithm has been trusted by the industry for nearly a decade to provide the most accurate DTC data available.

 

• Without transparency, that industry trust is misplaced. The data may actually be valid. But a simple “trust me” falls short. For decades, people trusted those who told them that the Earth was flat. Trust is emotional. “Trust but verify” is a rational process requiring information, disclosure and transparency on which trust can be merited.
  

The algorithm extrapolates DtC shipments from all transactions

 

• How many transactions from how many wineries from how many states? By DtC seller type and size?
  

filtered through the ShipCompliant by Sovos system in a given year.

 

• Filtered through? How do you define this? Does that equate to actual transactions?

The ShipCompliant by Sovos platform processes nearly 80% of the volume of the DtC channel,

 

• On what credible estimation or data is that 80% number based? Is that re-estimated each year?
  

or roughly 8 million individual transactions.

 

• From how many wineries, to how many states, By DtC seller type and size?

Our report includes data for shipments to all 45 states that allowed some kind of shipping of wine in 2018.

 

• Cases per state? By DtC seller type and size?
  

These include even states like Arkansas and Rhode Island whose DtC regulations may not yet meet the ideal system that most other states have adopted.

 

• That’s interesting, but vague. Tell us why this is important.
  

For reference, here is the complete quote, cut and pasted without any changes

How is the report created?

Wines Vines Analytics created a custom, proprietary algorithm that uses its database of U.S. wineries. Their database comprises 9,997 wineries, some holding multiple licenses. This algorithm has been trusted by the industry for nearly a decade to provide the most accurate DTC data available.

 

 

The algorithm extrapolates DtC shipments from all transactions filtered through the ShipCompliant by Sovos system in a given year. The ShipCompliant by Sovos platform processes nearly 80% of the volume of the DtC channel, or roughly 8 million individual transactions. Our report includes data for shipments to all 45 states that allowed some kind of shipping of wine in 2018. These include even states like Arkansas and Rhode Island whose DtC regulations may not yet meet the ideal system that most other states have adopted.