Beware of badly done surveys – This one about pot is just blowing smoke

Badly done opinion polls and surveys are a common affliction, especially on the Internet.

 

This poll — Eaze Insights: The Modern Marijuana Consumer — conducted by a marijuana delivery company with an obvious self-interest in the survey’s results — purports to offer data that can be applied to a much wider marketplace.

Sadly, that poll piece violates many of the foundations of valid polling.

 

This is not to say that it may ultimately have some value. However, its lack of standards and disclosure places it in the category of junk data.

 

The issue it raises is an important one for the wine industry, which deserves to have trustworthy data on market changes. Sadly, this one fails the credibility test.  Why does this fail the trust test? What standards should have been adhered to? Read on.

What makes a trustworthy survey?

Properly done polls — such as this recent one from Gallup always — ALWAYS — include the following specific information. Polls which do not include this should not be trusted.

 

Survey Methods

“Results for this Gallup poll are based on telephone interviews conducted June 7-11, 2017, with a random sample of 1,009 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia. For results based on the total sample of national adults, the margin of sampling error is ±4 percentage points at the 95% confidence level. All reported margins of sampling error include computed design effects for weighting.

Each sample of national adults includes a minimum quota of 70% cellphone respondents and 30% landline respondents, with additional minimum quotas by time zone within region. Landline and cellular telephone numbers are selected using random-digit-dial methods.”

In addition, trustworthy surveys, will include details allowing a close examination for validity: Gallup: View survey methodology, complete question responses and trends.

 

Again, surveys that do not include that degree of detail are always suspect.

 

Pot Poll Based On Condemned Practices

Condemned Survey Practices — This link is from the American Association for Public Opinion Research (AAPOR) which sets basic standards for surveys.

 

The following link notes the following AAPOR-condemned practice that is the foundation of the Eaze poll:

 

“4. Representing the results of a self-selected “poll” as if they were the outcome of legitimate research.

 

Write-in, call-in, and interactive polls have become increasingly common. These “polls” report the opinions of only those people who called in, and not those of the general public. AAPOR believes that any publicizing or promotion of such activities not only damages legitimate market and survey research, but can be very misleading when used to influence public policy or simply to disseminate information about the general public.”

 

In addition, it is unknown whether the people surveyed are representative of all marijuana users or whether those surveyed are a small niche. Valid decisions cannot be extrapolated from unknown samples.

Other sources that should be taken into account when examining the validity of poll data:

• Public opinion polls do not always report public opinion – Nieman Lab
• The New York Times’ Conformity to AAPOR Standards of Disclosure for the Reporting of Public Opinion Polls