“Drinking Highest Among Educated, Upper-Income Americans” — Can you trust this (or any other poll)?

“Drinking Highest Among Educated, Upper-Income Americans” — Can you trust this (or any other poll)?

Multiple polls and surveys hit the Internet and the media every day. Some come from professional polling companies like Gallup whose only business is polling. Gallup, like Pew and Harris have become trusted names with reliable data that have proved themselves over time.

 

Looking at those companies offers valuable lessons to consider when deciding whether to trust the polls and surveys offered by various companies, trade groups and other organizations.. How do you know which ones to trust? Sadly, many of those polls fall short from  a lack of standards, transparency and disclosure. That means many of those polls and surveys are simply junk data.

 

Read on to learn what you should look for in order to determine whether you should trust a poll or survey.

What makes a trustworthy survey?

Drinking Highest Among Educated, Upper-Income Americans

 

Properly done polls — such as this recent one from Gallup always — ALWAYS — include the following specific disclosure information.

Polls which do not include this should not be trusted.

Survey Methods

“Results for this Gallup poll are based on telephone interviews conducted July 8-12, 2015, with a random sample of 1,009 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia. For results based on the total sample of national adults, the margin of sampling error is ±4 percentage points at the 95% confidence level.

“For results based on the total sample of 664 drinkers, the margin of sampling error is ±4 percentage points at the 95% confidence level.

“All reported margins of sampling error include computed design effects for weighting.

“Each sample of national adults includes a minimum quota of 50% cellphone respondents and 50% landline respondents, with additional minimum quotas by time zone within region. Landline and cellular telephone numbers are selected using random-digit-dial methods.”

Learn more about how Gallup Poll Social Series works.

As the Gallup poll noted,  a trustworthy survey will disclose:

1. The date of the poll
2. The method of polling
3. The protocol for selecting a sample representative of a larger population
4. The number polled
5. A validly derived sampling error estimate and a confidence level.

 

In addition, trustworthy surveys, will include details allowing a close examination for validity: Gallup: View survey methodology, complete question responses and trends.

 

Again, surveys that do not include that degree of detail are always suspect.

Common Condemned Practices

Condemned Survey Practices — This link is from the American Association for Public Opinion Research (AAPOR) which sets basic standards for surveys.

The following link notes the following AAPOR-condemned practice that is the foundation of a junk survey.

 

Offering products or services for sale, or using participant contacts as a means of generating sales leads.
“A common practice is to gain entry or acceptance in order to make a sales pitch by initially defining the contact as being made for “research” purposes. This trades on the prestige of science, and it exploits the willingness of the public to reveal information about themselves in the public interest. In some cases, questions establish respondents’ susceptibility to sales pressure or their interest in some product or service. Follow-up contacts are then made to those so identified, all under the guise of “research” (sugging).”

 

Representing the results of a self-selected “poll” as if they were the outcome of legitimate research.

Write-in, call-in, and interactive polls have become increasingly common. These “polls” report the opinions of only those people who were motivated enough to fill out a form, and are frequently not representative of a specific population or industry segment.

 

“AAPOR believes that any publicizing or promotion of such activities not only damages legitimate market and survey research, but can be very misleading when used to influence public policy or simply to disseminate information about the general public.”

 

Valid results cannot be extrapolated from unknown samples.

Other sources that should be taken into account when examining the validity of poll data:

• Public opinion polls do not always report public opinion – Nieman Lab