How To Tell If A Survey Is Credible: Minimum Standards

Determining whether a survey is credible and the data valid as presented is often a difficult task.

 

Unfortunately, many surveys lack credibility because they fail to meet standards for design, implementation and transparency.

 

Significantly, sales and marketing surveys that present deliberately slanted information designed to elevate the issuing company or to denigrate competitors are a violation of the California Business and Professional Code and legally actionable by governmental bodies or affected private entities.

Issue #1 Conflict of Interest

Does the organization conducting the survey have a conflict of interest? This is especially of issue if the issuing organization stands to benefit from the results. Marketing surveys done by the marketer should always be viewed with skepticism.

 

The most credible polls and surveys are conducted by independent third-parties not affiliated with a company, political, or other group with a vested financial or other self interest.  Those may be somewhat credible if their methodology is conducted to industry standards and fully transparent.

What to look for

Whenever a poll is published, reliable surveys have a link — such as the one below — which details survey methodology.

 

Poor or incomplete methodology

In addition, some third-party surveys are also poorly done and cannot be trusted because they are “pay for play” in which a survey is designed to produce pre-conceived results on behalf of a client.

 

Credible standards exist to assess a survey

 

Minimum standards for credibility : Source: American Association for Public Opinion Research: Survey Disclosure Checklist

• Disclosure of conflicts of interest,
• The number of subjects surveyed,
• How the subjects were selected,
• How the questions were asked,
• When the questions were asked,
• The medium of conducting the survey (online, in person, by phone etc.),
• The exact wording of all the questions,
• When and where the questions were asked.
• A statistically credible margin of error.

 

All of those affect the validity of the results and should be available whenever results are presented as valid.  Standards for assessing data validity (see below for links to expert sources) require that a study must (at a minimum) reveal:

 

If a study does not publish that information along with its results, there is no way to tell if it’s fake data or not.

Trustworthy survey: case in study

Properly done polls — such as this recent one from Gallup always — ALWAYS — include the following specific information. Polls which do not include this should not be trusted.

Survey Methods

“Results for this Gallup poll are based on telephone interviews conducted June 7-11, 2017, with a random sample of 1,009 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia. For results based on the total sample of national adults, the margin of sampling error is ±4 percentage points at the 95% confidence level. All reported margins of sampling error include computed design effects for weighting.

Each sample of national adults includes a minimum quota of 70% cellphone respondents and 30% landline respondents, with additional minimum quotas by time zone within region. Landline and cellular telephone numbers are selected using random-digit-dial methods.”

 

In addition, trustworthy surveys, will include details allowing a close examination for validity: Gallup: View survey methodology, complete question responses and trends.

 

Again, surveys that do not include that degree of detail are always suspect.

Self-selected participant surveys among condemned practices

 

This link — AAPOR Code of Ethics — is from the American Association for Public Opinion Research (AAPOR) which sets basic standards for surveys. Also: Condemned Survey Practices

Other sources that should be taken into account when examining the validity of poll data:

• Public opinion polls do not always report public opinion – Nieman Lab
• The New York Times’ Conformity to AAPOR Standards of Disclosure for the Reporting of Public Opinion Polls
• Beware sloppiness when reporting on surveys

MORE BAD SURVEYS

This smelly vendor survey lacked data integrity, so USA Today flushed their stinky article down the drain.

Beware of badly done surveys – This one about pot is just blowing smoke