FREE! Subscribe to News Fetch, THE daily wine industry briefing - Click Here


Sponsored by:
Banner_Xpur_160x600---Wine-Industry-Insight[63]
InnoVint_WII_ad_portrait

Rating The Rating Systems

August 20, 2014 | Filed under Worth A Look | Posted by

This is part of a series on flaws in current wine recommendation systems originally published at Recommendation Insights:

For much more on recommendation systems, please visit: Recommendation Insights

How you ask a question can determine the answer you get.

The concept lies at the heart of journalists, scientists, those who interview job candidates, police detectives, opinion pollsters … and wine ratings.

In many settings — especially those where the vino-cognoscenti gather — the question, “What did you think of that wine?,” will frequently elicit a number between 70 and 100.

But that number, even if everyone were calibrated and always consistent, does not answer something even more basic to the average wine consumer, “would you buy it again.”

The gulf between the two question types is even wider than the 100-point scale’s credibility and consistency gap.

THE IRRELEVANCY OF THE 100-POINT SCALE

Numerous scholarly article as well as those in the general media have pointed out that, even if the 100-point scale were a good numerical system, the actual ratings are so inconsistent among the best experts that the numbers expounded are mostly worthless.

The best round-up of these flaws was recently published in The Guardian: “Wine-tasting: it’s junk science” which notes that, “Every year Robert Hodgson selects the finest wines from his small California winery and puts them into competitions around the state.

“And in most years, the results are surprisingly inconsistent: some whites rated as gold medallists in one contest do badly in another. Reds adored by some panels are dismissed by others. Over the decades Hodgson, a softly spoken retired oceanographer, became curious. Judging wines is by its nature subjective, but the awards appeared to be handed out at random.”

Hodgson’s most recent study (as are his oprevious ones) was published in the Cambridge University Press’s scholarly publication, Journal of Wine Economics An Examination of Judge Reliability at a major U.S. Wine Competition

100 POINTS FOR  PSYCHOLOGICAL BAGGAGE.

The 100-point wine-rating scale also carries psychological baggage from school where below 70 is an F and 100 is an A+.

Those rating/grade associations solidify the false perception that a rating is an objective, unassailable judgement on quality.

Beyond that, what is the “meaning” of an 83 versus 85 or 87? Is the “meaning” biased by personal expectations? Grades in school or performance at work?? What a wine “deserves?”

ALTERNATIVES TO 100 POINTS HAVE THEIR OWN PROBLEMS

Alternative scales — the substitution of 10 or 5 point scales as well as stars and other icons — have found widespread acceptance in wine as well as in other consumer products. But they also have psychological biases, possibilities for misinterpretation (“what does it mean?”) .

Research shows that when confronted with an odd-numbered scale, respondents tend to cluster around the neutral point with a bias to the positive. This reflects potential anxiety over extreme positions and a tendency to avoid those.

EVEN VERSUS ODD  —  DECISION VERSUS INDECISION

Odd-numbered scales allow for the expression of neutral opinions because if offers a middle point — neither negative or positive — that amounts to “undecided.”

Even-numbered scales force a choice into negative or positive territory because they have no “undecided” position.

Offering neutral or “undecided” positions serves a purpose in political polls and other surveys. But wishy-washy has little value in the market place where the decision is “buy or not buy?”

Thus, scales with an even number of choices provide the best, most valuable actionable data in a purchase decision environment.