The story begins with a campaign called “Feed me Better” launched by popular English chef Jamie Olivier back in 2004 in order to reduce obesity rates among school kids. The campaign is aimed at banning junk food from school canteens “and get the kids to eat fresh, tasty, nutritious food instead.” (see Jamie Oliver’s Manifesto). Oliver envisions a ten-year plan that gets all participants involved, school officials, teachers, parents, pupils and "dinner ladies." With public support the program was implemented in the
Here are some typical school food examples:
Before the change:
Mains: burgers and chips; sausage rolls; fish fingers; turkey drummers; chicken dinosaurs
Desserts: sponge pudding and custard; milk shake and home made biscuit; fruit salad
After the change
Main findings: “Our estimates show that the campaign increased the percentage of pupils reaching level 4 by 4.5 percentage points in English, and the percentage of pupils reaching level 5 by 6 percentage points in Science. We also find that authorised absences (which are likely to be linked to sickness) drop by 15% on average. These effects are particularly noteworthy since they only capture direct and relatively short-term effects of improvement in children’s diet on educational achievements.”
Does this sound too good to be true? In their paper the authors discuss potential shortcomings that theoretically could have biased their results.
(1) Self selection: “Mobility across LEAs [Local Education Authorities] could introduce a selection problem and bias our estimates, for example if those children who move towards healthier schools are relatively better pupils in terms of educational performance and presence at school.Unfortunately, we do not have data on the number of applications to primary schools, but one indication of possible selection effects is the mean IDACI score – the mean socio-economic index. Figure 2 [see the original Working Paper] shows the IDACI score remaining constant over the analysis period suggest that the composition of the households of
(2) Placebo effect: One concern is that the campaign affected educational outcomes not through the improvement in diet, but simply through a“placebo-effect”. Indeed, the schools were very well aware they were part of a pilot experiment and the campaign received a lot of media attention. Thus, we should worry that the effect we measure is a placebo effect rather than an actual effect of the campaign.”
However “On the other hand, the [media] attention was very much focused on the health benefits, and in particular on tackling the problem of obesity, rather than improving school performance. Also, we are looking at outcomes more than a year after the campaign and have excluded the year of the campaign itself. It is hard to believe that schoolchildren would remain motivated by a placebo effect more than a year after the campaign has been implemented.”
First, despite Mr. Matthews claim in one of his comments, I do not "convict" the Wine Spectator of anything. As I clearly state in the abstract, "I find that advertisers earn just less than one point higher Wine Spectator ratings than non-advertisers when I use Wine Advocate ratings to adjust for differences in quality. However, I find only weak evidence that the selective retasting of advertisers’ wines contributes to the higher ratings. Moreover, conditional on published ratings, Wine Spectator is no more likely to bestow awards upon advertisers. I conclude that while advertising may influence ratings on the margin, Wine Spectator appears largely to insulate reviewers from the influence of advertisers." In other words, despite the statistical evidence of a difference in ratings, based on Wine Spectator's use of blind tastings, and the preponderance of my empirical evidence, I conclude that the level of pro-advertiser bias is small to none. Given the conspiracy theories I’ve heard from fellow wine lovers, I expected Wine Spectator to be less defensive about my findings.
Second, to reiterate something Karl Storchmann wrote, whether advertisers earn the same ratings, on average, as non-advertisers tells us little about whether advertisers earn higher ratings from Wine Spectator (WS) than they deserve given the quality of their wines. To test for pro-advertiser bias, one needs to control for quality. My tests for differences in ratings assume that Wine Advocate (WA) ratings are noisy, but unbiased measures of quality. Given this assumption, I find that WS ratings are approximately one point higher for advertisers than I would have predicted given the published (and missing) WA ratings. To be clear, within the sample of wines rated by both WA and WS, the difference is less than half a point. However, the fraction of advertisers wines for which WA does not publish a rating is slightly higher than I would predict given the published WS ratings. Using a variant of a well-worn statistical technique, I find that removing one point from the ratings of advertisers' wines essentially explains away the difference in the fraction of advertisers wines for which WA does not publish a rating. I freely acknowledge that if I had access to unpublished WA ratings, I would have been able to conduct more powerful statistical tests, using a less-opaque-to-non-academics method. This is not say, however, that my empirical analysis is flawed.
Third, Mr. Matthews is correct to point out that the "just less than one point" difference can be interpreted in a variety of ways. I explicitly acknowledge this point in the paper when I consider the possibility that WS ratings reflect the tastes of WS reviews, WA ratings reflect the tastes of WA reviewers, and wineries are more likely to advertise in WS if they makes wines that appeal to WS reviewers. There is nothing sinister about this alternative interpretation, except that it calls into question, for example, the idea that Robert Parker and James Laube rate California wines using the same criteria.
Fourth, Mr. Matthews raises the possibility that WA is biased against the wines of WS advertisers. In particular, Mr. Matthews conjectures that WA ratings, because they are not always the result of blind tastings, may be more highly correlated with price than are WS ratings. This is a testable hypothesis. However, within the set of wines rated by both WA and WS, I find the correlation between ratings and price to be quite similar across the two publications. (The correlation between WA rating and the price reported in WS is 0.5461, the correlation between WS rating and the price reported in WS is 0.5710, and the difference between these correlations is neither statistically nor economically significant.) Another possibility, I suppose, is that some advertisers submit better-than-average bottles for review by Wine Spectator. But I have no idea if this is even feasible.
Finally, Matthews writes, "Perhaps Mr. Reuter is simply crunching the numbers beyond their breaking point." The fact that wines are rated 80, 81, 82, etc. does not rule out the possibility that some types of wines are consistently rated 1.0, 0.5, or 0.1 points higher than other types of wines. While it is true that small sample sizes generate noisy estimates, it is not true that they prevent valid statistical inference. Of course, if Wine Spectator would like to provide me with more comprehensive data on ratings, tasting notes (so that I can determine which wines were retasted), awards, composition of the WS Top 100, and advertising expenditures, I'm happy to gather other ratings and crunch the numbers again. Heck, I'm even happy to approach Wine Advocate about getting access to its published and unpublished ratings. Mr. Matthews' has my email address.
A Response from Wine Spectator
Wine Spectator states categorically that our reviews are not biased in favor of advertisers. In fact, our advertisers frequently complain about their ratings.And most of the highest-rated wineries have no budget for advertising. It’s a no-win world!
At Wine Spectator, every review of a newly-released wine is the result of a blind tasting, where neither producer nor price is known by the taster. This approach gives every wine a fair and equal chance to show its best, and guarantees that no bias can influence the scores. No other wine publication makes this claim.
Jonathan Reuter’s paper analyzes 713
Here are his comparisons (including average score):
WS advertisers = 87.50/ WS non-advertisers = 87.58
WA advertisers = 88.15/ WA non-advertisers = 88.65
I would like to make two points about this data. First, advertised wines actually received lower scores from Wine Spectator than non-advertised wines.Second, advertised wines actually received higher scores from Wine Advocate than from Wine Spectator.
Nonetheless, Mr. Reuter asserts that Wine Spectator scores are biased. However, this conclusion depends on his assumption that Wine Advocate scores are not biased. But since he has no proof of the latter, why should anyone accept his claim of the former?
A better explanation of the disparity in scores may be that Wine Advocate is biased in favor of higher-priced wines. (Since WA does not always taste blind,it is possible that the taster knows the price of the bottle when it is scored.)
I suspect this bias may exist based on the data Mr. Reuter presents relating price and score in both publications. For all wines, with an average price of $36.93 per bottle, the WA average score is 1.00 point higher than the WS average score. For wines that average $37.16 per bottle(non-advertisers), the WA average score is actually 1.07 points higher than the WS average score. In other words, a higher price results in a greater increase in score from WA than it does from WS.
Or, it may be thatneither publication’s scores are biased. After all, Reuter’s analysis dependson calculating ratings to two decimal places. While Wine Spectator believesthat the 100-point scale can accurately reflect the judgments of an informedtaster, we don’t insist scores are valid to the hundredth of a point. PerhapsMr. Reuter is simply crunching the numbers beyond their breaking point. (added by Thomas Matthews after original posting)
In the end, Mr. Reuter comes to the same basic conclusion that Wine Angels did. He writes, “I conclude that while advertising may influence ratings on the margin, Wine Spectator appears largely to insulate reviewers from the influence of advertisers."
In the real world, wine lovers will make their own determinations whether Wine Spectator, or any other critic, has the expertise and integrity to be a credible guide to the complex and fascinating world of wine.
Founded in 1976, Wine Spectator has become the most widely-read wine publication in the world, according to independent research, with more than 2.4 million readers in the
The expertise is embodied in a team of senior editors who have been writing about wine for a cumulative total of more than 150 years. The integrity rests on our Code of Ethics and our tasting methodology (both posted in detail on our Website, Wine Spectator.com). At Wine Spectator, we will continue to uphold the highest standards of truth and fairness, and to work diligently to avoid any bias or conflicts of interest. That is our pledge to our readers, and to ourselves.
Respectfully,
Executive editor
Wine Spectator
Are Wine Spectator points biased towards wineries that advertise with them?
This is the topic of a paper by Jonathan Reuter of
There are two questions here:
First, are Wine Spectator reviews biased? Second, who cares?
Let us begin with the latter question.
There are numerous economic papers that confirm the influence of critical points on the wine price. Wineries want to receive awards, medals or high critical ratings because they boost their reputation and will most likely allow to raise prices in the years to follow.
What is the money value of a Wine Spectator point? Or better, what is the value of getting one more point? First, the effect is probably not linear. An increase from 90 to 91points may be worth more than a jump from 80 to 81. Good, then we can simply regress a wine’s price on last year’s ratings using a non-linear specification or use different point brackets. That is what Jonathan Reuter did in his PhD thesis (this part is not published in the JWE). Result: an increase from 90 to 91 Wine Spectator points will increase next year’s bottle price by up to $5.84 (in $2000). But things are more complicated. A wine that is highly rated by Wine Spectator is probably also highly rated by other wine critics, e.g., by Robert Parker in his Wine Advocate.That is, simply regressing a wine’s price on last year’s Wine Spectator rating yields a biased price effect. Not the entire price effect is due to Wine Spectator alone. Therefore, Reuter controls for the wine’s quality by including Wine Advocate’s ratings. As a result, Wine Spectator’s price effect dropped to $3.45. But still, if a winery produces 5,000 cases of a 90-point wine, one more point would be worth about $200,000 in future revenues. Thus, one Wine Spectator point more or less can make a big difference.

Back to the first question. Is Wine Spectator biased towards its advertisers? In his analysis, Reuter exploits the fact that of the two major wine publications, namely Wine Spectator and Wine Advocate, only Wine Spectator accepts advertising. In fact, within his sample, about 10% of all wines reviewed by Wine Spectator advertise in it. In contrast, Wine Advocate does not accept advertisements and is entirely subscriber-supported.
In a first comparison, Reuter compares the average Wine Spectator points of advertisers and non-advertisers and finds that advertisers earn even slightly less points. Of course, if one does not control for quality that is meaningless. However, when including Wine Advocate ratings as quality control the picture changes. “Controlling for (censored) Wine Advocate ratings, I find a positive (partial) correlation between Wine Spectator ratings and lagged advertising intensity. The implication is that Wine Spectator ratings of advertisers’ wines are approximately one point higher than their ratings of comparable wines from non-advertisers.”
If Wine Spectator tastings are blind (and we do not doubt they are), why should the ratings be biased towards advertisers? Reuter examines several possibilities.
Possibility 1: Wine Advocate’s ratings are not unbiased.
Theoretically, this could have lead to Reuter’s results. But then Wine Advocate must exhibit a systematical bias against wineries that advertise in Wine Spectator. That seems to be far fetched.
Possibility 2: “Wine Spectator style” wines advertise in Wine Spectator because their wines are most likely to appeal to Wine Spectator readers.
Wineries that advertise in Wine Spectator tend to be large-production wineries. “ …while Wine Spectator may have a preference for reviewing large-production wines, it is not clear that large wineries should be more likely to produce wines whose qualities appeal to Wine Spectator’s reviewers.”
Possibility 3: Biased reviews could arise through selective retastings. As Wine Spectator states “We retaste all wines that score 70 points or less. We retaste many other wines to confirm impressions.” First, Reuter finds evidence that Wine Spectator is more likely to retaste an advertiser’s wine. Second, when controlling for retastings, Reuter finds that advertisers earn significantly higher ratings than non-advertisers. However, the number of retasted wines appears too small to explain a bias of one entire point.
What is the conclusion? “At worst, the tests for biased ratings suggest that Wine Spectator rates wines from advertisers almost one point higher than wines from non-advertisers. However, selective retastings can explain at most half of this bias…The remaining difference in ratings may simply reflect consistent differences in how the two publications [Wine Spectator and Wine Advocate, sic.] rate quality.” That is, a mix of Possibility 2 and Possibility 3.
Let us return to the 5,000 case wine. If selective retastings bias reviews by (at most) half a point --- that’s still (at most) a revenue boost of $100,000; probably more than the cost of advertising.
Reuter touches another question without analyzing it. Are advertisers more likely to be reviewed at all? For now, we can only speculate. But somebody will look into that. .....There are always plenty of research questions left.
AAWE Working Paper No. 51 Economics
Expert Opinion and
Cuisine Reputation in the Market for Restaurant Meals
James J.
Fogarty
*
AAWE Working Paper No. 52 Business
From Wine
Production to Wine Tourism Experience: the Case of Italy
Vincenzo Asero
and Sebastiano Patti
For more free AAWE Working Papers check http://wine-economics.org/workingpapers/
For more free AAWE Working Papers check http://wine-economics.org/workingpapers/
Research published in the Journal of Wine Economics is featured in an article by Leonard Mlodinow in the 14 November 2009 Wall Street Journal. The first few paragraphs of the story are reprinted below. Click on this link to read the entire article.
Excerpt from the WSJ article:
Acting on an informant's tip, in June 1973, French tax inspectorsbarged into the offices of the 155-year-old Cruse et Fils Frères wineshippers. Eighteen men were eventually prosecuted by the Frenchgovernment, accused, among other things, of passing off humble winesfrom the Languedoc region as the noble and five-times-as-costly wine ofBordeaux. During the trial it came out that the Bordeaux wine merchantsregularly defrauded foreigners. One vat of wine considered extremelyinferior, for example, was labeled "Salable as Beaujolais toAmericans."
It was in this climate that in the 1970s a lawyer-turned-wine-criticnamed Robert M. Parker Jr. decided to aid consumers by assigning winesa grade on a 100-point scale. Today, critics like Mr. Parker exertenormous influence. The medals won at the 29 major U.S. winecompetitions medals are considered so influential that wineries spendwell over $1 million each year in entry fees. According to a 2001 studyof Bordeaux wines, a one-point bump in Robert Parker's wine ratingsaverages equates to a 7% increase in price, and the price differencecan be much greater at the high end.
Given the high price of wine and the enormous number of choices, asystem in which industry experts comb through the forest of wines,judge them, and offer consumers the meaningful shortcut of medals andratings makes sense.
But what if the successive judgments of the same wine, by the samewine expert, vary so widely that the ratings and medals on which winesbase their reputations are merely a powerful illusion? That is theconclusion reached in two recent papers in the Journal of WineEconomics.
Both articles were authored by the same man, a unique blend ofwinemaker, scientist and statistician. The unlikely revolutionary is asoft-spoken fellow named Robert Hodgson, a retired professor who taughtstatistics at Humboldt State University. Since 1976, Mr. Hodgson hasalso been the proprietor of Fieldbrook Winery, a small operation thatputs out about 10 wines each year, selling 1,500 cases
A few years ago, Mr. Hodgson began wondering how wines, such as hisown, can win a gold medal at one competition, and "end up in thepooper" at others. He decided to take a course in wine judging, and metG.M "Pooch" Pucilowski, chief judge at the California State Fair winecompetition, North America's oldest and most prestigious. Mr. Hodgsonjoined the Wine Competition's advisory board, and eventually "begged"to run a controlled scientific study of the tastings, conducted in thesame manner as the real-world tastings. The board agreed, but expectedthe results to be kept confidential. ...
Cheap and Nasty
I’m talking about box wines or bag-in-box (theAustralians call them cask wines) that feature an airtight wine-filledplastic bladder inside a cardboard box. You use a built-in spigot toget to the wine. They can be found on the bottom shelf of the wine walland behind the bar and out of sight at your local restaurant. They comein several sizes — 3 liter and 5 liter containers are the most common.
Box wines have a bad reputation. They first appeared in the 1970sand were filled with generic bulk wines. They were one step down fromthe popular 1.5 liter “magnum” bottles of “Burgundy,” “Chabils” andthe notorious “Rhine” wine. Box wine was cheap, nasty stuff thatacquired a frequently deserved bad reputation.
[Re]-Thinking Inside the Box
It’s time to reconsider box wine. Screw caps had a bad reputation,too, until quite recently. We associated them with low grade swilluntil fine wines appeared under screw cap (the New Zealand producerswere in the vanguard) and we began to appreciate that that screw capshave many advantages. Now screw caps are actually associated with qualityfor some types of wine, especially youthful whites, and no one expectsto pay less or get less because of the screw-top closure.
The technology of bo
x wine is very solid. The airtight bladder is aneutral container that is well suited to holding wine for relativelyshort periods of time. (Don’t cellar box wine — consume within a yearof production — check out the “drink by” date on the box.) The bladderand spigot do in fact protect the wine from oxygen in the short run, soit will last longer once opened (especially if the box is stored in thefridge) than similar leftover wine in bottles.
Bladdersare so good at the particular thing that they do that they have becomean industry standard technology for bulk imported wines, which areshipped in huge bladders inside steel shipping containers (big bag in big box) and then bottled in the import market. So you may already be drinking box wine and not know it.
The Box Also Rises
The most recent Nielsen retail wine sales figures (reported in the October 2009 issues of Wine Business Monthly)suggest that box wine sales are growing. Wine sold in 3, 4 and 5 litercontainers (most of it is box wine, I think) accounts for just under 10percent of US supermarket wine sales, according to the Nielsen data(compared to 65% for standard bottles with the remainder in 1.5 literand other formats). Sales are rising in this category, with 3 literpackages up 8.7% in the last year on a dollar basis, for example, and 5liter packages are up 9.3% by value.
The total market for box wines rises if we include on-premisessales. Recent data (see previous posts) indicate that box wines (servedto customers in carafes and by the glass) are strong sellers in casualdining establishments.
The rise of box wine is part of the trading down effect, clearly,since most box wines fall into the two price categories that areexperiencing the highest growth. Sales of wines that are less than $3per 750ml bottle equivalent have risen 7.1 percent according to Nielsenand by 10% for wines between $3 and $5.99. Supermarket sales of $20+wines, on the other hand, have fallen by 3.4%.
Nasty, Brutish and Short?
Does this mean that Americans have traded down all the way to thebottom, back to the nasty box wines of the 1970s? The answer,incredibly, is no. Or at least not necessarily, according to theOctober 15 issue of Wine Spectator. You can’t miss this issueon the newsstand — it features a cover story on “500 Values for $20 orLess” and includes a set of box wine reviews that make interestingreading.
Wine Spectator purchased 39 box wines in packages thatranged from 1 liter to 5 liters. Twenty seven wines were rated as“good” (a score of 80-84) and ten “very good” (85-89). The names of the2 wines that scored below 80 were not reported.
The top box wine, going by the rating numbers, is a white: Wine CubeCalifornia Chardonnay, which sells in Target Stores for $17 per 3 literbox, which is $4.25 per standard bottle equivalent. It earned a veryrespectable 88 points. Wine Cube is a partnership between Target and Trinchero, the maker of a wide range of wines including Sutter Home.
The best red wine (at 87 points) is the Black Box Cabernet SauvignonPaso Robles 2006, which costs $20 for 3 liters or $5 per standardbottle equivalent. Black Box is a widely distributed ConstellationBrands product.
Good and Cheap?
Some box wine, apparently, is both pretty good and pretty cheap. Perhaps just to show that they really do rate wines blind, Wine Spectator gavea pretty good 84-point score to a non-vintage Carlo Rossi CabernetSauvignon California “Reserve” wine. Five liters for $13, in case youare interested, That’s $1.97 per standard bottle equivalent.
How can decent wine be this cheap? One answer, of course, is thatyou can choose to make the wine itself less expensive by economizing inthe cellar in many ways (less oak or none at all for red wines, forexample). But to a considerable degree the box itself is responsiblefor the savings.
The bag in box container costs less than $1, according to the Wine Spectator article,which automatically saves $4 to $8 compared with a similar quantity ofwine in standard glass bottles and the box they come in. Shipping costsare also less since the boxes weigh much less than glass bottles forthe same quantity of wine and are less likely to be damaged intransit. There are environmental benefits too, especially in areaswhere glass bottle recycling is problematic because the sour economyhas undermined the market for recycled glass.
Is box wine the future of wine? No. The wine market is too complexto be dominated by any single trend. But with better wine in betterboxes (and with consumers embracing a more relaxed idea of wine) boxwine deserves to play a bigger role in the future of wine. Anothertriumph for The Box!
In my field, we have something called the National Magazine Awards.Magazine writers tend to be both obsessed with who wins and convincedthe process is a pathetic joke. This isn’t just sour grapes, either.The last time The New Republic won a National Magazine Award,it was for publishing Betsy McCaughey’s infamous anti-Clintoncarescreed “No Exit,” which is probably the worst article in the history ofTNR. It’s as if the last American to win the Nobel Peace Prize wasTimothy McVeigh.
Are these cases unusually egregious? Perhaps. But they are notwildly out of character with how awards generally work. A recentstatistical analysis by Robert T. Hodgson, published in the Journal of Wine Economics(I kid you not), found that a wine that wins one competition is no morelikely to win another competition than any other wine. Which is to say,wine awards are handed out completely at random. If you listen to moviebuffs, they will tell you that the Academy Awards regularly commitunforgiveable sins of commission or omission. Look closely at any fieldthat gives out awards, and you will probably find that injustice ismore the rule than the exception.