Sometimes I wonder whether XKCD is just better at commentary than all of us. Today’s cartoon demonstrates a great understanding of equilibrium combined with a real issue with regard to consumer incentives. It raises the puzzle that if you like, say, a hotel you have an interest in not saying so because that either reduces vacancies or increases price. You have a similar issue with respect to negative reviews as you would like to steer other consumers there and away from potentially better hotels you would want to stay at. At the very least, one wonders why you would bother to write a review at all. Also, note that ratings of reviewers by other users doesn’t help. If you read a negative review and it turns out the hotel was good, you don’t want to rate the first reviewer poorly because they will help you steer customers away from the gem you have discovered.
Ratings appear to matter for these things — as I wrote about the restaurant evidence previously here. So if reviews were biased this could set in motion a poor equilibrium outcome as xkcd suggests. That said, if the system were explicitly broken, consumers would realise it. Ratings just wouldn’t matter well before the bad drove out the good.
But there is a more subtle bias I have wondered about; one that arises if we supposed consumers only wrote a review once. Suppose they write it honestly. If they have a bad experience, they write a bad review and move on to another hotel. If they have a good experience, they write a good review and return to the hotel. But they don’t review it again. That means that as we count up ratings and average over number of reviews rather than number of hotel stays, then rating averages will be biased downwards. That is, repeat customers soak up some capacity and one consequence of that is that they reduce the probability of an additional review. Those very popular hotels who are always at capacity would be hit by this bias more than unpopular ones and so the variance of average rating scores will be less than it should be. I don’t think this will break the system but it is interesting to ponder.
Digitopoly 
The cartoon, and xkcd in general, is indeed brilliant. But why then would I review something that I liked positively? For places I revisit, like restaurants, mainly because the danger of a restaurant I like going out of business seems greater than the danger of it becoming overcrowded. And, selfishly, if it does stay in business and knows I’m a customer that gave it one of those good reviews, I may get better treatment when I’m there next time. I think the same goes for hotels.
Your weighting concern seems more likely to be a problem in some contexts.
Oh, and if you get a chance you really (genuinely) should visit beautiful Swaledale, UK, and stay in the School House B&B at Gunnerside. One of my favourite places, but struggling economically and visitors would help to keep it thriving.
Three criticisms:
1) Not everyone is a dick (as put by XKCD). People are altruistic; or are not myopic but want to preserve a collective review system that works
2) This is even more so if people change holiday locations rather than choosing the same place every year – i.e., there is no point of being a dick in the first place – indeed, you want to make sure the place you liked has a steady flow of customers, so as to be able to maintain the same level of quality in the remote case you decide to go holidaying there again in the future
3) Three, it seems that being a dick would only pay off under capacity restraints. Moreover, with increasing returns to scale you want to sponsor product you like as greater volumes may lower the price paid for repeat purchases
With enough of a taste for variety (and enough restaurants to visit) one might conceivably want to drive other diners towards businesses you’ve already visited. If writing credible positive reviews is much easier with first-hand experience (and easier / more credible / more sustainable than writing many negative reviews), the selfish foodie might only write positive reviews. (But I don’t see an instrumental argument for those reviews being more positive the more the individual likes a given restaurant.)
Your story applies just as well to restaurants you’re considering visiting. Why wait until afterwards to give a negative review?
In another context, user “reviews” are more vulnerable to these incentives. A colleague told me that his strategy on Craigslist was to flag as spam anything he was going to pursue, and said that this was a common tactic for cars and housing rentals.
Yeah, I always wondered about this 🙂
But I think the opposite is true. When we like a store or restaurant or whatever, we want that joint to succeed. My favorite haunts are mom and pops, made great by the hard work of the owners to create something original. If I like that place, I want them to have business and stay in business. The better they do, the more likely my fav will still be around the next time I am hungering for some “Dog Collars” (unique french fries from Lost Dog Cafe in Arlington).
Except in a few markets, what is the greater risk? That my fav will get too much business or it will get too little business. I would rather take the risk that my fav will get too much. (maybe if I get lucky they will open a second store, closer to my office).
A classical prisoner’s dilemma in a ‘me vs rest of them’ world, where it’s always in the interest of the individual to cheat, lie or defect against others. When there are enough individuals defecting we’ll come to the point of collective destruction, which is exactly what we have in financial markets right now. my quick solution: encourage competition between the reviewers, allow people to rate their reviews
It would be interesting to compare successful and failed rating systems. I suspect that architects of successful systems, like Amazon and IMDB, do things i) to encourage pro-social behavior and ii) to inflate reviews generally. Come to think of it, these are also review systems for products that are non-rival or so distributed that reviews may not affect price or availability.
The xkcd comic is indeed brilliant. But I will disagree on the effect of the potential bias introduced by the lack of repeat reviews from repeat customers, mentioned in the last paragraph
There are much stronger survivorship effects in reviews that tilt the review biases the other way around. (From what I remember, on Amazon.com, the average product has a rating of 4.2/5.0, or something similar). Here are two biases that I can immediately recall:
a. Bad products and firms do not last for long in the market. Once they get enough negative reviews, it becomes more beneficial to for the firm/product to “die” and reinvent themselves with a different brand/product name.
b. The set of people that review a product are a subset of the people that buy the product. Even assuming that we get a random sample of the buyers to write a review, which is definitely not the case, we expect that positive reviews to be higher than the negative ones: The people who bought the product had already a high valuation for the product they bought, as they considered it to be the best available option among the competing products. Therefore, we expect more positive reviews than negative ones, for most of the products in the market.
The survivorship bias of not having repeat reviews from repeat satisfied customers should be (is?) more than compensated for, given the other two biases.
Am I misunderstanding, or does the bias appear in the average for all hotels, but doesn’t affect the average rating for any particular hotel?
What impact would there be if people who were picky and rated hotels negatively also switched hotels more often (in search of one that matched their unusual taste)?
What makes review systems generally work is that across a given website the biases all tend to push in one direction. For example on Ebay the bias is to leave positive feedback and it has become so much so that Ebay even removed the option for sellers to leave bad comments. People know this and therefore when negative feedback does appear it tends to be taken seriously.
On other sites like Newegg the bias swings towards people who have less than optimal outcomes because it is a mechanism to take action a misbehaving vendor and often times the vendor will see the comment and try to remedy the problem. Still, even if there is a negative bias one can simply compare the proportion of bad reviews a product has to similar products. If one has 80-90% 4 and 5 star reviews and another has 60-70% its not hard to see which product is more reliable.
Any large website with a coherent review policy can easily tailor its particular biases through common “nudge” tactics. Providing real rewards for reviews will, for example, tease out the silent majority, increase the sample size and make the reviews more consistent, no matter what the bias.
I think we also need to be cognizant of how the reviews weigh on a purchase decision. I believe that most people will find a product that they want to buy, then double check the reviews to avoid getting scammed. That’s important, the consumer WANTS to buy the product and the reviews will confirm or counteract that desire. Even if a newbie gets scared off at their first check of reviews, if the reviews are consistent between products they will come back to their first choice if other choices exhibit the same review pattern. Where reviews are sparse looking at the dates and reviewing the specific comments can provide consumers with good information on how to weigh them.
Regarding strategic reviewing I think that there are two strong incentives working against what Black Hat is doing in Xkcd. First, if you punish someone for good service in the hopes of getting better deals in the future, you take away your one tool to punish ACTUAL bad service and humans are set up to get a lot of satisfaction from punishing those that violate the social contract. Second, you risk making the place you enjoy either go out of business or change its service model so that you end up harming yourself. A bad review might become a self fulfilling prophecy where a vendor gives up trying to please people and just lowers their level of service to what customers are already expecting due to the reviews.