Posted Fri Aug 15, 2008, 12:38pm Subject: How to write a shill review (semi-serious)
... or maybe it should be "How to spot a shill review". Or maybe "How to rate a CG consumer review".
I've been debating whether to write anything about this topic for a while. Even though CG has rules about who can write reviews under what conditions, people still seem to violate this regularly. I think the moderators do a pretty good job of moderating the site, and am especially grateful since they do it for free, but I still see a lot of reviews that seem "shilly". I do spend a good amount of time rating reviews, although I don't know how much time others do the same.
The reason I've been debating whether to do it or not is because I might just wind up giving the "other side" ammunition to cloak their shilliness; but I'm going under the assumption that they don't read these forums too much and maybe we can have a semi-private discussion here. Besides, maybe they'll read it and decide to write something really in detail that turns out to be useful instead of a two-paragraph ad.
A shill review is bothersome because it represents company PR instead of a real user's experience with the product. Even if it comes from an employee of the company who really uses the product, it tends to be skewed to favor the product undeservedly. Here are some things I look for as "clues to a shill":
Review is short-- not decisive in itself, but is a clue to look for other problems.
Use of empty marketing-like phrases, like "makes a perfect cup every time!" without explaining exactly how they arrived at that decision, or what constitutes a "perfect cup".
Inclusion of official website links-- "see the full range of products at <link>".
Join-and-review in the same day. If they joined recently, and they have one post, or one review on the same day, then it's logical to conclude that they likely joined only to review their pet product.
Official-sounding descriptions-- "With this compact, maintenance free machine, even the non skilled user can make the best espresso and cappucino ." (Yes, I'm taking many of these quotes from a recent review, but this one pushed me to start the topic, and it's not possible to sort reviews by review quality.)
Obscure product line-- since the shiller is basically working to get free advertising, they are putting out the word on some new product in a crowded field, such as a drip maker or below-top-of-the-line espresso maker. In other words, you don't see shill reviews on the Mazzer Mini grinder.
Multiple reviews on the same new product in the same day or within a short timeframe-- this suggests that the same person or multiple people in the company joined to put up their short glowing and non-specific multiple reviews. I remember several months ago seeing the same product reviewed three times in one day!
Unexplained negative comments-- perhaps indicative of an anti-shill. "Arrived and did not work" but no explanation of what they did to get it fixed, no company response listed, etc.
Empty list of negatives-- or more frequently, a short list of extremely minor or obvious flaws, since the submission form encourages some negative comments. "Cannot steam and draw a shot at the same time" on a single-boiler espresso machine.
So am I worried about nothing here? Does the review quality system work well enough to spot this kind of phenomenon? How do you respond to these types of reviews? Do you have other clues that you look for? Do you even notice? Do you think we notice enough?
P.S. "Shill" is now an offical verb, as any noun can be verbed. I shill, I shillled, I have shilled, I will shill. shilly adj.; to shillify derived verb; shillification nounification of derived verb, etc.
MarkPrince Moderator Joined: 19 Dec 2001 Posts: 4,663 Location: Vancouver Expertise: Professional
Espresso: Frankenstein'ed LM Linea Grinder: Anfim Super Caimano Vac Pot: 1922 Silex Drip: Krups Moka Brew Roaster: Hottop
Posted Fri Aug 15, 2008, 12:59pm Subject: Re: How to write a shill review (semi-serious)
These are all good points, and I'd like to point out that this website has a zero tolerance policy for any kind of shill review - be it in forums or in the reviews section.
I occasionally go through the reviews looking for obvious shills, and occasionally taking various actions, either individually or in combination:
- deleting the review - contacting the author to get them to provide some sort of guarantee they don't represent the product or company in any way - disabling the user account - banning the IP address from the website
My motto is err on the side of caution, but in this case, caution means providing our readers with *real* reviews, not shills and attempts to sell a product.
The problem is, I can only do so much monitoring.
If you or anyone else spots what you believe is an obvious shill review, please email me with a link to the review and your reasons for it, and I'll review and take action where necessary.
Posted Fri Aug 15, 2008, 2:28pm Subject: Re: How to write a shill review (semi-serious)
"shill" is linked to being fictitious. It's difficult sometimes to decifer between a shill review and a really bad review, or even an uninformed reviewer. I'd say more reviews are nearly useless because of a bad/uniformed reviewer than it being shill. I've read some reviews here that just made me scratch my head. This is where the rating systems comes in handy (a review of the review if you will), but even at that, it has to do more with how WELL the review covered everything and NOT necessarily how accurate it was. When I read how people are more than happy with a blade "grinder" (if they like it it's fine), but to say it'll do anything from espresso to press it's gotta make you wonder. Or the one review I read where the only comments were that the grinder isn't available, or the one where baratza's customer service was the worst ever or... I could go on forever. I suppose ultimately the reviews are a tool and not a buyer's guide so buyers beware.
Posted Tue Aug 19, 2008, 8:06am Subject: Re: How to write a shill review (semi-serious)
Great thread! I agree that it's tricky knowing how far to trust a given review. I'll often click on the reviewer's name and see if I recognize their username. If they've never posted in the forums, that's a sign that they may not be terribly interested or informed (although it's not a dealbreaker - there are epic lurkers for example).
I'm less concerned with true shill reviews (disguised ads) than I am with those that are simply uninformed, light on experience or expertise or however you want to put it, if only because I think there are a lot more of those. Reviewers who don't know as much as they think they do, in other words (and who doesn't fit that description some time or other?). These can be very tough to spot in my experience.
For example, if I knew then what I know now, I'm not sure I would have started with a Virtuoso grinder for espresso. (Yes, you can go ahead and say "I told you so"... it's okay, it's almost like a rite of passage for espressogeeks on a budget, lol)
I do find the review ratings very helpful, but sometimes it seems like people haven't been rating very many of them. I also get the "review" and "rating" numbers confused sometimes. And I admit I'm also guilty of not rating most of the reviews I read. Many times I don't feel qualified to do so, but often I'm just lazy. :/
I think a "How to Use the Consumer Reviews" thing on the main review page could be a great help. And a "How to Write a Review". (And a pony... :P)
svyerkgeniiy Said:
Even though CG has rules about who can write reviews under what conditions, people still seem to violate this regularly.
Additionally, we have a set of review guidelines we ask you to follow:
>> Please write reviews for products you own, and avoid single line responses - details really do matter. >> Provide comparisons you might have with other products similar to this one you are reviewing. >> Be realistic with your ratings, and remember that others will be using your review to make purchase decisions. >> One Review per product. >> Outrageous or suspicious reviews, if detected, will be verified by email, and deleted if necessary. >> Commercial advertising within the product reviews not previously authorized by CoffeeGeek.com will be deleted. >> Please check spelling and avoid controversial language. >> Required fields really are required. Your review cannot be submitted without them.
You can also find information about writing reviews in the CoffeeGeek FAQ.
petuniaii Said:
I'm less concerned with true shill reviews (disguised ads) than I am with those that are simply uninformed, light on experience or expertise or however you want to put it, if only because I think there are a lot more of those. Reviewers who don't know as much as they think they do, in other words (and who doesn't fit that description some time or other?). These can be very tough to spot in my experience.
For example, if I knew then what I know now, I'm not sure I would have started with a Virtuoso grinder for espresso. (Yes, you can go ahead and say "I told you so"... it's okay, it's almost like a rite of passage for espressogeeks on a budget, lol)
I agree that this is a difficult aspect in general for user-written reviews. You never know what level of expertise or even enthusiasm ("seriousness") someone is at-- for instance, my Mom loooooves her Black-and-Decker low-temp brewer with Wal-Mart decaf coffee. I usually look at the tone of the review and the terminology the reviewer uses; usually you can discern how serious they are about good results from the way they write about it. I also try to see how it compares to other things I read on the site. But not everyone gets the same enjoyment or has the same appreciation for the same things. For example, I don't enjoy espresso too much, so I feel unqualified to write much about it. The only thing you can really do is to look over as many high-rated reviews as you can and get a general feel for it; there's no quick process here.
petuniaii Said:
I do find the review ratings very helpful, but sometimes it seems like people haven't been rating very many of them. I also get the "review" and "rating" numbers confused sometimes. And I admit I'm also guilty of not rating most of the reviews I read. Many times I don't feel qualified to do so, but often I'm just lazy. :/
I think anyone who appreciates fine coffee and would use the product in question is qualified. You should review based on what you want to see in the review, not so much on whether you are qualified in the product itself. Here are the questions I ask myself:
How detailed is the review? More detail is generally better.
Is normal operation explained? This usually gives insight into product quirks. Too often people gloss over this part, as if we all understood what it takes to make it work.
Is the writer's opinion explained in detail? If s/he doesn't like it, I want to know why, not just that s/he is displeased.
Does the writer talk about his specific experience with the product? Or do I see only generalities about operation and result quality? I want to know how it is to live with and use the product regularly, and any extra effort that can or should be expended to produce better results.
Are any negative aspects, such as broken components or bad-tasting results, investigated and attempted to be resolved? Or are they just complained about? I have to admit, specific detailed complaints in themselves cause me to give a higher quality rating than I would give without them, even if that's all the review is about. I want to know about the problems people have with a product. Often people only write when things are broken, but defects need to be pointed out. If enough reviews on a grinder say it produces too much dust, then that is a clue about the product in general.
I give an "average" review a 5. I take off points for unsupported complaining, suspected shillmanship, and lack of content. I add points for detail, supported negative points, and evidence of detailed understanding of how the product fits in with coffee in general. I sometimes do research into the person's profile to support my conclusions.
petuniaii Said:
I think a "How to Use the Consumer Reviews" thing on the main review page could be a great help. And a "How to Write a Review". (And a pony... :P)
How to write a review would be tough to write without it being too limiting I think. The problem seem to be that it takes a lot of effort to write a really good review, and some people are good at that and some are not. My reviews have often taken two hours of writing and editing, but why ask everyone to do that? A lot of people would be intimidated.
But I'd be glad to have more people rating reviews.
Symbols: = New Posts since your last visit = No New Posts since last visit = Newest post
Forum Rules: No profanity, illegal acts or personal attacks will be tolerated in these discussion boards. No commercial posting of any nature will be tolerated; only private sales by private individuals, in the "Buy and Sell" forum. No cross posting allowed - do not post your topic to more than one forum, nor repost a topic to the same forum. Who Can Read The Forum? Anyone can read posts in these discussion boards. Who Can Post New Topics? Any registered CoffeeGeek member can post new topics. Who Can Post Replies? Any registered CoffeeGeek member can post replies. Can Photos be posted? Anyone can post photos in their new topics or replies. Who can change or delete posts? Any CoffeeGeek member can edit their own posts. Only moderators can delete posts. Probationary Period: If you are a new signup for CoffeeGeek, you cannot promote, endorse, criticise or otherwise post an unsolicited endorsement for any company, product or service in your first five postings.
= No New Posts since last visit 