Technology Review
Researchers say online recommendation systems can be distorted by a minority of users.
When searching online for a new gadget to buy or a movie to rent, many people pay close attention to the number of stars awarded by customer-reviewers on popular websites. But new research confirms what some may already suspect: those ratings can easily be swayed by a small group of highly active users.
Vassilis Kostakos, an assistant professor at the University of Madeira in Portugal and an adjunct assistant professor at Carnegie Mellon University (CMU), says that rating systems can tap into the "wisdom of the crowd" to offer useful insights, but they can also paint a distorted picture of a product if a small number of users do most of the voting. "It turns out people have very different voting patterns," he says, varying both among individuals and among communities of users.
Kostakos studied voting patterns on Amazon, the Internet Movie Database (IMDb), and the book review site BookCrossings. The research was presented last month at the 2009 IEEE International Conference on Social Computing. His team looked at hundreds of thousands of items and millions of votes across the three sites. In each case, they found that a small number of users accounted for a large number of ratings. For example, only 5 percent of active Amazon users cast votes on more than 10 products. A handful of users voted hundreds of items.
"If you have two or three people voting 500 times," says Kostakos, the results may not be representative of the community overall. He suspects this may be why ratings often tend toward extremes.
Jahna Otterbacher, an assistant professor at Illinois Institute of Technology who studies online rating systems, says that previous research has hinted that rating systems can be skewed by factors such as the age of a review. But she notes that some sites, including Amazon, already incorporate mechanisms designed to control the quality of ratings--for example, allowing users to vote on the helpfulness of other users' reviews.
Kostakos proposes further ways to make recommendations more reliable. He suggests making it easier to vote, in order to encourage more users to join in.
Niki Kittur, an assistant professor at CMU who studies user collaboration on Wikipedia and was not involved with Kostakos's work, says that providing more information about voting patterns to users could also be helpful. Kittur suggests that sites could create ways to easily summarize and represent other users' contributions to reveal any obvious biases. "There are both intentional and unintentional sources of bias," says Kittur. "In the end, what we really need [are] tools and transparency."
Kostakos also suggests removing overly negative and overly positive reviews, so a site won't be too positive or too negative overall. But Otterbacher, who is examining reviews from IMDb, Amazon, and Yelp, worries that such a policy could discourage many people from taking part. "People who write reviews want to say something about the item, and they can be pretty passionate about their opinions," she says.
Applying the lessons of the $1 million competition to other recommendation systems will be a challenge.
VP Jim Bennett discusses how recommendation systems suggest your next movie and the challenges of building a better one.
Any complexity built into a review system is purely to prevent the system from being spammed. The best method to encourage people to review is to show them past purchases once logged in and give them a option to review that product at that time.
The article is assuming the personalities of people who are actively providing feedback are extreme in there views. If a person has a bad experience with a product they will likely write about it. If a product is far better than expected and they are quite happy they feel like telling other people. Many of these people are simply knowledgeable about the product and I find the advise they provide to be very useful. If 20 people review a TV and 5 say the remote is poor quality or broke soon after purchasing it then it is likely my remote will break as well. The reason I expect the remote to break is that 20% of the people experienced this in a short time frame.
In other words I find the reviews to be very accurate as long as they have enough reviews of the product. It is important to only let people review products they have purchased so the system won't be skewed by a agenda.
Isn't this minority called "art critic", "expert reviewer", "journalist", etc. in other contexts? ;-)
Yes, but with one significant difference: The professional critic is motivated by financial reward. The Amazon critic is motivated by something much more personal. That is the aspect I find most intrigueing, specifically what motivates the serial actors on venues like Amazon?
The true critic is NOT motivated by money or the subjective personal biases of the non-critic. A true critic is motivated by the objective enhancement of the human experience through careful and educated examination of the subject at hand.
We need more well-honed critical voices and less of the braying of the crowd. Crowd wisdom is built on a logical fallacy: just because a lot of people say it is so, doesn't make it so.
Democractic Intelligence = FAIL
Guest (wmclough)
Agreed. If you want evidence, look at California with its initiative process for enacting laws and changing the constitution. A perfect example of why the electoral college should NOT be done away with.
yes, participate, share, and learn
I strongly recommend greater participation to enhance sharing and learning, and wisdom.
New technologies like http://urtak.com/about are revolutionizing public opinion research and providing new tools that will promote and advance democracy.
There is another very common reason why ratings are so often in the extremes.
your opinion on a comment or product that you read and think neutrally of can easily be swayed if it has a high number of other people saying it is good or bad. It will prompt you to re-read it with a new slanted opinion on it, after which you input your vote along with the crowd, making it even more of an extreme for the next reader.
In addition, it generally takes time to sign up for a site to leave reviews. Often times a person will only sign up once they have something very extreme they need to say about a product. Before that point they don't feel the need to spend the time to sign up to leave their say. This coupled with the information from the article stating that most people only leave 1-2 reviews, means the majority of those reviews will be extreme.
It will be interesting to see how many more people comment/rate comments on this article versus the others.
I tend to believe that in most cases opinions are not necessarily based on "crowd Wisdom", but it can certainly be also. Human beens are the designers of their own judgement sometime-regardless of what goes on around them. Because of the way we are wired-it is true that so many elements can be a factors that influence how/what we say about a product or service. Culture, education, etc, are good examples.
The term "Crowd Wisdom" is too often an oxymoron. Take a gander at Jonathan Livingston Seagull if you disagree. The problem of a minority having too much power is older than the printing press. Deal with it!
Thanks a lot, Richard Bach's book is a good example and I have not made the connection.
The problem with Wisdom of Crowds is that it is used as a tool of anti-intellectualism and an
argument to belittle Experts. Really, its uses
are very limited and its importance is less than
claimed. We have to study how much WoC has contributed to the Crisis. Errare humanum est,perseverare diabolicum.
I question everything, especially opinions... do you guys think maybe I have trust issues or is this degree of skepticism healthy? The irony is hilarious, is it not?
I rated your comment 5 stars, just to add to your point. :-)
Interesting but incompete research
This is an interesting article that highlights that people submitting reviews or ratings of products on sites like Amazon do not provide acurate respresentation of the populations view. This as mentioned is due to the extreme lows and high reviews prevalence, super satisfied and horrible dis-satisfied people tend to comment and rate, while netural people do not.
The thing missing in this article is how the consumers interpretative these rating an rankings. I can speak for myself, where I look mainly at the negative review and determine if their is any product defects that are common, opinion usually mean little to me. Hence, I ignore the opinions and look for evidence of product failure.
I would love to see a follow up article explain how on average people interpret these ratings and rankings.
Dr. Brian Glassman
Ph.D in Innovation Management from Purdue University
Re: Interesting but incompete research
Agreed on the follow-up study. Additionally analysis (somehow) needs to be conducted to ensure the person reviewing the product or service has utilized/purchased it. Additionally background information should be taken into account; for instance a Digital Immigrant may not be as satisfied with an iPhone for instance as a Digital Native. Those that are low-level users that don't utilize all the features of a product versus those that push the limit of the technology or product are going to have different reviews and viewpoints, these need to be taken into consideration as well.
Re: Interesting but incompete research
I would opine that this research is not particularly interesting and full of urban legends:
1. Every online social media or collaboration platform studied, continually demonstrate the same pattern of behavior among their participants - 5%-15% contribute with the balance silently consume and rarely rate any of the activity or content that is being produced. That is somewhat higher participation rate than offline equivalents documented. People do it because they find it socially responsible. That is why I have done it for many years and I do not publish more than 5 product reviews per year;
2. Statistical analysis of 1,486,459 Reviews as of last update on March 18, 2010 2:10 PM PST published on http://www.amplifiedanalytics.com shows that 85% of these reviews are rated between 2.55 and 4.78 stars, which does not support the legend of extremist influence or mass gaming of reputation systems.
3. Purposeful analysis of statistically representative number of customer reviews provide very valuable information about customers experience and can offer the insight unparalleled to average Market Research survey.
Maybe I'm dense, but if you have enough access to the data provide voting transparency, couldn't you build in a mechanism that allows one person one vote? If you verified by IP address I'd think you could get a long way toward the one person one vote. That would solve that aspect of the problem. The other, the fact that some people express extreem views... so what? If that's their opinion then that's their opinion. What's the problem with that? Excluding their voices is a form of censorship. I'm against it.
Agreed. That seems like the most sensible way to address this issue.
You are misunderstanding what they are meaning unfortunately.
What the article is pointing out is that there is a minority of people who go around and vote on everything they have any reason to. They are not saying that a single product has 500 votes from one person, but 500 different products have votes from that one person.
Unfortunately, millions of people are dynamically allocated a different IP address from the limited pool that their ISP has available every time they connect to the Internet.
So, obviously, the same IP address could be used by different people at different times and one person will be allocated a different IP address each time they connect to the Internet.
Cookies also don't work as unique IDs because many people disable them. Perhaps making acceptance of a cookie to be able to vote (but then some people will just refuse to vote, which could also skew results if such people happen to tend to have similar opinions which will then be ignored).
Perhaps, weighting relevance of votes by voting frequency (assuming logins to help identify some degree of uniqueness - although multiple registrations could still occur) would help to ameliorate the skewing effect of extreme single - or seldom - voting members.
People who try to "beat the system" by registering multiple accounts would then automatically water down each of their responses with each identity overall - although they could still vote multiple times for one specific product, movie etc.
No simple silver bullet....
There are algorithms that use semantic analysis of style and phraseology to weight authenticity and uniqueness of each review. We are using them in our process.
The picture very well depicts how the crowd opinion and hence the crowd behavior can be wrongly skewed by a very few.Solution is that the voting engines need to mention the results as a percentage of the total users and need not give weightage/publish till at least 60% vote.
Comsumer’s Ratings of Products on the Internet
Several years ago, MIT Technology published an excellent article in its magazine about the internet and "group think", and how the more people involved in the process, it is more likely that the right political decisions will be made as opposed to a small number of "experts".
For this to be valid, one's information must contain a true cross section of society for its opinions... And the conclusion was that the internet provides that service with the above requirement.
Of course, the problem is that one has to separate the garbage from quality. And the conclusion was that bloggers that write crap, get lost in never-never land in time due to credibility.
Two examples. One, the ten or so advisors to LBJ and Nixon, during the Vietnam War, and the same with the Iraqi War with Bush illustrate the folly of “experts”.
The group think was way ahead of the "experts" in making the decision .
Hence, the internet, if used wisely, represents the third communication revolution.
The first, was controlled by the church and kings, they control of the knowledge, the second, moveable type which gave knowledge to the serfs. But still the second was under the control of the publishing houses and main stream newspapers, which propagated its own self interest... ie Time magazine, etc.
After Vietnam, the press was blamed for losing the war; however, when I wrote my book, about the sixties, covenantbetrayed.com, I read all the Newsweeks from 1958 to 1975. My conclusion was that the MSM was a strong proponent of the war, up until Tet. Throughout all of 1967, we got reports that we were winning, and than Tet. After Tet, no body believed anything regardless of whether or not the government was for once actually telling the truth... that Tet was a severe blow to the Viet Cong.(By their own admission.)
Regardless, the information was available if one searched for it, from where the antiwar movement gathered its information. Bernard Fall’s books and many others told us that the Domino Theory did not apply to Vietnam, since Vietnams history for the past 2000 years was to fight back against Chinese expansionism into Southeast Asia.
Now, we have the internet, the third, but we are seeing more and more control of the internet. ie, China. Even the United States want authority to close the Internet during a “national emergency”.
OK, I am digressing from the topic, but the point I want to make is that although user’s recommendations have serious flaws, a careful search, for other sources of information a consumer can still make correct decisions.
I am far more likely to reject a product based on just one or two poor ratings despite a ton of five star ratings. So, in some way, the ratings still have some value. The big bugaboo, is that often return policies are so short that one doesn’t have sufficient time to fully evaluate a product. Here again, I stay away from these internet sites.
So, make sure what the return policy is, and if there is a restocking fee. (something I also avoid.).
Hey, anyone wanta buy a Sangean HD radio? I got one I will sell at half price, never used, ‘cuz the software to run it is just too complicated and always crashes. Burned once, but not again. Actually, I would be ashamed to sell this to anyone. So, I am stuck with it. I don’t even want to donate it, because it will get sold to some other sucker.
ps, I didn't mean to plug my book, but just used the experience I had writing it to illustrate a point and felt I should give the reference for some validity as opposed to just pulling my comments without out documentation.
ron hansing 9.17.9
A 'social volume knob' comes into play
The conversation needs to be different when we are talking about social networks within an organization’s firewall. In that instance, it’s not about trusting the crowd’s wisdom. Rather, it's about managing the community by knowing whose input should be trusted, along with managing and moderating the community. This must all be done in the context of how the community relates to business initiatives and the information assets of the organization.
At our company, Inmagic, we call this the "social volume knob." As organizations roll out social technologies, they might want to start with the volume knob set "low" for certain classes of users. For example, some users might be allowed to tag one type of content, or other certain users can blog or comment.
In other instances a "high" setting on the social volume knob may be more appropriate because the wisdom of the crowd is more crucial to have. In these instances, users and members of the network may be given more freedom. In corporate settings, this is usually provided to domain experts, who can be authorized to write blog posts and have access to full social capabilities.
And then you have everything in between. Each contributor's access capabilities can be adjusted, so perhaps one user can blog, rate, and comment, while another can just comment -- and only on certain content. This lets vetted information retain its veracity, and provides control over what content gets socialized, and how.
Unmanaged social knowledge networking risks culture shock, or worse, information chaos that can undermine or overwhelm the knowledge management initiative.
Mike Cassettari
VP Marketing and Business Development, Inmagic
http://blog.inmagic.com
Re: A 'social volume knob' comes into play
cool! imbedded structure mediates participation. the ultimate in virtual censorship. sorry, I do understand the virtues and need, but also am intrigued by the implications for misuse. who then is the ultimate gate keeper?
Re: A 'social volume knob' comes into play
I second my colleague Mike’s thoughts here. To that I add: Remember the old adage "garbage in, garbage out." With a social volume knob, you can catch and eliminate the garbage before it finds its way into a business decision. In essence, a social volume knob increases what I call the signal-to-noise ratio. More signal (high-quality content), less noise (low-quality content). I elaborated on my theory on our blog: http://inmagicinc.blogspot.com/2009/09/signal-to-noise-ratio-finding-sense-in.html. Hope it’s ok to add that link!
Phil Green
CTO, Inmagic
http://blog.inmagic.com
This article is quite interesting to me - as a consumer, as a technology industry professional, and as a market researcher).
As new media continues to proliferate, it causes us as consumers (of news, services, products, etc.) to change the way we take in and process information. For most people, I think the feedback on a user-input site is but one piece of information that they process. If I am shopping for a TV, for instance, I would read the comments from users/purchasers, look at input from respected, independent sources, ask friends, and look at it myself - before making my own judgments.
The so-called Wisdom of Crowds (like practically any other market feedback system) can be influenced by a small group. I believe as time goes on, however, these systems will continue to be fine-tuned to improve the information provided.
However, opinions are opinions - whether they are from an individual that has written 5 of them or 500 of them. We just need to refine our system of making our own judgments to accommodate this type of feedback.
So, yes, the system can be influenced by a small group. But, if you wisely use it as just one source of input then you should be okay.
True, but the problem is not only the lack of diversity
I believe that the wisdom of the crowd only works in cases where one has to predict some thing that can be easily quantified (eg. number of jellybeans in a box). Using this principle for any thing that involves intangibles may be stretching it a bit too far.
I have started an experiment at( http://socialexperiment.heroku.com ) where people are asked to guess the weight of three objects, one is very light object, one is a very heavy object and the third is a person. The data collected so far shows that people are good at guessing the weight of a man but are really bad (way off) when guessing the weight of very heavy and light object. Which clearly shows that wisdom of crowd only works in cases when we can trust the intuition of the crowd even if the population is diverse.
Manufacturing in the United States is in trouble. That's bad news not just for the country's economy but for the future of innovation.
On Twitter
Become a Fan on Facebook
Subscribe to the Feed
peter193710
13 Comments
wisdom of crowds
Sorry, I think that crowds are not wise and what we call Wisdom of Crowds is nothing else and not more than the Power of Diversity. This paper shows
that it can be the Weakness of Diversity too.
Reply