Amazon reviewer rank and online community

So, I am - off and on - obsessed with my amazon reviewer rank. I’m not really sure why that should be, I think it started when I realized there was such a thing because my friend Sean started reviewing some Apress books. As a result, I’ve spent a bunch of time thinking about amazon (not to mention writing lengthy reviews).

There’s a lot to learn from (and like about) this ranking system, on a few levels. One - as a publicly viewable list sponsored by an extremely reputable site it is a gigantic motivator (for people of a certain sort) to contribute staggering amounts of content to the site. Two - the interface to this system is dead simple and immediately understandable by everyone. And three - they don’t publicise their ranking algorithm but it seems to be something at least mildly complex.

So in terms of public motivation, lists and top x anything are huge on these intertubes. I’ve seen it again and again if you publish a list that people can get on, they will try to get on it. People love to have their efforts recognized and if it is done well, ranking can be an incredible tool to build community on a site. Once people buy into the system and start contributing and rising through the ranks, they are most likely to stay with your site since they’ve spent so much time contributing to it and, as a result, developing their connection to the community.

Recognition is a key factor in developing this aspect of a site. Whether it is from the creators of the site or from the site’s audience - it is incredibly powerful. Amazon has embraced this idea by maintaining these ranks and handing out badges for the top reviewer, top 10, 50, 500 and 1000 reviewers. These appear on every review you make, should you be so august as to merit such a badge. They also show your rank on your profile page for the world to see. People devote a lot of time to writing reviews - these high ranking people often have hundreds of reviews (itself a topic for another post I hope to write).

The value of a good, public recognition system can not be underestimated in developing community.

Now the second bit is that when you have a ranking system (as opposed to other less algorithmic forms of recognition) you need to be careful about how you go about this. Amazon does not tell you exactly how your rank is determined (just like Google doesn’t). Fortunately there are not too many pieces to consider - there’s the number of reviews you have, how often you post and how many helpful/unhelpful votes each one received.

The thing I like best about this is that it’s incredibly easy, you immediately know what to do with it and intuitively know the basics of how your ranking is going to be determined. This is quite distinct from some other rating systems - like slashdot’s - where you have to have a phd to figure out your karma and whatever else is involved. Amazon hides all the complexity in the system behind the scenes with their rank calculating algorithms instead of making the voter need to understand and, more or less, implement the algorithm. For most sites this is the best tactic - slashdot’s works for their audience because that is a self-selected more technically inclined user base. Another beautiful fallout of this is that by putting the algorithm behind the scenes, they are free to change it and recalculate as they see fit. This allows them to learn how to tweak the system as time goes on.

In my search to see if there was any word on how amazon calculates the rankings I came across a page that tries to analyze how amazon might be doing it. I don’t know how accurate or how recent this page is, but it’s from a top 50 reviewer and he seems pretty confident. And many things he puts forth agree with my limited experience with reviewer rank, so let’s just assume he’s got at least the gist of it right.

First they have the ability to figure out “loyalty” votes. So they can identify and remove “false positive” votes that happen when you get all your friends to vote for all your stuff. You are not punished for this other than having those specific loyalty votes removed. The next thing they do, which is my second favourite thing about the calculations, is essentially ignore the negative voting. They may take some points away from you for excessive negative voting, but it is very limited. This is very important because angry or outraged people are much more likely to vote for something than happy people are, so every ranking system needs to take that into account or risk skewing negative. I personally like simply not having a negative option, but having the option and not paying too much attention to it works well for amazon.

The last bits of this are that amazon awards points at thresholds, after 3 helpful votes on a given item and then 10. This helps to prevent the self-fulfillment problem of spotlit reviews and reviewers. The problem is this, if the algorithm simply took into account how many helpfuls you received without capping things - by getting a review into the spotlit position (that is, always at the very top of the list) your rating would simply continue to rise as other reviews kept getting pushed down (and thus less and less likely to be read). Then, as your rating rises you are more and more likely to have other reviews of yours put into that premier position. This would result in your continued rise in rank (a virtuous cycle for you) at the same time that it becomes more difficult for others to do so (a vicious cycle for them). This is an elegant solution to that problem by both largely preventing it from happening as well as creating a greater incentive for people to write more reviews since the only way to continue rising through the ranks is to have more and more reviews for people to rate.

I think there’s a lot to like about amazon’s system and it does a pretty good job of presenting people with helpful reviews as well as motivating it’s users to contribute reviews to the system. It does that with a simple interface that hides the user from a lot of careful thinking the hallmarks of a succesful system.

← newer Three best tech essays of 2006  ↑  On the anti-perl bias in publishing older →