Not too long ago, on the Conquer Club website, they implemented a new system for player ratings. Previously it had been based on an eBay style positive/neutral/negative scale with a comment. You simply voted weather your experience playing them was positive, negative, or neither and then said something like “played all his turns in a timely manner” or “excellent team mate, worked well together” or “stupid dumbass only attacked his own team”. Anyway, they switch over to a 4 stat 5 star rating system. If you don’t care to follow the link, the short version is that after you play someone you can leave them a rating which is a rating from 0 (no rating given) to 5 (excellent) stars in 4 categories: Fair Play, Attendance, Attitude, Teamwork. And the rating isn’t revealed until the game goes into the archive (no more ratings can be made on that game).
Going back to the eBay style ratings system, I hated it on eBay, largely because it was heavily retaliatory. I once bought my wife a cello on eBay. I didn’t buy the best model out there, I just wanted to get her a practice cello so she could start playing again (seriously, I’m not going to spend five grand on anything for anyone if there is a chance its just going to sit in the corner collecting dust). The one I bought arrived and the bridge was not set up, which I expected. What I did not expect was to wind up driving around to about two dozen music stores in an attempt to get someone to set up the bridge and have none of them willing to work on it because it was an “off brand” they didn’t sell or support. I ended up setting it up myself, and did a very poor job of it, and after I gave it to my wife we finally found a music shop where someone relented and set up the bridge and tuned it. So, I went to eBay and left a review of my purchase experience. I was limited by the number of characters, but I said, “sale and shipment fine, but cello was ‘off brand’ and most music stores would not set up bridge or tune it.” and I gave him a neutral review. To which, he replied by giving me a negative review that said “clueless user, DO NOT SELL TO HIM AND DO NOT BUY FROM HIM, liar and cheat!” Now, because this was the only review I had, I ended up over the next 6 months unable to buy anything. All my bids were refused. The entire time, I was talking to the cello guy trying to get him to change his review. I had been totally honest and had not given him a negative review, I just felt that anyone who purchased his items might want to know about difficulty getting service on those items. Finally he relented and changed the review to neutral saying, “inexperienced eBay user, expects to get more than he pays for”. After having a few more bids of my dropped, I just stopped using eBay.
As this relates to Conquer Club… well, I played a game with some people and I rated most of them 3 out 5 in most categories, except attendance in which case missing zero turns got people a 5, the occasional missed turn got a 4 or 3, and every player who dead beat (missed three turns in a row and was kicked from the game) got a 1. Once the games got locked down and ratings were revealed, I got a flood of private messages from people complaining about getting a 3. But looking at the rating scale:
0 means No Rating. Or in other words, I don’t feel like I can judge you on this.
1 means Bad. A rating of 1 means you sucked at whatever it was.
2 means Below Average. This means you performed worse than I would expect.
3 means Average. Meaning you performed as expected.
4 means Above Average. You exceeded my expectations.
5 means Excellent. You are awesome.
In most cases, people are Average. To get Above Average in Attitude, for example, you just need to be gracious when you lose… or win. To get Excellent, you need to also chat and be a good natured guy. If you are silent the entire game, I cannot give you anything other than Average because there is nothing to base it on. The ratings are from Bad to Excellent, not from I Want To Kill Them to Didn’t Piss Me Off. There is a phrase where they call something “going above and beyond” and this is what they mean, you have to go Above and Beyond the Average to get better than an Average rating.
The problem is, people don’t see it that way. 3 out of 5 is a 50%, its halfway, and 50% is failing. Most user based rating systems end up being all but useless for the same reason. People being rated expect to be given the highest score possible unless there is a problem, and even then most of them want to still get the highest score possible after explaining or fixing the problem through other channels. People rating others get pulled into the retaliation loop where they are giving higher ratings than they should because they don’t want to be rated poorly themselves. And then most importantly, the site/game/whatever that is using the rating system, if they allow comments, usually have the length limited to the point where meaningful comments are not allowed.
I don’t know if there is any solution… you want to have the ratings to assist you in avoiding problem users, but the ratings can be, and are being, gamed so you don’t even know if a problem user is really a problem user.