Tag: score

The Ratings Game

Random Thoughts

starsThis isn’t about TV shows. That particular Ratings Game would take an entire blog unto itself to even begin to properly discuss. No, I’m talking about when people review things and rate them.

Does Size Matter?

Some people used a 5 point system. Usually 5 stars, but then they give out half star ratings, thus making it a 10 point system on a scale of 0.5 to 5. Then you get 10 point systems, and then they go and try to present averages between multiple reviewers and dish out things like 7.6 and 2.4, thus making them actually a 100 point system ranging from 0.1 to 10.0. Luckily, if a person starts with a 100 point system, they generally don’t do decimals (unless you are handing out grades and want to really rub it into the kid’s face that a 92 is an A and they got a 91.9, a B). Really though, most of these systems exist almost entirely to attempt to set expectations. If a movie review site uses a 5 star scale, 3 usually means “like”, 4 “really like” and 5 is “love”.

But I’m all about managing expectations, and large systems (even as large as 5) start to set them for me. A 5 star film isn’t going to be just good, it’s going to be great. A 1 star film isn’t just bad, it’s awful! And it happens with every scale.

And then you run into other people wanting to fiddle with your system. Tons of game sites rate on a 10 point or 100 point scale, but the vast majority of their scores will be in the 7-9/70-90 range. They save the top score for the absolute best games, and everything below 7/70 is complete trash, and even mediocre games get a “C”. I’m sorry, but on a scale of 1 to 10, 5 is the middle, the average, the “meets some minimum level of entertainment but I didn’t really enjoy it”. You wouldn’t know that from the way most sites work.

A Simpler System

For me, however, I prefer a binary system, a scale from 0 to 1. If I enjoyed something and would recommend it to other people it gets a 1. If I didn’t enjoy it and wouldn’t recommend it to other people it gets a 0. No fractions.

Of course, that’s just the score. Any long review would get into exactly why I liked it, and what sort of people I think would also like it. Or it would discuss why I think the thing failed for me, and perhaps try to understand what sort of people might enjoy this, because you have to assume if someone takes the time to make something they must intend for someone else out there to like it.

I’ve written about this before. In fact, twice before. At a glance, I need only one piece of information: did you like it. Then, based on that I can decide if I want to read your full review. More so, if I’ve read a bunch of your reviews in the past and I have a general sense of “if he likes it, then I like it” I may not even need to read every review. I can shorthand. If I’ve heard about a game and think I will like it and I see that a reviewer I generally agree with also liked it, I can pretty safely purchase it and avoid the possibility of getting spoilers in a review. On the other hand, if there is a game I’ve heard of and think I will like and that reviewer says he didn’t like it, now I definitely want to read his review because maybe there’s something I need to know.

Got it in 1

In the past, I’ve toyed with ratings systems on reviews here. I even invented a 13 point system just so that a 7 would be the middle/average score. And I’ve thrown out ratings systems, insisting that the review is the only thing that matters. In the future, I’m going to implement my more simple 0 & 1 system.

Now I just need to make some nifty graphics for my new rating system, and they need to be self-explanatory because I don’t want any passing creator see that I’ve given them a 1 and think I’m saying they suck.

The Value of Bad Press

Gaming

I first heard about it back in June.  But apparently it has recently been ruled on by the FTC that bloggers must disclose when they get free stuff.  Personally, I’ve never gotten a free game, and while I’ve gotten free movie passes they’ve never been sent to me specifically for review, I obtain them in other ways (more on that tomorrow).  But, other bloggers do get free games sometimes, or at least offers of free games, so there has been some discussion on the subject.

Being that free games are often sent out by marketing departments, and the goal of marketing departments is to try and get favorable review out in the wild where potential buyers can see them, the concern is that bloggers given free games might give a undeserved favorable review in order to continue getting free games.  Sadly, in part, this is why most game review magazines and sites tend to use the “7 to 9” scale of rating, reserving 10’s for truly astounding games and anything less than 7 for unmitigated pieces of crap, letting even a mildly entertaining game with numerous flaws still get a 7 out of 10.  Of course, the reality is that these places are actually reviewing on a 5 point scale: 6 or less, 7, 8, 9, and 10.  So getting a 7 there is kinda like getting a 2 out of 5.  Do you normally go see movies that get 2 out of 5 stars?

Now, while no game company really wants to get poor ratings, not all poor ratings are created equal.  Rather than trying to seek out favorable ratings, what they should be seeking is “fair” reviews.  By fair I simply mean that the game will be reviewed on its own merits, in detail, and then given a score intended to reflect the value of the game to that reviewer.  THEN companies should encourage people to read reviews instead of just viewing ratings.  Metacritic is the devil because it does the opposite, placing all the focus on the score and the reasons behind the score disappear.  The result is publishers pushing for higher ratings when what they should be pushing for is the abolishment of numerical ratings.  But players seem to demand “at a glance” ratings systems because its easier on them, even though in the end they are mostly being lied to.  Its all counter-intuitive and somehow self-reinforcing at the same time.

Personally, when I am trying to decide if I want to purchase something, I actually seek out poor reviews.  For example, on Amazon.com I may skim through the 5 star reviews, but I will read every single word of every 1 and 2 star review.  The reason is simple: people tend to more specifically describe their dislike of something than they do their like of something.  I’ve written about this before.  The main reason for this is that the only thing I can guarantee about a reviewer is that they are not me.

Take movie reviews for example.  Let’s say I was going to review Zombieland.  On a 10 point scale, I’d give the movie a 9, on a 5 star scale it would get a 4.5 or maybe even a 5 (I’m not a fan of the .5 in a rating scale).  But if you were looking for a movie to go see, the fact that I gave the movie a 9 out of 10 is less important than if you like zombie movies, gory movies, or comedies.  Even if you like zombie movies, Zombieland isn’t Dawn of the Dead, it is more like Shaun of the Dead, and that’s more important than my rating.  On the flip side, suppose I had given Zombieland a 2 out 10 and said, “This movie was just as dumb as Shaun of the Dead. If you liked that piece of crap, I guess you’ll like this too.”  And if you loved Shaun of the Dead, then despite my rating of a 2, you should probably see Zombieland.  When I get in to the idea of the value of bad press, well, I’ve already seen many dozens of reviews for Zombieland that were little more than “This was awesome! Must see!”  But all the negative reviews I’ve read have been very descriptive, and I think would actually be more helpful to people not sure about the film decide whether the film is something they’d enjoy or not.

In the end, though, I suppose I do agree with the FTC that people should disclose if they have gotten freebies, especially if they are going to comment on the value, like saying a game is or isn’t worth the $60 box price.  I’m not sure it should be a law with fines though.  At the same time, I don’t think people should discount a review just because the reviewer got a free copy, and I think marketing departments should push for quality reviews and not worry so much about just getting favorable ones.  They should reward reviewers who clearly spend time with the games and write well, not just those who blow smoke and tell them what they want to hear.

Quake Live

Gaming

Spent this morning playing some first person shooters on the PC… one was a beta, the other was the open beta Quake Live.

I had tried to play QL before, but the insane queue lengths kept me out.  I’d wait, then find something else to do long before I got into game.  But today I managed to get in and run through the tutorial and a couple of matches.  The tutorial started out alright, with me choosing the beginner level and quickly getting an 11 to 0 lead.  Then the AI adjusted and I lost 15 to 11.  Then I went and join some matches…

Either their skill levels are very broad or I somehow borked it up or everyone else is cheating.  First off, I hate deathmatch, and prefer team games where my personal frag count is less important than the team winning.  So I joined up with a capture the flag server.  Its been a very long time since I played bland CTF, usually sticking to Team Fortress, so I didn’t know any of the “standard” maps that were running, and I also didn’t know that I had to put flag-on-flag to capture it.  This coupled with the dumb ass on my team who was yelling at me to “go ahead and cap noob!” even though I was standing in the right place (the enemy had our flag too) confused me for a bit.  But that got sorted out, and we eventually won.  It was close, the score looks bad with an 8 to 1 victory, but it was much closer than that with a lot of good slugging it out for each hard won point.  However, I noticed while playing that even though I was doing alright, other players were fragging much more than me, and they were getting off air kills and other feats of awesome that I’m not so good at.  I really am of a beginner level, I know I suck, so how is it that I’m playing with frag gods when skill matching is supposed to prevent that?  Anyway, we won… then the second match started, and the other team picked up a few more frag gods while our team picked up a few more people like me.  We had to fall back into a pretty strong defence (the entire team, minus one guy) just to keep our flag on our side of the map.  In the end, we lost.  It wasn’t even close.  Sure, the score looks alright with an 8 to 5 loss, but we were winning at one point, all our caps were done pretty much by one guy and the other team got 5 of their points within just a few minutes, chain capping the crap out of us.  We got steamrolled.

Anyway, the game runs smooth, although now I need to go beat up on Comcast because I was getting “Connection Interrupted” every couple of minutes, just for a second, but it was enough to get me dead every time.  If you want to find me, I’m Jhaer.

Ratings Systems

Random Thoughts

I decided that I wanted to start rating things. When I review movies or games or TV shows or the lives of people I meet or customer service or my own stupidity, I want to have one of those goofy scales so that something can be a “9 on the ProbablyNot scale”. But I can’t just arbitrarily slap one on.

Or more to the point, I needed to make sure my scale was absolutely arbitrary.

First off comes the symbol. There are the overused stars and thumbs or even lawn chairs. I wanted mine to be confusing, so I looked long and hard for a symbol to properly represent the site but also to completely misrepresent the scale itself. Eventually, I settled on the wide spread “not” symbol, as seen on road signs and warning labels everywhere.

A Lone Not

With the symbol decided, I next needed a range. Typically these ranges go to 3, 4, 5 or 10. Each of these makes logical sense, so as with my symbol, I needed to make sure my range made no sense at all. Being that my name is Jason and thanks to a particular slasher movie franchise it will forever be associated with the number 13, and I happen to like the number 13, I am going to go with 13. Oddly enough, this actually will help reduce some confusion, especially when it comes to reviewing video games. Most game reviewers use a scale from 1 to 10, and since 70% is passing is most educational establishments, people (insane people) have come to expect that a 7 out of 10 means that it is just barely passing, and anything from a 6.9999999999 down to 1 is failure. Stupid, yes, but also completely understandable to magazines looking to sell issues and to get exclusive previews of new games. So many game sites and magazines actually rate games from 7 to 10, with lower scores being reserved for items that are complete and utter crap. Back to my scale, confusion will be lessened because a “middle of the road” score on a 13 point scale is 7. So game makers can feel good in getting a 70% on my scale, as long as they realize that I give out grades up to 130% for awesomeness.

But a straight linear scale would be too easy, so two elements are added.

First, on my 13 point scale, 7 will indeed be an “average score”. What a 7 means is that whatever I am reviewing was not a complete waste of time. If I am reviewing a movie and give it a 7, that means that after two hours I felt like I’d just spent two hours, but not wasted two hours. Enjoyable, but nothing to get excited about. Being on the low end of the scale is not always a bad thing. While 7 will be middle of the road, the “worst” score to get will be a 3. A 3 means that this thing is godawful bad, and had no redeeming qualities. To score a 2 or a 1 on the scale, your product must be so horrifically bad that it actually turns a corner and becomes something I will actually share my pain with others about. The kind of shitty movie or game that I insist other people must experience to truely understand the depths of the miserable quality contained therein.

Secondly, even with a confusing symbol and an unusual scale, the review score still isn’t odd enough for me. So I’m going to also steal an idea from the ESRB (the people who rate video games for content) and the MPAA (the Motion Picture Arbitrary Assessment, or something like that) and include verbiage for why the score is what it is, but make sure those words are vague enough or strangely worded so that no matter the rating you might still want to see the thing I’m reviewing just so you can get it.

As an example, recently I saw the movie The Mist, which I might have given, had this system existed at the time:

11 out of 13 nots
for Creepiness, Social Commentary, Religious Fanaticism and Clever use of Dog Food.

All reviews will be presented with the rating first followed by a more in depth write up. In depth write ups will likely contain spoilers.

Keeping up with a rating system can be a chore, so we’ll see how long I stick with it. I might get bored and give it up, or not… who knows…

A Few Good Men

Random Thoughts

“You can’t handle the truth!”
-Col. Nathan R. Jessep, A Few Good Men

It’s sad, but it’s true… there are a great many people who simply can’t handle the truth. And I mean simple truth, not ‘being brutally honest’ (which is more often a disguise for being honest, brutally). I had the misfortune of having a coworker ask about her own performance… Okay, let me back up.

I’m working on a program. And as I finish parts of it, I turn it over for requirements testing. The people who are testing it are also the people who gave me the requirements. More often than not, when they test, they complain about things the program doesn’t do, all of which are things they didn’t tell me it needed to do. As a result, I’m constantly rewriting my programs to include things after the fact. We have marathon email back-and-forths where we argue over the value of certain items. Their most common defense of a stupid business practice is “We’ve always done it that way.” And my most common attack is “We are writing a new program, so let’s take this opportunity to change the way its done and make it better.” And its not like its an alien idea… these are things they think SHOULD change, but they want the new program to work exactly like the old program and THEN change it. More work for everyone.

Anyway… so it comes that we are on the phone, and after we solve the latest fire she asks me how she’s been doing on the requirements and testing. I ask if she really wants to know and she says, “I want the truth.” I give her the truth, as kindly as I can. I don’t accuse her of giving me bad requirements, I instead explain that when working on requiements it would work better if she worked with the existing system for a couple or three weeks and documented every task she performed and later reviewed that log for missed steps or details to avoid the situation we have where daily tasks weren’t in the requirements. I explain that her testing should be testing of the requirements as written and not of desired features, and that things not in the requirements are enhancements for the next version, and if its discovered that essential requirements were missed they shouldn’t be reported as bugs, but should be brought up as requirements revisions. I explain that her testing is testing of my programming of the requirements as written, and when we move to phase two of testing, the user testing, her users will point out the missing details and at that point new requirements or revisions will need to be made. And lastly I say, that all this is for her sanity and mine, because if she’s constantly checking for items that weren’t in the requirements, then she’s going to be very unhappy; while on my end, if I spend every day rewriting my work for items that I wasn’t told about instead of working on remaining items or other projects, I get very unhappy.

In my opinion, I laid things out very clearly and kindly. I never yelled or accused, I just simply pointed out issues with the process she was using and how everyone would be happier if she did things differently. Well… except the users, but they’ll never be happy until we invent the “Do My Job” button so they spend their days like George Jetson or Homer Simpson, pushing one button when its needed.

So later that day I got pulled aside by my boss and told that I should never yell at anyone about how they do their job because its not within my authority as a contract programmer. I tried to defend myself, but he didn’t want to hear it. He only wanted to hear that I would never do it again (despite not having done it in the first place).

That’s my story… people, especially at work, can’t handle the truth… Now I will present you with a bastardized quote of my own:

“Work like you only need money. Love like you’ll never be happy. Dance like everyone’s got score cards. Sing only when nobody’s listening. And lie like your job depends on it.”

I don’t really mean that… but seriously, when someone asks for the truth, try to be sure they really want to hear it. And never ask for the truth unless you can’t handle it. And by “handle” I don’t mean “make them pay for telling it to you”.