By now you’re probably thinking metareview sites can’t get much worse, and you’re right. Gamerankings is functionally better than either Metacritic or RottenTomatoes. They automatically track tons of sites, they’re completely transparent with their results, and their averages actually are pretty good indications of whether or not you should research a game further.
Then again, sometimes they post scores for imports and don’t tell you the game you’re looking at is only available in Japan. And since their aggregates aren’t based on a minimum number of reviews, sometimes their aggregates are really just one review. They used to count Ziff-Davis reviews twice, and at this very moment they’re counting a Gamespot review that doesn’t even exist. Uh oh.
When we began this piece, the main thing that bothered us about Gamerankings was the fact that you’d see scores from sites like PGNx Media (who?) sandwiched in between GR and 1UP. We're thrilled they didn't invent a magic formula like Metacritic, but it also seemed like their standard for site inclusion was, in a word, nil. In some cases, half the scores Gamerankings included in their aggregates were from small, unheard of fan sites with terrible writing and ghastly design, and we were certain the opinions of these amateurs were inflating aggregate scores.
So, we used a site known as Alexa.com to sort the reviews for various games into two groups: reputable and not reputable. Alexa ranks every site on the Internet based in large part on the propagation of its nasty toolbar. According to Alexa, Gamespot is ranked around 200, while Game Informer is at 37,000 (Right now, GR is at 12,000) It’s certainly not precise, sort of a wonky Nielsen rating, but we felt comfortable doing this because Gamerankings’ own Site Statistics page has a defunct Alexa field. We thought this especially telling, and decided to fix it for them, GR style.
Using an Alexa ranking of 40,000th as our cutoff, we split the review sources for several games and averaged the scores. What we found shocked us. If those below 40,000 and those above didn’t entirely agree, they were always within one or two points. In fact, the less reputable group was frequently more conservative with its grades, serving to slightly check pubs like Game Informer, GameSpy and Play, all of whom love to dish out 100s. To our relative surprise, Gamerankings’ aggregate scores illustrate consensus among critics great and small.
Remember, though, that this is their job. In spite of the fact that they’re better at what they do than Metacritic or RottenTomatoes, their success seems to be less the product of virtue than vice. Whoever wrote their Help section included this little gem of an example:
“I used to include scores from Gamers.com in the average ratios, however, since their reviews come from Ziff-Davis magazines, it was including the same score twice. My Answer to people that used to complain about it was simple, you get a big magazine and a big website and your opinion can count twice too.”
So wait, if we ran the same review score on Game Revolution and, say, Game Evolution, Game Devolution and Game Helluvalution, it would count four times? Awesome. Indeed, Gamerankings displays a bias toward big publications even as their very own results prove them wrong. We should note that they eventually stopped counting Ziff-Davis reviews twice, albeit gracelessly:
“I finally gave in and list(ed) Gamers.com under other reviews and only use the score from the print version. Oh and the day I did that, I started getting all kinds of emails for not counting them. Any change = 1/3 Happy, 1/3 Unhappy, 1/3 Never Notice.”
Gamerankings must be in that last third, because they still haven’t noticed that the Gamespot review at the top of their LocoRoco list was taken down three weeks ago due to an issue involving Gamespot’s UK site and their import review policy. Far be it from us to criticize someone for keeping a sloppy database, but that's pretty much all Gamerankings needs to get right. Sites from Penny-Arcade to Joystiq have commented on the fact that Gamespot’s LocoRoco review was rescinded, yet there it sits, broken and fictitious. Instead of counting Ziff-Davis reviews twice, they’re counting a Gamespot review that doesn’t exist for a game that doesn’t come out in North America for a month. I could mention that Gamerankings and Gamespot are both owned by CNET, but why bother?
The reason Gamespot pulled their LocoRoco review was because it's based on an imported copy of the game. Since their site reviews North American releases, and the North American version is due in September (and under embargo, we might add), they decided to wait and “see if any content has been added/changed/removed and so on and so forth,” as Gamespot’s Alex Navarro told Joystiq.com. Good on you, Alex.
Gamerankings, on the other hand, makes no such distinctions. As of 8/4/06, every review in their LocoRoco list, aside from the one that doesn’t exist, is for either a UK or Japanese version. However, both are included in the same aggregate, in spite of the fact that one is in Japanese. When the North American version ships, it will presumably be lumped in with the UK and Japanese versions as though they were all North American reviews of North American games.
Before you burn the flag and protest, know that this isn’t about patriotism - it’s about the fact that different nations have different cultural standards. In Germany, for example, games that include either blood or Nazis are banned. Publishers know this and they make changes to their products accordingly. Gamerankings, on the other hand, doesn’t even seem to know what the word “standard” means. If they did, their minimum number of reviews for an aggregate score might be higher than one. Behold.
We aren’t fond of the way they break down letter grades, either, although this is admittedly the least of their troubles. Basically, they get A’s and B’s right, and then they skew C’s and D’s too high. Their system is arbitrary and a little retarded, and so is their defense of it as seen on their Help page:
“I stand by my system, trust me, I have put more thought and tried more things than any of you ever will, and I feel this is the best way to reflect the scores.”
Evidently, the only thing Gamerankings’ founders haven’t tried is high school English.
Okay, that’s harsh, but so is skewing our grades and featuring imaginary reviews of foreign video games. They’re the best aggregate site of the lot for their transparency and breadth of sources, yet occasionally they completely misrepresent games’ critical responses due to their lack of a review limit and their inability to tell an import from a final release. For that we give them an aggregate score of 50 out of 100. Even though that’s just our opinion, we want you to know it’s shared by everybody in North America, Marrakesh and Burma.