More Reviews
REVIEWS Cut the Rope: Triple Treat Review
Cutting the ropes has never before made gravity so… pricey.

Faster Than Light: Advanced Edit Review
FTL's Advanced Edition is a natural fit for the iPad. But whatever platform you're playing on, the new content adds depth and variety.
More Previews
PREVIEWS Sniper Elite 3 Preview
Sending bullets through the hate.
Release Dates
NEW RELEASES Teenage Mutant Ninja Turtles: Out of the Shadows
Release date: Out Now

Trials Fusion
Release date: Out Now

The Amazing Spider-Man 2
Release date: 04/29/14

Bound by Flame
Release date: 05/09/14


LATEST FEATURES 4 PAX Indie Titles to Look Out For
We've given attention to all the higher profile games at PAX EAST, but now it's time for the indies.

Ouya's Best Games Coming to the Platform
The Kickstarter console is slowly establishing itself with a couple of creative gems on the way.
 
Coming Soon

LEADERBOARD
Read More Member Blogs
FEATURED VOXPOP danielrbischoff
Peace in the Era of Call of Duty
By danielrbischoff
Posted on 04/15/14
In a world dominated by violent media, Americans are no more eager to go to war than they were in the 1980s or the 1960s or the 1940s. Hasn't it always been someone else's problem? The overwhelming majority would rather go on thinking it had nothing to do with them and there...

Mind Over Meta

Posted on Friday, July 14 @ 12:14:15 Eastern by Joe_Dodson

GameRankings.com

By now you’re probably thinking metareview sites can’t get much worse, and you’re right. Gamerankings is functionally better than either Metacritic or RottenTomatoes. They automatically track tons of sites, they’re completely transparent with their results, and their averages actually are pretty good indications of whether or not you should research a game further.

Sometimes.

Then again, sometimes they post scores for imports and don’t tell you the game you’re looking at is only available in Japan. And since their aggregates aren’t based on a minimum number of reviews, sometimes their aggregates are really just one review. They used to count Ziff-Davis reviews twice, and at this very moment they’re counting a Gamespot review that doesn’t even exist. Uh oh.

When we began this piece, the main thing that bothered us about Gamerankings was the fact that you’d see scores from sites like PGNx Media (who?) sandwiched in between GR and 1UP. We're thrilled they didn't invent a magic formula like Metacritic, but it also seemed like their standard for site inclusion was, in a word, nil. In some cases, half the scores Gamerankings included in their aggregates were from small, unheard of fan sites with terrible writing and ghastly design, and we were certain the opinions of these amateurs were inflating aggregate scores.

So, we used a site known as Alexa.com to sort the reviews for various games into two groups: reputable and not reputable. Alexa ranks every site on the Internet based in large part on the propagation of its nasty toolbar. According to Alexa, Gamespot is ranked around 200, while Game Informer is at 37,000 (Right now, GR is at 12,000) It’s certainly not precise, sort of a wonky Nielsen rating, but we felt comfortable doing this because Gamerankings’ own Site Statistics page has a defunct Alexa field. We thought this especially telling, and decided to fix it for them, GR style.

Using an Alexa ranking of 40,000th as our cutoff, we split the review sources for several games and averaged the scores. What we found shocked us. If those below 40,000 and those above didn’t entirely agree, they were always within one or two points. In fact, the less reputable group was frequently more conservative with its grades, serving to slightly check pubs like Game Informer, GameSpy and Play, all of whom love to dish out 100s. To our relative surprise, Gamerankings’ aggregate scores illustrate consensus among critics great and small.

Remember, though, that this is their job. In spite of the fact that they’re better at what they do than Metacritic or RottenTomatoes, their success seems to be less the product of virtue than vice. Whoever wrote their Help section included this little gem of an example:

I used to include scores from Gamers.com in the average ratios, however, since their reviews come from Ziff-Davis magazines, it was including the same score twice. My Answer to people that used to complain about it was simple, you get a big magazine and a big website and your opinion can count twice too.”

So wait, if we ran the same review score on Game Revolution and, say, Game Evolution, Game Devolution and Game Helluvalution, it would count four times? Awesome. Indeed, Gamerankings displays a bias toward big publications even as their very own results prove them wrong. We should note that they eventually stopped counting Ziff-Davis reviews twice, albeit gracelessly:

I finally gave in and list(ed) Gamers.com under other reviews and only use the score from the print version. Oh and the day I did that, I started getting all kinds of emails for not counting them. Any change = 1/3 Happy, 1/3 Unhappy, 1/3 Never Notice.”

Gamerankings must be in that last third, because they still haven’t noticed that the Gamespot review at the top of their LocoRoco list was taken down three weeks ago due to an issue involving Gamespot’s UK site and their import review policy. Far be it from us to criticize someone for keeping a sloppy database, but that's pretty much all Gamerankings needs to get right. Sites from Penny-Arcade to Joystiq have commented on the fact that Gamespot’s LocoRoco review was rescinded, yet there it sits, broken and fictitious. Instead of counting Ziff-Davis reviews twice, they’re counting a Gamespot review that doesn’t exist for a game that doesn’t come out in North America for a month. I could mention that Gamerankings and Gamespot are both owned by CNET, but why bother?

The reason Gamespot pulled their LocoRoco review was because it's based on an imported copy of the game. Since their site reviews North American releases, and the North American version is due in September (and under embargo, we might add), they decided to wait and “see if any content has been added/changed/removed and so on and so forth,” as Gamespot’s Alex Navarro told Joystiq.com. Good on you, Alex.

Gamerankings, on the other hand, makes no such distinctions. As of 8/4/06, every review in their LocoRoco list, aside from the one that doesn’t exist, is for either a UK or Japanese version. However, both are included in the same aggregate, in spite of the fact that one is in Japanese. When the North American version ships, it will presumably be lumped in with the UK and Japanese versions as though they were all North American reviews of North American games.

Before you burn the flag and protest, know that this isn’t about patriotism - it’s about the fact that different nations have different cultural standards. In Germany, for example, games that include either blood or Nazis are banned. Publishers know this and they make changes to their products accordingly. Gamerankings, on the other hand, doesn’t even seem to know what the word “standard” means. If they did, their minimum number of reviews for an aggregate score might be higher than one. Behold.

We aren’t fond of the way they break down letter grades, either, although this is admittedly the least of their troubles. Basically, they get A’s and B’s right, and then they skew C’s and D’s too high. Their system is arbitrary and a little retarded, and so is their defense of it as seen on their Help page:

I stand by my system, trust me, I have put more thought and tried more things than any of you ever will, and I feel this is the best way to reflect the scores.”

Evidently, the only thing Gamerankings’ founders haven’t tried is high school English.

Okay, that’s harsh, but so is skewing our grades and featuring imaginary reviews of foreign video games. They’re the best aggregate site of the lot for their transparency and breadth of sources, yet occasionally they completely misrepresent games’ critical responses due to their lack of a review limit and their inability to tell an import from a final release. For that we give them an aggregate score of 50 out of 100. Even though that’s just our opinion, we want you to know it’s shared by everybody in North America, Marrakesh and Burma.

 


comments powered by Disqus




More On GameRevolution