The Review Problem
Video games are a confusing form of media, one that has not yet fully been explored. This is especially the case when it comes to video game criticism and reviews. Every video game journalist has his or her own opinion on what video game reviews are and what they should be. Some believe that game reviews should primarily serve consumer interest. These reviews ought to answer the question, “Should I spend my limited financial and time resources on this game? Is it worth it?” Other reviewers value reviews as artistic critiques of games’ many gradable qualities, ranging from story to graphics and everything in between. For these reviews, the most important question is, “Is this game advancing the form?”
Most reviewers however, take the middle road. They review games by touching upon gradable qualities, spending equal amounts of time discussing gameplay and story. At the end of their reviews, they will often include a score. 4 stars! 90/100! Two thumbs up! These scores are the indicators of whether or not a consumer’s money would be well-spent on a new title.
A gulf sometimes forms between the content of the review and the final score the game receives. Commenters are often quick to point out the discrepancy and reviewers are just as quick to deliver a common refrain: “Don’t pay attention to the scores; read the review itself for what I actually think of the game.”
Why include a score at all if you don’t put any value in it? The response to that question is, of course, that many people don’t read the reviews too closely, and that the score is there to satisfy a certain segment of consumer. What the reviewers seem to ignore however, is that those scores are important to at least some people, and that those scores should be a measured reflection of the review’s content. So why is there sometimes a gap between the review score and the content of the review itself?
I contend that there is a fundamental flaw in video game reviews that has not been addressed by any community of game or mainstream journalists.
To illustrate this problem, let us look at a recent review of Far Cry 3 from The Polygon. In his review, Arthur Gies praises the game’s mechanics:
“Ubisoft Montreal has in turn built a world that allows for creativity and emergent strategy to unfold. The entirety of the game world is open to you as soon as the tutorial is over, though Far Cry 3 does a great job of slowly introducing it all to you via the early story missions. After that, though, you’re given carte blanche to wander or progress the story as you see fit.”
After his initial praise however, Arthur focuses his attention on what he considers to be a troubling aspect of the game: its story.
“…the story fails to sell Jason’s growing detachment or discomfort over that detachment, and what’s left often felt exploitative and pointless, dotted with misogyny and homophobia that only works for shock value. Is this ham-fisted presentation of problematic imagery and, honestly, gross stereotypes, an issue unique to Far Cry 3? Well, no — see Resident Evil 5, for example. It’s not even unique to video games. The portrayal of post-colonial themes and Western encroachment on other cultures is something that storytelling aimed at mass markets has had a lot of trouble with for decades. But that doesn’t make it easier for me to swallow it here.”
His appraisal of the story seems damning. Here is a game that contains themes of misogyny and homophobia. Arthur highlights these points, noting that he finds those themes to be hard to swallow.
His final verdict?
9/10
“The story’s sour notes mar what is, otherwise, one of the best games of the year. If you can look past its thematic problems, Far Cry 3’s story isn’t without genuine invention and surprise — there’s a hallucinatory aspect that allows for surprising, disorienting sections of narrative and character development, as well as gameplay moments that defy the basic reality of the rest of the game. When the story isn’t standing in its way, Far Cry 3 sees enormous success with its wide-open world and all the numerous things there are to do therein. Ubisoft has created a mechanically ambitious, exciting game.”
Even in the final text of the review, Arthur once again highlights his reservations about the story.
For reference, here is The Polygon’s review policy:
“Games are not scored until a review is written and finalized. Once a review is complete, the reviewer meets with a group of senior editors to determine which score on our scale properly reflects the text as written. We do not write with scores in mind.”
In other words, the score is supposed to match the content of the review, but there is a notable gulf in this situation. I believe that this gap between the content of the review and the final score can only be explained in one of two ways:
- Arthur does not actually place much value on the troubling undertones contained within the game. His motivations for mentioning these issues within the review—but not factoring them into the final score—are therefore unknown.
- Arthur chose to give the game a higher score based purely on its gameplay, not because he wasn’t troubled by the undertones, but rather because the review score is only supposed to be a reflection of gameplay, not story.
In both cases, the score is still clearly a misrepresentation of the content of the article.
It’s not hard to think of other situations where the score does not fully account for the gameplay and the art of the game. Think of a game with a terrible story, but amazing gameplay. Now think of a game with an amazing story, but terrible gameplay. Which game should get a higher score?
The answer is neither, but given the current review landscape, one of those two games would certainly get a higher score. In the eyes of the gaming public, one of those games would be quantifiable better than the other.

“The shooting was tight, but the homophobia was kind of grating. 9/10.”
The Solution
The best critics offer solutions, so here is my idea.
Games are fundamentally divided into two parts which are combined when reviewed: the curated art experience and the emergent gameplay experience.
The curated art experience includes everything that a developer presents to the player, but the player cannot manipulate through typical gameplay mechanics. These components include music, sound effects, graphics, art direction, narrative, characters, setting, themes and other artistic products. When we look at a painting, a sculpture or a building, we are consuming a curated experience. That is to say, when two people look at the same work, they are seeing the same colors, lines and patterns. Now, their impressions of the work are likely to differ—dramatically even—but the two observers can compare their experiences in a one-to-one manner.
The emergent gameplay experience is what sets games apart from art, but that quality is not limited to games alone. Sports and games have sets of rules in which players interact with each other or with the gameplay environment. Even though the rules are the same, the player experiences will differ dramatically. In video games, components of the emergent gameplay experience include all mechanics which the player must use to engage with the game world (i.e. control fidelity, non-aesthetic level design, ect.) and all mechanics which the player may affect or interact with directly (i.e. weapons, score systems, AI, skill trees, ect.).
These two distinct components of games must be accounted for in review scores. How? By separating them entirely.
In order for a review score to accurately reflect the text of a review, the score should consist of two scores, which are not to be averaged because the two parts are distinct and impossible to compare accurately. They need to be separated.
The Grubb Review Scale
Using the Grubber Review Scale,each game will receive two scores between -50 and 50. Think of the scale like a coordinate plane. (0,0) is the vertex of the scale and represents a game that is average in terms of both gameplay and the curated artistic experience. The x-axis represents the gameplay value of a given game; the y-axis represents that game’s artistic value.
Let’s look at games and see where they might fall on this review scale. Bear in mind that I believe that the value of a game is fundamentally subjective. Everyone has a different definition of “good” and “bad.” It is therefore completely reasonable for people to view my placement of certain games on the scale as being incorrect. My system is designed to expose a reviewer’s opinion more accurately, not to uncover a game’s objective worth—as if such a thing actually existed.

Superman 64, E.T.
Coordinates: (-50,-50)
The absolute worst of the worst. Completely lacking in coherent plot and without art direction that even comes close to standing the test of time, both of these games are artistically bankrupt. Add to this the fact that neither game has sensible mechanics, and you have the anchor of this scale.

Dear Esther
Coordinates: (-50,50)
An artistic triumph with wonderful level design, a fascinating plot and compelling themes not often explored in the video game medium, Dear Esther was a critical darling on linear review scales. Those scales were misleading however, because the game was almost completely devoid of mechanics. The player could walk around the world, but interaction was limited to such a degree, that the entire experience might very well have been a movie rather than an emergent player experience. I loved my time with this game, but I do think this example illustrates how my system might handle “mechanics-lite art house” games.

Pong
Coordinates: (50,-50)
Pong does everything it sets out to accomplish. It is the perfect gameplay experience, especially for its time. Having said this, the game is completely lacking in narrative and only offers rudimentary art and sound design. Some may argue that this minimalism would itself warrant a higher score, especially given that it is such an old game. However, I want to take into account the fact that the designers of the game were working with a limited palette and were more interested in creating a game as a proof of concept for mechanics, not in producing an art piece. I don’t think Pong is “saying” anything; I think it’s just trying to be a game.

Portal
Coordinates: (50,50)
I’m putting Portal as the ideal game on the Grubb Review Scale. Its art direction is minimalistic and sanitized, but the player understands that something ugly is going on behind the scenes. This contrast and duality are intentional artistic design choices. The main antagonist, GLaDOS, is a brilliant villain with a wicked sense of humor. The gameplay is unique and executed well. I believe that Portal is proof that the pinnacle of my scale can be reached, but as you might imagine, it does not happen often.

And what belongs at the vertex of the Grubb Review Scale? What game is so average that it can’t force itself away from the x or y axis?
Sniper Elite V2
Coordinates: (0,0)
Largely forgettable in every way, but also not particularly insulting in any way. Sniper Elite V2 was fun for a few levels, but got repetitive quickly. It also had some interesting set pieces but again, the entire experience, both from a gameplay and a an artistic experience, was not exactly inspired.
The irony of a game about snipers sitting at the vertex of my scale is not lost on me.

Will my system “fix” game reviews? No, absolutely not. There are a myriad of factors which contribute to a game being “good” or “bad.” I’m merely introducing two variables that I believe to be equally important to the overall worth of a game. Some people might only value multiplayer and polygon count. And those people are not necessarily wrong . After all, reviews are subjective.
I also believe that reviewers will likely not approve of a two-score system; they will think that it is too confusing for the average gamer. That might be true, but I like to believe that the average gamer might understand why two scores present a clearer final verdict.
Too many reviewers complain about scores and then resign themselves to the shortfalls of whatever system they are using. Even if reviewers disagree with this proposed alternative, I would still challenge them to improve existing scales. The consumer and the reviewer both have nothing to lose.
Ladies and Gentlemen, that does it for me for 2012. It’s been a pleasure writing, recording and drawing this year. Stay tuned for 2013. I’ve already got some great things planned!
Merry Christmas and Happy New Year!
Cheers,
Kevin
P.S. Follow me on Twitter @kpgrubb