posted 12-12-2001 03:03 PM               

Just a couple of points:

#1: MVC2 has 3 people per team, so the numbers don't mean the same thing as in say regular SF2. A tier of 10 characters in SF2 is very different than 10 in MVC 2 or CVS2.

#2: Paying play-testers well usually doesn't make a lot of economic sense IMO. Evaluating how good a playtester is going to be is difficult, they have to be good at the game and be able to properly express what they find. It's hard to find people like that that don't already have other jobs. And you need a fair number of playtesters. For example in Starcraft look at the number of maps, 3 races, different strategies and play-styles, etc. For something like a MMORPG it gets much much worse.

Wuth PC games the initial releases these days are essentially beta tests that people actually *pay* to test out! PC games, especially MMORPG, take a lot of flak at release time, but I would say that that is due more to the fact that the games are simply unfinished and a lack of planning than a lack of testing. For what I gather most of the major problems with MMORPG launches could be found with minimal playtesting or simply some foresight.

#3: MMORPG's also are a bit different in that in a Capcom game you can adopt new strategies and characters readily, whereas in MMORPG's people spend a long time building up and switching is a real pain. So the Capcom style balance doesn't really work out as well. You also have to beware of the effects of people who rush to judgement and like to whine. Depending on the game you may find many of them bailing out early before some counter-tactic is found. Also general perception usually lags way behind actual results. For example many people think that parries break SF3, *including* 3s, which is quite obviously false to anyone who plays the game or bothers to watch tourney videos.

"Zileas," a very good Starcraft player, says he has observed this with that as well, where perceptions about good strategies or units are often months behind expert results. (Many many people still swear by the Zergling rush...) So sometimes these corrections don't happen quickly enough. People complain about Cable in MVC2, or just helpers! Helpers are Too Good!"

(Zileas also has some very specific thigns to say about RTS balance, including some formulas, it's pretty interesting)

#4: You are exactly right that competitive games need much more bug testing and balancing. Info spreads faster and the incentive to break things is much much higher. I think there are a lot of professionals who for whatever reason aren't really aware of this. Also I would say the MMORPG's *are* competitive games, where the competition is raising levels faster, getting more stuff, etc.

posted 12-12-2001 09:52 PM            

quote:

---

#2: Paying play-testers well usually doesn't make a lot of economic sense IMO. Evaluating how good a playtester is going to be is difficult, they have to be good at the game and be able to properly express what they find. It's hard to find people like that that don't already have other jobs. And you need a fair number of playtesters. For example in Starcraft look at the number of maps, 3 races, different strategies and play-styles, etc. For something like a MMORPG it gets much much worse.

---

Maybe I'm misreading your post, but you don't want to pay testers well, and then you complain about difficulty finding qualified applicants to test full-time?

As for MMOGs, the complexity is a design issue and not really confined to MMOGs at all. Granted, there are network performance issues to deal with, but those usually aren't the big troublemakers. More often it's the interdepency of game objects that causes problems, which gets back to Sirlin's point about purity of purpose.

Oh, and I'm assuming you're only talking about balance testing, and not software quality testing as a whole. Unfocused design and will kill a game long before testing is even a factor.

posted 12-13-2001 01:21 PM               

MMORPG just require many more playtesters to uncover problems. The issue *is* complexity, but in general a MMORPG has much more complexity than any other game. You have all the different races, classes, skills, areas, level you can be at, items, etc.

How well can you pay a large team of people to test that? Actually some MMORPG's just have a test server people can choose to play on without getting paid anything. When a MMORPG gets released it is going to be well tested, unless is has a free beta test that lasts for a while.

There are problems with that too though. Ifthe beta test sucks you might scare off people and create bad word of mouth, and the longer you test the less new and exciting your game is. This is what happened to "Shatterred Galaxy," it was *more* popular during testing than after release. People played their fill while it was free, while others got pissed off by constant rule changed and bugginess.

As far as paying in general, it would be worth it to pay if you could find people that you *know* will be good. There are professional engineers and level designers and artists, but not really professional game testers. Larger places may have standing QA departments, but that doesn't make sense for a lot of projects and actual testers within QA divisions tend to come and go pretty rapidly.

Generally you can't really test for balance until tweaking is nearly done and all the systems are in place. So your balance testers are going to be around for only the very end of the project. That isn't the sort of schedule that attracts professionals, jumping from one location to the next every month. How many really articulate, thoughtful individuals are you going to find who are willing to work that sort of job? Why don't they have other, better jobs with some stability, benefits, etc?

I don't think payment really matters unless you start talkng about realy good pay. As an engineer I'm not going to be a tester unless they pay probably MORE than an engineering job. The same can be said of someone with design skills, writing skills, etc.

If your game is SF2, it may be possible to have a group of a few well paid fans who can cover all the basics between them. (At least figure out that Sagat and Blanka are too scrub-friendly...) But people's ideas of balance change over time. It took a while for people to perfect the Zergling rush, even with thousands of people playing constantly. How do you find a QA person competent enough to find that sort of thing on their own?

One of the things that invariably happens with Capcom testing is that they weaken things they find in testing and end up screwing that character over. Birdie in A1 had an overhead fierce in the beta test, but that was "too good." (As in, someone did well with Birdie one day) So he lost it, welcome to suck-ville. Same with SF3 Alex, Shuma-Gorath in MVC2,etc. Somebody hits on something "overpowered" which is in fact not overpowered at all; people just haven't gotten used to it yet. And these are expert players. If you *were* going to pay well, these are the people you would be paying!

Guile is too good, Akuma is too good...in MVC2!!! You want to spend your money on that? That isn't to belittle the players who said those things, they are good players. But to get good results out of them you would have to get a lot of them and employ them for a while. Add paying well on top of that and there goes your money.

posted 12-13-2001 02:01 PM            

I agree that public betas can be a great tool to help resolve balance issues, but I hate it when:

1.) developers ignore balance issues discovered by players during open beta, which basically means that...

2.) paying customers end up playing an unfinished game and uncovering major design flaws.

Sorry about the rant; it's just that I've participated in a couple of public betas that ended up not improving the retail product.

posted 12-13-2001 02:57 PM               

The article said to pay testers more and that it's unfortunate that they are usually treated poorly, but I really should have been more explicit about how testers should be treated, recruited, and used by a game company. They're usually second class citizens, and sometimes actually are second class employess, becasue the lifestyle associated with the job sucks so much. As margalis says, smaller companies can only keep on a testing team for short periods of time at the end of the project.

Next, I should have been more explicit on how the testers have to basically simulate the entire gaming community. They have to be able to "cover all the bases" in terms of play style. Some need to be innovators, others need to perfect tactics discovered by the innovators. Some have love running away, stalling, being annoying. Others have to love rushdown and offense.

The biggest thing pointed out by margalis was beta testing. How did I manage to leave it out completely? Oops. With it comes the usual caveats of "you often have to do it since you can't come close to simulating the gaming community with your 5 testers" and "it sucks to do it since you ruin the newness of the game and its first impression to many players, etc."

The fact that experts are ahead of the pack when it comes to strategy is also worthy of note. That's why your testers need to be super-elite, but yeah, there needs to be some semblence of balance at low levels of skill, too.

So basically, I should revise that article. Too many things left out and too many things unclear. It will probably end up being pretty damn long though. I wonder if that matters.

And just think...I even left out some *other* stuff on purpose so I could write Part 2 and Part 3!

PS--I know Zileas and I even wrote an article about him! http://www.sirlin.net/Features/feature_AoW4-DivideAndConquer.htm

The "zilean school of asskicking" link is broken though. Anyone have a good link to it?

Thanks for the discussion guys!

posted 12-13-2001 08:12 PM               

I agree about Beta testing. The thing you have to remember is that many companies are willing and in fact planning to ship products that are unfinished. There is even a vocabulary starting to build up around that concept, for example people will say "we won't have that when we go live..." When the hell WILL you have it then, and why am I paying for it?

Essentially they give you an unfinished trial version that lacks the features they have promised, but you pay full price. At least they could be honest about what you are getting, rather than making you buy the game to realize that skills and classes they said were there either aren't at all or don't work properly.

There *is* something to be said for adding features to a game later, especially with rising dev times and costs. But the features they promise you get IN THE BOX better be in the damn box and working properly.

It's funny if you look at the Anarchy Online launch, during the beta everyone said it was horribly broken. When previewers pointed out how broken it was the company (Funcom) wrote nasty letters to them saying it was disengenious to give opinions of a preview copy. (What exactly is the point of a preview if not impressions then?) They said that the beta test was running old versions of the code in "debug mode" and that all the problems were fixed...uh huh...of course there was no test of these "fixes" as you figure there must be to make sure they *do* actually fix something.

And surprise surprise, every complaint about the beta applied to the final version as well! Who would have guessed it?!

Beta testing isn't really the problem in these cases. The problem is that developers bite off more than they can chew and that most developers are fairly clueless. This fact dooms them way before beta testing ever takes place.

posted 12-13-2001 08:18 PM               

> Give every character something “so good that it’s broken.” Include so much variety that by the time anyone ever figures out which broken thing actually does ruin the game—the game will be dead by then anyway.

I know this is a theory, but I'm not sure if this is particularly accurate. I think Capcom does design things around the idea of "give everyone something useful", but they clearly don't give everyone "something so good it's broken". It seems to be quite arbitrary on their part. Sentinel, for example, has many more good things going for him than, say, Charlie.

posted 12-13-2001 11:30 PM               

I don't know the first thing about Samurai Showdown 3, so I'll take your word for it.

About MvC2, yes Sentinel has much more going for him than Charlie. And they certainly didn't give EVERY characters something brokenly good. Maybe they just try to give everyone radically different stuff and they don't even know which of it is incredibly broken. Or maybe they do know. I have no idea. But there is a definite "Capcom flavor" of "highly diverse brokenness" they have going.

posted 12-13-2001 11:59 PM               

KI had good balance, everyone had the same "low block" and that was all that was really important.

I do know what Ultima is saying though. There are games where a lot of characters have really stupid and frustrating strategies. Since enough have them it is "balanced" in a way but it sure as hell isn't fun.

As far as more recent Capcom games, there are so many characters and grooves and helpers and whatever I think they just throw up their hands. In a way I don't blame them. Just give everybody some stuff and hope for the best. If I was designing a modern Capcom game I would probably do the same thing. For each character come up with *something* that sets them apart and then throw them to the wolves. In other words as long as one character isn't worse in *every* way than some other character that would almost have to be good enough.

I *do* think they could do a better job of course. If you make a character that takes 75% normal damage and another that takes 130% you'd better think of some real advantage character b has over a.

In the case of Blanka/Sagat, it isn't obvious that Blanka would be as good as he is. You can punish the ball, he has no projectile, no dp like special AA, no command throw. Why *IS* Blanka so good? And Sagat doesn't look any different on paper than in CVS1 or in the Alphas. Remember when CVS2 first came out everyone was saying that Blanka got toned down and they didn't see him as a powerhouse anymore...

It's tough because part of makes Sagat good is that his competition got worse. As I mentioned earlier, it takes people time to adjust. Remember when Daigo was using Ken/Ryu/Akuma. Comparing uppercuts, the shotos got worse and Sagat's stayed the same, so whereas before Sagat had priority issues now he is tied or ahead of the shotos. Add in parry and just defend and Sagat uppercut comes out ahead. But it takes people time to realize things like that, the shotos are typically good and people clung to that. There were a lot of people saying "I'm not sure, but I figure there is still a lot to the shotos that makes them worth playing."

Vega is another good example. (Claw and mask vega that is) Vega basically stayed the same. Sounds like no big deal until you realize a lot of chars got worse in some ways. Shotos got worse, Nak got worse, King got worse, Guile got worse...Vega improves a bit and so relatively speaking he improves a fair amount overall. Same with Geese, Bison, Rugal. They stayed the same or went up in a game where a some other characters dropped. But people will always think "character x was good in the previous game so they should be good here too, I just know it!"

So what I am saying is that there were a lot of people saying "sure Blanka/Sagat is good but once we adapt to the shotos they'll be good too." So even if that imbalance was found it's not clear it would have been seen as a problem and fixed...who knows it may still pan out.

posted 12-20-2001 03:53 AM            

-The "Capcom Principle". I would be more inclined to call this the SNK principle. My favorite SNK game, Samurai Shodown 1, is a perfect example of the "give everone something totally broken" genre.

*Earthquake cannot be thrown. He can get ahead with one hit and turtle the rest of the game.

*Charlotte can lock you in block stun with repeated crossover triangle slashes

*Gen-an has an unretaliable, high block damage version of the blankaball. He can just do this over and over.

*Haohmaru has beyond insane damage and dizzy potential.

*Jubei has a redizzy, and an unbeatable air attack (jump short).

*Wan fu can use low strong to beat any air attack.

-You can use the CvS ratio system to balance any fightng game "perfectly". Get the best players to play a matrix of fights, each char vs each char. If A beats B 6-4, then increase B's damage until it is 5-5. Once all fights have been balanced, normalize damage.
I'm pretty sure you can make an entire fight matrix go 5-5 for every fight by continually applying this method.
In the case that you end up unable to raise B's damage because that would imbalance the fight of B-C, then you can just increase B's damage those fights where B is underpowered. So there's an algorithm for balancing any fighting game. MVC2 is no longer an excuse

-Julien Beasley