2.0 Vs 2.01
RyoOhki
Join Date: 2003-01-26 Member: 12789Members
Join Date: 2003-01-26 Member: 12789Members
<div class="IPBDescription">Playtesting methods</div> *prays this won't turn into a flame-fest*
2.0 was playtested by a fairly select group of people in the community, namely official playtesters and "veteran" NS players. We all know this of course. We also all know that 2.0 wasn't balanced.
2.01 was playtested by the community as a whole. The result was the most balanced version of NS to date. Yes, it still has some issues to iron out, but it's the closest thing to perfect balance we've managed so far.
Hence, the question must be asked, is it best to playtest builds of NS with the community? Certainly a wider range of players, with a wider range of skills, will be availible to examine and test the patch.
Now it may be that the results of the two differant forms of playtesting are unconnected and occured by chance. But I think there is a definite link between the two types of playtesting and the results that occured.
This isn't ment to poar scorn on the playtesting abilities of the pts or vets. Simply that, when using only the very best players, you're likely to end up with a product that doesn't play well with the community. People will likely say the reverse is also true. However, I would maintain that by playtesting with a very wide range of players, you end up with not only a much wider range of feedback, but a patch that is more likely to satisfy the community as a whole.
2.0 was playtested by a fairly select group of people in the community, namely official playtesters and "veteran" NS players. We all know this of course. We also all know that 2.0 wasn't balanced.
2.01 was playtested by the community as a whole. The result was the most balanced version of NS to date. Yes, it still has some issues to iron out, but it's the closest thing to perfect balance we've managed so far.
Hence, the question must be asked, is it best to playtest builds of NS with the community? Certainly a wider range of players, with a wider range of skills, will be availible to examine and test the patch.
Now it may be that the results of the two differant forms of playtesting are unconnected and occured by chance. But I think there is a definite link between the two types of playtesting and the results that occured.
This isn't ment to poar scorn on the playtesting abilities of the pts or vets. Simply that, when using only the very best players, you're likely to end up with a product that doesn't play well with the community. People will likely say the reverse is also true. However, I would maintain that by playtesting with a very wide range of players, you end up with not only a much wider range of feedback, but a patch that is more likely to satisfy the community as a whole.
This discussion has been closed.
Comments
-edit- Alright i'll add a bit more substance,
Advantages to a public beta system:
- Greater number of testers means more likely to run into obscure bugs, or hardware/platform specific bugs.
- Larger amount of statistical data
- Good way to stress test, or test delivery systems / ease of use / problems that less able users may run into.
Dissadvantages to a public beta system:
- Large amount of redundant or useless feedback to wade through
- Difficult/Impossible to stage quick tests of ideas or tweaks without a dedicated testing team. (Ex. during 2.0 testing it was not uncommon for a dev to call out for a small group of players, to get them on a server build he had recently made a small modification to. These changes would be tested, and in many cases adjust mid-game while testers played with the new changes. In this way, a developer can go through many different ideas, and have all of them tested in the space of an afternoon.)
- Any public beta released, regardless of the disclaimers shipped with it, will immediately become the standard version of the game being run by servers. As well as problems with new players sampling a version of the game that is only intended as a beta test, this also brings up problems of support. Beta server operators have to be supported, as well as end user problems.
- Since it takes alot of time and planning to get a beta build out to the public, along with the above factors, it is not worth releasing a new build just to test a small change. And it would be an incredible waste of time to release a build that was immediately deemed unplayable because of a bug that could have easily been spotted by a small, closed testing team. (This happens more than you could imagine).
Advantages of a closed beta system:
- Testing can be made on very small tweaks, very regularly. The time from a new build being created, to a test server having it up and running, to a testing team playing it and pointing out any obvious problems is far, far shorter than the time it would take to roll a release out to an entire community. This means if you release a build that just happens to have a game halting error, you lose very little in time or public relations.
- Testers can be instructed to test specific events or perform specific tasks that a developer has queries about.
- 'Special builds' that would never be suitable for public release can be tested easily, eg. builds with certain cheats enabled, with placeholders or with issues that would destroy a public game.
- You aren't swamped with useless feedback. If a build crashes everytime you try to go fade, this is spotted within 30 minutes, reported on a single centralised forum, and goes onto the known bugs list. As opposed to having to deal with 300 emails from faceless NSPlayers who think they're being helpful by reporting a bug they just discovered.
- Feedback regarding gameplay is far more objective
Dissadvantages to a closed beta system:
- Not great at running into the more obscure minor problems that only show up after a large deal of testing.
- Doesn't give you a test run of trying to get a thousand different users and a hundred different servers, all with varying hardware/platform/level of competance up and running.
Summed up, public beta testing is completely unsuitable for building a game from the bare bones, upwards. The game you finally get to play on your servers is the well-polished final result of millions of small code changes and slow evolution of features. An unfinished game in its early stages is not a pretty sight, and is certainly not suitable for public consumption. In its early stages, a game is more than likely unplayable.
Where public testing can be used, is later on in development, when you have a solid working foundation, and want to make tweaks that mainly influence gameplay, as well as possibly track down the more obscure problems. The public testing of 2.01 was in imo a good move, but calling it beta testing is a little misleading. These were more like official builds, mostly bug-free and definately playable, the only part of these builds that was really 'beta' were the gameplay changes. These changes were experimental, but since the builds themselves were quite solid, it was easy for the public to accept them. They became more like an official stopgap patch before 2.1.
people whine too much.
even the changelog for 2.1 was removed a few days ago.. so there's no way flayra gonna release beta for the public..
If people were perfectly satisfied with everything they were given, we would still be throwing rocks at our evening meals.
people whine too much.
even the changelog for 2.1 was removed a few days ago.. so there's no way flayra gonna release beta for the public..
<!--QuoteEnd--></td></tr></table><span class='postcolor'><!--QuoteEEnd-->
Maybe you missed the rather large public testing of 2.01. Looking at the results of that, it went brilliantly. People don't whine too much, people tested the patch and gave feedback. The result was 2.01, argueably the most balanced NS version ever released.
There were no such changes in 2.0->2.01, and the underlying patches were relatively minor overall. Personally, I'd like to see a version of 2.0z released (possibly as NSv1.1?) just to show what it had going at the time. It was seen more as 'stagnant' though, as Marines and Kharaa actually had to defend a resnode for two whole minutes, meaning taking over an area and defending it, making it very area-control oriented.
I'm still of the opinion that removing Marine RFK (which would kill 45-minute-plus Marine turtling endgames, as well as decrease the painful over-effectiveness of a shotgun rush) and adding a 30-90 second Kharaa RT activation time (to counter the full-map Gorge rushes that happen now) would be leaps and bounds toward stabilizing the game, and giving yet another distinct difference between the sides.. making Marines oriented on RTs for res (which they can afford to be, with a communal res pool), and Kharaa more oriented toward attacking constantly (allowing Skulks to go back to being suicide-soldiers, as well as virtually guaranteeing that you will not have people just sitting around waiting for res to drop Onos... at least, not successfully).
The game isn't designed for pro players. Think again.
It's tested by 3 members of every pro clan.
<b>Why</b>?
Fairly simple. Who are the people that find bugs, exploits, unbalances and flaws in a game? The public, or the select few that play the game hardcore. Daily. For hours and hours. The answer to that question's pretty obvious. Oh, and don't go I PLAY HARDCORE TOOO!!111, that's just besides the point. Most of the time, simple bugs like processing sieging and unbalances like that, are found by a few select people and then spread to the populace who see it in use. It's tested by pro players, it's designed for the public. The fact that the public is too stupid in most games to listen to their commander, is a sad fact.
A public playtest would be pretty horrible. Check the tech support forum, there's people that download the .dll of the server side 2.01 everywhere, messing up their game, there are LOADS of people screaming DIS IS UNBALANCT in the open beta forum, there's speculations everywhere about balance, and everyone's complaining rather than trying to contribute.
I don't see how the public playtesting could bring NS more balance. True, at some times I've seen bugs in versions of NS that I thought "How the hell did the PT's miss that?", but it happens. I sincerely doubt that even if the public tested every release, it would be bug free. Leave the playtesting to the PT's and vets.
No i didn't miss it, i was running the 2.01x patch on our server.
and the 2.1 changelog was removed for a reason...
I've heard this stated so many times and never really bothered rebutting it before. However, to put it plainly, it's nothing more than a nice story. Z wasn't balanced. The win ratios in the 'pub style' test games might have been pretty close, but the few scrimms played in the builds around the 'Z' era pointed out its serious problems. It wasn't really any better balanced than the alternate build, infact both suferred from practically identical issues, despite the significant changes to the resource system.
Everyone likes to point out that months were spent on balancing the game, many people seem to think that after so much testing it must be impossible to not end up with a perfectly balanced final build. The nice way of explaining this is to say we had a build that was well balanced, but decided to go for a more fun build instead. That's a really nice story, and yes za was chosen because it had far better flow than z, but Z was never close to a balanced game, and given a few weeks of solid clan play would have fallen apart just as 2.0final did.
The reality is, people don't understand just how much there is to change, test, tweak, retest, change and test again when you completely rework the entire foundations of a game, as was done with 1.1 over 1.04. 2.0 isnt the result of small tweak after small tweak to steadily to try balance 1.04, it was a case of throw everything out the window and work from there. It also didn't start with one idea and steadily become balanced, new factors were being introduced all throughout testing that added more to the game yet in the same instant threw the whole idea of balance into a muddle again.
"Months worth of testing", had you started with a solid build such as 1.04, and your only intention was to balance out win ratios, would surely of resulted in a pretty well balanced build. But this isnt even remotely close to the 2.0 development proceedure. It wasn't a steady progression from chaos to order, it was chaos, then a different brand of chaos, then chaos with a smidgen of order, then chaos mark-B, everything on its head again. It isnt untill very late in development, when Flay starts to say 'Ok i think we've got the foundations of the game down now, lets start tweaking' that you can really start to refine the game and squeeze out the balance issues. And even then, as happened in 2.0 development, something comes along like -za and you're set back a few steps again.
Oh, and i edited my first post, which is so far back now that most people wont notice if i dont mention it.
If you want a mod focused on veterans and screwing the newbs, well you've found the right way to do that with asking only veterans to playtest. There is a good reason why 2.0z is more balanced.
2.0 was playtested by a fairly select group of people in the community, namely official playtesters and "veteran" NS players. We all know this of course. We also all know that 2.0 wasn't balanced.
2.01 was playtested by the community as a whole. The result was the most balanced version of NS to date. Yes, it still has some issues to iron out, but it's the closest thing to perfect balance we've managed so far.
Hence, the question must be asked, is it best to playtest builds of NS with the community? Certainly a wider range of players, with a wider range of skills, will be availible to examine and test the patch.
Now it may be that the results of the two differant forms of playtesting are unconnected and occured by chance. But I think there is a definite link between the two types of playtesting and the results that occured.
This isn't ment to poar scorn on the playtesting abilities of the pts or vets. Simply that, when using only the very best players, you're likely to end up with a product that doesn't play well with the community. People will likely say the reverse is also true. However, I would maintain that by playtesting with a very wide range of players, you end up with not only a much wider range of feedback, but a patch that is more likely to satisfy the community as a whole. <!--QuoteEnd--> </td></tr></table><span class='postcolor'> <!--QuoteEEnd-->
The "whole community" didn't have to take a hive race restriction game, and make it wide open to have onos at 1 hive. They didn't have to balance that. They didn't have to test marine tech lines, alien gorge sucking of empty res nodes, etc etc. There is no way in hell a public 2.0 would have worked. All that happened in 2.01 was public beta servers and tweaked what we worked on for many months. That is the easy part. Getting a game to work is the hard part. Not only that but during the 2.01 beta, the veterans are really the ones that still made the input for the changes. Noone really worked hand in hand with flayra but he still came to the veteran / private beta forum and talked to us (And the playtesting channel on IRC). It just made it easier to get beta games to play since they were on so many servers.
And You are completely wrong about 2.01 being the most balanced and fun version. A) It is not the funnest version by any means. B) It is not balanced by any means. Every top team goes into their alien half and says "Alright we got a free win here"...then when they play their marine half its like "Ok guys play your best, we have a chance at this....just play good". WHY? Because the alien half is 100x easier to win then the marine half.
Thus.....the beta test going on right now. 2.1H (NS:Combat and NS changes). And 2.1H IS the most balanced version there are just some things I don't like about it. =)
Balance you might be able to argue about, though with the two teams being so different it's actually quite hard to accurately measure that as well.
Back on topic:
The public testing only went so well because it had such a strong foundation from the private testing. It's a standard progression of testing in software development. Both steps are valuable and need to be done in order. If you skip one, your development cycle and end product will suffer (or you can luckily hit a home run and get it right the first time, heck who needs testing? <!--emo&:p--><img src='http://www.unknownworlds.com/forums/html/emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif'><!--endemo-->)
Teoh's above edited post explains it well, and his comment on standing on the shoulders of giants is fitting.
Sheer Brilliance <!--emo&:D--><img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'><!--endemo-->
speculation about a version that you have not even played or even understand is best avoided.
The .0X changes (1.01, 1.02, 1.03, 1.04, 2.01) are opened.
The rough of the work is done in closed beta, and the smaller tunes are done by the mass. Just imagine how slow it would be if everyone had to download 2.1, test it, say his own little word, have the devs reading all this, starting again...
The vet program was a sign of Flayra's brilliance as it's obvious that we needed community members to balance the game as well as obviously good players to force out the very imbalancing strats, such as the old 1.0x jetpack/hmg combo. Now, where would Flayra find such players? Perhaps the ones who have outshined as quality players, both in skill and teamwork.
With the added help of the playtesters, the 2 perform a symbiotic testing of alpha style material besides just game balancing.
Perhaps next time before I make inane comments about the quality the veterans program gives us, I'll stop a moment to think about how insignificant my resources/information/ and blatantly ignorant ideas are.
Oh, and I <3 Frikk.
We never have forced our opinions upon anyone. We do not have Flay and the PT's locked up in a room, only doing what we want. We do not have any more control over this game than the PT's. Probably less. Flay does what HE wants, not what we want. Simply because a program was started to add a higher skill base to the testing does not mean that the whole game is focused on organized play.
Would you rather have someone totally new to the game try and test? Have to explain to them how the armory works? How to command? Do you see how that would be totally detrimental to testing?
You want public testing. Yet, when a small change is made, you leap to arms. You complain enough that we can't post change logs. Public testing of the builds that we have now would be horrible. Combinations of bugs, poorly written 'OMG NURF THIS IT SUCKS', the typical anger that gets displayed here quite often would make for a totally unsavory experience. Experiments like making all gorges spawn neon purple would be impossible, because everybody jumps to conclusions here so fast.
For those of you that hate us because we have beta testing privileges and you don't, I'm sorry. This isn't something that I asked for. Ever. Just because we come from clans, does not mean we can't be both dedicated to the game and our clan. On several occasions when Grep has gone bug hunting recently it's been him, a pt or two, and a vet or two. We really do try and help in what way we can. Irrational jealousy is far too juvenal, grow up some please.
We don't hate veterans...or at least I don't. I just see them as a special interest group.
Playtesting with the vets having the majority of the strong voices is like having any one politicallly active special interest group (READ: SBA, Oil companies, Naturalists, Tobacco, Feminists, KKK, NAACP, ect.) take up most the seats in the judicial executive and legislative branches of the US.
It will make things pleasurable for that group in no time flat and the rest of the population will be left out to dry. After a while the group in control will wonder why people get so whiny, since it all works out for them.
But this is just my opinion.
I find it rediculous, and damaging to NS. The entire reason it is not balanced, regardless of anyones feelings, it is NOT balanced for pick up and play. A game shouldn't have to meet very exact conditions for it to be fun and playable. Thanks for listening.
/edit If this gets locked without a damn good reason I want an explanation otherwise I might be lead to be believe its just meant to silence the thinkers
I find it rediculous, and damaging to NS. The entire reason it is not balanced, regardless of anyones feelings, it is NOT balanced for pick up and play. A game shouldn't have to meet very exact conditions for it to be fun and playable. Thanks for listening. <!--QuoteEnd--> </td></tr></table><span class='postcolor'> <!--QuoteEEnd-->
* Bingo *
Example: A while ago someone suggested making areas able to be 'capturable' via nanogridlock, making them infested or clean. I don't think I read any replies that DIDN'T like it, and it spawned what? 2 new topics discussing it?
Now, what would that idea do? Aside from providing a new dynamic to the game, it would be an already comfortable feature (Capture and hold) in the game to make newer players feel less alienated. I can see that as one of the reasons the devs would think NOT to implement it.
Six days before the release of 2.0 everyone collectively came together and said "this feels pretty balanced, but its not that fun." The hive emitted spawn umbra when aliens spawned, and it put a cloud of umbra around itself at frequent but random intervals. A turret factory cost 10, turrets were 5, electricity was 10(and had 4 targets). You had to wait 90 seconds after building a node before you could get any resources from it. So the collective idea was, theres no way to balance this in the amount of time left, but get it close enough, release it, then fix it up.
Basically, the public *didn't* playtest 2.01. The same people who playtested 2.0 playtested 2.01, just about every single change to 2.01 was something that was really discussed and debated in detail on the pt/vet forums. Whereas if you look at any post in the public beta forum you'd be lucky to find one constructive reply in a 10-pager. Playtesting went on in the normal fashion except that the public was allowed to download it and try it if they wanted. I'm not trying to be mean or discount anyones opinions or ideas, and I definitely think that the ability to instantly see what a really large group of people's reaction to a change was interesting. All i'm really saying is if the PTs deserved the blame for 2.0 then they deserve just as much credit for 2.01.
And just for the record, the "public" beta was composed of minor server-side changes. A public beta the way the real 2.0 beta went would be impossible, huge client patches every other day, major changes all the time, tons of bugs, it'd really be a disaster. Thats not to say that anyone here couldn't handle it, just that the community as a whole couldn't handle it.
One last note for the people who hate the vets, I dont think people understand why we're here and what we do. Basically, we play the game. Flayra watches us play the game. We observe what we feel are imbalances in clan and public play, and we discuss them. Flayra comes up with a way to try and get rid of the imbalances and gives us a new version. Rinse and repeat.
As for your post TempDeleteMe, I actually agree with one of your principles. We arent bad people and we genuinely tried harder to build the game around the pub game than the clan game. But, the few people who were commanders in the beta were for the most part very skilled RTS players and experienced commanders who know what theyre doing, and we all felt that commander skill should be an important part of the marine game, and still do. I dont think its a good reason to kick 50-100 people who've really devoted alot time and energy to the game, but maybe commanding should be easier. I think you're on to something there, but the question is, is it our fault or yours? Maybe the next time you get on a server and you have a newbie commander, you let him play and dont eject him and let him learn, tell him good try even if he loses? It might be a pain for a month or two but i'd bet after that you'd end up with alot more people who know how to command. Its really not that hard to command, people could learn if you gave them a chance. Are people scared to command because its slightly more work than 1.04, or are they scared because of the cruelness that is displayed towards new commanders on every single server?
You do NOT balance a game for clan play.
You balance a game for pub play. The job of the clans is to work with what the game has to offer to the best of their abilities. But in no way should the game bend to them. If the vets complain that something isn't powerful enough for their tactic, too bad. Use a different tactic -- you're good at that.
Are there any games with pubbers but no vets? Sure. Are there any games with vets but no pubbers? No!
I don't want to play a game that assumes that, in order to play correctly, I have to have the skills of a clanner. For example, the fade. No doubt it was balanced to fit the needs of ultra-skilled clanners... but NOT PUBBERS. If you aren't a god with the fade, it's a wasted 50 res. This is not the way it should be. All facets of the game should be intuitive and balanced in a pub environment.
And anyway, there's a "tournament" mode for a reason. Why not let vets PT tourney mode? Keep them out of pub mode, thx.
<!--QuoteBegin--></span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> We observe what we feel are imbalances in clan and public play, and we discuss them.<!--QuoteEnd--></td></tr></table><span class='postcolor'><!--QuoteEEnd--> This is the problem right here. How can clanners analyze imbalances in pub play?
I think the last playtest scrim was seriously held when the game was still being called 1.1. It has almost entirely been a pub style environment. It's not a clan fighting another clan, unless specifically asked.
<!--QuoteBegin--></span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->You do NOT balance a game for clan play.<!--QuoteEnd--></td></tr></table><span class='postcolor'><!--QuoteEEnd-->
Ironically, they didn't. Neither did Flayra. About 99% of the balancing that was done was done in a pubbing environment. The only difference between this pubbing environment and your local favorite server's pubbing environment is the fact that 99% of the vets can shoot, know the map, know strategy, know how an armory works, and are generally not the newbies you can run into.
1) The vets are all clanners.
2) The vets all played together.
Therefore it's a clan environment. Where was the 'pub style environment' when they made Fade completely and utterly impossible for your average player to use? It's a good tool, but it's a tool that requires six fingers to use. Where was the pub style environment when they balanced the economy? Compare a 5 vs 5 player game to a 12 vs. 12 player game. Interesting, most pubs have at LEAST around 10 players on each team.
The list goes on and on.
BTW: If a mod thinks of locking this topic for being hurtful to vets: The vets playtested and balanced the game. They are 100% resposible and should take the blame for everything wrong with the game. Rather then do what would be SMART: Balance in a small select group for bug testing (Internal beta, which I agree is necessary) and then roam the popular pub servers and steal away players from there for final testing, they just did the internal beta and released it. Imagine if 2.0 was delayed a month, and 2.0 was ACTUALLY released as 2.01 we have now. Guess what? That would've happened IF the vets were scrapped as dedicated beta testers.
Am I sore? It might seem that way. But I'll be damned if I'm going to sit back and watch this DAMN FINE game slip into an elitist hellhole.
guys, I don't hate the vets... to be honest I don't even know them =P
but... the posters have a point. If clan play is so different from pub play then having clan members do balance work will be all well and good for clan matches but won't help pub play much at all. Don't get me wrong, I'm not going 'AAAHHH FIRE THE VETS!!!' because all the poor guys did was win a competition. The suggestion for hand-picked pubbers however is probably better for the case of balancing for the general public.
With it having the requirement that they know how to play the game at least (ie: no explaining anything, it's easy to check for this before allowing an applicant into the 'focus group') and don't just sit about shouting 'OMG SUXXOR NOOBS' then it should be a lot nicer for balancing out the good old pubs somewhat =D
To give an example, it's like having a wine company that makes most of it's sales to the public but they hold a competition and select some of the top wine tasters in the country to do tasting for them. Sure the wine tasters who buy that line of wine will be thrilled with the delicate bouquet of the fruity little number but the general public will probably either spit it out going 'what the feck?' or not really notice the difference between this £3 wine and the £1 wine down at the market.
No hate for you vet guys, I just think it was bad judgement on the NS team's part that's all <!--emo&:)--><img src='http://www.natural-selection.org/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'><!--endemo-->
<b>edit:</b> wow... lots of posts while I typed that one out o.O
the pub tests were done with clanners right? Isn't that a bit of an odd pub test? ^^;
Do you see what I am saying? I mean that none of those vets are at the skill level of a pubber, and thus issues that are specific to pubbers are lost because vets simply don't run into them.
Secondly, we don't control what Flay does. At all. Flay does what he wants, we simply make suggestions on what would be the next logical steps. It's the same with the PT's. You CANNOT say that we made this game exactly the way we want it. Infact, a couple of changes were very much against what the majority of the vets wanted. It's not our call.
[...] This topic has been flame bait ever since it was posted back when the Vet program started. It was flame bait when it jumped up again several months ago, and was batted down in a few days by the mod team. It's still flame bait, and I'm sick of it.
[Added because several posts have popped up: We did not balance 2.0 Flay did. BTW, the test server WAS 11 v 11. We played like that all the time.]