Organized Playtesting

Cowswin · June 2003

<div class="IPBDescription">Playing or testing</div> As a programmer myself, we are required to put in a certain amount of time toward testing. I also playtest games "un-officially" on the side. Depending on the game/program/boss, I am sometimes required to test or log a certain amount of time toward testing per week. Depending on how many people you have, the average length of a game and the amount of things you have to test, I would suggest coming up with a formula or length of time required for testing each week.

Of course exceptions can be made and the time might be different for some people. You might even let them come up with schedules and whatnot (Group A is testing on even days, Group B tests on odd days and Group C tests on the weekends with Group A and B).

You could also go into "hardcore" mode and set times every night at a specified time that players are expected to log on (8PM est is game time). Players can be grouped into not only Group A, Group B, etc, but also based on their time alotments, so Group A could be say...the people who have average 10 hours a week for testing time, Group B is 20, Group C is 40+.

If you don't have them (which I'm sure you do), I'd suggest having server "monitors", people who were always watching the servers from spec mode to catch bugs, possible exploits and general other annoyances that might be in the game. These people could have their own seperate forum and gather information collected by all the testers/vets from the other forums and summarize in yet another forum for Flayra to persue in easily readable un-opinionated format. The only problem with this is finding people willing to do this on a volunteer basis (although there's a large amount of players I'm sure there's a couple people with enough brains and time to do such a thing).

I'm sure you have at one time or another kicked PT/Vets from their positions so to speak but I might suggest watching the PTs (since the vets are really there just to play for the most part as I understand), time in game and constructive feedback and kicking them on this basis. Albeit what they might have had to say was already posted by someone else but if they're not posting and not playing...you might want to get more active or more expressive PTs. Personally I might invite more PTs based on the fact that 300 PTs, vets and devs might not express the opinion of the community and/or get the testing done that's needed. 300 sounds more like the number a small ALPHA game might use. Even Beta's use more people than that most of the time. With a playerbase of 15K+, I'd suggest an increase in testers.

P.S. I also forgot to add (but of course you already know) that having an open beta wouldn't be all that helpful at this point. You'd get too much useless info from newbie players that just joined and think it's the official version and ideas that have been repeated umpteen times since the dawn of NS 1.0.

Just my <img src='http://www.unknownworlds.com/forums/html/emoticons/skulk.gif' border='0' style='vertical-align:middle' alt='skulk.gif'> & $0.02

Niteowl · June 2003

twisted, these are excellent suggestions.

I'd also like to add a few ideas of my own to better the PT experience.

really open it up to a ton more players. but have the primary PT have access to the beta forum only. have as many primary PT that want to, each moderate a group of 2nd lvl PTs, who's comments go to the primary, and the primary decides whether or not to bubble it up to an area flay and co can see.

if necessary, have secondary PT be responsible for tertiary, etc etc etc. that way, you get more eyes on the bugs, and more info coming thru. due to the filtering system, it'll be better quality info from the dredges (quartenary?) up to the top (primary).

does this post make any sense? lemme know.

Clefns · June 2003

while this could be good, at the same time it could be bad. i don't know about you but i consider myself to be the average gamer by stats. i'm a 21 male college student that works. college is out so i game more :-p but yeah due to the fact of that my work schedual is ever changing. i can't dedicate myself to game EVERY day at 8 or any given time. one day i work at 9am. next day at 5pm. granted there should be a required amount of playtime but not a required time to do it. i'd say i game at least 20 hrs a week minimum. but the times are before 5pm est or after midnight est. and i do see a lot of clanners playin on red's pit :-p that seems to be a popular one but i don't even see the other ones much. Well just thought i'd give info to ur post.
[TDA]Clef

Clefns · June 2003

oh and opening it up too much is a bad idea :-p anything over 500 could be overwhelming. they want to be able to read the feedback and work on it. if they have over 1000 testers they could get 200 responses all saying the same thing like "OMG LOOK SOME GUY WAS SKULK AND HE WAS INVISIBLE AND BIT ME! TOOO POWERFUL NERF IT PLEEEZ" in hopes that the people picked would be more intelligent but there still would be a lot of posts saying the same thing and they would have to filter through all of em or they could miss something important.
[TDA]Clef

MagiTek · June 2003

Reading recent posts by MonsE and others, it seems like it would be a very good idea to increase the number of PTs. Proper gameplay testing requires a minimum of 6 on 6, which means that nearly half of the (currently 29) testers must be ready to play at the same time, on the same server for anything to get done. With the players spread out over so many different timezones, it's no surprise that getting a game together on a weekday is nigh-impossible.

I would suggest that the number of PTs be expanded significantly, and that their ranks be organized by their location. Get 20+ people on the west coast of the US, 20+ on the east coast, 20+ in the UK, 20+ in Australia/NZ, etc. I'm using 20 as an example, but it would be any number large enough to support self-contained games at the most convenient time for that zone. That way you could have multiple games going on every weekday at say, 9pm local time. Then on the weekends you would have the international marathon sessions that are happening now. As a side effect, each "cell" might develop their own set of special strats and playing style, and when the weekends roll around we could find how these variety of styles work against one another.

Also, is there currently an organized system for relaying gameplay test info to the devs? Any serious matchup not being observed by a dev should probably have a mandatory, brief writeup describing the facts of how the game flowed. "Aliens started here, marines did this, aliens did that, marines attacked here with this, aliens countered with that, aliens win in 13 minutes." Personal tweak suggestions are nice, but they're often tainted by anecdotal experiences that may make the player feel that something should be nerfed more than necessary. Having a large database of statistics is more useful in the long run.

littlewild · June 2003

I suppose the problem is that the play testers are currently occupied and unable to play much.
To solve this problem, you can either make it a public beta or reselect the group of PTs you currently have.

Then again, increasing the number of PTs will increase the number of feedbacks proportionally. Flayra's development cycle is probably something like: Code -> Release -> Play Test -> Feed back. I don't think he can cope with the amount of feedbacks alone. There is a need for a group of people that will digest the information and compile a list for Flayra.

Ph0enix · June 2003

It seems as if there is a slight issue with time zones.

As all the vets are American it's not a problem for them but with the PTs scattered everywhere it's relegating them getting together to weekends, which obviously curtails their PT'ing opportunities somewhat.

As to simply increasing PT numbers, i'd guess thats up to Flay and Grendel to decide. I'm not sure if it's actually needed as things <b>seem</b> to be progressing OK as they are.

If it ain't broke...

Cowswin · June 2003

Niteowl - Yes, you have the general idea down pretty good. The only thing with opening it up is that you still have to limit and filter the people you let in for feedback reasons. Not to bash newbies to the forum or anything but some normally intelligent people not used to the community joining playtesting would produce more useless posts than if you stuck an infinite number of pink monkeys in a room with an infinite number of typewriters.

Filtering doesn't neccesarily have to be done by age either as some 16 year old's are quite competant in expressing opinions, ideas, suggestions and bug fixes while some 21 year old's post things like "The ali3nz r 2 powrful, u need 2 fix teh cl4w". You could use a multitude of systems to organize the testing, not specifically requiring them to have X amount of hours per week but perhaps show up at a certain amount of scheduled games per week. Out of 7 games you must show up at 3 or something of that nature. This would solve your problem of not being able to show up for every game or even half of them Clefns.

You'd have to pick your PT server monitors (spectators) wisely though as you don't want to create elitist groups within groups. Aka, powertrip. Mainly because you don't want these people to feel as if they're being babysat. They'd take "attendance" for scheduled games and other than that rely on the players themselves to report their weekly hours on the honor system. It actually works quite well with the self policing system believe it or not. You can require them to post their hours on boards open to only them or someone could even make a simple site that they can submit their end of week hours on ala a metrics system.

Cowswin · June 2003

A simple tester application page would solve most of the problems with easily filtering for the good bits and pieces of the NS community. It could be written in a day, asking players obvious questions that would be useful as a tester, reason why, timezone, age, time of day usually used to play (multiple select fields if needed), past testing experience if any, computer experience, etc. Basically the setup most major game companies use to get their own beta testers.

You could get the widest range of possible testers with a minimum amount of hassle in this fashion. Sound like a good idea to anyone else? I'm also curious since I didn't hear from them before word of 1.1 started spreading around but do you playtest at all outside of version releases? Or is the development process not ongoing so after a release the game sortof...sits around for a while before going back into testing? I wasn't quite sure if Flayra was the only actual coder to the game itself or not.

Dread · June 2003

Good ideas there. Though I think Veterans and PT's shouldn't be mixed. Veteran only and PT only servers would be better to test the whole scrim vs public play scheme.

and absolutely no to open beta *shrugs* <img src='http://www.unknownworlds.com/forums/html/emoticons/confused.gif' border='0' style='vertical-align:middle' alt='confused.gif'>

Twisted Master</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>A simple tester application page would solve most of the problems with easily filtering for the good bits and pieces of the NS community. It could be written in a day, asking players obvious questions that would be useful as a tester, reason why, timezone, age, time of day usually used to play (multiple select fields if needed), past testing experience if any, computer experience, etc. Basically the setup most major game companies use to get their own beta testers.</td></tr></table><span class='postcolor'>

Actually I believe age doesn't have anything to do with playtesting since all you show to other is your virtual image. It really doesn't matter what you really are as long as your virtual image behaves and acts the right way. I think application pages wouldn't be so great either, as I smile at the thought of Monse going through 16 000 applications it would take ages. I think the best way is just to contact proper persons directly and then ask the timezone-questions etc.

Grendel · June 2003

There are a lot of people who want to test.

There are relatively few people whose input it useful, for a number of reasons.

Since we don't want the code widely distributed, this makes life hard.

X_Stickman · June 2003

You could recruit from these forums. While a lot of the time people get into flame wars (which seems to be at an all time low at the moment, w00t!) a lot of people show that they can write up interesting and understandable things, put accross points in a clear way, and are BRILLIANT at spotting faults in anything, even heaven or Mars Bars, which are just undefeatable. And while it is generaly better to have good people in the beta, at least some people should be n00bs or at least medium-skill players, to see how steep the learning curve is for the less skilled and so on.

Snidely · June 2003

Just out of curiosity - how can you trust people (such as new PTs) not to leak the code? I'd have thought that would be something that's hard to prevent.

And Mars Bars have a bad chocolate:inside stuff ratio, thus rendering them sickly.

Dread · June 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Snidely @ Jun 3 2003, 05:00 PM)</td></tr><tr><td id='QUOTE'> Just out of curiosity - how can you trust people (such as new PTs) not to leak the code? I'd have thought that would be something that's hard to prevent. </td></tr></table><span class='postcolor'>
I think leaking the code is not so hazardous as you might think. They can't practically do anything with it anyway because your WonID is checked before joining to server. Also I would be more afraid of Vets leaking it to their clanmates but I think devs/grendel know who to pick.

MagiTek · June 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Snidely @ Jun 3 2003, 05:00 PM)</td></tr><tr><td id='QUOTE'> Just out of curiosity - how can you trust people (such as new PTs) not to leak the code? I'd have thought that would be something that's hard to prevent.  </td></tr></table><span class='postcolor'> 
Snidely, because it would be useless. They don't have access to the source, and you can't play on 1.1 beta servers without having your WONid manually added to the list of authorized testers. If you or I got ahold of the latest beta, we wouldn't be able to do anything with it.

X_Stickman · June 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (MagiTek @ Jun 3 2003, 05:06 PM)</td></tr><tr><td id='QUOTE'> </span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Snidely @ Jun 3 2003, 05:00 PM)</td></tr><tr><td id='QUOTE'> Just out of curiosity - how can you trust people (such as new PTs) not to leak the code? I'd have thought that would be something that's hard to prevent. </td></tr></table><span class='postcolor'>
Snidely, because it would be useless. They don't have access to the source, and you can't play on 1.1 beta servers without having your WONid manually added to the list of authorized testers. If you or I got ahold of the latest beta, we wouldn't be able to do anything with it.  </td></tr></table><span class='postcolor'> 
/me throws down plans to raid Grendels house.

BUGGER! <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'> <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'> <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'> <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'>

Snidely · June 2003

Oh, thanks. That's cool. It just seemed a bit strange to trust people over the internet, I should have known that they'd have taken precautions. (:

Cowswin · June 2003

To go along Grendel's comment, many people want to 'test' but for the wrong reasons, which is why filtering is required to scoop out the good the bad and the ugly. Most players are disillusioned and believe that playtesting is playing the game. Some people might BELIEVE they are testing while they are in all actuality just playing the game (i.e. those that don't provide any feedback or useless feeback). No matter how much you filter you'll still get the few here and there that don't provide constructive feedback and even good playtesters are bound to slip off track now and then.

This is where the levels of playtesting would come in. The comments/suggestions from the testers would go to a forum where a server monitor or forum monitor could parse and summarize the data into a more complete easily readable form, even an web page if they had the skillz <img src='http://www.unknownworlds.com/forums/html/emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif'> From that point it could be sent to the Playtester Lead, being Grendel, who would just do a double check over it for accuracy and understanding and submit it to Flayra with his comments on the feedback and Flayra could pursue a much more organized base of information and have it to fall back on when needed. That way no one person in the chain would be bombarded with reports, PMs, emails, etc as the testers themselves would only have to post their reactions/bugs/ideas and have them sorted out by other people for completeness. So those people below them don't whine about not having their idea submitted, the server monitor could even display his/her page for them to look over before it was sent to Grendel.

Having it in this fashion also still keeps the route open for Flayra to come down to the testers and get first hand experience (play under a different handle Flayra), without being bombarded with "Fix teh bug here" or "I think u should have this". The individual server monitors could also be monitoring tester activity level if you don't want to go with the whole "submit hours per week" or "# of games played this week" things and submit that so and so hasn't shown up for 20 games in a row and hasn't posted in a month or whatever.

Cowswin · June 2003

*bump* because the other related thread had been derailed <img src='http://www.unknownworlds.com/forums/html/emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif'>

X_Stickman · June 2003

Yes, but playing the game is a good test as well. Testing servers capabilities and so on. If you are always trying <b>completely</b> different ideas each time, you can't <b>evolve</b> individual ones. For instance, a JP rush might be deemed balanced, but perhaps there's a vent somewhere that allows easy access to a hive that they all missed the first time they tried it, but then they never tried it again because they moved on. I know that isn't a major issue, it's just an example.

Cowswin · June 2003

True, but I'm sure we could find enough constructive players to fill those empty holes instead of just random people to fill the server up and "play". There would be scheduled games that are watched by server monitors but other than that the players would be free to play to their hearts content. The big problem of just letting a whole mess of people in even if they didn't have access to the Beta forums or any private forums for that matter is that they will still have access to THESE forums, and you'd have a huge spam fest of suggestions and XXX doesn't work fix plz anyway. Which of course defeats the whole purpose of selecting those that can provide decent feedback.

Cowswin · June 2003

*bump*

Because I feel due to the recent...missing persons...from NS beta v1.1 this thread may be of use. Amazing how the people suddenly flowed away right around the release of the free Wolfenstein III game came out...

Snidely · June 2003

Do you really need to keep bumping this thread? And is it really fair to accuse PTs/Vets of slacking of based on second hand evidence.

Grendel is the lead tester. If changes are to be made, I'm pretty sure he has them in hand. (I notice Voogru's a PT now. Not before time. (: )

Cowswin · June 2003

Ok, first off, I'm not accusing them of slacking. They left. Plain and simple, it's all over the forums and quite evident if you have ASE and take a look at the empty servers. I'm also not blaming Grendel for anything, he just recently became the lead tester and probably hasn't had time to set up any type of formal testing as suggested above. However, before he came along, it seemed that <b>nobody</b> had them in hand, which seems to be the reason a couple of the vets left <u>AMOUNG a compilation of others</u>. Some of them just weren't tester material.

Lastly, these are suggestions, not telling anyone how to do anything, just offering up my personal experience which I figured would be helpful. Which is why I bumped the topic. Which brings us here <img src='http://www.unknownworlds.com/forums/html/emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif'> I just gave them a suggestion, the rest is up to them <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>

Tiax · June 2003

Id be happy to simply spectate games and report findings. I sometimes join a server for no other reason then to watch (though usually when I know one team is doomed because the teams were stacked so poorly....I mean, who wants to waste their time on a team thats gonna be spawn killed over and over for the next 15 minutes until finnally the aliens deside to go onos and end it?

Snidely · June 2003

On re-reading my post, I think I sounded a lot crankier than I actually am. So apologies if I offended you.

I'm still a bit reluctant about the idea of us drawing any conclusions from these boards, but you're right in that there has been a lot about this floating around. Maybe they should make an announcement and quash it.

I'm pretty sure they're recruiting playtesters - like I said, unless I'm mistaken, Voogru is a fairly new PT. Anyway, yeah, let's see what happens. (:

Cowswin · June 2003

On the "open beta" issue, this whole fiasco pretty much sums up why just grabbing random peeps and/or a semi-open beta wouldn't work. It seems that they thought the game was supposed to be "finished" while they were playing beta. Exactly what I said would happen if you had people who didn't understand the meaning of testing the game, which is the whole point of the original part of this thread <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'>

Just so we're all on the same page, my comments are not directed toward the PTs and Vets that are still here <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>

Frikk · June 2003

/me huddles in the corner.

Make them stop, mommy. Make the hurting stop.

Organized Playtesting

Comments