Map Change Times
ZEROibis
Join Date: 2009-10-30 Member: 69176Members, Constellation
Join Date: 2009-10-30 Member: 69176Members, Constellation
Can someone please look into why the requirements for changing the map are so crazy. The issue is not really with how long it takes by itself but how the server process handles I/O priority. For example:
Server Process A <1min to change maps
Server Process B <1min to change maps
CS:S server A <15sec to change maps
CS:S server B <15sec to change maps
When both CS:S server A and B change maps at the same time it takes <30 sec as one would logically expect. However, if you do a NS2 Server Process A and B at the same time you would expect it to take maybe 2min 3 at most but in reality it is like a black hole opens up on your server and it suddenly takes like 5+min to load.
It is like for some reason the NS2 server process is just like a kid trying to push all the other kids cups away from the water fountain and as a result everyone gets water a lot slower b/c they are all fighting for it.
Is there something we server admins need to do to the server process so that it will get in line and order its I/O requests properly?
I can understand that there is a lot of stuff for NS2 to load but I do not understand why it appears to just tank when placed in a situation where it needs to queue for I/O resources. Perhaps there is some setting I am missing or the game just needs some programing optimizations in future builds to improve I/O management.
Server Process A <1min to change maps
Server Process B <1min to change maps
CS:S server A <15sec to change maps
CS:S server B <15sec to change maps
When both CS:S server A and B change maps at the same time it takes <30 sec as one would logically expect. However, if you do a NS2 Server Process A and B at the same time you would expect it to take maybe 2min 3 at most but in reality it is like a black hole opens up on your server and it suddenly takes like 5+min to load.
It is like for some reason the NS2 server process is just like a kid trying to push all the other kids cups away from the water fountain and as a result everyone gets water a lot slower b/c they are all fighting for it.
Is there something we server admins need to do to the server process so that it will get in line and order its I/O requests properly?
I can understand that there is a lot of stuff for NS2 to load but I do not understand why it appears to just tank when placed in a situation where it needs to queue for I/O resources. Perhaps there is some setting I am missing or the game just needs some programing optimizations in future builds to improve I/O management.
Comments
What's in the server log during map loading?
In the mean time I am going to take everything down to make sure the defrag is not ignoring any files in use ect and run it a few times to help things as much as possible.
As I said before the normal load times are not long but when you get 2 servers changing at the same time is when things start to get out of hand.
As for the HDD it is just a normal 7200rpm drive. During the test I just ran it took about 6 min for 2 servers to change at the same time, virtually 0% cpu usage ram at 40% usage HDD has 40% free space and was fully defragmented.
Wait, you're running two NS2 server applications on one physical server?
I guess the combined load might be causing the long loading times then.
What's the disk load during loading?
When the server changes by itself it only takes a few seconds but if anything else is using disk to change maps at the same time it takes like 5min. There has got to be something wacky with the way that the server process is accessing the disk that causes these massive load times.
Can this issue please be investigated?
Why are you never answering this question?
And i repeat it again. You server has other problems than loadingtimes:
http://ns2servers.devicenull.org/servers/273675/174.37.194.20:27015
Your server cant handle the game, thats it.
Should the devs possibly state that the game is intended and requires individual disk drives for each server instance or can only be run on SSD? Given that it takes 60 times longer for 2 servers to change maps at the same time vs twice as long clearly indicates a problem.
I had 4 ns2 servers running on a SAS drive previously and was not suffering at all from this problem. Must be on your end.
There are programs that allow you to defragement in a way so that a specific folder has all its items in a space on the HDD, so it doesn't have as much seek time. Other solution would be a new partition (which simlar) or try to put the files on a ram.
Other issue might be NTFS being shitty.
If you are running a lot of servers you should consider a HDD upgrade.
Those of you who regularly play on the Tactical Gamer servers know them to be far and away capable server instances, despite our many mods (see below; repro is stock), and you've watched countless mapchanges happen without issue or delay.
I've repro'd this problem many times running both instances completely stock, w/ no mods at all, but I've yet to find any input/timing/etc which will /reliably/ repro the problem (very frustrating!).
My best luck reproducing the behavior is when both servers (again: different install folders) change maps at the same time (within 1sec), and it doesn't happen every time. In fact, it can take tens of attempts to intentionally repro it in testing. And so, of course, It can take tens and tens of games to repro in normal usage. Or just a few games. It's terribly intermittent.
But, when it happens, one or both instances will take 5+ minutes to change maps. I've clocked times as long as 15 minutes, and 10 minutes is common if you leave it alone. The server has not crashed. Given enough time, it /will/ release and continue its operation.
During this extended hang, I see in the server console the routine messages (config files loaded, consistency hashed, meshes cooked/built, "Finished loading 'maps/ns2_soandso.level', etc, etc).
But then the server console stops all output, allowing no clients past "WAITING FOR SERVER". Clients see "WAITING FOR SERVER" whether they were present since the mapchange started, or if they try to connect several minutes into the hang. When the server is hung like this, they're not getting past "WAITING FOR SERVER" no matter what.
If you already had webadmin loaded, it continues to show NaN for tickrate (as it briefly would during any mapchange). If you didn't already have webadmin loaded, and you browse to it, the only response you see is "Loading server..." while the server is hung.
I can reliably and immediately resume the operation of one of the hung server instances by completely shutting down the other instance.
Think of it like 2 kids fighting to use a water fountain rather than take turns. Creating a virtual second fountain might help but as long as the kids keep fighting or they each get their own private fountain you will have the same issue.
Isn't it true that plenty of server owners/hosters are running multiple servers per box? If so, and insofar as your disk-per-instance requirement is accurate, shouldn't 5+ minute mapchanges be a much more common (even if only intermittently) experience for the average NS2 player than it is?
@ZEROibis, I know I'm just asking you to guess, but... how might you resolve three seemingly conflicting observations? 1) must have a disk per instance to avoid 5+ minute mapchanges 2) most players in the community at large don't experience 5+ minute mapchanges 3) plenty of server owners/hosters are running multiple instances per disk
Is one of those observations broken? Am I missing something?
My fear is that my problem (which, granted, might not be identical to yours) is more complicated (and not yet fully understood, despite my best efforts and your knowledge sharing).
I was running 4 servers off normal HDD before and it never happend. Not even when I restarted all servers at the same time.
Still if you guys have issues try following:
1) Use a single shared directory for the server files (symlink, or directly run off that folder)
1.a) Use a program to put those files most closely together on the HDD to reduce seek time
1.b) or, if you have plently of ram, try to load them into memory
2) Try linux, it handles files differently then windows and has plently of file systems
3) Upgrade your HDD