Server Performance

A[L]C · July 2011

How are people finding the server performance? We are hosting a server on p180 and finding it really quite laggy. Is this to do with cysts? The server is a quad xeon and ns is only using around 30% cpu and 350mb ram.

Is there anything we can do to improve it server side?

Thanks
A[L]C

btw I understand the code isnt optimised yet :D

McGlaspie · July 2011

From what I've seen of B180, server performance has indeed dropped. Not by a huge amount (say, 12-18%). The problem is that means the tick rates are staying below 10 again, on average, during a full game. I suspect this is due to the Cysts, and the sheer amount of them required to properly infest an area.

A[L]C · July 2011

Im seeing tick rates of 5 sometimes 3 with 12 players. :( Does seem to be as the game progresses, its >20 at round start

Quantum_Singularity · July 2011

it was suggested that the game was retaining information about the cysts even after they had been destroyed, causing more and more lag. Not sure if thats true or not, can we maybe get confirming on that from a developer?

The last game I was in became unplayable after about 1/2 an hour of playing, until the end of the round that is. We ended up having to recycle just to start a new 'playable' game

MOOtant · July 2011

Slow NS2 is slow but you should remember to set affinity to single core because otherwise Windows process scheduler will keep moving server.exe between cores making it get 10 ticks/s with 2 players.

matso · July 2011

<div class='quotetop'>QUOTE (Quantum_Singularity @ Jul 9 2011, 11:34 PM) <a href="index.php?act=findpost&pid=1859189"><{POST_SNAPBACK}></a></div><div class='quotemain'>it was suggested that the game was retaining information about the cysts even after they had been destroyed, causing more and more lag. Not sure if thats true or not, can we maybe get confirming on that from a developer?

The last game I was in became unplayable after about 1/2 an hour of playing, until the end of the round that is. We ended up having to recycle just to start a new 'playable' game</div>

Actually, from looking at the profiler, the biggest performance problem right now seems to be the minimap blip updates. Spends about 30% cpu on the server to update the minimap blips ( at the 500 ents count level)

cysts is hardly visible in the profiler.

Start the server as a listen server and type "profile" in console. "cheats 1" and then spread cysts all over the map. Use 'ents' to see how many entities the server has.

endar · July 2011

<div class='quotetop'>QUOTE (MOOtant @ Jul 10 2011, 08:23 AM) <a href="index.php?act=findpost&pid=1859200"><{POST_SNAPBACK}></a></div><div class='quotemain'>Slow NS2 is slow but you should remember to set affinity to single core because otherwise Windows process scheduler will keep moving server.exe between cores making it get 10 ticks/s with 2 players.</div>
Interesting. I don't see that occur, which OS specifically are you referring to? I have affinity across all cores (default), but it strictly never leaves core 0.

MOOtant · July 2011

Windows 2008 R2 Server.

maesse · July 2011

Checked out the minimap blips and here's my stab at an improvement:
<div class='codetop'>CODE</div><div class='codemain' style='height:200px;white-space:pre;overflow:auto'>
function NS2Gamerules:UpdateMinimapBlips()
PROFILE("NS2Gamerules:UpdateMinimapBlips")
if GetGamerules():GetGameStarted() then

// create table of map blips from last frame
local mapBlips = EntityListToTable(Shared.GetEntitiesWithClassname("MapBlip"))

// @maesse: new code start
// Cache ownerid -> mapblip lookups for CreateUpdateMapBlip
local blipDict = {}
for index, mapBlip in ipairs(mapBlips) do
local ownerid = mapBlip:GetOwnerEntityId()
blipDict[ownerid] = mapBlip
end
// @maesse: new code end

// grab all scriptactor entities
local allScriptActors = Shared.GetEntitiesWithClassname("ScriptActor")
for entIndex, entity in ientitylist(allScriptActors) do

// Decide if entity should create a blip and what type
local success, blipType, blipTeam = self:GetMinimapBlipTypeAndTeam(entity)

if success then
// update the blip
CreateUpdateMapBlip(mapBlips, entity, blipType, blipTeam, blipDict) // @maesse: send blipDict
end

end

self:DeleteOldMapBlips(mapBlips)

end

end</div>

My modifications are a bit more spread out here:
<div class='codetop'>CODE</div><div class='codemain' style='height:200px;white-space:pre;overflow:auto'>function CreateUpdateMapBlip(mapBlips, entity, blipType, blipTeam, blipCache) // @maesse: added blipCache
// Update MapBlip entity if exists, else create new one
local updated = false
local found = false
local blipResult = nil

// @maesse: make use of blipCache
if (blipCache ~= nil) then
local index = entity:GetId()
blipResult = blipCache[index]
if(blipResult ~= nil) then
// got it
found = true
end
end

// @maesse: this is the codepath we want to avoid
if not found then
//Shared.Message("Blip not cached!")
for index, mapBlip in ipairs(mapBlips) do
if mapBlip:GetOwnerEntityId() == entity:GetId() then
found = true
blipResult = mapBlip
break
end
end
end

// update
if found then
blipResult:Update(entity, blipType, blipTeam)
updated = true
end

if not updated then
// Create new MapBlip
local mapBlip = CreateEntity(MapBlip.kMapName)
mapBlip:Update(entity, blipType, blipTeam)

end

end</div>

Minimap usage went from 4ms to 2ms on my test-setup. It might be possible to further optimize it by not recreating the dictionary every server-tick. If anyone tries this out or want the lua files to test it on your servers, just let me know.

tl:dr; changed a for(for()); to create-dictionary(); for();

A[L]C · July 2011

If you can tell me what to change, i'll test it for you

Gunter · July 2011

I've got 3x 12 man servers running on a Quad Core Xeon 2.4Ghz w/ 2GB of memory.. It stays stable (above 10 ticks/s) until late game turret/hydra/cyst spam is rampant throughout the map. Then it will spike between 3-12 ticks randomly.

I am seeing the server.exe processes use about 550-575mb and will only ever use one core (so 25% CPU usage per process) when under load.

I also cannot actually type <u>anything</u> into the server console..
Are you using server.exe and launching with switches, or running via a full-game client to do your profile testing?

matso · July 2011

Run a dedicated server with the -console switch, you get up a black screen. Use your normal console toggle key and you get the console running. Type 'profile' and exit the console, voila! you get the profile output from the server. Space to pause, [] to move left/right in the profiling history.

Plasma · July 2011

<div class='quotetop'>QUOTE (Gunter @ Jul 11 2011, 01:19 PM) <a href="index.php?act=findpost&pid=1859592"><{POST_SNAPBACK}></a></div><div class='quotemain'>It stays stable (above 10 ticks/s)</div>
I'd consider that awful as a player FYI, needs to hang around 30 ticks/sec

maesse · July 2011

Here's the updated <a href="http://dl.dropbox.com/u/11425965/mapblipmod/NS2Gamerules.lua" target="_blank">NS2Gamerules.lua</a> and <a href="http://dl.dropbox.com/u/11425965/mapblipmod/MapBlip.lua" target="_blank">MapBlip.lua</a>. They go into ns2/lua/. Might want to backup the old files just in case.

I've only changed NS2Gamerules:UpdateMinimapBlips() and CreateUpdateMapBlip in MapBlip.lua as seen in my last reply.

My test setup is only an empty listen-server (with a ton of cysts and drifters), so It's possible the minimap blips aren't actually causing any slowdown in a real game. If one of you guys manage to get the profiler running on the dedicated server, I'd love some screenshots or text output so I can compare the hotspots against my own setup.

Raneman · July 2011

That's very nice, maesse, but wouldn't both the client and server need the modification for it to work?

Devs, I'd appreciate it if we could actually have maesse's work in game.

Gunter · July 2011

<div class='quotetop'>QUOTE (Plasma @ Jul 11 2011, 08:06 AM) <a href="index.php?act=findpost&pid=1859689"><{POST_SNAPBACK}></a></div><div class='quotemain'>I'd consider that awful as a player FYI, needs to hang around 30 ticks/sec</div>

What server are you playing on that stays at 30, while full and under load? (full map of structures)

I think that that is currently just a dream and not reality.

Gunter · July 2011

<div class='quotetop'>QUOTE (matso @ Jul 11 2011, 07:29 AM) <a href="index.php?act=findpost&pid=1859677"><{POST_SNAPBACK}></a></div><div class='quotemain'>Run a dedicated server with the -console switch, you get up a black screen. Use your normal console toggle key and you get the console running. Type 'profile' and exit the console, voila! you get the profile output from the server. Space to pause, [] to move left/right in the profiling history.</div>

Very nice! Been trying to figure that out for awhile. I will check the profiler when the server is full (hopefully tonight)

autograder · July 2011

My server has been running great, a lot has been cleaned up in the last few patches.

The changes to summit are quite nice, definite improvements, although I haven't run any empirical tests yet.

If your ping is over 180, find a different server (it's your connection), best to be under 80.

luns · July 2011

since recent patch my fps stay within 20-30 range, mostly usually somewhere 25 to be fair.

performance hasn't improved at all for me since last patch.

A[L]C · July 2011

<div class='quotetop'>QUOTE (Raneman @ Jul 11 2011, 07:16 PM) <a href="index.php?act=findpost&pid=1859756"><{POST_SNAPBACK}></a></div><div class='quotemain'>That's very nice, maesse, but wouldn't both the client and server need the modification for it to work?

Devs, I'd appreciate it if we could actually have maesse's work in game.</div>
I have put it on my server

GamingDeluxe.co.uk NS2
109.70.148.28:27025

Could do with some testers!

Gunter · July 2011

<div class='quotetop'>QUOTE (autograder @ Jul 11 2011, 02:03 PM) <a href="index.php?act=findpost&pid=1859767"><{POST_SNAPBACK}></a></div><div class='quotemain'>My server has been running great, a lot has been cleaned up in the last few patches.

The changes to summit are quite nice, definite improvements, although I haven't run any empirical tests yet.

If your ping is over 180, find a different server (it's your connection), best to be under 80.</div>

Details on this..?

How many players? What is the servers tickrate late game?

What is the hardware specs? Do anything special with affinity or priority?

Players ping to server is important yes, but if the server is running at 5 tickrate even if you have 10ms - the game is absolutely unplayable.

gamester_5 · July 2011

I am seeing LUA errors of "Belly slide in" and "belly slide out". This is one of the most common errors I am seeing to date. I will monitor further.

Playa_2b

A[L]C · July 2011

im afraid to say it didnt seem to help much. We were still getting <5 ticks at points near end game. Memory useage seemed a little lower

maesse · July 2011

Yeah, I imagine it's something else. 5 ticks is 200ms pr frame, so even if my patch cuts of 5ms, it wont change much.. I have one more patch like that, dealing with ComputeLOS (line of sight) bringing it from ~8ms to ~4ms on my test setup.

Playing around today though, it seems that pathfinding for MACs and Drifters are possible causes for the slowdowns. Spamming waypoints for MACs is a great way to bring the server to it's knees. Check out this screenshot:
<img src="http://i.imgur.com/AChoF.jpg" border="0" class="linked-image" />

Look at that insane call count. I wonder if builders sometimes gets stuck in these pathfinding loops where they just start eating up CPU resources... Either way, that code is off-limits for me so it's not something I can play around with.. Could be interesting to see some profiler output from a struggling realworld server, to see if that's the reason.

Typhon · July 2011

Wow, looks like server problems would be instantly solved by optimizing collisionscene::sweeppoint and collisionobject::sweeppoint.

autograder · July 2011

<div class='quotetop'>QUOTE (autograder @ Jul 11 2011, 03:03 PM) <a href="index.php?act=findpost&pid=1859767"><{POST_SNAPBACK}></a></div><div class='quotemain'>My server has been running great, a lot has been cleaned up in the last few patches.

The changes to summit are quite nice, definite improvements, although I haven't run any empirical tests yet.

If your ping is over 180, find a different server (it's your connection), best to be under 80.</div>

<div class='quotetop'>QUOTE (Gunter @ Jul 11 2011, 03:06 PM) <a href="index.php?act=findpost&pid=1859771"><{POST_SNAPBACK}></a></div><div class='quotemain'>Details on this..?

How many players? What is the servers tickrate late game?

What is the hardware specs? Do anything special with affinity or priority?

Players ping to server is important yes, but if the server is running at 5 tickrate even if you have 10ms - the game is absolutely unplayable.</div>

Sure, here are the answers to your questions:
<ol type='1'><li>14 Players</li><li>Server Tick rate is ~30, I'll check again when we have a full server and edit this</li><li>Haven't messed with affinity or thread priority, I haven't even disabled Windows Aero Theme</li><li>Overmind mod</li><li> Hardware:</li></ol>
First let me start by saying that I purchased this to act as a server through beta only, eventually it will become a media/extra pc:
<ul><li>Dell XPS 8300 (<a href="http://www.dell.com/us/p/xps-8300/pd" target="_blank">clickity click</a>)</li><li>Intel Quad Core i5-2300 processor (6MB Cache, 2.8GHz)</li><li>6 GB DDR3 SDRAM at 1333MHz, (2X2G/2X1G)</li><li>1 TB SATA II Hard Drive (7200RPM)</li><li>460 Watt Power Supply</li><li>Intel HD Graphics 2000</li><li>Integrated 7.1 with THXÂ® TruStudio PCâ„¢sound</li><li>16X DVD +/- RW Drive</li><li>Windows 7 Home Premium 64-Bit Operating System</li></ul>
Scratch and Dent for $445.19 shipped.

Let me know if you have more :D

endar · July 2011

<div class='quotetop'>QUOTE (MOOtant @ Jul 10 2011, 09:28 PM) <a href="index.php?act=findpost&pid=1859375"><{POST_SNAPBACK}></a></div><div class='quotemain'>Windows 2008 R2 Server.</div>
Interesting, me too. Have you tried not binding(for lack of a better word) to a single core? I know that the server does have some multithreading capability, so you would be missing out on that by binding. Here is what I have currently with 18 players.
<a href="http://imageshack.us/photo/my-images/809/cpuje.png/" target="_blank"><img src="http://img809.imageshack.us/img809/4186/cpuje.png" border="0" class="linked-image" /></a>

What about windows Processor scheduling? I believe default for server editions is "Background Processes" which is what I currently use. You?

matso · July 2011

<div class='quotetop'>QUOTE (Typhon @ Jul 12 2011, 04:35 AM) <a href="index.php?act=findpost&pid=1859903"><{POST_SNAPBACK}></a></div><div class='quotemain'>Wow, looks like server problems would be instantly solved by optimizing collisionscene::sweeppoint and collisionobject::sweeppoint.</div>

Nah. Better to optimize MoveToTargets so it doesn't do 6000 calls to TraceRay. TraceRay is probably really hard to optimize further because its part of the basic toolkit used by the code all over the place.

player · July 2011

<div class='quotetop'>QUOTE (Typhon @ Jul 12 2011, 04:35 AM) <a href="index.php?act=findpost&pid=1859903"><{POST_SNAPBACK}></a></div><div class='quotemain'>Wow, looks like server problems would be instantly solved by optimizing collisionscene::sweeppoint and collisionobject::sweeppoint.</div>
It looks like he's just testing MACs\Drifters there though. I recall there are also issues with the turrets-targetting, the hydra-targetting and the DI, and I bet if you solve all of those issues another one will crop up.

Raneman · July 2011

Maesse is my hero.
I want you to bear my children, Maesse.

A[L]C · July 2011

<div class='quotetop'>QUOTE (Raneman @ Jul 12 2011, 03:13 PM) <a href="index.php?act=findpost&pid=1860057"><{POST_SNAPBACK}></a></div><div class='quotemain'>Maesse is my hero.
I want you to bear my children, Maesse.</div>
did his change make much difference?

Server Performance

Comments