Linux Ns Server Takes Lot Of Cpu, Not Windows One
CrK0
Join Date: 2004-07-09 Member: 29815Members
<div class="IPBDescription">Linux server, 16 players, 70% CPU</div> I've a big problem, my NS server under Linux , takes me a lot of CPU.
For example, in a P4, 1 mb L2 Cache, 1 GB RAM, 2.8 GhZ and HT, with 16 slots, it takes 67% Of CPU.. i think it's a lot.
I've tested also in a DualXeon, 2 GB ram, 2.8 GhZ 2 micros, and It seems not work well.., becouse virtual stats are collapsed and idle is 0%.
I've seen windows NS too, it takes 14 players about 3% in a dual xeon 2.4 ghz processor..
Is there a fix about my linux server ? or the solution is to install windows ?..
Thnx a lot, mi mail is : crk01@newlightsystems.com
For example, in a P4, 1 mb L2 Cache, 1 GB RAM, 2.8 GhZ and HT, with 16 slots, it takes 67% Of CPU.. i think it's a lot.
I've tested also in a DualXeon, 2 GB ram, 2.8 GhZ 2 micros, and It seems not work well.., becouse virtual stats are collapsed and idle is 0%.
I've seen windows NS too, it takes 14 players about 3% in a dual xeon 2.4 ghz processor..
Is there a fix about my linux server ? or the solution is to install windows ?..
Thnx a lot, mi mail is : crk01@newlightsystems.com
Comments
Recently, I've upgraded my system to Fedora Core 2 and now my usage is around 60% for a 14 player server. Over the next couple of days I'm going to stick another Hard disk in my system and reload RedHat 7.3 to see if the usage goes back down as an experiment.
I'd suggest you look into some other distros, slackware or debian being prime examples of distros with a light footprint and a good stock kernel. Even if you insist on runnng redhat you can always rebuild your kernel and strip out all the crap your computer will never have in it. For example, if your computer lacks SCSI you can remove a massive amt of code from the kernel that you will never need. Same goes for USB, on most servers the USB code will never be needed.
Check out the kernel howto for instructions on how to rebuild your kernel. Running slackware I never see more than about 20% cpu usage so you really should try a few other distros. All those slick looking GUIs and graphics that you can't avoid running redhat really slow your server down.
No, I haven't done it yet. Rather I'm playing with custom kernel compiling in hopes I can figure out a universal way to make this better. If I find something that works I'll write up a guide to help others. If nothing works I might go back to trying the reload of 7.3 again.
Right after I starting the HLDS server it would be sitting there taking up .3 to 4% of the cpu with no one on the server. When I joined the server, it would jump up to 5%
So, I took a spare hard disk, downloaded the RedHat 7.3 ISO images and installed that again. Guess what, I'm back to the same CPU usage I was before, about 30-35% for a 14 player (on Classic) server.
From here on I'm just going to be talking out loud, maybe this will click with someone or maybe I'll just be babbling:
I have a real hard time blaming this on Distro bloat since I pretty much stripped out everything except the drivers, file systems and CPU optimizations when I recompiled the kernel for FC2. I think there is something else going on here. Could it have something to do with the version of the C libraries on a system (unless they are all statically linked)?
I noticed something different between the two systems. First, in the HLDS console if I type "stats" on the Redhat 7.3 system I show 35-49 FPS. When I do that on the FC2 system I show 70-95 fps. Could this be an indicator of cpu usage? The FPS is doubled or nearly doubled on the FC2 system vs the RH7.3 system and so is the CPU usage.
I think this might have something to do with the HZ settings in the kernels. The HZ setting increases the interactivity of the server. Basically the amount of times per second the Kernel is interrupted to do something else. At a HZ=100 setting the Kernel is interrupted every 10ms, at 512 it's approx 5ms and 1000 its every 1ms.
<a href='http://kerneltrap.org/node.php?id=464' target='_blank'>In this KernelTrap article</a> they talk about how previously the 2.4 kernel (the one in my 7.3 box) is typically set at HZ=100. On Redhat 8 and 9 Kernels this is set to HZ=512. Nearly all 2.5/2.6 Kernels are set at HZ=1000. Could this one change affect the perfomance of Linux systems so dramatically? I think I'm going to see what happens if I recompile the FC2 kernel with HZ=100 setting...this might take a while so don't hold your breath.
Also in one of the valve mailing lists someone (I believe steven hartland) found that some versions of the linux kernel (2.4 I believe) incorrectly report the cpu usage via top.
<a href='http://www.mail-archive.com/hlds_linux@list.valvesoftware.com/msg27542.html' target='_blank'>http://www.mail-archive.com/hlds_linux@lis...m/msg27542.html</a>
This is perhaps not so likely in your case considering the fps is lower as well. If it was the same fps then I'd be more inclined that it was being reported incorrectly.
<!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->
Well, alot has changed. First off, using patches are the best way to get performance.
<a href='http://ck.kolivas.org/patches/2.6/' target='_blank'>http://ck.kolivas.org/patches/2.6/</a>
patch your kernel with it. any ck# will do. They include the staircase Process Schedular. Info available <a href='http://members.optusnet.com.au/ckolivas/kernel/' target='_blank'>here.</a>
Keep in mind that the amount of CPU latency more important that actual CPU usage. For example, your latency was 5ms and mine was <1ms, then if we both used identical systems and util was 50%, my pings would be lower than yours and my system would be more responsive via SSH.
ck patch sets are geared for desktops...however,
<!--c1--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'><!--ec1-->
Wed Jun 16 21:11:13 EST 2004
Server usage?
The staircase scheduler includes two special sysctl settings which allow you to optimise it's behaviour for different workloads
echo 0 > /proc/sys/kernel/interactive
will disable interactive tasks from having bursts, thus being even stricter about nice levels (suitable for non gui desktop usage or a server)
echo 1 > /proc/sys/kernel/compute
makes round robin intervals much longer, delays task preemption and disables interactive mode to optimise cpu cache usage - suitable for computational intensive tasks.
<!--c2--></td></tr></table><div class='postcolor'><!--ec2-->
Meaning this might be something to experiement with. I haven't run a NS server in quite a while. But in theory, HL isn't a server. <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif' /><!--endemo--> hahah, Silly I know, but listen. HL isn't a realtime process either, so WTH is it? Its an interactive process, meaning if you leave ineractivity on, it *should* get more time slices (lower latency) per cycle. If you renice it, it will be given a minumum amount of time slices in the schedular. And if bursting is enabled (/proc/sys/kernel/interactive) then we should see a large performance boost by using more time slices.
If you have alot of I/O activity, in your lilo or grub append evelator=cfq (i spelled it wrong, but you know what I mean). This will lighten disk activity and lower disk iowait. It is a Fair Queuing Async IO schedular. meaning disk reads are very fast and writes are placed when they have a chance.
Another big problem with servers is memory. Cron jobs and old file handles being cached into memory. 2.6 includes a really cool feature called swappiness (/proc/sys/vm/swappiness) echo # > /proc/sys/vm/swappiness where # is a value between 0 and 100. 100 means commit alot of stuff to VM, 0 means hold out as long as you can. default is 60. You might want to set this to 30, this way map changes are swapped out and the ingame memory growth doesn't require any swap commits.
HZ settings should be 1000 or atleast 500, this is due to the fact that HL needs more clock cycles that a db server. So, in short, turn off pre-emption, bump HZ to 1000, enable "Use register Arguments", :MTRR support", use ACPI if you have it.
Now, you might not see a change in cpu utility, but HL should run better <!--emo&;)--><img src='http://www.unknownworlds.com/forums/html//emoticons/wink.gif' border='0' style='vertical-align:middle' alt='wink.gif' /><!--endemo-->
Sorry It took me so long to check this thread, I just haven't been checking my email <!--emo&:(--><img src='http://www.unknownworlds.com/forums/html//emoticons/sad.gif' border='0' style='vertical-align:middle' alt='sad.gif' /><!--endemo--> I hope this was informative and helpful.
<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
Now, with that all said. Vada was onto somthing.
<!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->
I noticed something different between the two systems. First, in the HLDS console if I type "stats" on the Redhat 7.3 system I show 35-49 FPS. When I do that on the FC2 system I show 70-95 fps. Could this be an indicator of cpu usage? The FPS is doubled or nearly doubled on the FC2 system vs the RH7.3 system and so is the CPU usage.
<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
CPU usage is not an indicator of HLDS performance. FPS is all that matter. 100%, 40%, who cares really. 20% CPU and 14FPS = The suck. 100% and 75FPS = l33t lag free server. For example. SETI and Dnet clients take 100% CPU usage, but their nice level is VERY low. If you renice to -20 or so, then it occupies more minimum time slices reducing other processes abliltiy to use those slices. HLDS can utilize more CPU time (resulting in higher FPS) because it occupies more slices. HZ has to do with the number of ticks per timeslice I believe. maybe want to google for it.
edit: HT now has a penality in SMP evalutation now as of 2.6.5 or 2.6.6. you can download schedutils (by Robert Love) to set CPU Affinity for HLDS
<b>Utter BS to system lag:</b><ul><li>System "bloat"</li><li>crappy distros</li><li>windows is better</li><li>NS needs -O3 -march=athlon-xp</li><li>NS was written for windows and not linux</li><li>linux is becoming more unstable with 2.6</li><li>the 2.4 is better for servers than 2.6</li></ul>Hope this was informative and gets sticky.
then, if you turn hl booster off and the fps drops back down to 60 the cpu usage drops down to 1-2%
and since the hl engine is based on fps, that would be a good assumption.
System Specs
AMX XP2800+
GB DDR 400 Ram (Corsair CL2.5)
120GB hdd (SATA)
Redhat 9
HLDS_L 1.1.1.2 (Insecure RCON)
I have the latest admin mod and amx 9.7
then, if you turn hl booster off and the fps drops back down to 60 the cpu usage drops down to 1-2%
and since the hl engine is based on fps, that would be a good assumption.
<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
Proportain...maybe...but not enough to even make a graph by it. There is a 'correlation' between FPS and CPU usage. But like I said, they aren't truely proportional to each other. Too man factors play into running this type of usage. (meaning the HLserver).
Ghozer:
What NS are you running? You probably don't have a memory leak if it is being "freed" when you kill the pid. Look at these and see if it helps:
<a href='http://forums.gentoo.org/viewtopic.php?t=175419&highlight=ram+usage' target='_blank'>http://forums.gentoo.org/viewtopic.php?t=1...light=ram+usage</a>
<a href='http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html' target='_blank'>http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html</a>
Fresh Boot 33% Used
Start HLDS_L with NS - 56% Used
Change Map - 99% Used
Kill HLDS_L with NS - 75% used
IT Doesnt free the full 66% that it should free if It freed ALL that NS used...
if its 33% without HLDS_L and 99% With it (after map change) - when i kill it it should free up 66% of the ram, returning it back to 33% -- which it doesnt... :\
Fresh Boot 33% Used
Start HLDS_L with NS - 56% Used
Change Map - 99% Used
Kill HLDS_L with NS - 75% used
IT Doesnt free the full 66% that it should free if It freed ALL that NS used...
if its 33% without HLDS_L and 99% With it (after map change) - when i kill it it should free up 66% of the ram, returning it back to 33% -- which it doesnt... :\ <!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
Type in free -m and you'll see that its not all really *used*. You should really read those links that I gave you. I promise, its very informative <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->
For example, check out my system info on <a href='http://users.evilsquid.net/phpsysinfo/' target='_blank'>http://users.evilsquid.net/phpsysinfo/</a> and <a href='http://littlenemo.evilsquid.net/sysinfo/' target='_blank'>http://littlenemo.evilsquid.net/sysinfo/</a>
Both of which are production servers and you'll see that ran is using "100%". But If it were really that high, I'd be swapping, which I'm not using any. Read those articles I gave you, then explain the VM system quite well.
windows reported
cpu#1 26%
cpu#2 9%
hlds reported
cpu#1 90%
maybe we can con valve into releasing the hlds source so we can start optimizing it. <!--emo&:p--><img src='http://www.unknownworlds.com/forums/html//emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' /><!--endemo-->
Due to this, I am going to go back to windows since there is really no motivation to "fix" the problem on linux....
sigh
I tested the server on a P4 ( now without booster ) with preemtible activated and SMP ( P4 with hipertrheading ) and the CPU Load on kernel 2.6 is "lower" at TOP but it gets lower performance than booster -1 or -3 of course.
I think that there must be a logical combination of lots of parametres starting with kernel and finishing in tickrate.
I assume also that NS tooks anytime, anyway more resources than any HL mod.
So, now i run it at sys_ticrate 300, kernel 2.6 and it goes normal, fps are up and down, but i think i can optimize this touching sys_ticrates.
I tested the server on a P4 ( now without booster ) with preemtible activated and SMP ( P4 with hipertrheading ) and the CPU Load on kernel 2.6 is "lower" at TOP but it gets lower performance than booster -1 or -3 of course.
I think that there must be a logical combination of lots of parametres starting with kernel and finishing in tickrate.
I assume also that NS tooks anytime, anyway more resources than any HL mod.
So, now i run it at sys_ticrate 300, kernel 2.6 and it goes normal, fps are up and down, but i think i can optimize this touching sys_ticrates. <!--QuoteEnd--> </td></tr></table><div class='postcolor'> <!--QuoteEEnd-->
I'm curious, did you enable the SMP(HT) or just SMP ? And which version of 2.6? 2.6.8.1 is latest as of this writting.
NS is way over the top with the information about each and every bullet and this causes a nasty hard disk thrashing. Switching log files off dropped my ping from 250 to 60 and my cpu from 99% to
<!--c1--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'><!--ec1-->PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND USER
1896 14 -1 215m 114m 10m R 66.9 12.9 2105:26 hlds_i686 Balrgsnapper1<!--c2--></td></tr></table><div class='postcolor'><!--ec2-->
The above is after a long game of ns_hera with 16 players on debian kernel 2.6.8 (and no patches or tweaks, which it does need for other reasons)
Hows about the mod team assign a ram drive of about 10 meg for log files which are swapped to disk end of the round while every one is in the ready room or is that too sensible and involves too few kernel tweaks?
RAM is cheap, CPUs are not so cheap
<!--emo&:0--><img src='http://www.unknownworlds.com/forums/html//emoticons/wow.gif' border='0' style='vertical-align:middle' alt='wow.gif' /><!--endemo-->
<!--emo&:angry:--><img src='http://www.unknownworlds.com/forums/html/emoticons/mad-fix.gif' border='0' style='vertical-align:middle' alt='mad-fix.gif' /><!--endemo-->
Linux is a modular operating system used mainly by programmers...
Each module is compiled specificly for compatibility and optimizations with the hardware that the software is being run on.
What -insert whoever the hell runs steam here- is doing is they are realeasing one package that is supposed to run on all machines...
And thats not even the worst thing, they are using shortcuts and and hackeyed strategies to "get things done" at the cost of performance, truth is they dont much care for linux.... Steam runs bad with ALL halflife mods when run under the linux platform, its not just natural-selection.
As much as i love linux, and as much as I hate using windows for ANY kind of server, i would say that it just dosent make sense to run a steam server on linux if you can do it on windows.... mind you this is a very recent change of opinion on my part.
Who cares about the mass security holes? Its a gameserver for gods sake, it dosent hold any creditcard numbers. Besides, you can strip windows down so much that theres barely anything left to hack, and you can be anal about ports and such.
With that said, i run my server linux =P the only reason right now is that many other software applications are running on it and are more critical than the gameserver, and they were programmed to work on linux only.
its just the more players that join, the higher the ping gets, even if they on lan.
From reading the licensing terms on microsoft's website it appears not, however I have no experience with this so I was wondering if someone has a MS Server OS running game servers and knows what the situation is with licensing.
Cpu is a XP3200+, even though sys_ticrate is at 1000 (For better ping) it shouldnt be that high am i correct ?
i read the kernel recompile guide from VadaKill and havent tried it yet cause the server is in a datacenter not close to my home <!--emo&:p--><img src='http://www.unknownworlds.com/forums/html/emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' /><!--endemo-->
So is this cpu usage normal at sys_ticrate of 1000 ?
Cpu is a XP3200+, even though sys_ticrate is at 1000 (For better ping) it shouldnt be that high am i correct ?
<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
Jeeze! The default 100. I never modify my ticrate and everything works fine. Jacking the rate up as high as it can go will of course increase your CPU overhead. Try reducing it. HL2DM's default ticrate is 66, and CS:S is 33.
From reading the licensing terms on microsoft's website it appears not, however I have no experience with this so I was wondering if someone has a MS Server OS running game servers and knows what the situation is with licensing. <!--QuoteEnd--> </td></tr></table><div class='postcolor'> <!--QuoteEEnd-->
If your worried about it - use 'nix
I m running linux for my ns server. I op 2 servers. 1 is ns 1 is co.
HW config both same. P4 2.4(socket 478)HT enable, 512MB ram, 80G SATA HD.asus p4p800.
SW config. Fedora core 2 running now. plugin, metamod 1.17.4p20,amx0.99,AMXMODX1.0 on 1 server.
B4 I use FC1(running 3.0b5). when over around 1x players on the server playing, it lag like in hell. & i checked the stats. it nvr up over 50fps(48.x-49.x), cpu loading around 6x-7x%.
Then I upgraded to FC2 kernel 2.6.9-1.11_FC2smp(running 3.0b5) I can have over 20 players(server max 24 players) online without any lag. the stats was 8x-99.3fps. cpu loading maximum is 5x%. it works fine.
Now. use same version of kernel running ns3.0, when over 15 players online. it also lag to 1xx-2xx ping.(no plugins has changed.) & the stats normal(no one on server) is 83.x-100fps. but when over 1x players, it go from 3-100 fps, very unstable. then I updated the kernel to kernel 2.6.10-1.14_FC2smp & the result is
stats
<!--c1--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'><!--ec1-->17:42:42 CPU In Out Uptime Users FPS Players
97.50 52.76 61.40 270 213 20.41 21<!--c2--></td></tr></table><div class='postcolor'><!--ec2-->
<!--c1--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'><!--ec1-->17:43:23 CPU In Out Uptime Users FPS Players
97.00 49.01 56.64 270 213 11.91 21<!--c2--></td></tr></table><div class='postcolor'><!--ec2-->
<!--c1--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'><!--ec1-->17:44:23 CPU In Out Uptime Users FPS Players
95.00 58.35 75.65 271 216 50.02 21<!--c2--></td></tr></table><div class='postcolor'><!--ec2-->
CPU loading
<!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->top - 17:48:11 up 9:42, 1 user, load average: 0.88, 0.88, 0.83
Tasks: 54 total, 3 running, 51 sleeping, 0 stopped, 0 zombie
Cpu(s): 47.3% us, 0.3% sy, 0.0% ni, 51.3% id, 0.0% wa, 0.2% hi, 0.8% si
Mem: 513868k total, 408008k used, 105860k free, 38256k buffers
Swap: 1052216k total, 0k used, 1052216k free, 246964k cached<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
Hope the developer team can find out how to fix it soon. <!--emo&???--><img src='http://www.unknownworlds.com/forums/html/emoticons/confused-fix.gif' border='0' style='vertical-align:middle' alt='confused-fix.gif' /><!--endemo-->
Thanks a lot.
Full 16 player server on Beta 5 = 40-50% cpu.
Full 16 player server on v3.0 = 65-70% (MINIMUM) and above.
Theres definately something wrong with it :/
I have also seen with just 5 players on the start of a map that the cpu usage is at 27% with NS v3.0, on Beta5 it was 10-20%....