Weirdest tick rate issue on Windows Server 2012 R2
d4rkAlf
Sweden Join Date: 2013-11-12 Member: 189309Members, Squad Five Blue, Reinforced - Shadow
Sweden Join Date: 2013-11-12 Member: 189309Members, Squad Five Blue, Reinforced - Shadow
CPU: Intel i5 760@4 GHz (Tested for 16 h with Prime 95's torture test blend. Not a single error and temps are below 70°C. The frequency remains at a constant 4 GHz.)
RAM: 12 GB@1600 MHz (These are NOT error correcting so a bit flip is to be expected every now and then. 1 error during 12 h of Memtest86+. I could test for longer be I really don't think this is because of defective RAM.)
MB: Asus P7P55D-E
HDD: 2x500 GB 7200 RPM in a RAID1 (Using the motherboard's firmware RAID.)
PSU: Corsair 400W
OS: Windows Server 2012 R2
Should easily be able to run a 12+ slot NS2 server, right? Well. . . No.
The problem: The tick rate averages around 15 ticks per second while the CPU is nowhere near being pressed for cycles. A more or less constant 95% idle.
The confusion: Simply starting a "magic" program called Throttlestop removes the issue. Throttlestop is used for removing throttling from (mostly) laptops. The weird thing is that I do not have to actually enable the program's features. Simply launching it is enough. Another thing to note is that once the program is started I can't close it since that crashes the computer. (I also suspect it may be the culprit for other system crashes I have been having.) When I start Throttlestop HwInfo64 reports that the processor's turbo feature gets enabled (goes from red to green).
The "solution": Trying to figure out where the problem was stemming from I moved my SSD with Windows 7 on it over to the server machine. Starting the exact same Server.exe binary with the exact same configuration (except for the drive letter) including BIOS and server settings makes it run flawlessly. I didn't install any extra drivers with the exception of those that Windows does automatically.
This is an incredibly annoying problem and despite my best attempts at googling it I have not managed to find a proper solution.
Here's a list of the things I have tried:
I actually don't expect anyone here to be able to help me. This is just a last-ditch effort before I scrap WS2012R2 and install something else. (Probably Debian, but we'll see.) But I would still really appreciate not having to go through that hassle so I wanted to check with you guys and gals first.
Update 4-2-2015: I installed Debian on a spare HDD and as expected everything is working just beautifully with a very stable 30 tick rate on the empty server.
Update 2: Solution! Adding more players to the server will, somewhat counter-intuitively, increase the tick rate. Only applies at low player numbers naturally.
RAM: 12 GB@1600 MHz (These are NOT error correcting so a bit flip is to be expected every now and then. 1 error during 12 h of Memtest86+. I could test for longer be I really don't think this is because of defective RAM.)
MB: Asus P7P55D-E
HDD: 2x500 GB 7200 RPM in a RAID1 (Using the motherboard's firmware RAID.)
PSU: Corsair 400W
OS: Windows Server 2012 R2
Should easily be able to run a 12+ slot NS2 server, right? Well. . . No.
The problem: The tick rate averages around 15 ticks per second while the CPU is nowhere near being pressed for cycles. A more or less constant 95% idle.
The confusion: Simply starting a "magic" program called Throttlestop removes the issue. Throttlestop is used for removing throttling from (mostly) laptops. The weird thing is that I do not have to actually enable the program's features. Simply launching it is enough. Another thing to note is that once the program is started I can't close it since that crashes the computer. (I also suspect it may be the culprit for other system crashes I have been having.) When I start Throttlestop HwInfo64 reports that the processor's turbo feature gets enabled (goes from red to green).
The "solution": Trying to figure out where the problem was stemming from I moved my SSD with Windows 7 on it over to the server machine. Starting the exact same Server.exe binary with the exact same configuration (except for the drive letter) including BIOS and server settings makes it run flawlessly. I didn't install any extra drivers with the exception of those that Windows does automatically.
This is an incredibly annoying problem and despite my best attempts at googling it I have not managed to find a proper solution.
Here's a list of the things I have tried:
- At first I thought it had something to do with the power saving options so I disabled all of them in BIOS. This includes Speedstep, C1E, the other C-states and even stuff such as EuP Ready. Tick rate goes up from average 12 to average 15 while the idle percentage also goes up. Close, but no cigar.
- I installed the chipset drivers for the P55-chip in the motherboard. No change.
- WS2012 seems to have an issue with CPU speed when running Hyper-V, so despite never having installed any of those roles/features I disabled every service I could find that mentioned Hyper-V. No change.
- Since HwInfo64 reports that turbo gets enabled when I launch Throttlestop I tried changing how Windows Server handles the turbo mode. No change.
- Removed all overclocking from the machine. No change.
- I haven't, technically, done a complete re-install of WS2012R2. However, since the issue has been present from the very beginning I seriously doubt it would make a difference.
- Setting Windows's power plan to "High performance".
- Probably even more stuff that I have forgotten about by now.
I actually don't expect anyone here to be able to help me. This is just a last-ditch effort before I scrap WS2012R2 and install something else. (Probably Debian, but we'll see.) But I would still really appreciate not having to go through that hassle so I wanted to check with you guys and gals first.
Update 4-2-2015: I installed Debian on a spare HDD and as expected everything is working just beautifully with a very stable 30 tick rate on the empty server.
Update 2: Solution! Adding more players to the server will, somewhat counter-intuitively, increase the tick rate. Only applies at low player numbers naturally.
Comments
http://ns2servers.devicenull.org/servers/1602767/82.199.102.133:27055
http://ns2servers.devicenull.org/servers/1433106/82.199.102.133:27045
http://ns2servers.devicenull.org/servers/1432876/82.199.102.133:27015
If the cpu load is low, it goes into slow-ass clock to save energy, interestingly resulting in low idle tr. This is normal behaviour. The moment some players join and play, it'll throttle back up. Not ideal for a server but "should not impact performance".
Disable these features in the bios to fix it. It will use more energy tho.
These are really simple, almost irrelevant, things. Try asking fellow ops before wasting that much time on it.
There is no reason the server should run with full tick-rate if there are no player at it. As there is no need for extra cpu power in that situation of an empty server the cpu throttles itself.
Actually i don't see any real reason to disable it as it saves energy (and who cares about the idle tickrate going down as long as it has no influence onto the actual server perf.)
So overall throttling saves energy and cpu power.
Must some sort of "magic".
Anyways, what makes you think I didn't consult with other server operators before making this thread? Of course I already spoke with a couple. They didn't have a clue either and it's not like I'm going to track down every single NS2 server op. and interrogate them personally. And I, of course, also googled it but was unable to find anything.
Since you have something against fledgling NS2 server operators asking questions in the official server discussion forum I kindly suggest that you head over to the community wiki and start adding all of the information that is missing or ambiguous.
No I totally agree, there is no reason to disable Speedstep et al. However the energy savings from the "optimized lowering of tick rate" seem, at first glance, to be negligible. Between the Debian system and the WS2012 system the power consumption differed by only half a watt (0.5 W). And yes, I am very well aware that making a comparison in that manner is a flawed method. It still intrigued me though since I thought the difference would be much bigger. Do you have any real world data that backs up the claim that a lower tick rate saves energy?
Yes, not only have I convinced myself that the Yggdrasil server is the proud host of NS2's best hit registration. I have also managed to swindle persuade the other members of Ygg that this is a fact.* ;-)
*I'm just poking a bit of fun here. Please don't ban me!
Edit: I just realized I forgot to thank you all for your help. A big thank you to the people who helpfully replied in this thread and an even bigger thank you to all the community members who joined up earlier today and helped me stress the server!
Thats what i called "real magic"
Type in perfmon and watch the serverscore. Every score above 20 is good and has the same result on hitreg or whatever.
The only thing that could have an negative impact on the quality then is the net connection.
And thats why a home hosted server cannot be better than one located in an data center.
But feel free to believe your voodoo.
And have fun to play on your srver with a ping of 4. Maybe thats why it feels better for you?
Only speculation for sure.
But who i am, only an admin who was hosting the best NS2 server for over 3,5 years.
Surely you meant to type 12 threads. Instead of 2.
This behaviour happened long before we worked towards the server performance improvements.
You're going very far. It got worse.
It is true tho the effects on performance should be negligible. It's more of an annoyance. For edge cases it's well worth disabling just to save the trouble.
Are you trying to create the perfect server? Nice. I hope it works out. It worries me tho, many have wasted time setting up servers and they remained empty. Seeing you spend ridiculous amounts of time on it makes me even more worried. Yes, you'll learn from it but you'll be very sad if it doesn't become at least somewhat popular.
You'll want to spend some of that time thinking about what makes a server a place to return to.
Personally, I haven't noticed any persistent differences in hit reg between our servers, at least not something that can't be explained by the fact I have 10x the ping on your server. I can't talk for my clan mates of course.
Finally, I hate to admit it, but yes I'm a bit ignorant about what exactly goes into "net quality". However, what I do know is that I'm hosting this server from a 1000 Mb/s (both ways) connection in a country with one of the best internet infrastructures in the world. It is of course not a "dedicated" connection but the few speed tests I've done seem to indicate no real problems.
Edit: The biggest upside of using an actual server provider is in my opinion the much better uptime. My system has multiple points of failure that will bring the entire NS2 server down. Such as only a single internet connection, no UPS, and only a single machine with a single PSU (hardware errors). Since the HDDs are in a RAID1 I can lose one of them and just keep on going, but there is no redundancy for anything else. A server provider should have multiple spares of all those things. I'm a bit uncertain about how they handle major power outages though.
Yes indeed the performance impact from Speedstep doesn't seem to effect any servers except those running heavy SQL databases (from what I've read so far).
What do you mean by "didn't try hard enough"?
And no, I'm not trying to create the perfect server. I just want a clan server for me and my clanmates to have our PCWs and drills on. I'm going to host a public server as well from the same machine because I really have no reason not to, but I will be quite alright even if nobody ever plays on it. And yeah, this is also a learning experience for me. This is my first time running a server of any kind and there is so much for me to learn! :-D
Oh, it'd be nice to be full every evening tho.. Would be waste for that connection otherwise...
By the way @GhoulofGSG9, should a server like this still be fine for a 1 versus 1 training session? Or will the reduced tick rate hold that back in any way? My understanding so far is that the send rate will still be 20 and that it should not interfere with training in any way.
Edit: Forgot the "not" when writing this. Doh!
Ok here is what exactly is going on at the server when it is idling. When the server is idling the cpu per core usage goes pretty much down to a point were the cpu's / windows energy-saving functions kick in.
And some windows energy saving magic causes the tick-rate drop in combination with "sleeping" threads.
About 1 vs 1 training at 20 ticks. The tick-rate is mainly controlling the ai update rate. As in the training it's mostly and basically only pvp and not pve players should not notice any difference.
In fact testing this at my own server it's back up to 30 ticks with the third player joining.
So everything should be fine.
But maybe ghoul is able to enlight us here.
Is it possible that some servers can have a better hitreg than others? And we are talking about servers with a constant score over 20 here.
The only real reason I had for wanting to switch was that I wanted to put the Yggdrasil server through its paces.
That was why I found your initial post so funny. I thought you were mocking me light-heartedly for my eagerness to switch server because of the "bad" hitreg. (Which, of course, wasn't bad because of the server and hence must have been bad due to "magic".)
Move calculation time goes up and it throttles tickrate which controls ai? It'd fall apart if that's how it worked.
It actually throttles mr and tr follows. Yes, in a 1vs1, you'll play on a degraded server.
The low tr during idle is something that occurred before the throttling was adapted (in 272) to what it is now.
@d4rkAlf No, really, if you actually properly disabled any cpu throttling functions, it will not show the low tr when idle. Something is missing.
Yeah after looking up the source code i noticed the adjustment code never made it actually into the public build. So it has to be some energy saving vodoo function of win2012 r2.
But actually as said before it is only affecting the networked tick-rate (the actual tick-rate goes up to it's real value as soon as a player has connected) a short while and has basically no real affect. Beside that the tick-rate graph might look a bit weird.
I know this might not be the problem at all, but it's worth checking. It seems to be that you have a fresh install of 2012 so you may have just overlooked it.
Hi there! Thanks for your suggestion. Unfortunately I have already tried that. I only remembered recently (I have tried so many things) so it only just got added to the list of stuff I have tried.
Your home hosted server was a perfect example.
Red/yellow plugs and ping spikes nonstop.
Maybe your machine was able to run a server, your connection wasnt.
I see CPU TM Function is on, why?
Because CPU TM Function stands for CPU Thermal Management Function. I.e. the motherboard will signal to the CPU that the CPU package is above 100°C and that it should lower its clock speed in order to cool down. When I run Prime 95 the cores stay below 70°C so it is not an issue during normal operation. (In case you don't know, the cores can be, and usually are, hotter than the package.)
The reason I have this enabled is in case of catastrophic CPU fan failure (maybe even during a heat wave). Sure, the CPU will shut itself off when/if it reaches 130°C but running above 100°C can cause long term damage any way.
Running another executable jar file with nothing but an infinite loop does not cause the tick rate to go up. So it seems that it's something specific to the Minecraft server application.
The plot thickens. . .