Random server crashes (build 268)
Amad
Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
Any other server admin getting this? I basically tried to fully reinstall all my mods and configurations and yet my server still keeps crashing. After hours and hours of constant testing I eventually sent a ticket to my server provider and they're fully aware of the issue, apparently waiting for a hotfix. Anyone else having the same problem? Sometimes it occurs during a map change while sometimes the server just redplugs in the middle of a match. Highly frustrating. :c
And no, logdir.txt doesn't really tell me any reason why it would crash all the suddenly. I just want to know if anyone else is having similar experiences after the new patch.
And no, logdir.txt doesn't really tell me any reason why it would crash all the suddenly. I just want to know if anyone else is having similar experiences after the new patch.
Comments
But it looks like we have 4 admins here with modfied servers (serverrates, slotammount)
Maybe the crashes are more often there, dont know.
So the server.exe does crash and not just freezed ( you can test this by checking if the webadmin is still reachable ).
Webadmin is not reachable until the server finishes booting up.
Would be interisting to now if an Vanilla server has the same issues.
[23:11:49]P[47135802] ran command sh_vote with arguments: ns2_summit
Downloading mods
Finished downloading and installing mods
Mounting mod from c:/thirsty1/mods/m706d242_1408976514/
Mounting mod from c:/thirsty1/mods/m812f004_1408894912/
Mounting mod from c:/thirsty1/mods/mc6fbbb0_1408734207/
Mounting mod from c:/thirsty1/mods/maa8da06_1408729446/
Loading Shine config...
Loading extensions...
bandwidth limit: 102400 bytes/sec per player
- Extension 'basecommands' loaded.
- Extension 'votesurrender' loaded.
- Extension 'voterandom' loaded.
- Extension 'unstuck' loaded.
- Extension 'adverts' loaded.
- Extension 'mapvote' loaded.
- Extension 'namefilter' loaded.
- Extension 'crossspawns' loaded.
- Extension 'ns2stats' loaded.
- Extension 'ban' loaded.
- Extension 'logging' loaded.
- Extension 'afkkick' loaded.
- Extension 'adminmenubutton' loaded.
- Extension 'pregameplus' loaded.
- Extension 'serverswitch' loaded.
- Extension 'motd' loaded.
Completed loading Shine extensions.
Loading Shine users...
[23:12:18]Shine started up successfully.
[23:12:18]Loading config://ServerConfig.json
[23:12:18]Loading config://ReservedSlotsConfig.json
[23:12:18]Loading config://ServerAdmin.json
[23:12:18]Loading config://BannedPlayers.json
[23:12:18]Loading config://MapCycle.json
[23:12:18]Loading config://ConsistencyConfig.json
[23:12:18]Hashed 1 game_setup.xml files for consistency
[23:12:18]Hashed 980 *.lua files for consistency
[23:12:18]Hashed 2737 materials/* files for consistency
[23:12:18]Hashed 38 sound/* files for consistency
[23:12:18]Hashed 12 maps/*.level files for consistency
[23:12:18]Hashed 20 shaders/*.hlsl files for consistency
[23:12:18]Hashed 24 models/*.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Basic.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Emissive.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Level.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Level_alpha.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Level_emissive.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Model.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Model_alpha.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Model_emissive.surface_shader files for consistency
[23:12:18]Hashed 1 shaders/Model_emissive_alpha.surface_shader files for consistency
[23:12:18]Hashed 30 models/alien/skulk/* files for consistency
[23:12:18]Hashed 28 models/alien/lerk/* files for consistency
[23:12:18]Hashed 20 models/alien/fade/* files for consistency
[23:12:18]Hashed 15 models/alien/onos/* files for consistency
[23:12:18]Hashed 20 models/alien/gorge/gorge* files for consistency
[23:12:18]Skipped 100 *.py files for consistency
[23:12:18]Skipped 75 materials/dev/* files for consistency
[23:12:18]Skipped 21 materials/editor/* files for consistency
[23:12:18]Skipped 15 materials/hud/* files for consistency
[23:12:18]Skipped 85 materials/skybox/* files for consistency
[23:12:18]Skipped 4 materials/system/* files for consistency
[23:12:18]Skipped 19 materials/effects/mesh_effects/* files for consistency
[23:12:18]Skipped 98 materials/effects/*.dds files for consistency
[23:12:18]Skipped 0 materials/effects/*.py files for consistency
[23:12:18]Skipped 100 materials/effects/*.polygons files for consistency
[23:12:18]Skipped 0 cinematics/environment/* files for consistency
[23:12:18]Skipped 0 cinematics/test/* files for consistency
[23:12:18]Skipped 9 models/alien/skulk/*view* files for consistency
[23:12:18]Skipped 13 models/alien/lerk/*view* files for consistency
[23:12:18]Skipped 6 models/alien/fade/*view* files for consistency
[23:12:18]Skipped 6 models/alien/onos/*view* files for consistency
[23:12:18]Skipped 6 models/alien/gorge/gorge*view* files for consistency
[23:12:18]Skipped 7 models/alien/fade/*vortex* files for consistency
[23:12:18]Skipped 1 shaders/Blur.hlsl files for consistency
[23:12:18]Skipped 1 shaders/DarkVision.hlsl files for consistency
[23:12:18]Hashed 1 lua/entry/*.entry files for consistency
[23:12:18]Consistency checking took 0 seconds
[23:12:18][Elixer (Server)] Loading Utility Scripts v.1.72
[23:12:18][Elixer (Server)] Using Utility Scripts v.1.72
[23:12:18]Loading config://NS2PlusServerConfig.json
[23:12:18][NS2+] Mod updater is: Enabled
[9120.143] Worker 05 : Loading 'maps/ns2_summit.level'
Building pathing mesh for level maps/ns2_summit.level
Finished loading 'maps/ns2_summit.level'
Client connected (81.27.12.236)
[23:12:28]Client Authed. Steam ID: 11611431
Strego connected.
Client connected (84.251.27.140)
[23:12:30]Client Authed. Steam ID: 13123664
oh connected.
Client connected (149.172.173.218)
[23:12:31]Client Authed. Steam ID: 1839897
Isoterik connected.
Client connected (217.250.77.238)
[23:12:32]Client Authed. Steam ID: 98485725
dino connected.
Client connected (85.224.182.104)
[23:12:32]Client Authed. Steam ID: 36770742
[OAG] HYAan connected.
Client connected (193.83.228.148)
[23:12:34]Client Authed. Steam ID: 75715334
Flugstein connected.
Client connected (85.179.188.7)
[23:12:35]Client Authed. Steam ID: 46347502
vertigo connected.
Client connected (82.74.66.214)
[23:12:35]Client Authed. Steam ID: 47135802
P connected.
Client connected (88.66.135.248)
[23:12:39]Client Authed. Steam ID: 1663296
Deanosaurus connected.
Client connected (81.227.93.97)
[23:12:40]Client Authed. Steam ID: 475075
Mr Boz connected.
Client connected (78.69.38.192)
[23:12:42]Client Authed. Steam ID: 84783956
Anakie connected.
Client connected (89.2.9.68)
[23:12:43]Client Authed. Steam ID: 21396444
Euregan connected.
Client connected (89.67.145.58)
[23:12:46]Client Authed. Steam ID: 10534949
Friend Computer connected.
Client connected (77.179.199.118)
[23:12:46]Client Authed. Steam ID: 165422768
sta connected.
Client connected (109.192.50.85)
[23:12:47]Client Authed. Steam ID: 296310
~C|-|!LL3|2~ connected.
Client connected (81.217.47.122)
[23:12:47]Client Authed. Steam ID: 80592649
Bluewind connected.
[23:12:58]Bluewind[80592649] ran command sh_voterandom with no arguments.
Client connected (176.199.153.26)
[23:13:00]Client Authed. Steam ID: 108208717
exe connected.
[23:13:04]sta[165422768] ran command sh_voterandom with no arguments.
[23:13:05]~C|-|!LL3|2~ was killed by oh
[23:13:08]vertigo[46347502] ran command sh_voterandom with no arguments.
[23:13:12]Euregan was killed by oh
[23:13:12]Deanosaurus[1663296] ran command sh_voterandom with no arguments.
[23:13:13]Mr Boz[475075] ran command sh_voterandom with no arguments.
Client connected (94.30.48.182)
[23:13:16]Client Authed. Steam ID: 45835001
America connected.
Client disconnected (213.64.0.238) Timeout
[23:13:21]Strego was killed by oh
[23:13:21]oh was killed by [OAG] HYAan
[23:13:36]vertigo was killed by [OAG] HYAan
[23:13:36]Bluewind was killed by Strego
Client connected (83.89.38.70)
[23:13:42]Client Authed. Steam ID: 70529433
GanjaGhost connected.
Sometimes it crashes right after map change and sometimes it takes a couple of minutes. The server runs NS2+, Shine admin, Shine NS2Stats and Shine extra plugins.
For us we started getting it some builds ago, I'd say 265 (don't hold me to that). And then it was fine with say 266. With 267 and beyond the Pregame Crash is back as it was before.
Our prime suspects for this (hold on to your seat) are:
- The randomizerr and voteforceeven votes from vanilla
- The badgesrequest (see /lua/Badges_Server.lua line 6 set to false)
Try disabling those three (we haven't gotten any further in figuring out which one it is because we, too, are struggling with crashes. However we try not to make a fuss since our server is borderline insane. Unless the world turns upside down, we will always see more crashes than smaller servers.)
Crash
The only error before the mapchange was:
[7045.768] Main : Error: Failed to add an obstacle at -79.65, -100.89, 45.65!
Strange, I disabled the randomizer and voteforceeven votes and the crashes has been greatly reduced. Still getting them but not as often as before.
We have the above mentioned three things disabled since a couple days and i don't think i've seen another pregame crash.
Ah, yes, I should speculate in detail that the crash does not occur the moment the vote succeeds and puts players on the team. Instead, it occurs right when someone starts the vote and before the vote gets shown on clients screens and before it gets logged in the log!
Shine does things to those votes (make them leave afks in rr) afaik. It's entirely possible that shine or shine's afk system (not just it's mapvote) causes it.
Regrettably we can't disable shine for testing as we *still* need the extraips extension. A smaller server could, potentially disable shine entirely to test this.
Your crash might be a different one, though, dePARA. If it does not get at least to the point where the first players have finished connecting, it's not the 'pregame crash' I speculate. Not that this helps... :-<
The error you noted, we get 3-20 of those after every round and consider them 'normal'.
@ATF
How did you guys come to these conclusions? I'm probably going to take the advice but I'm just curious. Also I more so mean the badges_server change.
Exactly how I managed to crash my server the first time it occured, tried to start a map vote myself and what do you know, redplug. However, I haven't had a single crash in a whole day because of those little things you suggested. This seems like a possible work around though I'll keep monitoring the situation.
Much appreciated, thank you.
We are ancient magic wizards with silver grey hair and tall pointy hats!
In all seriousness, it's a lot of guesswork! During pregame, the server essentially isn't doing anything. It should absolutely not crash at that time. So we tried to think of what are the things that do happen during pregame.. and that came up so trial and error...
Regarding the badges request, that came from a different angle. Since a 42 player server pushes absolutely every limit, it also sent the Hive servers more requests than they accepted at the time.
(When a player connects, it contacts UWE's Hive server and checks if that player has any badges that need displaying).
This caused a lag spike right when players load after a mapcycle.
We talked with Acedude who manages the Hive API server and he doubled their capacity. The lagspike was fixed.
However, during all of that, we found a way to en-/disable the badgesrequest and since that was also at the time we had the pregame crashes, it -appeared to play a role- in it.
The badges request is the other thing the server does during pregame, once for every player that connects / loads the map.
Glad you asked
I have disabled these features since ages and the server is crashing more since 267.
And since 268 there can be crashes every 3rd mapchange.
I was alone on the server @ one test using the sv_cyclemap map command and the server crashed at first mapchange.
My suggestion is the new cache mechanic in 268.
And again, we have still no feedback from an admin of an vanilla server.
seems like there are some critical issues at map loading. To investigate into this we need to get crashdumps.
How to enable crash dumping for your windows server:
1. Open the cmd
2. Type in regedit
3. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\
4. Create a new key called LocalDumps
5. Wait for the server.exe to crash
6.Open run ( with win + r ) and type in %APPDATA%\CrashDumps now there should be a file.
7. Zip it, upload it somewhere and post the link here
Thanks for your help
Did you try the three things I mentioned? Does it help, not help?
Whenever we enable even one of the three things mentioned (2 votes & badgereq) we get the pregame crash. So if you disable just one... oh well see for yourself.
If you can't figure it out, add me on steam and I'll take a look (provided you have vnc installed).
Regarding the pregamecrash, you need to disable all three things mentioned above, not just one.
EDIT:
As a matter of fact, we've just now had a pregame crash with 8-10 players on the server. And one player reported specifically to have hit the randomizerr vote just before. (We activate those after every build release to check if it's fixed. . .)
We do have a dump of it and from the last dumps of this type, a dev said it's a luaj crash which likely originates from the older lua scripts and libs.
I am starting to suspect the built in voting system. I have had several players reporting server crashes just as they called a vote. Just now a reset vote was called and then the server crashed.
This is what works for us.
Well if even the reset vote causes it, that would make sense. According to the devs all those votes call the votemanager parent class which checks for vote validity. The crash is then caused by an luaj call (taken from dumps). Only @fsfod can tell which luaj call it is, since he wrote the lua vms.
Prepare your offerings. [-O<