We want your Server crash reports!

IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
edited June 2016 in Server Discussion
Dear NS2 Server Operators,

During the time of rebuilding our new build system (which is now online), we’ve also reviewed and restructured our working process.

While doing so we’ve realized that in the past, mod related issues connected to the LUA VM or Spark API may at some point cause major server performance and stability issues with vanilla NS2.
These issues slipped by or didn’t get the needed attention by the dev team as it pertained to mod usage.

From now on the CDT will attempt to address these issues to the best of our ability by using the following process to collect and manage server issue reports.
This means from now on we will accept any kind of report that is submitted to this thread on our official tech support forums. Your report must follow this protocol:

Date of crash:
Used NS2 Build Version:
Map the crash occurred on:
Server details:
Name:
IP:
All Mods:
Description of the situation the crash occurred :
Files to link (do not upload them directly to the forum!) :
  • Server log
  • Crash dump / gdb backtrace (large files must be compressed as zip file)
  • (Optional) server config files
Information how to fetch the needed files are at the end (appendix) of this message.


Any report not containing ALL of the needed details will be ignored!!
Also reporting an issue to us does not guarantee a fix will be developed or help given.


All valid reports will be reviewed as soon as possible by the following CDT members: @Asraniel , @GhoulofGSG9 , and @fsfod
If you found an exploit, do not share it with the public, instead send your report to ns2cdt[AT]gmail[DOT]com .

But please remember that we are still only a volunteer team! Therefore have patience if we don’t answer you right away, or inform you that we have prioritized something else in the meantime.
We will attempt to update you on every matter as soon as we are able.

We also still won’t fix any script errors related to mods. However we will try to inform you what mod is causing the given issue and provide technical details.
In those cases, it’s up to you to contact the mod author for help and make sure that he/she is fixing the issue.

Thank you for your reports!




Appendix:

How you fetch the log.txt :
  • The log should be in your logdir or by default in %appdata%/Natural Selection 2/ (Windows) or ~/.config/Natural Selection 2/ (Linux)


How to create a Dump / GDB backtrace:
  • On Windows:
    You can either create a dump manually with a tool like the process explorer
    Or follow this guide to enable that full crash dumps get auto generated if an application crashes.
  • On Linux (debian-based):
    In some cases you have to install gdb first for your system via “sudo apt-get install gdb”

    After that you have to find out the PID of your ns2_server process. For this use
    pidof <name of the ns2_server process>
    

    Then start gdb with:
    gdb 2>&1 | tee gdb-<program name>.txt
    (gdb) handle SIG33 pass nostop noprint
    (gdb) set pagination 0
    (gdb) attach <PID>
    
     gdb-<program name>.txt
    
    is name of the text file the gdb output will be save into later.

    Next let the ns2 server process continue to work via
    (gdb) continue
    

    If the server crashes now run these commands:
    (gdb) backtrace full
    (gdb) info registers
    (gdb) x/16i $pc
    (gdb) thread apply all backtrace
    (gdb) quit
    

    Attach the
    gdb-<program name>.txt
    
    to your report.

Of course all of the above steps could be also put into a startup script so you don’t have to run them every time yourself.



If you need for further help with fetching the needed data, either ask in this thread or try to google for an answer.

«1

Comments

  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 26.06.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_honorguard
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.38.164:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri, "7404cfb" ns2_honorguard
    Description of the situation the crash occurred:
    Happens exclusively during a pregame phase, sometimes on it's own, often when starting a vote or when a vote succeeds.
    Affected maps can be officials or customs. For us it's almost exclusively the following maps: orbital, yana, temple, uplift. We have to exclude them :'(, making the described crash incidents rare.
    This crash report shows it happen on honorguard.

    Link with dump, log and config.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 14.07.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_forgotten
    Server details: win7pro 64bit, Z77, i5-2550k oc'd, 12Gb Ram oc'd
    Name: Wooza's Hamster Wheel
    IP: 188.63.38.164:27017
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri, "16A96DC0" rr picture, "136d050a" ns2_forgotten
    Description of the situation the crash occurred:
    Second round on that map, crash out of the blue after ~12 days uptime (notice the mem usage). No other details known. It was not detected as a hc, i.e. it exited on its own.
    We moved away from daily restarts a good while ago since they have zero effect on the likelyness of a server to crash, at least for us.

    Link with dump, log and config.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 30.06.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_gorgon
    Server details: win7pro 64bit, Z77, i5-2550k oc'd, 12Gb Ram oc'd
    Name: Wooza's Hamster Wheel
    IP: 188.63.38.164:27017
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri, "16A96DC0" rr picture, "16c11635" ns2_gorgon
    Description of the situation the crash occurred:
    After map change, most players have finished loading. Someone started a FET vote.

    Link with dump, log and config.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 17.06.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_descent
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.38.164:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    Random crash mid round. No details known.

    Link with dump, log and config.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 15.07.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_docking2
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.38.164:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri, "DDAB434" ns2_docking2
    Description of the situation the crash occurred:
    After map loaded and most clients had finished loading, the start of a FET vote instantly crashed it.

    Link with dump, log and config.
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    map - veil
    running - ns2+, shine, epsilon,
    ver : 274
    date : july 16th


  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    edited July 2015
    List of the until now reported issues:
    http://forums.unknownworlds.com/discussion/138140/list-of-known-server-crashes

    Rusty_1 wrote: »
    map - veil
    running - ns2+, shine, epsilon,
    ver : 274
    date : july 16th
    Please follow the in the OP given protocol!
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    edited July 2015
    Date of crash: 16.07.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_summit
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.38.164:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    Hc mid round on summit. All the well known hc symptoms.

    [EDIT]: Apparently the symptoms require elaboration once more:
    With this crash, the only part that still works after it entered this state is the server status query. I.e. when you refresh server list with empty ones filtered out, it will briefly display with the playercount at the time of crash then disappear from the list.
    When all filters are off, it'll show the server as if nothing is wrong at all. The last state of the players is still visible in details, with their connection time counting up while the score never changes.
    All clients time out the moment the server enters hc state.
    We detect the hc by using a tiny lua script that writes a ping every second into a text file. When the server enters the half crash state, this file stops being updated.
    The server process does not exit (maybe 1 in 500 cases it will exit) and never recover. No manual recovery methods are known.
    Some servers that run into this crash remain there for months until someone notices.
    The 42 slot attracts this crash, averaging 2-3 times per day.
    In contrast, the 32 slot will run into the half crash only once a week on average.

    Link with dump, log and config.

    [EDIT2] Why isn't this appearing in known server crashes? This has been around for years.
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    Hope this is a better format Ghoul.

    Date of crash: July 16th 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: Unknown
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Unknown
    Files to link (do not upload them directly to the forum!) : http://pastebin.com/PGw5kTPu
    Server log : http://pastebin.com/xqR6LZQD
    Crash dump / gdb backtrace (large files must be compressed as zip file)
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    Rusty_1 wrote: »
    Hope this is a better format Ghoul.

    Date of crash: July 16th 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: Unknown
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Unknown
    Files to link (do not upload them directly to the forum!) : http://pastebin.com/PGw5kTPu
    Server log : http://pastebin.com/xqR6LZQD
    Crash dump / gdb backtrace (large files must be compressed as zip file)

    Thanks you just confirmed that the Issue with ID 3 affects both OS
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 18.07.2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_nexus
    Server details: win7pro 64bit, Z77, i5-2550k oc'd, 12Gb Ram oc'd
    Name: Wooza's Hamster Wheel
    IP: 188.63.38.164:27017
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri, "16A96DC0" rr picture, "17b0e28c" ns2_nexus
    Description of the situation the crash occurred:
    After map change, most players have finished loading. Crash without a round having started.

    Link with dump, log and config.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    Date of crash: July 20, 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: Biodome
    Server details: Windows 7
    Name: [SWE] The Thirsty Onos #22
    IP: 85.229.117.11
    All Mods: NS2+, Shine Admin, Epsilon, Disabled first person spectate, Badges+
    Description of the situation the crash occurred : Empty server, looks like it happened just after a map change.
    Crash dump and server log zipped on Dropbox.
    Server log on Pastebin.

    Let me know if you want the config files or anything else.
    Thanks.
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    Date of crash: July 22nd 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: ns2_veil
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Unknown

    GDB dump : http://pastebin.com/NTWnGRjM
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    edited July 2015
    Date of crash: July 22nd 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: unknown
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Unknown

    gdb dump - http://pastebin.com/WniDKwf3
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    edited July 2015
    Thanks everyone, categorizing and looking over these.
    Be sure to look at our List of Known Server Crashes before reporting, to see if they differ at all from yours (from OS to map to mods etc) and if they don't at all, just feel free to mention "I experienced issue #3 with no new data".
    The idea being we don't want you to have to spend the time to upload dumps etc if there is no difference at all... but if you have _any_ difference then your data could still provide more useful information.
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    Date of crash: July 26th 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: veil
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Tried to do a vote reset - immediately crashed server.

    http://pastebin.com/rQ9ieFLm
  • RustyRusty Canada Join Date: 2013-07-02 Member: 185850Members, NS1 Playtester, Constellation, NS2 Playtester, Reinforced - Silver
    Date of crash: July 27th 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: unknown
    Server details: Linux, Debian 64 bit
    Name: DiamondGamers - Dallas1
    IP: 45.35.1.10:27015
    All Mods: NS2+, Badges+, Shine, Epsilon,
    Description of the situation the crash occurred : Unknown.

    But this error comes up as it crashes.
    0.166442871) at ../Source/Engine/Model.cpp:3162
    3162 ../Source/Engine/Model.cpp: Permission denied.
    Not sure if that helps.

    http://pastebin.com/C8M7N1gb
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    edited July 2015
    Date of crash: July 22, 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: Eclipse
    Server details: Windows 7
    Name: [SWE] The Thirsty Onos # 16
    IP: 85.229.117.11
    All Mods: NS2+, Shine Admin, Epsilon, Disabled first person spectate, Badges+
    Description of the situation the crash occurred : Empty server, looks like it happened just after a map change.

    Crash dump and log
    Log
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    SupaFred wrote: »
    Date of crash: July 22, 2015
    Used NS2 Build Version: 274
    Map the crash occurred on: Eclipse
    Server details: Windows 7
    Name: [SWE] The Thirsty Onos # 16
    IP: 85.229.117.11
    All Mods: NS2+, Shine Admin, Epsilon, Disabled first person spectate, Badges+
    Description of the situation the crash occurred : Empty server, looks like it happened just after a map change.

    Crash dump and log
    Log

    Link to dump file is not working.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    Link to dump file is not working.

    Sorry, it's fixed now.

  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 18.10.2015
    Used NS2 Build Version: 276
    Map the crash occurred on: -
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.57.183:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    This happens ~5 times per week for both servers when they are restarted:
    We have significant delays set up to avoid having bugs arise from a very immediate server restart after crash/termination. Despite that, it sometimes just remains stuck at this point:
    Date: 10/18/15
    Time: 15:47:21:
    --------------------------------------------------------------
    [  0.000] Main : Filesystem initialized, cache enabled
    Build 276
    Windows 7 64-bit (build 7601)
    [  0.056] Main : Error: PhysX: Invalid Cuda context!
    Server tickrate 30, client sendrate 20, bandwidth limit per player 25600
    
    It will not continue and not exit on it's own. Just like the hc, lol. Usually it gets terminated by our restartscript once over which is a lot of fun for players that want to reconnect after server crash <3

    This is decidedly not a 276 issue, it's happened multiple builds before.

    Haven't been able to capture a dump yet.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    edited October 2015
    Got lucky, here's a dump of a server stuck like this, it's from the wheel but same issue.

    It happened on first start after applying the ninja patch.

    Just to show how often this happens:
    oCw54Ab.jpg
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 20.10.2015
    Used NS2 Build Version: 276.1
    Map the crash occurred on: ns2_descent
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.57.183:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    Unknown, about 10minutes into first round on this map it crashed and a dump was autosaved by OS.
    Server was not at capacity, around 31 players.

    It's not a pregame crash and not a half crash.

    Dump & Log

  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Woah. Same as above, again on descent. New type of crash?
    Dump & Log
    I wonder if anyone is even looking at these...
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    edited October 2015
    To clarify: we won't comment on each report.

    If we found a new bug it will be added to http://forums.unknownworlds.com/discussion/138140/list-of-known-server-crashes.
    Otherwise your report points towards an already known issue.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    edited October 2015
    Id 3 then? Alright. But what about the stalling and not starting ones? Check out my awesome drawing.


    Edit: Just after I made that picture, it happened again.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    Date of crash: 08.11.2015
    Used NS2 Build Version: 277
    Map the crash occurred on: ns2_descent
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.57.183:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    Typical pregame crash which was claimed fixed multiple times but never actually was. Here on descent during pregame (some players still connecting). This is often either on initiation or success of an ingame vote.
    Log is not included as it says nothing.
    What did happen some weeks before 276 (not 276.1) was released is that the target of the pregamecrash changed (away from most custom maps to descent and summit).
    It also became much more rare.
    Since then ns2_yana is running twice a day on average. It was previously unplayabe due to this crash.
    We did not change anything about the hardware, the only thing changed is a corrupted perfhist.bin replaced with a new one...

    Dumpy dump.

    It's obvious that you only add issues based on personal agenda. Meaning half are left in pretense of their non-existence. Good going.
  • FursFurs Join Date: 2013-09-20 Member: 188365Members, NS2 Playtester, Reinforced - Shadow
    Date of crash: 18/11/2015
    Used NS2 Build Version:277
    Map the crash occurred on: ns2_fracture (didnt even exist on the server)
    Server details: Ubuntu 12
    Name: Wraith :: NSL :: Amsterdam #1 - Hosted by Furs
    IP: 89.105.209.250
    All Mods: a2ddae8 812f004 e5ffa15 706d242
    Description of the situation the crash occurred : sv_changemap ns2_fracture (map doesnt exist, and crashed on cycle)
    Files to link (do not upload them directly to the forum!) : http://f0rc3.net/dump/ns2-server-linux-connect-hang.tgz

    Seems a admin changed it to a map that didnt exist, resulting in a unuseable server.
  • BabblerKingBabblerKing Join Date: 2015-07-12 Member: 206120Members
    edited November 2015
    Date of crash: 19.11.2015
    Used NS2 Build Version: 277
    Map the crash occurred on: ns2_veil
    Server details: win7pro 64bit, Z97, i7-4790k oc'd, 16Gb Ram oc'd
    Name: Wooza's Playground
    IP: 188.63.57.183:27015
    All Mods: "706d242" Shine, "812f004" NS2+, "15BE2873" Potpourri
    Description of the situation the crash occurred:
    Typical pregame crash. Not posting every single one of these.

    Dump+Log
    https://www.dropbox.com/s/fw67pe8wmhwacav/Server.exe.4920-pregamecr-ns2_veil.7z?dl=0
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    edited November 2015
    @Furs check mantis, I think that old bug (that i am sure was fixed at some point?) has just returned or there is some other parameter that was not fixed.

    @babblerking So repro steps are random, during pre game only?
Sign In or Register to comment.