Random server crashes (build 268)

2»

Comments

  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    I haven't had any crashes at all since 269. All votes enabled etc.
  • The_RangerThe_Ranger So.Cali Join Date: 2003-01-27 Member: 12800Members, Constellation, WC 2013 - Shadow
    I had a few crashes in 269 while votes were on, 0 while off. I plan to turn them on again and see if if it stays the same, as I have also made a few other changes.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    We are currently testing with the shine afkkick extension disabled.
    Although looking at its lua it only hooks into VoteRandomizeRR, not VotingForceEvenTeams.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    dumping this here as I'm in a hurry:

    [372.322] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #5: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VoteRandomizeRR"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    [375.670] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #6: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VotingForceEvenTeams"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    Got those during a pregame, they might help point to the origin.
  • the_tickthe_tick Netherlands Join Date: 2014-01-20 Member: 193352Members
    I think this is a good spot to look at, just now, ns2_summit just loaded in. and after about 2 minutes ( which is also my afk kick timer) the server crashed
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    The shine afkkick extension disabled does not stop the pregame crash, so we can rule it out as the culprit!

    What remains are the built-in votes. Those need to be looked at.
  • the_tickthe_tick Netherlands Join Date: 2014-01-20 Member: 193352Members
    I've disabled built in votes for 6 months or so now, can't be that...
  • remiremi remedy [blu.knight] Join Date: 2003-11-18 Member: 23112Members, Super Administrators, Forum Admins, NS2 Developer, NS2 Playtester
    ATF wrote: »
    dumping this here as I'm in a hurry:

    [372.322] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #5: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VoteRandomizeRR"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    [375.670] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #6: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VotingForceEvenTeams"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    Got those during a pregame, they might help point to the origin.

    That is a really odd call-stack. Wonder if it's some issue with luajit doing something weird.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    I scrabbled about in our log archive and found some info on these:
    1. They started appearing on September 2nd.
    2. They appear during pregame only, maybe at a 3% rate. Meaning mostly they don't appear.
    3. There's not just those two but also for the other 3 votes. So it happens for all vanilla votes.

    There's a lot in common with the preconditions for the pregame crash. I'd say they are a hint.

    You are already in posession of 15-20 dumps of those crashes, and yes the last thing in those is a luaj call according to Ghoul.

    Additionally -and this is like a ghost story-, we've always noticed a slight tendancy for the pregame crash to happen on veil and veil five specifically. It also happens on random other maps, albeit much more rarely.
    Recently we've added veil five again and out of 8 times it got voted, 4 times pregame crash. Now the map is out again.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    Finding a bunch of these in the log today:
    [9258.987] Main : Error: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value)
    [Server] Script Error #1: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value)
        Call stack:
        #1: StartVote lua/MixinDispatcherBuilder.lua:230
            functionName = "OnGetIsSelectable"
            allFunctionsTable =  {1=function, 2=function }
            self = "VoteKickPlayer"
            count = 2
            className = nil
        #2: lua/Voting.lua:114
            client = ServerClient { }
            message =  {client_index=0, expireTime=0, kick_client=14, voteId=0 }
    
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    So far no pregame crash since 270, with all but one ingame-vote enabled. :)>-
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    edited November 2014
    Having the pregame crash on ns2_yana. Yep, it's confirmed the exact same symptoms and repro. We're investigating.

    EDIT: Same on ns2_orbital.
  • TermFragsTermFrags Tampa, FL Join Date: 2012-08-30 Member: 156877Members, NS2 Playtester
    Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:

    Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
    Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
    Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]


    I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server. ;)
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    TermFrags wrote: »
    Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:

    Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
    Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
    Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]


    I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server. ;)

    Ironically, I've also had tons of crashes in the past of couple of weeks. Especially during mapchanges. Decided to take a proper look today and what do you know, disabling [SHINE]NS2stats solved the problem. Not a single crash in over 24 hours. ;o
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    @Amad‌ Same here. Ghoul runs NS2Stats without problems though. Perhaps it's one of the settings in ns2stats.json causing it?
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    edited November 2014
    I'm refactoring ns2stats just another time to see if it can fix it at my side. Fun note thanks to my new IDE tools ns2stats already runs now 4-5 times faster.

    But the io stream issues seem to still be there when it comes to reading larger json files.



  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    SupaFred wrote: »
    @Amad‌ Same here. Ghoul runs NS2Stats without problems though. Perhaps it's one of the settings in ns2stats.json causing it?

    Have you tried changing the settings yet? I'm going to toy around with them tonight.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    @Amad‌ No not yet. The settings I changed from default are
    "Awards":false,
    "LogChat":true,
    "StatusReport":true

    What settings did you use?
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    The only settings I've ever changed from that .json file was awards, set it to false. Now I tried all different options but no go, server keeps crashing between map changes, randomly but crashes nevertheless. Can't use ns2stats at all. :/
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    Get me some crash dumps please ;)
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    So apparently Compmod doesn't like NS2stats. With compmod enabled and NS2stats disabled, everything works (Obviously works other way around too). These two mods enabled however...server will crash eventually between mapchanges.

    Saw this in the previous server log:

    [66295.006] Main : Error:
    EntityChangeMixin.optionalCallbacks =

    {

    OnEntityChange = "Called when an entity changes into another entity or is destroyed"

    }



    if Server then



    /**

    * Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.

    */

    function EntityChangeMixin:SendEntityChanged(newId)



    // This happens during the game shutdown process, so don't force a new game

    // rules to be created if one doesn't already exist.

    if GetHasGameRules() then


    [Server] Script Error #1:
    EntityChangeMixin.optionalCallbacks =

    {

    OnEntityChange = "Called when an entity changes into another entity or is destroyed"

    }



    if Server then



    /**

    * Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.

    */

    function EntityChangeMixin:SendEntityChanged(newId)



    // This happens during the game shutdown process, so don't force a new game

    // rules to be created if one doesn't already exist.

    if GetHasGameRules() then


    Call stack:
    #1: read [C]:-1
    #2: OnScriptLoaded lua/EventTester.lua:169
    fileName = "lua/EntityChangeMixin.lua"
    file = userdata
    #3: Load lua/EventTester.lua:193
    fileName = "lua/EntityChangeMixin.lua"
    reload = nil
    #4: lua/CompMod/NewTech/TunnelExit.lua:16
    #5: scriptLoad [C]:-1
    #6: Load lua/EventTester.lua:191
    fileName = "lua/CompMod/NewTech/TunnelExit.lua"
    reload = nil
    #7: lua/CompMod_Shared.lua:11
    ModFiles = {1="lua/CompMod/NewTech/GorgeTunnelEntranceAbility.lua", 2="lua/CompMod/NewTech/GorgeTunnelExitAbility.lua", 3="lua/CompMod/NewTech/HeavyMachineGun.lua", 4="lua/CompMod/NewTech/TunnelExit.lua", 5="lua/CompMod/NewTech/WalkMixin.lua", 6="lua/CompMod/NewTech/Welder.lua" }
    (for index) = 4
    (for limit) = 6
    (for step) = 1
    i = 4
    #8: scriptLoad [C]:-1
    #9: Load lua/EventTester.lua:191
    fileName = "lua/CompMod_Shared.lua"
    reload = nil
    #10: lua/CompMod_Server.lua:4
    #11: scriptLoad [C]:-1
    #12: Load lua/EventTester.lua:191
    fileName = "lua/CompMod_Server.lua"
    reload = nil
    #13: lua/ModLoader.lua:116
    modEntries = {1= {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }, 2= {Client="lua/NS2Plus/CHUD_Client.lua", ModName="CustomHUD", Predict="lua/NS2Plus/CHUD_Predict.lua", Priority="50", Server="lua/NS2Plus/CHUD_Server.lua" }, 3= {Client="lua/Badges+_Client.lua", ModName="Badges+", Priority="4", Server="lua/Badges+_Server.lua" } }
    entryFiles = {1="lua/entry/Badges+.entry", 2="lua/entry/CompMod.entry", 3="lua/entry/CustomHUD.entry" }
    ParseEntryFile = function
    SortByModPriority = function
    (for index) = 1
    (for limit) = 3
    (for step) = 1
    i = 1
    modEntry = {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }
    #14: scriptLoad [C]:-1
    #15: Load lua/EventTester.lua:191
    fileName = "lua/ModLoader.lua"
    reload = nil
    #16: lua/PostLoadMod.lua:12
    #17: scriptLoad [C]:-1
    #18: Load lua/EventTester.lua:191
    fileName = "lua/PostLoadMod.lua"
    reload = nil
    #19: lua/Server.lua:462
    chatMessageCount = 0
    reservedSlots = {amount=0, ids= { } }
    kMapEntityLoadPriorities = {ns2_gamerules=1 }
    GetMapEntityLoadPriority = function
    DumpServerEntity = function
    LoadServerMapEntity = function
    CheckForDuplicateLocations = function
    OnMapPostLoad = function
    OnCanPlayerHearPlayer = function
    OnCheckConnectionAllowed = function
  • babblerblingbabblerbling Join Date: 2015-05-27 Member: 204951Members
    Newest intel on this one is that it appears to be dependant on the server hardware. Or at least, when you run the very same server on two different boxes, there will be different maps affected.

    let's say there's server [A] and it's crashing in pregame on maps like veil veil_five mostly and oddly on some customs +summit.
    We move server [A] to hardware and suddenly there's yana, orbital that crash everytime and oddly uplift.
    These are real observations.

    Here comes server [C] and crashes in pregame a lot on mineshaft, almost always when one of the ingame votes is called/succeeds. Disabling all votes helps... a bit.

    It's just like how the now fixed (worked around) workshop issues affected servers (in alternating geographical locations) differently.
Sign In or Register to comment.