Help us reproduce the "phantom players" bug

Flayra

Hey everyone,

We've been seeing "phantom players" in servers and really want to fix it. However, we can't even reproduce the dang thing. The problem is something like this: possibly after a game end, round reset, there are players in game that have the names of players that were just playing, but they are floating in mid-air and can't be killed. They seem to be totally inert. They take up spaces on the scoreboard and often those players will be renamed. Ie, if I was playing, suddenly we'd see a phantom player called "Flayra" and I would be renamed to "Flayra (2)".

Let's use this thread to figure out how to reproduce this...then we'll have it fixed!

Thanks,

Runteh

When the game crashes and I restart it and rejoin the same server, sometimes there are 2 versions of me. I have seen this for other players as well.

Hope this helps?

ZelDerX

I may be worng at this but there is a similar thing to this in tf2

If a player suddenly DC from the server there is a chance that his "player" will still remain at the server taking a slot but do nonthing,

I think also you cant kill him with all weapons.

luns

this happened to me when my pc crashed, and when I reconnected I was still connected. Other times if my NS2 crashed, I still show up connected when I rejoin. happened to me maybe twice.

Papayas

I think it could be because of the player's game crashing. I am not fully sure though.

Runteh

Is there a message sent to the server from the client indicating a disconnect somehow? I haven't a clue when it comes to programming, but if a player is crashing and the message is not being sent to the server, perhaps the server is not registering the disconnect - yet no info is being sent from the client.

throttle

If I had to hazard a guess this is a networking/socket issue combined with peoples firewall/routers. There's some dissonance on what an open connection is once the game crashes and the socket disappears. Depending on peoples rules, the router might forward packets to the client, but the client drops the packets -- thus you have to wait for the full timeout on the connection for connection to go away. Meanwhile the client starts up again with a fresh range of ephemeral source ports....

OR

The router/firewall is being too smart for it's own good and keeps sending keepalives for non-existent connections until it times out it's own connection clientside entry...

weezl

yep, this is what i was gonna say too...

this happened to me once:
i was playing normally and suddenly the game starts to connect/load (just as if admin changes map in NS1) so i think an admin changed map.
but it loads and connects me to the exact same game that hasn't stopped running or reset. there, almost all the other players are left standing frozen where they were when this happened. Those who reconnect get a (2) or (3) added to their name.

Simon493

Not sure if you saw this I posted in another thread but here is copypasta:

Not sure if it helps but I noticed that when I crashed and rejoined the same server there was a clone myself that was a "phantom" player. I think I crashed immediately when I joined the marine team although I'm not certain. If I had to guess it is the client shutting down but not telling the server it is disconnection, server sits there thinking the phantom player is still there because nothing has told it the client disconnected. Maybe after 30s-1 min if a client doesn't send anything to the server the server pings the client and if there is no response it times them out?

Runteh

I was going to mention that as well Weezl. Not sure if the devs have experienced this but you get the server doing a few different types of auto restarts.

1. The game just restarts mid game at spawn positions, like at the beginning of a round start. (Also I was able to automatically upgrade to L2 CC, w/ exisiting upgrades)
2. The same has happened before but it restarts with all buildings/structures in existing positions.
3. As Weezl mentions.

MadMakz

sounds like a timeout/session bug / server keeps lost or respective improper closed connections

Lazer

I really hope this isn't from servers not setting a timeout to check connectivity. That would be a horribly obvious thing to overlook and one of the first features that should be added into any networked game. I am going to guess that is not the case but it really does look like it, or something related to it's logic is incorrect.

I wanted to say maybe somehow the client receives the information of these ghosts and doesn't know to remove them from the list of entities to render, but if it's taking up slots on a server it has to be a server problem so it clearly has to do with the server not timing out clients properly.

Alpha.

1. I'm playing on some server
2. Then I try to connect to another server [or just reconnect]
3. Game crashes while map is still loading [30% times]
4. After restarting a game and connecting to same server - there's a ghost

Mr.InTeLeX

I've seen this bug many times within other fps games&mod (especially with hl1 and hl2 mod, also included some other ones as well). All of those bugs occured when your game crashed while you're still playing in a server. After you join that server again, the ghost appeared and you are renamed.

1stToast

I joined a server as alien and there were several floating marines. I was able to bite kill them. I have seen a similar effect in DODS while playing around with the config file. I disabled the rag doll function and when anyone was killed a floating body appeard and couldn't be removed.

Asraniel

in that case, a partial solution is to send a ping to the player with the same name, and kick him if he does not respond. Perhaps this should not be made with NSplayer players.

Skie

<!--quoteo(post=1834839:date=Feb 24 2011, 11:49 PM:name=Alpha.)--><div class='quotetop'>QUOTE (Alpha. @ Feb 24 2011, 11:49 PM) <a href="index.php?act=findpost&pid=1834839"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->1. I'm playing on some server
2. Then I try to connect to another server [or just reconnect]
3. Game crashes while map is still loading [30% times]
4. After restarting a game and connecting to same server - there's a ghost<!--QuoteEnd--></div><!--QuoteEEnd-->
This. The game crashes quite often when you join a server while playing/after having played on another server. Then when you connect back there, there will be a ghost.

Also happened to me once when the game 'crashed', ie the screen went black during a game and it just said "Connection problems...", then I disconnected using the console and connected back - there was a ghost.

BomBom

I have had this issue when crashing and rejoining aswell, i then decided to use overmind to kick the 5 players i had on my server which were ghosts with different names and the same steamid from what overmind was saying, instantly crashed the server, if you add some sort of timeout options it could fix the issue?

Alpha.

<!--quoteo(post=1834847:date=Feb 25 2011, 12:19 AM:name=Skie)--><div class='quotetop'>QUOTE (Skie @ Feb 25 2011, 12:19 AM) <a href="index.php?act=findpost&pid=1834847"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Also happened to me once when the game 'crashed', ie the screen went black during a game and it just said "Connection problems...", then I disconnected using the console and connected back - there was a ghost.<!--QuoteEnd--></div><!--QuoteEEnd-->
happened here too, must be related to bsod in some way


<!--quoteo(post=1834850:date=Feb 25 2011, 12:32 AM:name=BomBom)--><div class='quotetop'>QUOTE (BomBom @ Feb 25 2011, 12:32 AM) <a href="index.php?act=findpost&pid=1834850"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->(...)then decided to use overmind to kick the 5 players i had on my server which were ghosts (...) instantly crashed the server<!--QuoteEnd--></div><!--QuoteEEnd-->
happens every time I try it, on every server [tried at least 3 different machines]. sometimes OM does not show them though

Mordrake

Not sure it will help but I see that this bug happens on a LAN game too.
When changing team when the round hasn't begun. I join the Marines, I type "RR" in the console for the Ready Room, then I join the Aliens and the bug appears sometimes.

Camron

Here's my copy paste from <a href="http://getsatisfaction.com/unknownworlds/topics/map_reload_ghost_clients_server_bug" target="_blank">Get Satisfaction</a>

"The server I was playing b161 on randomly initiated a map reload for about 3-4 players. This was completely out of no where and the game was not over or anything. We then found ourselves back in the Ready Room while the game was going on fine for everyone else (they didn't get the map load screen). While we were in the RR, we could see our old names on the scoreboard still(we were still on our old teams) and our "current" selves had names like "Camron(3)". After we rejoined the teams, our old ghost selves were still spawning and were still acting as clients (taking up a server slot). When looking at the scoreboard, my name would appear on my old Alien team as "Camron", and then 3 seconds later it would move and appear on the true Marine side as "Camron(3)". After another 3 seconds it would go back to the other side and this kept repeating. This was visible to everyone and not just me. I exited the game and rejoined but the ghost clients were still there. After the game was won and everyone was transported back to the RR, the ghost clients were still there. This happened on a server named "SitYour-fAsstInternet[SP]" on 1/18/2011 around 9:30 pm. First it should be checked if the server is running "NS2-GmOvrmind" server mod as that may be at fault. Does anyone know how to contact the operator of that server? A server log would be helpful."

endar

I have had what Camron just described happen on my server multiple times, but not for quite a few builds. It appears like a mapchange, but yea no change, game still going. I am running GM overmind, and can confirm that I did not do any commands to cause it, or preceding it. I don't think it is related to GMO, but who knows.

This is not the only way to cause the ghost players.

NurEinMensch

<!--quoteo(post=1834817:date=Feb 24 2011, 09:44 PM:name=weezl)--><div class='quotetop'>QUOTE (weezl @ Feb 24 2011, 09:44 PM) <a href="index.php?act=findpost&pid=1834817"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->yep, this is what i was gonna say too...

this happened to me once:
i was playing normally and suddenly the game starts to connect/load (just as if admin changes map in NS1) so i think an admin changed map.
but it loads and connects me to the exact same game that hasn't stopped running or reset. there, almost all the other players are left standing frozen where they were when this happened. Those who reconnect get a (2) or (3) added to their name.<!--QuoteEnd--></div><!--QuoteEEnd-->


I have experienced exactly the same thing.

Pathdrc

<!--quoteo(post=1834842:date=Feb 24 2011, 05:03 PM:name=Mr.InTeLeX)--><div class='quotetop'>QUOTE (Mr.InTeLeX @ Feb 24 2011, 05:03 PM) <a href="index.php?act=findpost&pid=1834842"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->I've seen this bug many times within other fps games&mod (especially with hl1 and hl2 mod, also included some other ones as well). All of those bugs occured when your game crashed while you're still playing in a server. After you join that server again, the ghost appeared and you are renamed.<!--QuoteEnd--></div><!--QuoteEEnd-->

I haven't seen this happen in TF2, like a previous post mentions, and I have not seen it happen in HL1 or HL@ (but I have not played either in over a year). Neither have I seen it in NS!.
Having said all that, though, if it does happen in these games, then maybe the problem has something to do with Steam?
That would be the commonality between HL1, HL2, and NS2.

player

Yes this bug again, It's been causing me a bit of aggrivation. GmOvrmind operates under the assumption that multiple clients with identical SteamIDs are not allowed, and thus behaves unpredictably (read: crashes) when that does happen. Though v4 is supposed to at least keep players out while a 'phantom'-version of them is still present, preventing errors to occur in the admin-mod (v5 will kick phantoms). Reproducing this problem can be tricky, as it only happens to me when my client dies (crashes), which isn't something that you can reproduce reliably. Perhaps you can ask Max to build in a trigger to cause a nasty access-violation on your client just for this occasion. This of course begs the question why the engine doesn't kick clients that have obviously timed-out (a round ping-time of 15+ secs pretty much constitutes a dead client), or even allows the same SteamID twice in the same server (this shouldn't be acceptable behaviour).

[EDIT]
Forgot to mention, it's highly unlikely GmOvrmind causes this, as I can think of no logical reason for it to, and these phantom-players occur in just about every server (with or without GmOvrmind).

SN.Wolf

By removing phantoms do you mean checking to see if client is really there or looking for duplicate names as when several people join at once they are listed with same default name followed by a # until they load completely in.

And, yes this does happen in servers without GmOvrmind on them so it is not related to the mod (plugin?).

Thaldarin

I find if you make an 'open world' it's much easier to re-create. Start flying as a lerk or jumping on and around near the edges of faces as a skulk, that was how I have encountered it twice. I had to create the server by loading console typing 'map name' to get this, loading via the actual UI screen didn't work for me trying this.

player

<!--quoteo(post=1834984:date=Feb 25 2011, 09:34 PM:name=SN.Wolf)--><div class='quotetop'>QUOTE (SN.Wolf @ Feb 25 2011, 09:34 PM) <a href="index.php?act=findpost&pid=1834984"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->By removing phantoms do you mean checking to see if client is really there or looking for duplicate names as when several people join at once they are listed with same default name followed by a # until they load completely in.<!--QuoteEnd--></div><!--QuoteEEnd-->
Not sure if this was directed to me, but it checks for duplicate SteamIDs.

But again, I'm not sure why this thread is even here, as the problem is pretty obvious:
<u>Client crashes, therefore doesn't exit gracefully</u>; <b><- Ideally crashes shouldn't happen, but we'll let this one slide.</b>
<u>Server didn't get a disconnect-msg from the client, thinks it's still there</u>. <b><- This isn't out of the ordinary, but a TCP-connection alongside the UDP' can fix this.</b>
<u>Server pings client, doesn't recieve a ping back within a reasonable amount of time, sets ping to naught and does nothing else</u>. <b><- WRONG: kick clients that do not respond to pings.</b>
<u>The client rejoins the server, and there are now 2 instances of the same client (and SteamID).</u> <b><- WRONG: kick clients already present on the server when a new instance of them joins.</b>

Regarding the last remark, I'm not sure whether they intend to allow multiple SteamIDs on 1 server, but they really shouldn't, as Steam isn't designed to do that (which is apparent as steam doesn't allow multiple instances of it's Gldsrc\Source games) and chances are it'll screw up the ingame-overlay\friends-system and whatnot.

My mod already does the last point, and I could have it kick clients with a ping of naught as well, but I really don't want to hack together fixes for what for all intents and purposes is an engine-problem.

[Ninja EDIT again]
Above is a discription of just ONE of the causes of ghost-players, but I think it's responsible for most occurances.

Dalin Seivewright

Solution: Set a server timeout from responses from the client. If no response after X amount of seconds, remove client from server. If the client is legit but somehow going through a lag spurt or something, their client will see lack of response from the server as a disconnect and move on from there.

That is, if this isn't already in place...

Edit: If a rejoin happens in between this interval and a duplicate player is there, verify that the STEAMID is the same. If the connection is the same, remove one player entity from the game and use the other player entity as the controlled player. If the connection is not the same, find the wrong connection and remove its entity.

Alpha.

<!--quoteo(post=1835096:date=Feb 26 2011, 05:56 PM:name=Dalin Seivewright)--><div class='quotetop'>QUOTE (Dalin Seivewright @ Feb 26 2011, 05:56 PM) <a href="index.php?act=findpost&pid=1835096"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Edit: If a rejoin happens in between this interval and a duplicate player is there, verify that the STEAMID is the same. If the connection is the same, remove one player entity from the game and use the other player entity as the controlled player. If the connection is not the same, find the wrong connection and remove its entity.<!--QuoteEnd--></div><!--QuoteEEnd-->

OM doesn't show ghost players <b>sometimes</b> [using om_players, when ghost is shown on scoreboard]. It may mean that server doesn't recognize them as connected [not sure if its servers or OM's fault] = problem may be more complicated, i.e. few bugs independently, not only one is causing ghosts [<b>sometimes</b> OM shows them]

MCMLXXXIV

The most common cause of this that I've seen so far is clients typing "quit" into the console and then NS2 crashes instead of exiting normally. I noticed that that often causes the player to stick around. Usually they do timeout after a while though (not sure if that is Overmind doing the kicking or not).

I'd definitely advise implementing player's strategy and working from there if you still see issues. In particular, the running of a TCP connection in conjunction with UDP should allow you to have a clean distribution mechanism for packets that *need* to arrive (reduces server cpu load for processing lost UDP messages) and for pings between client and the server.