Raised This Month: $12 Target: $400
 3% 

[Neotokyo] Reporting serious Memory Leak to the SM team!


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
glub
AlliedModders Donor
Join Date: Feb 2015
Old 02-26-2015 , 00:57   [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #1

I'm running several Neotokyo Source dedicated server, mostly on Windows but also on Linux (through WINE).



We have had (other Neotokyo server admins too) this huge memory leak issue for years now, and it seems to be caused by SM. SRCDS by itself, with only MetaMod loaded, doesn't leak memory. Once SM is loaded though, it starts eating memory like crazy until a point where the server crashes (well, most likely killed by the operating system).

This screenshot is for a server running on a Windows server box. The "NT (memory) fix" is from this thead consisting of unloading all plugins, adding a bot, then refreshing the plugins. This fix used to work for a few weeks (see this screenshot, the RAM usage is fluctuating but not climbing nonstop, that's the ideal situation as the server never crashes). Now it's not working anymore and I don't know why. I have tested nearly all configurations possible by now.

What does Basecomm have to do with all this? We don't know but we have problems with this base plugin as when it is reloaded, it messes up voice chat in game (Asherkin should know about it, I filed a bug on that plugin recently). From this graph it also appears that it contributes to the memory leak somehow.

Now, after loading manually SM into MetaMod (by renaming the SM folder before starting the server, then renaming again and loading manually), you can see on this screenshot that the memory leak doesn't start straight away, only after a (few?) map changes. This is new to me and very strange.

I tried using UMDH (Windebug) to debug but I'm a complete idiot who don't know anything about computers, and I don't know what I'm doing. Although I noticed stack overflows when starting the server.

I also did sm_dump_handles a few times out of curiosity (with all plugins loaded): 4 dumps.

Does anyone from the dev team have a idea about this? If we can't fix it for whatever reason (Neotokyo is a niche game now), does anyone have any idea how to work around this?

If someone wish to test this on their end and debug, here's the quick install procedure with SteamCMD:
login anonymous
force_install_dir .\NT\
app_update 313600 validate

Additional comments: this memory leak happens both on Linux(wine) and Windows.
It happens with all version of SM and MM, only base plugins too.
You need actually players to join the game for this memory leak to happen. With bots only, the memory usage is more or less stable.
plugins used on regular basis

Last edited by glub; 10-06-2019 at 18:48.
glub is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 02-26-2015 , 06:08   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #2

This all sounds like related problems with the game's handling of the voice flags, can you just remove basecomm (and anything else that manipulates voice flags) and see if this still happens? basecomm's hooks break for you when reloaded, which is the cause of your other bug, so it wouldn't be doing any more flag changes, which would explain the memory leak "fix".
__________________

Last edited by asherkin; 02-26-2015 at 06:09.
asherkin is offline
glub
AlliedModders Donor
Join Date: Feb 2015
Old 02-26-2015 , 21:10   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #3

Thanks for the reply, asherkin.
Yes! I did just that (can't believe I never tried that before) and the memory leak doesn't seem to happen anymore! See below:



So now, what are the options? Do you think Basecomm can be fixed in some way, or do I have to look for another workaround?
Basecomm allowing voice chat among dead players is really important for Neotokyo, as it's a big part of what kept the game alive up until now.
glub is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 02-26-2015 , 21:17   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #4

I'll take a look when time permits (will need to grab a copy of the game binaries and poke around, but it might be a month or more away), but it is very likely the game doing something really buggy. The best end result might be a NT-specific rewrite of basecomms that does what you need (and only that), which might at least mitigate it a bit.
__________________
asherkin is offline
glub
AlliedModders Donor
Join Date: Feb 2015
Old 02-26-2015 , 21:56   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #5

Thanks asherkin. I hope this won't take too much of your time.
I've been looking at the Basecomm source with my eyes wide open for the past few days already. My head is still spinning.
If there is anything I can do to save you time, let me know. I'll keep investigating on my side. Wish I could be more useful. >.<
glub is offline
glub
AlliedModders Donor
Join Date: Feb 2015
Old 02-27-2015 , 20:18   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #6

A quick update: the Dead Alltalk plugin (discontinued) seems to work fine and doesn't cause memory leak at all. Could be useful to compare the source code?
It does however, break just like Basecomm in regards to opposite team voice comm.
Unfortunately, we lose the gag/silence/mute and ungag/unsilence/unmute SM features. Although the neat Permamute plugin doesn't work without Basecomm, I think it could be achieved through other plugins with a bit of tweaking around.

Edit: additionally, unloading SDK Hooks from MetaMod (with meta unload), then reloading it manually yields the same problem as when reloading Basecomm: voice chat flag are broken and players rejoining the server can't hear the opposing team anymore.

Last edited by glub; 03-06-2015 at 06:53. Reason: added unloading SDK hooks manipulation
glub is offline
neatek
AlliedModders Donor
Join Date: Jul 2010
Location: Russia
Old 02-28-2015 , 02:25   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #7

Quote:
Originally Posted by glub View Post
Unfortunately, we lose the gag/silence/mute and ungag/unsilence/unmute SM features. Although the neat Permamute plugin doesn't work without Basecomm, I think it could be achieved through other plugins with a bit of tweaking around.
You can test it -> GameVoting.
Its works without basecomm.

I need to add admins commands to handle mute, ban, mute, gag.

Last edited by neatek; 02-28-2015 at 02:28.
neatek is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 02-28-2015 , 07:48   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #8

I presume you mean SDKTools rather than SDKHooks? Although you should never be unloading SM extensions from MM:S.

EDIT: SDKTools does indeed have to maintain internal state about who can hear each other, so reloading it breaking things is expected, it could probably be slightly less broken though.

EDIT 2: Just to clear something up, the memory leak isn't in SourceMod - the voice hook code doesn't dynamically allocate any memory. It is the game doing something non-standard that isn't playing nice with SM.
__________________

Last edited by asherkin; 02-28-2015 at 09:03.
asherkin is offline
glub
AlliedModders Donor
Join Date: Feb 2015
Old 02-28-2015 , 21:14   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #9

Thanks for the suggestion neatek, actually I had already check out your plugin previously, but as you said, we are missing the admin handles to unmute/ungag etc.
Anyway, voting to mute still works fine with the VoteMute/VoteSilence 1.0.105 plugin so this is not really a problem, however lifting sanctions is not possible because of the same issue: missing admin handles.

@asherkin: I really meant SDKHooks. Unloading SDKTools breaks other things but doesn't seem to have any direct impact on the voice chat.

I forgot to add this: when (as an admin) I use sm_enemyvoice from Adminsentinel, I could hear the ennemy team fine after rejoining the server (and the hooks broke). Although after a map change, it doesn't work anymore and I have to switch it off, and on again for it to work again.

Second edit: well after further testing, apparently even the older DeadAlltalk seems to mess up voice flags just like Basecomm, even if I don't reload it at all somehow... So just like with Basecomm, have to restart the server to reset it to normal. Very intriguing.

Third edit: somehow Deadalltalk was causing a memory leak too while it was breaking voice chat
Edit:
scratch that, it doesn't break all the time, could have been something else causing it to be reloaded.

Last edited by glub; 03-02-2015 at 01:55. Reason: added more details
glub is offline
glub
AlliedModders Donor
Join Date: Feb 2015
Old 03-05-2015 , 14:52   Re: [Neotokyo] Reporting serious Memory Leak to the SM team!
Reply With Quote #10

I have tested some more things. At first I suspected that joining the server or switching teams would be the cause of the problem as clients execute vmodenable 1 and vban 0 automatically after successful connection. But it seems that transmitting voice is simply what triggers the memory leak, however some details to consider:



1) I have 2 "human" players switching teams NSF/Unsassigned/Jinrai/Unassigned constantly in a loop for several minutes, with a few bots added on top: no visible effect.
2) 30 bots added, 2 human players now transmitting voice (with +voicerecord) all the time. Clearly leaking memory.
3) 2 human clients only on the server (one in each team), one only transmitting voice (the other can't hear). Then a few minutes later, I used Adminsentinel to listen on the other: a slight increase but nothing too concerning.
4) Added 4 bots: very slow but steady increase in memory usage. Edit: at this point I also rejoined the server a bunch of times in a row.
5) Added 10, then 20, then 30 bots with +voicerecord still active for the two human players, the memory leak starts happening there.
6) Stopped +voicerecord on both players, but kepts bots around, memory usage stopped increasing (I think?)
7) Kicked all bots, no voice transmission, stable no leak.
8) Added bots again, still no voice: mem usage increases again.
9) Unloaded Basecomm, no voice, and bots loaded: memory usage still increases!

Sorry for the sloppy report. I thought I'd still try things on my own.


EDIT: I can confirm that transmission of voice from one team to the other is causing the memory increase! Here is what I did:
-started a local server with 30 bots, and 2 human clients. (wanted to use WinDbg, doesn't yield intelligible results for me)
-only base plugins loaded
-my 2 human clients rejoined a bunch of times (I suspect after a set amount of people joining the server, it starts leaking)
-started +voicerecord to broadcast voice from one to the other (each in a different team)
====> Memory leak happens! If only one client receives voice broadcast, it doesn't increase as fast as when both clients receive each other's broadcast!
Once I also had Basecomm glitch for whatever reason, and I couldn't hear anything from the other team, so I used Adminsentinel to receive the voice broadcast. The memory leak doesn't happen when I can't hear the other team, start happening as soon as I use Adminsentinel.
Kicking the bots out seems to make memory usage stable again. So the more clients are connected, the faster the memory incrases.
Even with only the 2 human clients connected, as soon as the player in opposite team hears the voice broadcast from the other, memory increases as a steady pace.
Interesting fact however, stopping broadcasting voice doesn't stop the memory from increasing at the same pace! (about 16Kb per second with only 2 players connected).
Although after a while, it seems to stabilize by itself.I have created a WinDbg .mdmp dump of 300Mb if anyone is interested. Super intriguing.


TLDR: If I can't hear the ennemy team's voice chat, the memory usage is stable, whenever opposite team receive voice broadcast, memory leak happens.

Last edited by glub; 03-06-2015 at 06:49. Reason: confirmation that mem leak happens with voice reception from other team
glub is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 04:26.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode