Raised This Month: $51 Target: $400
 12% 

Solved TF2 server started crashing on CForward::Execute since Debian upgrade


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
nosoop
Veteran Member
Join Date: Aug 2014
Old 07-09-2019 , 01:57   TF2 server started crashing on CForward::Execute since Debian upgrade
Reply With Quote #1

I'm stumped here; hopefully someone can chime in (at least to verify that it's not just an issue on my particular setup, because the other TF2 servers I manage are running Ubuntu 18.04 16.04).

I updated my Debian instance from Stretch to Buster yesterday, and since then, at a worst case frequency of 15~20 minutes (the time between level changes -- the server has bots running 24/7), the server segfaults with a SEGV_MAPERR on this line. The error indicates an attempt to access unmapped memory.

Didn't expect a Debian upgrade to hose my setup in this way, haha.

No changes to the TF2 server install were made for me to feasibly bisect against, and to my knowledge the code triggering this fault doesn't have external dependencies. The stack trace doesn't provide any good information for me to work with, either; it doesn't look like a forward based on a hook.

Here are a few accelerator links: [1] [2] [3] [4] [5] [6]

Other information:
  • meta: 1.10.7-dev:963:20c72b5, sm: 1.9.0.6281
  • Server is a KVM VPS
  • Kernel is 4.19 (also tested with 4.9 to no success)

I'll probably spin up a VM and try to get it reproducible, but otherwise suggestions short of rolling back and nuking the entire system are welcome (though I do have configuration backups to get things up and running again if it comes down to it).

----

Edit: Alright, so I've discovered that at least one of the causes on an empty server is map voting (with NativeVotes' mapchooser replacement). The server instantly segfaults with a forced invocation of sm_mapvote. Odd, but I suppose I can switch that off for the time being while I isolate what's causing issues with it.

I'm still not sure why that would be an issue, and this solution isn't quite satisfactory, so I don't consider this completely solved (but marked as such since the culprit is known). Hopefully I don't have any other things throwing up segfaults.
__________________
I do TF2, TF2 servers, and TF2 plugins.
I don't do DMs over Discord -- PM me on the forums regarding inquiries.
AlliedModders Releases / Github / TF2 Server / Donate (BTC / BCH / coffee)

Last edited by nosoop; 07-10-2019 at 04:51.
nosoop is offline
nosoop
Veteran Member
Join Date: Aug 2014
Old 07-10-2019 , 04:27   Re: TF2 server started crashing on CForward::Execute since Debian upgrade
Reply With Quote #2

After a hardcore session of adding PrintToServer() calls all over the place, I think I've sorted out the issue.

The NativeVotes' menu handler frees up the private forward handle that invokes it when it receives MenuAction_End (which matches what the normal Menu API does with the Menu handle). I suppose this isn't well-defined behavior for forwards, as this wasn't an issue on Debian 9 nor Ubuntu 18.04.

Here's some example code that causes a similar crash on Debian 10 (or at least my instance):

Code:
#pragma semicolon 1
#include <sourcemod>

#pragma newdecls required

public void OnPluginStart() {
	RegAdminCmd("call_freed_forward", CallFreedForward, ADMFLAG_ROOT);
}

public Action CallFreedForward(int client, int argc) {
	Handle fwd = CreateForward(ET_Ignore, Param_Cell);
	AddToForward(fwd, INVALID_HANDLE, ForwardCall);
	
	Call_StartForward(fwd);
	Call_PushCell(fwd);
	Call_Finish();
	
	PrintToServer("Finished call");
}

public void ForwardCall(Handle fwd) {
	// this is similar to what the nativevotes menu does -- it frees the handler on its callback
	delete fwd;
}
The plugin-sided fix on NativeVotes for this would be to CloneHandle the forward before calling it to increment the refcount, then freeing it afterwards. There probably should be a SourceMod-side fix for it as well, though.
__________________
I do TF2, TF2 servers, and TF2 plugins.
I don't do DMs over Discord -- PM me on the forums regarding inquiries.
AlliedModders Releases / Github / TF2 Server / Donate (BTC / BCH / coffee)

Last edited by nosoop; 07-10-2019 at 13:09.
nosoop is offline
Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 08:53.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode