Raised This Month: $420 Target: $400
 105% 

DispatchSpawn + 2^17 + sqlite = obscure crash


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
sarysa
Senior Member
Join Date: Mar 2014
Old 06-26-2015 , 23:25   DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #1

I'm wondering if any of the back-end gurus have any ideas what causes this issue. I am working on a plugin that spawns lots of projectiles. I mean lots. It's heavily regulated and never comes close to the end-all limit of 2048 (and has a hard maximum of 1750), but projectiles are constantly being created and destroyed. I'm nearing the playable alpha stage for this mod and I did some stress testing with puppet bots, giving them logic and all.

I noticed right away that the server crashed after about 10 minutes with an unintelligible error with dbi.sqlite.ext.dll. Since I haven't added any sort of client prefs or database support to my plugin yet, this baffled me. After a bit of wasting time SHADoW pointed me to asherkin's Accelerator tool. Using that, I got this stack trace:

Spoiler


The important part is the crash happens with DispatchSpawn. Early on I'd figured I hit some unknown limit other than the entity limit. So I decided to have a print at every spawn, but also keep track of the number of spawns.

Both time I tested, the crash happened when trying to spawn my 131069th rocket, which is very close to 2^17. I'm guessing the other 3 were internal calls to DispatchSpawn or something.

I'm somewhat resigned to the notion I'll store rockets in an empty room (which is no problem since this mod requires specific map parameters anyway) and recycle them over and over. It wouldn't be that difficult or taxing to the CPU relative to the current level of management already present. (though always having a near-peak entity count could be a concern -- it usually stays around 400-500 with the bots going at it) What bugs me though is that it's a sourcemod extension that's crashing.

I don't know if it's logging every artificial spawn or something and then giving up at 2^17, but is there any way to disable this in code?

UPDATED, PLEASE READ BEFORE REPLYING SAYING THIS IS AN ENTITY SLOT PROBLEM
Alright, I finally have something substantial. Oddly the crash happens right around 2^16 DispatchSpawn calls instead of 2^17 as was happening with my plugin. I can reproduce the crash from a different module. Here's the error I get without accelerator, the one that faults sqlite:
Spoiler


Then here's the error I get with accelerator, which faults datacache.dll (plus the printout from the accelerator dump):
Spoiler


I'm attaching the .sp that will allow you to reproduce the bug. (edit note: it's in reply #6) It has dependencies so you'll need to compile it manually. I'm hoping that even though you lack the model, the bug will happen.
Fun fact, if you comment out these two lines, the bug never happens:
Code:
	SetEntProp(rocket, Prop_Send, "m_nModelIndex", DC_PROJECTILE_MODEL_INDEX);
	SetEntPropFloat(rocket, Prop_Send, "m_flModelScale", GetRandomFloat(20.0, 50.0) / DC_PROJECTILE_MODEL_RADIUS);
So what do you make of this weird situation? An obscure Valve limitation or is it some bad call that's being made to sqlite?
__________________

Last edited by sarysa; 06-27-2015 at 13:09.
sarysa is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 06-27-2015 , 00:03   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #2

You know that the game itself already uses a lot of entities, RIGHT?
__________________
So long and thanks for all the fish.

Last edited by Powerlord; 06-27-2015 at 00:03.
Powerlord is offline
sarysa
Senior Member
Join Date: Mar 2014
Old 06-27-2015 , 00:09   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #3

Yep. I'm sure everyone's going to read this and immediately think I've hit the 2048 entity limit, but for this particular issue, that's not the case. Guess I didn't preface my post profusely enough.

You could probably reproduce it on every server by creating a rocket, destroying it next frame and create another, destroy, create, etc...keep counts, print out each one, and you'll probably hit the same limit I did. (with the same sqlite crash)

(actually, maybe I should make and test that now, then post the source. heh)
__________________

Last edited by sarysa; 06-27-2015 at 00:12.
sarysa is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 06-27-2015 , 00:24   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #4

datacache.dll is part of the game engine, not SQLite. It lives in /bin (not /tf/bin, but /bin).
__________________
So long and thanks for all the fish.
Powerlord is offline
sarysa
Senior Member
Join Date: Mar 2014
Old 06-27-2015 , 00:44   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #5

The sqlite DLL is the one that faults in the windows error.

I'm working on reproducing this outside of my plugin though so far it's looking to be more complicated than just "spawning X rockets in a single map". (though I also removed some code like model swapping, that doesn't happen until after DispatchSpawn...bah) Not only does this bug eat up my evening but it also turns me into a liar.

One of those days, eh?
__________________

Last edited by sarysa; 06-27-2015 at 00:45.
sarysa is offline
sarysa
Senior Member
Join Date: Mar 2014
Old 06-27-2015 , 01:59   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #6

Alright, I finally have something substantial. Oddly the crash happens right around 2^16 DispatchSpawn calls instead of 2^17 as was happening with my plugin. I can reproduce the crash from a different module. Here's the error I get without accelerator, the one that faults sqlite:
Spoiler


Then here's the error I get with accelerator, which faults datacache.dll (plus the printout from the accelerator dump):
Spoiler


I'm attaching the .sp that will allow you to reproduce the bug. It has dependencies so you'll need to compile it manually. I'm hoping that even though you lack the model, the bug will happen.
Fun fact, if you comment out these two lines, the bug never happens:
Code:
	SetEntProp(rocket, Prop_Send, "m_nModelIndex", DC_PROJECTILE_MODEL_INDEX);
	SetEntPropFloat(rocket, Prop_Send, "m_flModelScale", GetRandomFloat(20.0, 50.0) / DC_PROJECTILE_MODEL_RADIUS);
So what do you make of this weird situation? An obscure Valve limitation or is it some bad call that's being made to sqlite?

(and fair warning to anyone outside of this discussion, the download below is meant to make your server crash within minutes. it's the point. )
Attached Files
File Type: sp Get Plugin or Get Source (rocket_sqlite_crash_repro.sp - 12 views - 3.9 KB)
__________________

Last edited by sarysa; 06-27-2015 at 02:06.
sarysa is offline
RedSword
SourceMod Plugin Approver
Join Date: Mar 2006
Location: Quebec, Canada
Old 06-27-2015 , 02:14   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #7

I remember reading somewhere that it takes time for the engine to free entity slots (READ: No instant). I've absolutely no idea how long though.

Red
__________________
My plugins :
Red Maze
Afk Bomb
RAWR (per player/rounds Awp Restrict.)
Kill Assist
Be Medic

You can also Donate if you appreciate my work

Last edited by RedSword; 06-27-2015 at 02:15.
RedSword is offline
sarysa
Senior Member
Join Date: Mar 2014
Old 06-27-2015 , 02:27   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #8

Quote:
Originally Posted by RedSword View Post
I remember reading somewhere that it takes time for the engine to free entity slots (READ: No instant). I've absolutely no idea how long though.

Red
It's not an entity slot problem, I swear. It does take time, but it's no more than seconds. Even with my test program which spawns 5 rockets per frame, I've never seen the entity indexes over 599. (I printed them out, and watched the scrolling text carefully) Even if you doubt my ability to distinguish a deluge of repeating lines of text, you can be absolutely certain that none of the entity indexes were 1000 or greater. Anyone could notice the sudden change in line width.

Whether or not this is sourcemod's problem, this is a unique issue. Not the typical "2048 entity limit. don't go over it" issue. (hell, the 2048 limit is one of few errors that actually tells you why the server's about to crash)
__________________

Last edited by sarysa; 06-27-2015 at 02:30.
sarysa is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 06-27-2015 , 05:09   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #9

The SQLite crash is a red-herring caused by a faulty termination (I'm guessing datacache is catching the exception it triggers), focus on the other.

EDIT: Yes, that exception code is a hardware breakpoint, so you're hitting some debug code in datacache.
EDIT 2: Actually that could be from Accelerator's handler, please link the actual crash on Throttle.
__________________

Last edited by asherkin; 06-27-2015 at 05:17.
asherkin is offline
sarysa
Senior Member
Join Date: Mar 2014
Old 06-27-2015 , 13:01   Re: DispatchSpawn + 2^17 + sqlite = obscure crash
Reply With Quote #10

Quote:
Originally Posted by asherkin View Post
EDIT 2: Actually that could be from Accelerator's handler, please link the actual crash on Throttle.
https://crash.limetech.org/yyljllxiozcp

Also, keep in mind you can reproduce it yourself with the SP I posted in reply #6. It might be more useful since you have a far better means to trace crashes than I do. Just compile it, run a TF2 server for about 5 minutes, then remove it once it's no longer needed.
__________________

Last edited by sarysa; 06-27-2015 at 13:07.
sarysa is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 08:00.


Powered by vBulletin®
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Theme made by Freecode