DispatchSpawn + 2^17 + sqlite = obscure crash
 

I'm wondering if any of the back-end gurus have any ideas what causes this issue. I am working on a plugin that spawns lots of projectiles. I mean lots. It's heavily regulated and never comes close to the end-all limit of 2048 (and has a hard maximum of 1750), but projectiles are constantly being created and destroyed. I'm nearing the playable alpha stage for this mod and I did some stress testing with puppet bots, giving them logic and all.

I noticed right away that the server crashed after about 10 minutes with an unintelligible error with dbi.sqlite.ext.dll. Since I haven't added any sort of client prefs or database support to my plugin yet, this baffled me. After a bit of wasting time SHADoW pointed me to asherkin's Accelerator tool. Using that, I got this stack trace:



The important part is the crash happens with DispatchSpawn. Early on I'd figured I hit some unknown limit other than the entity limit. So I decided to have a print at every spawn, but also keep track of the number of spawns.

Both time I tested, the crash happened when trying to spawn my 131069th rocket, which is very close to 2^17. I'm guessing the other 3 were internal calls to DispatchSpawn or something.

I'm somewhat resigned to the notion I'll store rockets in an empty room (which is no problem since this mod requires specific map parameters anyway) and recycle them over and over. It wouldn't be that difficult or taxing to the CPU relative to the current level of management already present. (though always having a near-peak entity count could be a concern -- it usually stays around 400-500 with the bots going at it) What bugs me though is that it's a sourcemod extension that's crashing.

I don't know if it's logging every artificial spawn or something and then giving up at 2^17, but is there any way to disable this in code?

UPDATED, PLEASE READ BEFORE REPLYING SAYING THIS IS AN ENTITY SLOT PROBLEM
Alright, I finally have something substantial. Oddly the crash happens right around 2^16 DispatchSpawn calls instead of 2^17 as was happening with my plugin. I can reproduce the crash from a different module. Here's the error I get without accelerator, the one that faults sqlite:


Then here's the error I get with accelerator, which faults datacache.dll (plus the printout from the accelerator dump):


I'm attaching the .sp that will allow you to reproduce the bug. (edit note: it's in reply #6) It has dependencies so you'll need to compile it manually. I'm hoping that even though you lack the model, the bug will happen.
Fun fact, if you comment out these two lines, the bug never happens:
So what do you make of this weird situation? An obscure Valve limitation or is it some bad call that's being made to sqlite?

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

You know that the game itself already uses a lot of entities, RIGHT?

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

Yep. I'm sure everyone's going to read this and immediately think I've hit the 2048 entity limit, but for this particular issue, that's not the case. Guess I didn't preface my post profusely enough.

You could probably reproduce it on every server by creating a rocket, destroying it next frame and create another, destroy, create, etc...keep counts, print out each one, and you'll probably hit the same limit I did. (with the same sqlite crash)

(actually, maybe I should make and test that now, then post the source. heh)

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

datacache.dll is part of the game engine, not SQLite. It lives in /bin (not /tf/bin, but /bin).

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

The sqlite DLL is the one that faults in the windows error.

I'm working on reproducing this outside of my plugin though so far it's looking to be more complicated than just "spawning X rockets in a single map". (though I also removed some code like model swapping, that doesn't happen until after DispatchSpawn...bah) Not only does this bug eat up my evening but it also turns me into a liar.

One of those days, eh?

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

Alright, I finally have something substantial. Oddly the crash happens right around 2^16 DispatchSpawn calls instead of 2^17 as was happening with my plugin. I can reproduce the crash from a different module. Here's the error I get without accelerator, the one that faults sqlite:


Then here's the error I get with accelerator, which faults datacache.dll (plus the printout from the accelerator dump):


I'm attaching the .sp that will allow you to reproduce the bug. It has dependencies so you'll need to compile it manually. I'm hoping that even though you lack the model, the bug will happen.
Fun fact, if you comment out these two lines, the bug never happens:
So what do you make of this weird situation? An obscure Valve limitation or is it some bad call that's being made to sqlite?

(and fair warning to anyone outside of this discussion, the download below is meant to make your server crash within minutes. it's the point. :P )

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

I remember reading somewhere that it takes time for the engine to free entity slots (READ: No instant). I've absolutely no idea how long though.

Red

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

It's not an entity slot problem, I swear. It does take time, but it's no more than seconds. Even with my test program which spawns 5 rockets per frame, I've never seen the entity indexes over 599. (I printed them out, and watched the scrolling text carefully) Even if you doubt my ability to distinguish a deluge of repeating lines of text, you can be absolutely certain that none of the entity indexes were 1000 or greater. Anyone could notice the sudden change in line width.

Whether or not this is sourcemod's problem, this is a unique issue. Not the typical "2048 entity limit. don't go over it" issue. (hell, the 2048 limit is one of few errors that actually tells you why the server's about to crash)

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

The SQLite crash is a red-herring caused by a faulty termination (I'm guessing datacache is catching the exception it triggers), focus on the other.

EDIT: Yes, that exception code is a hardware breakpoint, so you're hitting some debug code in datacache.
EDIT 2: Actually that could be from Accelerator's handler, please link the actual crash on Throttle.

Re: DispatchSpawn + 2^17 + sqlite = obscure crash
 

https://crash.limetech.org/yyljllxiozcp

Also, keep in mind you can reproduce it yourself with the SP I posted in reply #6. It might be more useful since you have a far better means to trace crashes than I do. Just compile it, run a TF2 server for about 5 minutes, then remove it once it's no longer needed.