Server Crashes Frequently

[daleGEND]

So I've loaded the store onto 3 of my servers. 1 of my servers is a Mini Games server that runs on CentOS 6.4 64-bit. This server has been crashing between 7-8 times a day since the store was loaded on it. Here is a list of information which I hope will help figure out why it's crashing. I have a few core dump files but I'm unsure how to grab information from them:

meta list
[01] SourceMod (1.5.0-dev+3791) by AlliedModders LLC
[02] SMRCon (1.2.1) by psychonic
[03] Server Secure - Files Only (1.0.0) by AzuiSleet (mod by asherkin)
[04] CS Tools (1.5.0-dev+3791) by AlliedModders LLC
[05] SDK Tools (1.5.0-dev+3791) by AlliedModders LLC
[06] Connect (1.2.0) by Asher "asherkin" Baker
[07] SteamTools (0.8.3) by Asher Baker (asherkin)
[08] SDK Hooks (1.5.0-dev+3791) by AlliedModders LLC
[09] DHooks (1.0.12-alpha) by Dr!fter

sm version
SourceMod Version: 1.5.0-dev+3791
SourcePawn Engine: SourcePawn 1.1, jit-x86 (build 1.5.0-dev+3791)
SourcePawn API: v1 = 4, v2 = 4
Compiled on: Mar 2 2013 11:33:20
Build ID: 3791:8a2bcfd6aa8a
http://www.sourcemod.net/

meta version
Metamod:Source version 1.10.0-dev
Build ID: 819:506c71b4ff73-dev
Loaded As: Valve Server Plugin
Compiled on: Mar 4 2013
Plugin interface version: 15:14
SourceHook version: 5:5
http://www.metamodsource.net/

sm exts list
[01] Automatic Updater (1.5.0-dev+3791): Updates SourceMod gamedata files
[02] Webternet (1.5.0-dev+3791): Extension for interacting with URLs
[03] SMRCon (1.2.1): Tools for SM plugins to interact with rcon auth and requests
[04] BinTools (1.5.0-dev+3791): Low-level C/C++ Calling API
[05] Server Secure - Files Only (1.0.0): The finest defence
[06] CS Tools (1.5.0-dev+3791): CS extended functionality
[07] SDK Tools (1.5.0-dev+3791): Source SDK Tools
[08] Top Menus (1.5.0-dev+3791): Creates sorted nested menus
[09] Connect (1.2.0): Forward for early connection
[10] Client Preferences (1.5.0-dev+3791): Saves client preference settings
[11] MySQL-DBI (1.5.0-dev+3791): MySQL driver implementation for DBI
[12] Socket (3.0.1): Socket extension for SourceMod
[13] SteamTools (0.8.3): SteamWorks for SourceMod.
[14] SDK Hooks (1.5.0-dev+3791): Source SDK Hooks
[15] GeoIP (1.5.0-dev+3791): Geographical IP information
[16] Regex (1.5.0-dev+3791): Provides regex natives for plugins
[17] SMJansson (2.3.1/3): JSON parser/writer
[18] <FAILED> file "builtinvotes.ext.so": /home/tcagame/bgofficers/2/css/cstrike/addons/sourcemod/extensions/builtinvotes.ext.so: cannot open shared object file: No such file or directory
[19] cURL Extension (1.3.0.0): cURL Extension
[20] DHooks (1.0.12-alpha): Allows basic dynamic hooks
[21] SQLite (1.5.0-dev+3791): SQLite Driver

sm plugins list
01 "Anti-Flood" (1.5.0-dev+3791) by AlliedModders LLC
02 "Players Votes" (1.5.0) by The Resident, pZv!
03 "SM SKINCHOOSER" (2.3) by Andi67
04 "SMAC Rcon Locker" (0.8.4.0) by SMAC Development Team
05 "SMAC ConVar Checker" (0.8.4.1) by SMAC Development Team
06 "Immunity Reserve Slots [CONNECT]" (2.0.7) by Jamster
07 "SMAC Command Monitor" (0.8.4.0) by SMAC Development Team
08 "SourceMod Anti-Cheat" (0.8.4.0) by SMAC Development Team
09 "Client Preferences" (1.5.0-dev+3791) by AlliedModders LLC
10 "SM warn" (1.1.0) by ecca
11 "[Any] SteamTools Game Description Override" (1.1.3) by Dr. McKay
12 "Basic Votes" (1.5.0-dev+3791) by AlliedModders LLC
13 "SMAC Client Protection" (0.8.4.0) by SMAC Development Team
14 "SMAC AutoTrigger Detector" (0.8.4.0) by SMAC Development Team
15 "Sourcebans Checker" (1.0.0) by psychonic & Ca$h Munny
16 "Admin File Reader" (1.5.0-dev+3791) by AlliedModders LLC
17 "SourceComms" (0.9.7) by Alex
18 "gameME Plugin" (4.3) by TTS Oetzel & Goerz GmbH
19 "Admin Menu" (1.5.0-dev+3791) by AlliedModders LLC
20 "NoBlock" (1.4.2) by Otstrel.ru Team
21 "RandomCycle" (1.4.6) by AlliedModders LLC
22 "Map Nominations Extended" (1.9.2) by Powerlord and AlliedModders LLC
23 "SourceBans" (1.4.10) by SourceBans Development Team
24 "UpdateRadar Fix" (fix) by GoD-Tony
25 "Network Tools" (1.3) by Kyle Sanderson
26 "AllChat" (1.1.1) by Frenzzy
27 "Admin loggin" (1.1) by vIr-Dan
28 "SMAC Aimbot Detector" (0.8.4.0) by SMAC Development Team
29 "Sound Commands" (1.5.0-dev+3791) by AlliedModders LLC
30 "Spray Tracer" (5.8a) by Nican132, CptMoore, Lebson506th
31 "Mapchooser Extended Sounds" (1.3) by Powerlord
32 "SMAC Eye Angle Test" (0.8.4.0) by SMAC Development Team
33 "AdsQL Advertisements System" (2.3.1) by PharaohsPaw
34 "Advanced admin commands" (0.16b) by 3sigma
35 "Player Commands" (1.5.0-dev+3791) by AlliedModders LLC
36 "Basic Chat" (1.5.0-dev+3791) by AlliedModders LLC
37 "[Store] Kill Awards" (1.1.1-alpha) by eXemplar
38 "[Store] Inventory" (1.1.1-alpha) by alongub
39 "[Store] Gifting" (1.1.1-alpha) by alongub
40 "[Store] Backend" (1.1.1-alpha) by alongub
41 "[Store] Chat Colors" (1.0-alpha) by Twisted|Panda
42 "[Store] Tracers" (1.0.1) by Franc1sco steam: franug
43 "[Store] Refund" (1.1.1-alpha) by alongub
44 "[Store] Distributor" (1.1.1-alpha) by alongub
45 "[Store] Shop" (1.1.1-alpha) by alongub
46 "[Store] NadeSkins" (1.1-alpha) by Phault
47 "[Store] Logging" (1.1.1-alpha) by alongub
48 "[Store] Name Colors" (1.0-alpha) by Twisted|Panda
49 "[Store] Loadout" (1.1.1-alpha) by alongub
50 "[Store] Core" (1.1.1-alpha) by alongub
51 "[Store] NadeTrails" (1.1-alpha) by Phault
52 "[Store] Trails" (1.1-alpha) by alongub
53 "[Store] Equipment" (1.1-alpha) by alongub
54 "[Store] Titles" (1.1-alpha) by alongub
55 "SMAC Anti-Speedhack" (0.8.4.0) by SMAC Development Team
56 "Server Hop" (0.8.2) by [GRAVE] rig0r
57 "Admin Help" (1.5.0-dev+3791) by AlliedModders LLC
58 "Map configs" (1.1.1) by Berni
59 "Nextmap" (1.5.0-dev+3791) by AlliedModders LLC
60 "Fun Commands" (1.5.0-dev+3791) by AlliedModders LLC
61 "Basic Commands" (1.5.0-dev+3791) by AlliedModders LLC
62 "Weapon Restrict" (3.0.9) by Dr!fter
63 "MapChooser Extended" (1.9.3) by Powerlord, Zuko, and AlliedModders LLC
64 "Server Clean Up" (1.1.9) by Jamster
65 "Updater" (1.1.5) by GoD-Tony
66 "Fun Votes" (1.5.0-dev+3791) by AlliedModders LLC
67 "Web Shortcuts" (1.1) by Kyle Sanderson, Nicholas Hastings
68 "SM File/Folder Downloader and Precacher" (1.4) by SWAT_88
69 "Radio Spam Blocker" (1.2) by TheAvengers2
70 "Simple Chat Processor (Redux)" (1.1.4-fix2) by Simple Plugins, Mini
71 "Show Damage" (1.0.7) by exvel
72 "Basic Comm Control" (1.5.0-dev+3791) by AlliedModders LLC
73 "Napalm Lag Fix" (1.0.1) by GoD-Tony
74 "Basic Info Triggers" (1.5.0-dev+3791) by AlliedModders LLC
75 "Rock The Vote Extended" (1.9.2) by Powerlord and AlliedModders LLC

No errors in the error log. No sourcemod fatal logs.

[alongub]

Try to disable NadeTrails, Trails, Equipment, NadeSkins and Tracers and check if it's still happening. If yes - try to enable them back and disable each one until you find the problem.

[daleGEND]

Try to disable NadeTrails, Trails, Equipment, NadeSkins and Tracers and check if it's still happening. If yes - try to enable them back and disable each one until you find the problem.

Ok, I've removed them for now and will let you know if it crashes again.

[daleGEND]

After removing sm_skinchooser and the downloader and precacher from the server I've noticed very few to no crashes on the server. Just a heads up to anyone else having similar issues.

[daleGEND]

Actually I lied, still crashes unfortunately.

[alongub]

And if you disable the entire store?

[daleGEND]

It's fine. Before we had the store enabled the server had no crashes. I made no other changes to the server.

[alongub]

It's fine. Before we had the store enabled the server had no crashes. I made no other changes to the server.

You should check again.

[winterghost]

i am getting a lot of server crashes when the store is enabled on tf2 :/

[alongub]

Unfortunately, saying that you're getting server crashes with the store does not help. For example, I run the store on my server with no crashes at all. The store core itself isn't the problem, it's probably one of its modules, or a completely different plugin that you're running.

[daleGEND]

Unfortunately, saying that you're getting server crashes with the store does not help. For example, I run the store on my server with no crashes at all. The store core itself isn't the problem, it's probably one of its modules, or a completely different plugin that you're running.
Hey alongub,

I understand that vague requests aren't helpful but I've offered to give you a lot of information including dump files and all the information provided above. All I'm getting is requests to remove the plugin and see if it still crashes. I removed the plugin and the server has not crashed for 24-hours. Please look into this issue and I'll provide you the dmp info via PM. Thanks.

[alongub]

Hey alongub,

I understand that vague requests aren't helpful but I've offered to give you a lot of information including dump files and all the information provided above. All I'm getting is requests to remove the plugin and see if it still crashes. I removed the plugin and the server has not crashed for 24-hours. Please look into this issue and I'll provide you the dmp info via PM. Thanks.

That post wasn't intended to you, it was intended to winterghost.

Dumps don't help. I just need you to see if the server is still crashing without the store at all.

[daleGEND]

That post wasn't intended to you, it was intended to winterghost.

Dumps don't help. I just need you to see if the server is still crashing without the store at all.

I know it wasn't, but I'm not getting much feedback. Also I have tested the server without the store for 24-hours. No crashes have been reported.

[winterghost]

That post wasn't intended to you, it was intended to winterghost.

Dumps don't help. I just need you to see if the server is still crashing without the store at all.

I was running basic store, distributor and trails. Server has not crashed once since store was removed.

[daleGEND]

I had someone look at the crash dumps to see where it was crashing and it was:
CEntitySphereQuery::GetCurrentEntity(void)

Not sure if that helps at all, but so far the server has been crashless without the store enabled.

[winterghost]

i added the store modules back, with the exception of trails. solid server.

[daleGEND]

i added the store modules back, with the exception of trails. solid server.

I'll give that a shot for my server as well and see if it performs well.

[daleGEND]

Unfortunately, still crashes for me even without trails, so it must be some other conflict somewhere. Anyone else have any issues?

[winterghost]

dale, you tried just the core store plugins (all alongub's ones)?

[daleGEND]

dale, you tried just the core store plugins (all alongub's ones)?
Nope, but I'll give it a shot.

[daleGEND]

Still crashes unfortunately with only default store plugins enabled.

[winterghost]

Still crashes unfortunately with only default store plugins enabled.

try updating SM. it's one of the things I did.

if this helps, these are the plugins you are running that are different to the ones I am using:


02 "Players Votes" (1.5.0) by The Resident, pZv!
03 "SM SKINCHOOSER" (2.3) by Andi67
04"Immunity Reserve Slots [CONNECT]" (2.0.7) by Jamster
07 "SM warn" (1.1.0) by ecca
11 "[Any] SteamTools Game Description Override" (1.1.3) by Dr. McKay
15 "Sourcebans Checker" (1.0.0) by psychonic & Ca$h Munny
17 "SourceComms" (0.9.7) by Alex
20 "NoBlock" (1.4.2) by Otstrel.ru Team
21 "RandomCycle" (1.4.6) by AlliedModders LLC
24 "UpdateRadar Fix" (fix) by GoD-Tony
25 "Network Tools" (1.3) by Kyle Sanderson
26 "AllChat" (1.1.1) by Frenzzy
27 "Admin loggin" (1.1) by vIr-Dan
28 "Spray Tracer" (5.8a) by Nican132, CptMoore, Lebson506th
31 "Mapchooser Extended Sounds" (1.3) by Powerlord
34 "Advanced admin commands" (0.16b) by 3sigma
37 "[Store] Kill Awards" (1.1.1-alpha) by eXemplar
56 "Server Hop" (0.8.2) by [GRAVE] rig0r
58 "Map configs" (1.1.1) by Berni
62 "Weapon Restrict" (3.0.9) by Dr!fter
63 "MapChooser Extended" (1.9.3) by Powerlord, Zuko, and AlliedModders LLC
64 "Server Clean Up" (1.1.9) by Jamster
67 "Web Shortcuts" (1.1) by Kyle Sanderson, Nicholas Hastings
68 "Radio Spam Blocker" (1.2) by TheAvengers2
71 "Show Damage" (1.0.7) by exvel
73 "Napalm Lag Fix" (1.0.1) by GoD-Tony
75 "Rock The Vote Extended" (1.9.2) by Powerlord and AlliedModders LLC

[alongub]

The only way to find the source of these crashes is to find the plugin that's causing them. Just try to enable/disable modules until you find something.

If it's a module by me, let me know so I could fix the problem.

[daleGEND]

The only way to find the source of these crashes is to find the plugin that's causing them. Just try to enable/disable modules until you find something.

If it's a module by me, let me know so I could fix the problem.

Would it also help that the players in the server notice it happens typically on maps with explosions. That's when the server tends to crash. Doesn't crash when the store is disabled. So I'm not sure if there is a conflict on some sort.

[asherkin]

I had someone look at the crash dumps to see where it was crashing and it was:


Not sure if that helps at all, but so far the server has been crashless without the store enabled.

Just the top of the stack isn't helpful, need the full trace.

[daleGEND]

Just the top of the stack isn't helpful, need the full trace.
I just updated my version of SM 1.5 so let's see how it goes. I also have your Accelerator plugin on the server. If I see any new crashes would you mind if I send you the detail links?

[asherkin]

I just updated my version of SM 1.5 so let's see how it goes. I also have your Accelerator plugin on the server. If I see any new crashes would you mind if I send you the detail links?

Fire away. Accelerator processing requires some manual work at the moment until I have a change to rewrite most of the site.

[daleGEND]

Interestingly enough, since updating to the latest snapshot I haven't seen a single crash. O_o'

[daleGEND]

Sorry to dig up this thread. I originally thought that the server was fine with the latest snapshot of 1.5, but it seems that the crashes are still happening. Just to verify I disabled the store completely and for the past 24 hours there have been no crashes.

I plan to re-add modules each day to see if the crashes continue to happen. I'll update this thread with additional information.

[asherkin]

Sorry to dig up this thread. I originally thought that the server was fine with the latest snapshot of 1.5, but it seems that the crashes are still happening. Just to verify I disabled the store completely and for the past 24 hours there have been no crashes.

I plan to re-add modules each day to see if the crashes continue to happen. I'll update this thread with additional information.

Do you have an Accelerator ID?

[daleGEND]

Do you have an Accelerator ID?

I do not, I completely rebuilt my servers and moved back to Windows. I can try and re-add it though. Just for testing purposes as I re-add modules.

Edit: Oh looks like you've updated it. Nice :)

I've added it to all my servers and I will see if I can get any additional information today.

[asherkin]

Oh looks like you've updated it. Nice :)

Indeed. The new plugin fully supports Windows as well as Linux, and the backend has symbols for at least all of SourceMod and my extensions, so should give a good hint if the crash is caused by a plugin directly (make sure you have a very recent snapshot in order to get SourceMod symbols).

[daleGEND]

Indeed. The new plugin fully supports Windows as well as Linux, and the backend has symbols for at least all of SourceMod and my extensions, so should give a good hint if the crash is caused by a plugin directly (make sure you have a very recent snapshot in order to get SourceMod symbols).

Would SourceMod v1.5.0-dev+3841 be good enough?

[asherkin]

Would SourceMod v1.5.0-dev+3841 be good enough?

As that's the latest, yeah.

[daleGEND]

As that's the latest, yeah.

Oh sorry, I didn't check the snapshots to see if anything new has been released. I'll let you know if anything comes up.

[thetwistedpanda]

On an unrelated note daleGEND, I stopped by bor3d's MG server a few days ago and noticed the majority of my community members started calling it home after I shut mine down, so if you need/want assistance >;3.

On topic, when I ran a MG server I couldn't run any type of supporter plugin that created entities as they did NOT play well with other maps. In addition, plugins that modified a player's m_iName value also seemed to cause issues as maps assumed that a specific name would be available. That being said, MG maps are usually not of the highest quality and are prone to crashing regardless.

[daleGEND]

On an unrelated note daleGEND, I stopped by bor3d's MG server a few days ago and noticed the majority of my community members started calling it home after I shut mine down, so if you need/want assistance >;3.

On topic, when I ran a MG server I couldn't run any type of supporter plugin that created entities as they did NOT play well with other maps. In addition, plugins that modified a player's m_iName value also seemed to cause issues as maps assumed that a specific name would be available. That being said, MG maps are usually not of the highest quality and are prone to crashing regardless.

Thanks Panda, I appreciate the support. Feel free to add me on Steam: http://steamcommunity.com/id/b0r3d

[thetwistedpanda]

I ended up joining Bor3d Gaming as their Server Tech. After five days of tracing and debugging all other potential crash reasons, I've narrowed it down to one of four Store plugins. I'll narrow it down further over the next week (to ensure there's no crashes between each modification), but it's looking to be one of the following: Equipment, Trails, Grenade Trails, and Grenade Models. I'm fairly certain it's Equipment, as the other three I have experience within not causing crashes. A good number of MiniGame maps (for CS:S) have entities that effect the player's general area, such as env_explosions, as well as maps that attach additional items to players. The interesting fact is when the server crashes, it's entirely within server.dll and no hints of plugin involvement, which further hints to Equipment being the cause. It's likely no fault of the actual plugin, just an incompatibility with the map/feature type. I'll update once a definite is confirmed.

Edit: Grenade Trails and Grenade Models have not caused crashes.
Edit2: Trails has caused crashes, where as Equipment actually has not... That shoots my theory to shit.