AlliedModders - Accelerator - Crash Reporting That Doesn't Suck

AlliedModders (https://forums.alliedmods.net/index.php)

- Extensions (https://forums.alliedmods.net/forumdisplay.php?f=134)

- - Accelerator - Crash Reporting That Doesn't Suck (https://forums.alliedmods.net/showthread.php?t=277703)

Accelerator - Crash Reporting That Doesn't Suck

Accelerator replaces the default SRCDS crash handler with one that is a lot more reliable and uploads the crash reports to a community-accessible processing backend.
This is useful since Valve's processing system discards crash reports from modded servers - and they still receive a representative sample of crash reports for actual game issues.

The processing backend (Throttle - https://crash.limetech.org/) analyses crash reports to extract useful information, and in the case of common issues affecting many servers where manual analysis has been done, tags them with helpful notices with information on resolving the cause of the crash.

Previous versions of Accelerator have been in use on hundreds of servers since 2010 (as of this thread's creation, Throttle has processed over 30,000 crash reports in the last 90 days), this is a re-release as a lot has changed since then.

If you're running any earlier version (including the previous extension alpha build), please upgrade to 2.4.3+.

Installation

Click the big Download link below.
Download the latest version for your server's OS.
Extract the Zip file and merge the addons/sourcemod directory with the one on your server.

Configuration

Edit addons/sourcemod/configs/core.cfg to add the following key:

Code:

"MinidumpAccount" "<insert your steamid here>"

Your SteamID should be in the full 64-bit format, you can use a site like https://steamid.io/ to get yours, it'll look something like 7656119798781xxxx.

So your core.cfg will end up looking like (with your own steamid):

Code:

"Core"

{

  [...]



  "MinidumpAccount"  "7656119798781xxxx"

}

Obviously, you will have all the other options in there as well.

The full set of available options with their default values and documentation are below, you should only put ones you intend to change into core.cfg:

Code:

"Core"

{

  /**

   * SteamID64 (Community ID) that will have ownership of uploaded crash reports.

   * You can share your crash reports with additional users from the website.

   *

   * If unset, your crash reports will be uploaded anonymously and you will not be

   * able to see all of the information.

   */

  "MinidumpAccount"        ""



  /**

   * Controls which binaries will be eligible to be processed for symbols and uploaded.

   * Only modules loaded by the server at the time of the crash can be considered.

   *

   * 0 = Disabled: No binaries will be processed or uploaded.

   * 1 = System Only: Only binaries outside of the game directory (where the srcds binary is).

   * 2 = System + Game: Loaded modules outside of the addons/ directory.

   * 3 = System + Game + Addons: All loaded modules.

   */

  "MinidumpSymbolUpload"        "3"



  /**

   * Controls whether Accelerator can upload complete module binaries when explicitly requested

   * by the processing server. This also respects the value of the MinidumpSymbolUpload setting.

   */

  "MinidumpBinaryUpload"        "yes"



  /**

   * Controls whether Accelerator does local processing of crash reports before upload.

   * This should only be changed if local processing causes issues such as crashes,

   * the processing server may reject crash reports that have not been presubmitted.

   */

  "MinidumpPresubmit"        "yes"



  /**

   * URL to upload crash dumps to. Should not be changed.

   */

  "MinidumpUrl"        "http://crash.limetech.org/submit"



  /**

   * URL to upload processed symbols to. Should not be changed.

   */

  "MinidumpSymbolUrl"        "http://crash.limetech.org/symbols/submit"



  /**

   * URL to upload binaries to. Should not be changed.

   */

  "MinidumpBinaryUrl"        "http://crash.limetech.org/binary/submit"

}

Usage

Once running on your servers, whenever they crash they will write a crash dump into sourcemod/data/dumps/.

Next time the server starts it will upload any pending crash dumps in that directory to Throttle (https://crash.limetech.org/), you can login to that site using Steam OpenID authentication to view any processed crash reports owned by you.

Accelerator logs all upload activity to sourcemod/logs/accelerator.log, from there you can see all uploaded crash IDs and error messages if any fail to upload.

The crash dump uploader is threaded so will not impose any startup delay on your servers. Crash dumps plus metadata are generally less than 500kb combined. There may be additional uploads occasionally if your server has symbols available that Throttle does not have yet.

Security

Crash dumps can contain sensitive information such as RCon / Database credentials, as they are produced by saving key parts of SRCDS' memory space.

As standard, crash reports uploaded to Throttle will contain at least the following information:

The process command line.
The last 500 lines of console output (only for supported games).
A list of all loaded native modules.
The contents of the stack memory for each thread.

As of version 2.4.x, Accelerator can now partially preprocess crash reports locally before upload, and process modules loaded into SRCDS' memory for symbol files (this is a hard restriction in the code, it won't touch any other files). The latter, if enabled, will cause files to be processed locally and uploaded to Throttle. See the MinidumpSymbolUpload and MinidumpBinaryUpload options above for more information.

Throttle maintains strict security controls, only the owner of the crash (the SteamID specified in core.cfg), people the owner selects to share their reports with on the website, and a select set of SourceMod Core team members can access full crash reports.

Everyone else can only see the crashing thread's call stack and any assigned notices, so please do link Throttle crash reports directly if you're seeking help rather than just copying the stack trace / taking a screenshot.

That said, please do not post in this thread looking for help diagnosing a crash.
Create your own thread in the Source Servers section where you'll get much better help and will not fill up this thread with transient trash.

Supported Games

At a basic level Accelerator should work with all games on both Linux and Windows.
Games with enhanced support will additionally get console output included in crash reports.

Windows (and CS:GO) crash reports are generally useless if they do not directly crash due to SourceMod, this is because Valve continue to refuse to provide stripped symbol files (like Microsoft provide for Windows).

Download

Source Code

Build Status
https://travis-ci.org/asherkin/accel...?branch=master https://ci.appveyor.com/api/projects...aster?svg=true

Re: Accelerator - Crash Reporting That Doesn't Suck

Nice release! Updated my servers, Accelerator has always been a great help for me. Keep up the good work!

Re: Accelerator - Crash Reporting That Doesn't Suck

sounds nice! can't wait to try this great job master!

Re: Accelerator - Crash Reporting That Doesn't Suck

What about backend sorucecode? For example start this service in private network.

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by spumer (Post 2383452)

What about backend sorucecode? For example start this service in private network.

There is a link to the source code at the bottom of every page, and there has been since launch. There are no complete installation instructions and I don't have time to walk you through setting it up - it is also not advised since a number of the advantages come from operating at scale, if you don't know how to analyse a crash dump yourself you're going to get very little out of it.

Re: Accelerator - Crash Reporting That Doesn't Suck

Nice update, but I cant understand.

When server crash, nothing happen ! I mean, I dont have any logs on the website ! I just have to install extension and add my steam in core file ?

log sm : Failed to upload crash dump: The requested URL returned error: 400 (22)

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by poel (Post 2383717)

That means that Breakpad failed to generate a crash dump for your crash and it tried to upload an empty minidump.

I've never been able to replicate the issue and it appears to be an OS-level problem, I would need access to your host to look into it.

Re: Accelerator - Crash Reporting That Doesn't Suck

Is it possible to revert a375772 commit?

Serveral servers of my friend was damaged by this commit. He realy enjoyed Accelerator because it saved him a lot of times! He is using "thing, that cannot be spoken" not on his own, but because all of his local competitors use it and without it his online will suffer.

The amount of CSS players is dramatically decreasing and by now, the amount of players using "thing, that cannot be spoken" and not using it is almost the same.

Re: Accelerator - Crash Reporting That Doesn't Suck

No, by far most of Throttle's processing capacity was being used by no-steam servers.
While that capacity is seriously constrained, I wish to make it available to legitimate servers.

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by asherkin (Post 2383992)

No, by far most of Throttle's processing capacity was being used by no-steam servers.
While that capacity is seriously constrained, I wish to make it available to legitimate servers.

If the problem is in capacity, then maybe it would be better to just prioritize reports processing?

1. Legit
2. Legit ES crash
3. no-steam
4. no-steam ES crash

(Almost all servers, that uses ES is a WCS servers, and they are crashing every 1-2 hours)

Re: Accelerator - Crash Reporting That Doesn't Suck

@Asherkin,

So, what Can I do to fix that : [CRASH] Accelerator failed to upload crash dump: The requested URL returned error: 400 (22)

??

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by ejk499 (Post 2384018)

If the problem is in capacity, then maybe it would be better to just prioritize reports processing?

You were presented with the AlliedModders Rules when you signed up today. Did you read them at all?

Supporting "no-steam" is a no go.

Quote:

Originally Posted by ejk499 (Post 2384018)

1. Legit
2. Legit ES crash
3. no-steam
4. no-steam ES crash

(Almost all servers, that uses ES is a WCS servers, and they are crashing every 1-2 hours)

If you won't run a legitimate server, then I suggest you leave this forum alone.

Re: Accelerator - Crash Reporting That Doesn't Suck

is there a way to log what map the server was on when it crashed, it would really help in rooting out bad maps

Re: Accelerator - Crash Reporting That Doesn't Suck

Sure, is there anything else you would find useful?
(I have a long list, but it makes sense to prioritize - and some of it is just not possible without SM changes.)

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by PresidentEvil (Post 2385904)

is there a way to log what map the server was on when it crashed, it would really help in rooting out bad maps

Well, its ez.

Crash log can tell you the time when server crash, then just check your "log" files, and see at the same time which map was used.

About my error Asherkin, What can I do to fix that ?

[CRASH] Accelerator failed to upload crash dump: The requested URL returned error: 400 (22)

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by poel (Post 2386010)

no I rather have accerator note the map in the crash log, making it easy to view on crash.limetech.org instead of opening up ftp and digging through large ass log files..

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by PresidentEvil (Post 2386162)

no I rather have accerator note the map in the crash log, making it easy to view on crash.limetech.org instead of opening up ftp and digging through large ass log files..

i agree with presidentevil this feature would be awesome! (hopefully this post won't be deleted :))

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by PresidentEvil (Post 2385904)

is there a way to log what map the server was on when it crashed, it would really help in rooting out bad maps

This is now in 2.2.1, you wont find it displayed on Throttle yet, but if you append "/metadata" to the URL you can see the raw data and it's at the top.

Quote:

Originally Posted by 8guawong (Post 2386239)

i agree with presidentevil this feature would be awesome! (hopefully this post won't be deleted :))

No, it's just spammy. I already said I would add it, further evangelizing is just pointless and taking up space.

Quote:

Originally Posted by poel (Post 2386010)

About my error Asherkin, What can I do to fix that ?

I've already answered you, the problem is in Breakpad and there is nothing I can do about it since I can't reproduce it, and never have been able to - I only know of 3 people it affects, which is tiny and implies an obscure OS misconfiguration.

Re: Accelerator - Crash Reporting That Doesn't Suck

Its possible to copy logs in a txt file maybe on ftp ?

Re: Accelerator - Crash Reporting That Doesn't Suck

Throttle now shows you how many other people are getting similar crashes, this should help to identify actual bugs (especially post-update) vs wierdo issues.

Re: Accelerator - Crash Reporting That Doesn't Suck

hello i try to compile but i get:

Code:

git clone https://github.com/asherkin/accelerator.git accelerator

cd accelerator

mkdir build && cd build

python ../configure.py --enable-optimize

Quote:

Checking c compiler (vendor test gcc)... ['gcc', 'test.c', '-o', 'test']
found gcc version 4.7
Checking cxx compiler (vendor test gcc)... ['g++', '-fno-exceptions', '-fno-rtti', 'test.cxx', '-o', 'testp']
found gcc version 4.7
git rev-list --count HEAD
git log --pretty=format:%h -n 1
Adding job breakpad.
Adding job accelerator.ext.
Adding job package.

Code:

ambuild

Quote:

Error: folder was not configured for AMBuild.

Re: Accelerator - Crash Reporting That Doesn't Suck

Accelerator is using AMBuild1, not AMBuild2 - see the TravisCI (.travis.yml) or AppVeyor (appveyor.yml) configs for what dependencies you need and commands you need to run.

Re: Accelerator - Crash Reporting That Doesn't Suck

So I just learned how to find windows signatures and how to decipher windows stack traces.

Does this extension cause any noticeable overhead while it's being run on the server or none?

Thanks!

Re: Accelerator - Crash Reporting That Doesn't Suck

Effectively none, it only does work on a crash and when the server first starts (which is all offloaded to a thread in current versions).

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Uploaded crash dump: Crash ID: XXXX-XXXX-XXXX
Uploaded crash dump: Crash ID: XXXX-XXXX-XXXX
Uploaded crash dump: Crash ID: XXXX-XXXX-XXXX

Maybe timestamp for logs ?

Re: Accelerator - Crash Reporting That Doesn't Suck

You can find timestamped log messages in the SourceMod error log.

Re: Accelerator - Crash Reporting That Doesn't Suck

My server crashes evertyime map changes during the game not like on start, accelerator says dump was created with ID ( number) but when i look at dumps folder its empty

Re: Accelerator - Crash Reporting That Doesn't Suck

They are deleted after upload, enter the ID you were given into the web page.

Re: Accelerator - Crash Reporting That Doesn't Suck

Just saw this in a random crash report:

"The server that this crash is from has been modified to allow pirated game clients to join."

Why aren't you just outright discarding and refusing to process these crashes? Don't we reject no-steam servers? Even if you must fully process them to determine it is a cracked server, I still don't see any harm in just discarding the processed data and saying "no-steam servers are not supported by this application".

I could be talking out of my ass so I don't know.

Re: Accelerator - Crash Reporting That Doesn't Suck

Subsequent reports from the same server are discarded for the next 90 days.

Re: Accelerator - Crash Reporting That Doesn't Suck

asherkin can you gather a list of map that are causing the server.so!CCSPhysicsPushEntities::EnsureValid PushWhileRiding(CBaseEntity*, CBaseEntity*) + 0x8b crash?

the least we can do is remove the maps from the rotation to prevent the servers from crashing every so often...

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by 8guawong (Post 2394504)

It's not very complete due to so many people not bothering to update Accelerator.

Code:

    [ze_farmhouse_v9_fix] => 5

    [ze_sky_athletic_v5_3] => 4

    [ze_elevator_escape_p2] => 4

    [ze_Boatescape_Extendido_p4] => 4

    [ze_Random_p2] => 4

    [mg_course_clear_v1fix] => 3

    [ze_Pirates_Port_Royal_v4_2] => 3

    [ze_potc_p4] => 3

    [ze_rollermine_factory_2010_p] => 3

    [ze_arctic_escape_p2] => 3

    [ze_LOTR_Helms_Deep_v5_p6] => 2

    [ze_30_seconds_go_v2] => 2

    [bhop_playboy_csgo] => 2

    [ze_icecap_escape_v5_sgfix] => 2

    [mg_savetheisland_course_final] => 2

    [ze_mytex_escape_v3_p] => 2

    [mg_Sketchy_course_go1] => 2

    [mg_airmap_run_v2_1] => 2

    [ze_fapescape_p5] => 1

    [ze_trainescape_final_p2] => 1

    [mg_circle_course] => 1

    [ze_infected_lab_v4_1] => 1

    [jb_clouds_beta01] => 1

    [Surf_Castle_v6_esportcenter] => 1

    [mg_minecraft_adventure_final_fix] => 1

    [ze_ice_hold_p1] => 1

    [mg_go_for_it_csgo] => 1

    [mg_lego_course_3_csgo] => 1

    [ze_farmhouse_v9] => 1

    [ze_licciana_xmas_v1_e1] => 1

    [ze_random_p2] => 1

    [ze_halo_poa_p4] => 1

    [ze_30_seconds__b21_p2] => 1

    [deathrun_cb_egypt_csgo_v1] => 1

    [ze_minecraft_adventure_v1_3d] => 1

    [ze_NY_marathon_p] => 1

    [deathrun_cb_egypt_csgo_v1_esportcenter] => 1

    [ttt_nexus_v2_csgo] => 1

    [ze_Halloween_House] => 1

    [workshop/278917934/mg_The_Factory_course] => 1

    [ka_trains_enw] => 1

    [mg_randomizer_reborn_v1] => 1

    [mg_minecraft_multigames_csgo_v1] => 1

    [mg_kc_course_v1-1] => 1

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by asherkin (Post 2394549)

It's not very complete due to so many people not bothering to update Accelerator.

Code:

    [ze_farmhouse_v9_fix] => 5

    [ze_sky_athletic_v5_3] => 4

    [ze_elevator_escape_p2] => 4

    [ze_Boatescape_Extendido_p4] => 4

    [ze_Random_p2] => 4

    [mg_course_clear_v1fix] => 3

    [ze_Pirates_Port_Royal_v4_2] => 3

    [ze_potc_p4] => 3

    [ze_rollermine_factory_2010_p] => 3

    [ze_arctic_escape_p2] => 3

    [ze_LOTR_Helms_Deep_v5_p6] => 2

    [ze_30_seconds_go_v2] => 2

    [bhop_playboy_csgo] => 2

    [ze_icecap_escape_v5_sgfix] => 2

    [mg_savetheisland_course_final] => 2

    [ze_mytex_escape_v3_p] => 2

    [mg_Sketchy_course_go1] => 2

    [mg_airmap_run_v2_1] => 2

    [ze_fapescape_p5] => 1

    [ze_trainescape_final_p2] => 1

    [mg_circle_course] => 1

    [ze_infected_lab_v4_1] => 1

    [jb_clouds_beta01] => 1

    [Surf_Castle_v6_esportcenter] => 1

    [mg_minecraft_adventure_final_fix] => 1

    [ze_ice_hold_p1] => 1

    [mg_go_for_it_csgo] => 1

    [mg_lego_course_3_csgo] => 1

    [ze_farmhouse_v9] => 1

    [ze_licciana_xmas_v1_e1] => 1

    [ze_random_p2] => 1

    [ze_halo_poa_p4] => 1

    [ze_30_seconds__b21_p2] => 1

    [deathrun_cb_egypt_csgo_v1] => 1

    [ze_minecraft_adventure_v1_3d] => 1

    [ze_NY_marathon_p] => 1

    [deathrun_cb_egypt_csgo_v1_esportcenter] => 1

    [ttt_nexus_v2_csgo] => 1

    [ze_Halloween_House] => 1

    [workshop/278917934/mg_The_Factory_course] => 1

    [ka_trains_enw] => 1

    [mg_randomizer_reborn_v1] => 1

    [mg_minecraft_multigames_csgo_v1] => 1

    [mg_kc_course_v1-1] => 1

Well, I crashed on other maps not listed here, so, I can remove all my maplist lel

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by poel (Post 2394582)

Well, I crashed on other maps not listed here, so, I can remove all my maplist lel

Well it isn't magic, I just provided the data that 8guawong asked for.

I've spent some time today working with ImACow, who was having the same problem as you with empty crash dumps - we tracked it down to the grsecurity patch set being included in his kernel by his hosting provider. He's going to post some details for other OVH customers on how to switch to a standard kernel.

Since you refused to go into details on your kernel modifications, I'm guessing you're in a similar position. Breakpad takes measures to work with the newer restrictions included in the mainline kernel, but grsecurity is too opinionated to be supported.

I'm going to see what I can do to improve Breakpad's failure logging so everyone at least gets more information in these cases.

Re: Accelerator - Crash Reporting That Doesn't Suck

Quote:

Originally Posted by asherkin (Post 2394632)

About maplist, I just said, this shit update fuck 40% of maps so, its a real bullshit.

About the extension, the problem is, im using the same kernel since 2 years. I just changed my dedicated server, and now, not working anymore. So, I dont think so its from the kernel

Edit : So, im guessing, some library are missing (maybe)

Edit : maybe some settings on the latest dedicated servers ovh are fucked ?

Re: Accelerator - Crash Reporting That Doesn't Suck

It's the kernel - if you provide your kernel config I can probably help you.

Re: Accelerator - Crash Reporting That Doesn't Suck

Will be updating Accelerator but this is terrible news. Minigame and zombie servers are screwed :(

What about WIndows servers (also ovh)

Re: Accelerator - Crash Reporting That Doesn't Suck

"crash" engine commands create a dump which is uploaded to throttle. Can't you ignore theese report?

I'm using it to restart my server because "_restart" command will somehow shutdown.