Raised This Month: $7 Target: $400
 1% 

Accelerator - Crash Reporting That Doesn't Suck


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-14-2016 , 11:22   Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #1

Accelerator replaces the default SRCDS crash handler with one that is a lot more reliable and uploads the crash reports to a community-accessible processing backend.
This is useful since Valve's processing system discards crash reports from modded servers - and they still receive a representative sample of crash reports for actual game issues.

The processing backend (Throttle - https://crash.limetech.org/) analyses crash reports to extract useful information, and in the case of common issues affecting many servers where manual analysis has been done, tags them with helpful notices with information on resolving the cause of the crash.

Previous versions of Accelerator have been in use on hundreds of servers since 2010 (as of this thread's creation, Throttle has processed over 30,000 crash reports in the last 90 days), this is a re-release as a lot has changed since then.

If you're running any earlier version (including the previous extension alpha build), please upgrade to 2.4.3+.

Installation
  1. Click the big Download link below.
  2. Download the latest version for your server's OS.
  3. Extract the Zip file and merge the addons/sourcemod directory with the one on your server.

Configuration

Edit addons/sourcemod/configs/core.cfg to add the following key:
Code:
"MinidumpAccount"  "<insert your steamid here>"
Your SteamID should be in the full 64-bit format, you can use a site like https://steamid.io/ to get yours, it'll look something like 7656119798781xxxx.

So your core.cfg will end up looking like (with your own steamid):
Code:
"Core"
{
  [...]

  "MinidumpAccount"  "7656119798781xxxx"
}
Obviously, you will have all the other options in there as well.

The full set of available options with their default values and documentation are below, you should only put ones you intend to change into core.cfg:
Code:
"Core"
{
  /**
   * SteamID64 (Community ID) that will have ownership of uploaded crash reports.
   * You can share your crash reports with additional users from the website.
   *
   * If unset, your crash reports will be uploaded anonymously and you will not be
   * able to see all of the information.
   */
  "MinidumpAccount"	""

  /**
   * Controls which binaries will be eligible to be processed for symbols and uploaded.
   * Only modules loaded by the server at the time of the crash can be considered.
   *
   * 0 = Disabled: No binaries will be processed or uploaded.
   * 1 = System Only: Only binaries outside of the game directory (where the srcds binary is).
   * 2 = System + Game: Loaded modules outside of the addons/ directory.
   * 3 = System + Game + Addons: All loaded modules.
   */
  "MinidumpSymbolUpload"	"3"

  /**
   * Controls whether Accelerator can upload complete module binaries when explicitly requested
   * by the processing server. This also respects the value of the MinidumpSymbolUpload setting.
   */
  "MinidumpBinaryUpload"	"yes"

  /**
   * Controls whether Accelerator does local processing of crash reports before upload.
   * This should only be changed if local processing causes issues such as crashes,
   * the processing server may reject crash reports that have not been presubmitted.
   */
  "MinidumpPresubmit"	"yes"

  /**
   * URL to upload crash dumps to. Should not be changed.
   */
  "MinidumpUrl"	"http://crash.limetech.org/submit"

  /**
   * URL to upload processed symbols to. Should not be changed.
   */
  "MinidumpSymbolUrl"	"http://crash.limetech.org/symbols/submit"

  /**
   * URL to upload binaries to. Should not be changed.
   */
  "MinidumpBinaryUrl"	"http://crash.limetech.org/binary/submit"
}
Usage

Once running on your servers, whenever they crash they will write a crash dump into sourcemod/data/dumps/.

Next time the server starts it will upload any pending crash dumps in that directory to Throttle (https://crash.limetech.org/), you can login to that site using Steam OpenID authentication to view any processed crash reports owned by you.

Accelerator logs all upload activity to sourcemod/logs/accelerator.log, from there you can see all uploaded crash IDs and error messages if any fail to upload.

The crash dump uploader is threaded so will not impose any startup delay on your servers. Crash dumps plus metadata are generally less than 500kb combined. There may be additional uploads occasionally if your server has symbols available that Throttle does not have yet.

Security

Crash dumps can contain sensitive information such as RCon / Database credentials, as they are produced by saving key parts of SRCDS' memory space.

As standard, crash reports uploaded to Throttle will contain at least the following information:
  • The process command line.
  • The last 500 lines of console output (only for supported games).
  • A list of all loaded native modules.
  • The contents of the stack memory for each thread.

As of version 2.4.x, Accelerator can now partially preprocess crash reports locally before upload, and process modules loaded into SRCDS' memory for symbol files (this is a hard restriction in the code, it won't touch any other files). The latter, if enabled, will cause files to be processed locally and uploaded to Throttle. See the MinidumpSymbolUpload and MinidumpBinaryUpload options above for more information.

Throttle maintains strict security controls, only the owner of the crash (the SteamID specified in core.cfg), people the owner selects to share their reports with on the website, and a select set of SourceMod Core team members can access full crash reports.

Everyone else can only see the crashing thread's call stack and any assigned notices, so please do link Throttle crash reports directly if you're seeking help rather than just copying the stack trace / taking a screenshot.

That said, please do not post in this thread looking for help diagnosing a crash.
Create your own thread in the Source Servers section where you'll get much better help and will not fill up this thread with transient trash.


Supported Games

At a basic level Accelerator should work with all games on both Linux and Windows.
Games with enhanced support will additionally get console output included in crash reports.

Windows (and CS:GO) crash reports are generally useless if they do not directly crash due to SourceMod, this is because Valve continue to refuse to provide stripped symbol files (like Microsoft provide for Windows).

Download

Source Code

Build Status
__________________

Last edited by asherkin; 01-14-2019 at 18:22.
asherkin is offline
sneaK
SourceMod Moderator
Join Date: Feb 2015
Location: USA
Old 01-14-2016 , 12:11   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #2

Nice release! Updated my servers, Accelerator has always been a great help for me. Keep up the good work!
__________________
sneaK is offline
braak0327
AlliedModders Donor
Join Date: Dec 2012
Location: NORT....SOUTH KOREA
Old 01-14-2016 , 19:24   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #3

sounds nice! can't wait to try this great job master!
__________________
I'M KIMCHI
braak0327 is offline
Send a message via Skype™ to braak0327
spumer
Senior Member
Join Date: Aug 2011
Old 01-15-2016 , 01:57   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #4

What about backend sorucecode? For example start this service in private network.
__________________
spumer is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-15-2016 , 05:38   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #5

Quote:
Originally Posted by spumer View Post
What about backend sorucecode? For example start this service in private network.
There is a link to the source code at the bottom of every page, and there has been since launch. There are no complete installation instructions and I don't have time to walk you through setting it up - it is also not advised since a number of the advantages come from operating at scale, if you don't know how to analyse a crash dump yourself you're going to get very little out of it.
__________________
asherkin is offline
poel
Veteran Member
Join Date: Mar 2013
Old 01-16-2016 , 08:31   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #6

Nice update, but I cant understand.

When server crash, nothing happen ! I mean, I dont have any logs on the website ! I just have to install extension and add my steam in core file ?

log sm : Failed to upload crash dump: The requested URL returned error: 400 (22)

Last edited by poel; 01-16-2016 at 08:36.
poel is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-16-2016 , 08:57   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #7

Quote:
Originally Posted by poel View Post
Nice update, but I cant understand.

When server crash, nothing happen ! I mean, I dont have any logs on the website ! I just have to install extension and add my steam in core file ?

log sm : Failed to upload crash dump: The requested URL returned error: 400 (22)
That means that Breakpad failed to generate a crash dump for your crash and it tried to upload an empty minidump.

I've never been able to replicate the issue and it appears to be an OS-level problem, I would need access to your host to look into it.
__________________
asherkin is offline
ejk499
New Member
Join Date: Jan 2016
Old 01-17-2016 , 05:44   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #8

Is it possible to revert a375772 commit?

Serveral servers of my friend was damaged by this commit. He realy enjoyed Accelerator because it saved him a lot of times! He is using "thing, that cannot be spoken" not on his own, but because all of his local competitors use it and without it his online will suffer.

The amount of CSS players is dramatically decreasing and by now, the amount of players using "thing, that cannot be spoken" and not using it is almost the same.

Last edited by ejk499; 01-17-2016 at 05:45.
ejk499 is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-17-2016 , 06:07   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #9

No, by far most of Throttle's processing capacity was being used by no-steam servers.
While that capacity is seriously constrained, I wish to make it available to legitimate servers.
__________________

Last edited by asherkin; 01-17-2016 at 06:07.
asherkin is offline
ejk499
New Member
Join Date: Jan 2016
Old 01-17-2016 , 08:10   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #10

Quote:
Originally Posted by asherkin View Post
No, by far most of Throttle's processing capacity was being used by no-steam servers.
While that capacity is seriously constrained, I wish to make it available to legitimate servers.
If the problem is in capacity, then maybe it would be better to just prioritize reports processing?

1. Legit
2. Legit ES crash
3. no-steam
4. no-steam ES crash

(Almost all servers, that uses ES is a WCS servers, and they are crashing every 1-2 hours)
ejk499 is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 22:06.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode