View Single Post
Author Message
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-14-2016 , 11:22   Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #1

Accelerator replaces the default SRCDS crash handler with one that is a lot more reliable and uploads the crash reports to a community-accessible processing backend.
This is useful since Valve's processing system discards crash reports from modded servers - and they still receive a representative sample of crash reports for actual game issues.

The processing backend (Throttle - https://crash.limetech.org/) analyses crash reports to extract useful information, and in the case of common issues affecting many servers where manual analysis has been done, tags them with helpful notices with information on resolving the cause of the crash.

Previous versions of Accelerator have been in use on hundreds of servers since 2010 (as of this thread's creation, Throttle has processed over 30,000 crash reports in the last 90 days), this is a re-release as a lot has changed since then.

If you're running any earlier version (including the previous extension alpha build), please upgrade to 2.4.3+.

Installation
  1. Click the big Download link below.
  2. Download the latest version for your server's OS.
  3. Extract the Zip file and merge the addons/sourcemod directory with the one on your server.

Configuration

Edit addons/sourcemod/configs/core.cfg to add the following key:
Code:
"MinidumpAccount"  "<insert your steamid here>"
Your SteamID should be in the full 64-bit format, you can use a site like https://steamid.io/ to get yours, it'll look something like 7656119798781xxxx.

So your core.cfg will end up looking like (with your own steamid):
Code:
"Core"
{
  [...]

  "MinidumpAccount"  "7656119798781xxxx"
}
Obviously, you will have all the other options in there as well.

The full set of available options with their default values and documentation are below, you should only put ones you intend to change into core.cfg:
Code:
"Core"
{
  /**
   * SteamID64 (Community ID) that will have ownership of uploaded crash reports.
   * You can share your crash reports with additional users from the website.
   *
   * If unset, your crash reports will be uploaded anonymously and you will not be
   * able to see all of the information.
   */
  "MinidumpAccount"	""

  /**
   * Controls which binaries will be eligible to be processed for symbols and uploaded.
   * Only modules loaded by the server at the time of the crash can be considered.
   *
   * 0 = Disabled: No binaries will be processed or uploaded.
   * 1 = System Only: Only binaries outside of the game directory (where the srcds binary is).
   * 2 = System + Game: Loaded modules outside of the addons/ directory.
   * 3 = System + Game + Addons: All loaded modules.
   */
  "MinidumpSymbolUpload"	"3"

  /**
   * Controls whether Accelerator can upload complete module binaries when explicitly requested
   * by the processing server. This also respects the value of the MinidumpSymbolUpload setting.
   */
  "MinidumpBinaryUpload"	"yes"

  /**
   * Controls whether Accelerator does local processing of crash reports before upload.
   * This should only be changed if local processing causes issues such as crashes,
   * the processing server may reject crash reports that have not been presubmitted.
   */
  "MinidumpPresubmit"	"yes"

  /**
   * URL to upload crash dumps to. Should not be changed.
   */
  "MinidumpUrl"	"http://crash.limetech.org/submit"

  /**
   * URL to upload processed symbols to. Should not be changed.
   */
  "MinidumpSymbolUrl"	"http://crash.limetech.org/symbols/submit"

  /**
   * URL to upload binaries to. Should not be changed.
   */
  "MinidumpBinaryUrl"	"http://crash.limetech.org/binary/submit"
}
Usage

Once running on your servers, whenever they crash they will write a crash dump into sourcemod/data/dumps/.

Next time the server starts it will upload any pending crash dumps in that directory to Throttle (https://crash.limetech.org/), you can login to that site using Steam OpenID authentication to view any processed crash reports owned by you.

Accelerator logs all upload activity to sourcemod/logs/accelerator.log, from there you can see all uploaded crash IDs and error messages if any fail to upload.

The crash dump uploader is threaded so will not impose any startup delay on your servers. Crash dumps plus metadata are generally less than 500kb combined. There may be additional uploads occasionally if your server has symbols available that Throttle does not have yet.

Security

Crash dumps can contain sensitive information such as RCon / Database credentials, as they are produced by saving key parts of SRCDS' memory space.

As standard, crash reports uploaded to Throttle will contain at least the following information:
  • The process command line.
  • The last 500 lines of console output (only for supported games).
  • A list of all loaded native modules.
  • The contents of the stack memory for each thread.

As of version 2.4.x, Accelerator can now partially preprocess crash reports locally before upload, and process modules loaded into SRCDS' memory for symbol files (this is a hard restriction in the code, it won't touch any other files). The latter, if enabled, will cause files to be processed locally and uploaded to Throttle. See the MinidumpSymbolUpload and MinidumpBinaryUpload options above for more information.

Throttle maintains strict security controls, only the owner of the crash (the SteamID specified in core.cfg), people the owner selects to share their reports with on the website, and a select set of SourceMod Core team members can access full crash reports.

Everyone else can only see the crashing thread's call stack and any assigned notices, so please do link Throttle crash reports directly if you're seeking help rather than just copying the stack trace / taking a screenshot.

That said, please do not post in this thread looking for help diagnosing a crash.
Create your own thread in the Source Servers section where you'll get much better help and will not fill up this thread with transient trash.


Supported Games

At a basic level Accelerator should work with all games on both Linux and Windows.
Games with enhanced support will additionally get console output included in crash reports.

Windows (and CS:GO) crash reports are generally useless if they do not directly crash due to SourceMod, this is because Valve continue to refuse to provide stripped symbol files (like Microsoft provide for Windows).

Download

Source Code

Build Status
__________________

Last edited by asherkin; 01-14-2019 at 18:22.
asherkin is offline