Raised This Month: $90 Target: $400
 22% 

Accelerator - Crash Reporting That Doesn't Suck


Post New Thread Reply   
 
Thread Tools Display Modes
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-05-2019 , 19:37   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #291

Static linking isn't sane inside SRCDS, just build it on your target system.
__________________
asherkin is offline
kadet.89
Veteran Member
Join Date: Nov 2012
Location: The term is used to iden
Old 01-05-2019 , 19:43   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #292

asherkin, these flags work with standard extensions. There is no way to build the extionsion on the target system. Is there a way to set these "static build" flags to make ambuild understend them?

Last edited by kadet.89; 01-05-2019 at 19:44.
kadet.89 is offline
Send a message via Skype™ to kadet.89
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-05-2019 , 19:55   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #293

You would need to make the same modifications to Breakpad's build system, which is not built with AMBuild - it isn't something I can help you with.
__________________
asherkin is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-06-2019 , 19:40   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #294

Quote:
Originally Posted by asherkin View Post
I've been working on some features for the next generation of the Accelerator extension, and they're ready for a bit of wider testing (Linux servers only at the moment).

The big addition is partial local processing of crash dumps, to do deduplication with and figure our what binaries are missing symbols server-side.
This has been working well for the people using it for quite a while now - symbol coverage went up from an average of 60% to 80%, which makes for much more complete crash reports.

The "finished" version of the local crash processing changes are now complete, and the following options have been added (shown with their default values):
Code:
"Core"
{
	/**
	 * SteamID64 (Community ID) that will have ownership of uploaded crash reports.
	 * You can share your crash reports with additional users from the website.
	 *
	 * If unset, your crash reports will be uploaded anonymously and you will not be
	 * able to see all of the information.
	 */
	"MinidumpAccount"	""

	/**
	 * Controls which binaries will be eligible to be processed for symbols and uploaded.
	 * Only modules loaded by the server at the time of the crash can be considered.
	 *
	 * 0 = Disabled: No binaries will be processed or uploaded.
	 * 1 = System Only: Only binaries outside of the game directory (where the srcds binary is).
	 * 2 = System + Game: Loaded modules outside of the addons/ directory.
	 * 3 = System + Game + Addons: All loaded modules.
	 */
	"MinidumpSymbolUpload"	"3"

	/**
	 * Controls whether Accelerator can upload complete module binaries when explicitly requested
	 * by the processing server. This also respects the value of the MinidumpSymbolUpload setting.
	 */
	"MinidumpBinaryUpload"	"yes"
}
I hope that this branch will become the mainline version later this week, along with becoming the recommended version by Throttle. (Unless anything critical comes up, you won't need to update again after release.)

Download (Source)
__________________
asherkin is offline
Dragokas
Senior Member
Join Date: Nov 2017
Location: Ukraine
Old 01-06-2019 , 19:48   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #295

Asherkin, nice! Good job.

Can you please answer the first part of my question?

offtop.
Are there any settings in sm - my server doesn't create crash logs sometimes (but, I see RCON console show that crash happens). The case of infinite server rebooting (possibly, incompatiblity between 1.10 and my plugins, 1.9 - work ok).
__________________
Expert of CMD/VBS/VB6. Malware analyst. L4D1 Spartan Witch //^.^\\ server

Last edited by Dragokas; 01-06-2019 at 19:49.
Dragokas is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-07-2019 , 12:13   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #296

Quote:
Originally Posted by Dragokas View Post
Asherkin, nice! Good job.

Can you please answer the first part of my question?
Your crash doesn't have any plugin code on the stack - there is no plugin directly involved.

You can see that it passed though SM in 2 places: the hook for the OnGameFrame forward, and the hool for the OnPlayerRunCmd forward, but the server has continued much deeper than those and is busy executing the games internal PlayerRunCommand code.

Now, the stack may not be 100% correct, generally things are more likely to be wrong the further away from #0 you look (which is always correct), and that one is extra questionable because of the odd recursive behaviour with CountdownTimer::Now at the top and the fact that all the frames have been found using stack scanning (L4D must be one of the only Valve games compiled with frame pointers omitted, and without debug symbols it has had to do some guesswork to reconstruct the stack, so it is very possible for it to go off on some wild tangent.)

A directly plugin-caused crash looks a lot more like this: https://crash.limetech.org/fqoikydcdxdc

Quote:
Originally Posted by Dragokas View Post
offtop.
Are there any settings in sm - my server doesn't create crash logs sometimes (but, I see RCON console show that crash happens). The case of infinite server rebooting (possibly, incompatiblity between 1.10 and my plugins, 1.9 - work ok).
On Windows, Accelerator can miss some crashes due to how it is implemented, but on Linux it will catch all abnormal terminations. There are cases in the game/engine where it exits cleanly rather than crashing, which may be what you're seeing. There are also cases where Breakpad cannot generate a crash report due to OS issues, but that is generally not intermittent. In any case, Accelerator doesn't upload any crash reports until the server has started normally again, but they're all saved even when it is restarting in a tight loop (as long as it is getting far enough to load Accelerator) and will be uploaded later.
__________________
asherkin is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-10-2019 , 20:00   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #297

So it turns out the information the Breakpad project has been providing for years on processing "linux-gate.so" is bogus. Many of you will have seen crashes with linux-gate.so at the top of the stack and then completely bogus information from there on - it turns out this is mostly due to the symbols for it being quite wrong for modern Linux kernels.

I have removed all of the garbage linux-gate.so symbol files from Throttle and will be re-processing all crash dumps affected to remove any garbage / misleading data from the database (sadly this will probably result in very short stack traces for anything crashing in a syscall - but no information is better than wrong information).

I've also written a small program to dump the correct symbols from a running system, which'll shortly be incorporated into the "local-processing" beta version of Accelerator. As those symbols come back in from real systems, Throttle will automatically re-process crash dumps as-needed and we'll all have better stacks, woo.
__________________
asherkin is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-11-2019 , 22:00   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #298

Quote:
Originally Posted by asherkin View Post
I've also written a small program to dump the correct symbols from a running system, which'll shortly be incorporated into the "local-processing" beta version of Accelerator. As those symbols come back in from real systems, Throttle will automatically re-process crash dumps as-needed and we'll all have better stacks, woo.
This has now been done and is in the 2.4.2 version of the local-processing branch which is available at the link above / below.

Quote:
Originally Posted by asherkin View Post
__________________
asherkin is offline
Lux
Veteran Member
Join Date: Jan 2015
Location: Brexit
Old 01-12-2019 , 14:21   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #299

Quote:
Originally Posted by asherkin View Post
This has now been done and is in the 2.4.2 version of the local-processing branch which is available at the link above / below.
Thankyou for Accelerator!
__________________
Lux is offline
asherkin
SourceMod Developer
Join Date: Aug 2009
Location: OnGameFrame()
Old 01-14-2019 , 19:32   Re: Accelerator - Crash Reporting That Doesn't Suck
Reply With Quote #300

Hi all!

Coincidentally on the 3rd anniversary (and 300th post) of this thread (and a little after 8 years of Accelerator in all), the local-processing beta branch has been merged into master and is now the recommended version. Thanks for all the testing everyone, with your data I managed to make quite a few fixes and improvements. Sadly that does mean we ended up with a 2.4.3 version so it'll be updates all around.

All the details, documentation, and download links can be found in the OP in the normal places.

On the Throttle side, there's been a few minor improvements while this work on Accelerator has been ongoing. Mostly around optimisation and performance (Throttle recently passed 15,000,000 uploaded crash reports), but a useful change has been made: Crash reports will no longer be deleted (either due to age or per-server limits) if they have been viewed recently. This should mean less dead links here on the forum and in bug reports.

There is some more stuff in the works (when isn't there?), but that requires the new client-side functionality in Accelerator 2.4.3+, so get updating!

As always, post here if you run into any issues with the new version.

Thanks,
Asher
__________________

Last edited by asherkin; 01-14-2019 at 19:32.
asherkin is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 01:15.


Powered by vBulletin®
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Theme made by Freecode