Hey, did you ever solve this?
We also have lots of crashes on our servers (2-10 per day). These are mod-servers with gungame, wc3ft and a bunch of other small plugins, so I am guessing that a plugin is causing the crashes. Still, I would like to know
how to find out which plugin is to blame.
I have now tried to get some debugging with gdb done, here is how it is set up. Please alert me if there is anything configured incorrectly, this is mostly stuff scraped together from these forums and the hlds mailing list:
To start the server I run the command:
screen -A -m -d -S ag3 ./loop.script
loop.script
Code:
#! /bin/bash
export LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH"
while [ true ]; do
./hlds_run_gdb -binary ./hlds_i686 -autoupdate -debug -timeout 7 -pidfile super_cs.pid -game cstrike +map de_dust2 +maxplayers 16 +ip <removed> +port 17015
echo "Server crashed at '`date`' - Restarting"
echo "Server crashed at '`date`' - Restarting" >> crash.log
sleep 5
done
This again calls hlds_gdb which is a copy of hlds_run but with this as the command in the run() method:
Code:
# Run the server
echo $HL_CMD
gdb -x ag_batch_options -batch --args $HL_CMD
This uses the file ag_batch_options which contains
Code:
set logging on
run
info sharedlibrary
bt
gcore
quit
Phew. Thats the setup of how it all works. It actually seems to work quite well also, except for mentions of the "gcore" command "You can't do that without a process to debug". So we get core dumps of 80-120 MB in size, and a gdb.txt in the server folder. Here is one of both:
Example from the generated gdb.txt (instead of debug.log)
Code:
[New Thread 0xb47dfb90 (LWP 25989)]
Program received signal SIGHUP, Hangup.
[Switching to Thread 0xb4fd9b90 (LWP 9873)]
0xffffe410 in __kernel_vsyscall ()
From To Syms Read Shared Object Library
0xb7fd5880 0xb7fea8ef Yes /lib/ld-linux.so.2
0xb7fcca70 0xb7fcd964 Yes /lib/libdl.so.2
0xb7fb9270 0xb7fc4164 Yes /lib/libpthread.so.0
0xb7e9add0 0xb7f806b0 Yes /lib/libc.so.6
0xb7975c30 0xb79f62b4 Yes /home/ag/ag3/engine_i686.so
0xb79365f0 0xb7944ec4 Yes ./libsteam_api_c.so
0xb790e470 0xb7928a24 Yes /lib/libm.so.6
0xb78f2a60 0xb7900b4d Yes /home/ag/ag3/filesystem_stdio_i386.so
0xb58bd184 0xb58d06e0 Yes ./cstrike/addons/metamod/dlls/metamod_i386.so
0xb569809c 0xb57c1668 Yes /home/ag/ag3/cstrike/dlls/cs_i386.so
0xb55fd2d0 0xb5611618 Yes /home/ag/ag3/cstrike/addons/logmod/dlls/logmod_mm_i386.so
0xb55478b0 0xb55c1cb4 Yes /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libstdc++.so.6
0xb54fb8b0 0xb5502874 Yes /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
0xb54f1920 0xb54f7704 Yes /lib/libnss_files.so.2
0xb54eaba0 0xb54ecf84 Yes /lib/libnss_dns.so.2
0xb54da180 0xb54e44e4 Yes /lib/libresolv.so.2
0xb544b5e0 0xb54a5854 Yes /home/ag/ag3/cstrike/addons/amxmodx/dlls/amxmodx_mm_i386.so
0xb528b290 0xb530d114 Yes cstrike/addons/amxmodx/modules/mysql_amxx_i386.so
0xb5247be0 0xb524a6e4 Yes cstrike/addons/amxmodx/modules/fun_amxx_i386.so
0xb52370e0 0xb52410b4 Yes cstrike/addons/amxmodx/modules/engine_amxx_i386.so
0xb51ffac0 0xb522ab74 Yes cstrike/addons/amxmodx/modules/fakemeta_amxx_i386.so
0xb51cc090 0xb51d1f34 Yes cstrike/addons/amxmodx/modules/nvault_amxx_i386.so
0xb51c1150 0xb51c68f4 Yes cstrike/addons/amxmodx/modules/cstrike_amxx_i386.so
0xb514cca0 0xb5152e24 Yes cstrike/addons/amxmodx/modules/csx_amxx_i386.so
0xb511a9f0 0xb51453a4 Yes cstrike/addons/amxmodx/modules/hamsandwich_amxx_i386.so
0xb31520f0 0xb3687534 Yes ./steamclient.so
0xb50d7d80 0xb50f8e94 Yes ./libtier0_s.so
0xb50768e0 0xb50a7d14 Yes ./libvstdlib_s.so
0xb4b604b0 0xb4c41a04 Yes ./libsteamvalidateuseridtickets.so
0xb4fe32d0 0xb4febe84 Yes cstrike/addons/amxmodx/modules/regex_amxx_i386.so
0xb4ecbdd0 0xb4eccdf4 Yes cstrike/addons/amxmodx/modules/sockets_amxx_i386.so
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7f3f7d1 in select () from /lib/libc.so.6
#2 0xb315b5f7 in CPipeWaiter::BWait () from ./steamclient.so
#3 0xb315c0d8 in CIPCServer::Thread_MainLoop () from ./steamclient.so
#4 0xb50daf6f in CatchAndWriteMiniDumpForVoidPtrFn () from ./libtier0_s.so
#5 0xb315b820 in CIPCServer::CThreadClient::Run () from ./steamclient.so
#6 0xb50df2e6 in SteamThreadTools::CThread::ThreadProc () from ./libtier0_s.so
#7 0xb7fba18b in start_thread () from /lib/libpthread.so.0
#8 0xb7f4609e in clone () from /lib/libc.so.6
Saved corefile core.9868
The program is running. Exit anyway? (y or n) [answered Y; input not from terminal]
[Thread debugging using libthread_db enabled]
[New Thread 0xb7dd56c0 (LWP 26275)]
[New Thread 0xb4730b90 (LWP 26278)]
[New Thread 0xb3f2fb90 (LWP 26279)]
[Thread 0xb3f2fb90 (LWP 26279) exited]
[New Thread 0xb4f2ab90 (LWP 26280)]
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e656c0 (LWP 26297)]
Program exited with code 0377.
warning: Temporarily disabling breakpoints for unloaded shared library "/lib/libpthread.so.0"
warning: Temporarily disabling breakpoints for unloaded shared library "/lib/libc.so.6"
From To Syms Read Shared Object Library
0xb7fb6880 0xb7fcb8ef Yes /lib/ld-linux.so.2
No stack.
/home/ag/ag3/ag_batch_options:5: Error in sourced command file:
You can't do that without a process to debug.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e646c0 (LWP 26305)]
Backtrace from the above mentioned corefile core.9868
Code:
gdb hlds_i686 core.9868
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /home/ag/ag3/engine_i686.so...done.
Loaded symbols for /home/ag/ag3/engine_i686.so
Reading symbols from /home/ag/ag3/libsteam_api_c.so...done.
Loaded symbols for ./libsteam_api_c.so
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /home/ag/ag3/filesystem_stdio_i386.so...done.
Loaded symbols for /home/ag/ag3/filesystem_stdio_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/metamod/dlls/metamod_i386.so...done.
Loaded symbols for ./cstrike/addons/metamod/dlls/metamod_i386.so
Reading symbols from /home/ag/ag3/cstrike/dlls/cs_i386.so...done.
Loaded symbols for /home/ag/ag3/cstrike/dlls/cs_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/logmod/dlls/logmod_mm_i386.so...done.
Loaded symbols for /home/ag/ag3/cstrike/addons/logmod/dlls/logmod_mm_i386.so
Reading symbols from /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libstdc++.so.6...done.
Loaded symbols for /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libstdc++.so.6
Reading symbols from /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1...done.
Loaded symbols for /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/dlls/amxmodx_mm_i386.so...done.
Loaded symbols for /home/ag/ag3/cstrike/addons/amxmodx/dlls/amxmodx_mm_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/mysql_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/mysql_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/fun_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/fun_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/engine_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/engine_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/fakemeta_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/fakemeta_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/nvault_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/nvault_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/cstrike_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/cstrike_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/csx_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/csx_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/hamsandwich_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/hamsandwich_amxx_i386.so
Reading symbols from /home/ag/ag3/steamclient.so...done.
Loaded symbols for ./steamclient.so
Reading symbols from /home/ag/ag3/libtier0_s.so...done.
Loaded symbols for ./libtier0_s.so
Reading symbols from /home/ag/ag3/libvstdlib_s.so...done.
Loaded symbols for ./libvstdlib_s.so
Reading symbols from /home/ag/ag3/libsteamvalidateuseridtickets.so...done.
Loaded symbols for ./libsteamvalidateuseridtickets.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/regex_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/regex_amxx_i386.so
Reading symbols from /home/ag/ag3/cstrike/addons/amxmodx/modules/sockets_amxx_i386.so...done.
Loaded symbols for cstrike/addons/amxmodx/modules/sockets_amxx_i386.so
Core was generated by `/home/ag/ag3/hlds_i686 -binary ./hlds_i686 -autoupdate -debug -timeout 7 -pidfil'.
Program terminated with signal 1, Hangup.
[New process 25989]
[New process 9873]
[New process 9868]
#0 0xffffe410 in __kernel_vsyscall ()
(gdb) backtrace full
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0xb7fbe576 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0xb5292069 in ?? () from cstrike/addons/amxmodx/modules/mysql_amxx_i386.so
No symbol table info available.
#3 0xb2eb4cd4 in ?? ()
No symbol table info available.
#4 0xb2eb4d04 in ?? ()
No symbol table info available.
#5 0xb2d18ca8 in ?? ()
No symbol table info available.
#6 0x00000000 in ?? ()
No symbol table info available.
(gdb) info frame
Stack level 0, frame at 0xb47df330:
eip = 0xffffe410 in __kernel_vsyscall; saved eip 0xb7fbe576
called by frame at 0xb47df350
Arglist at 0xb47df328, args:
Locals at 0xb47df328, Previous frame's sp is 0xb47df330
Saved registers:
ebp at 0xb47df320, eip at 0xb47df32c
(gdb)
Linear Mode
