[staaat]

Hi,

Fairly new to troubleshooting server side problems - I'm probably not looking in the right place for more details on understanding my problem. I'm hoping someone might be able to point me in the right direction.

I have a CSGO DS with two GOTV instances (0s and 90s delays) running.

I did a performance test by adding 9 bots to the server and I attempted to play a couple of rounds, but roughly a minute after the first round the server crashes.

I get this text spitting out at me, can anyone point me in the right direction?

Waited 14.7ms for SteamNetworkingSockets lock [ServiceThread]
**** WARNING: Watchdog timer exceeded, aborting!
PreMinidumpCallback: updating dump comment
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 149ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 1223.8ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
crash_20211129112255_1.dmp[23680]: Uploading dump (out-of-process)
/tmp/dumps/crash_20211129112255_1.dmp
crash_20211129112255_1.dmp[23680]: Finished uploading minidump (out-of-process): success = no
crash_20211129112255_1.dmp[23680]: error: libcurl.so: cannot open shared object file: No such file or directory
crash_20211129112255_1.dmp[23680]: file ''/tmp/dumps/crash_20211129112255_1.dmp'', upload no: ''libcurl.so: cannot open shared object file: No such file or directory''
Aborted (core dumped)

My assumption here is that the server's hardware performance cannot handle the amount of activity that is going on here.

Thanks in advance

[Bacardi]

There is 3 topic of this 5minute then crash.

Answer?
https://forums.alliedmods.net/showthread.php?t=335370

[staaat]

Hey Bacardi - thanks for the response.

I had seen this thread but didn't appear to help unfortunately. To be fair, I say ~5 minutes after playing, it actually crashes depending on how much is going on in the server, so for example, after about 10 deaths have happened it appears to crash.

I've posted the most recent output I got of reproducing this issue today.

I feel like this is related to an unnecessary CPU consumption issue, I don't foresee this lacking in resource requirements to perform such actions. I did monitor the server using "top" and noticed that CPU usage hits near 100% when this issue happens.

• O/S - Ubuntu 18.04.6 LTS
• CPU - Intel Xeon Platinum 8000 series processor (Skylake-SP) with a sustained all core Turbo CPU clock speed of up to 3.1 GHz
• Memory - 4GB
• HDD - 53GB

Waited 3.6ms for SteamNetworkingSockets lock [ServiceThread]
Waited 11.1ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 573ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 1649.5ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Net channel ratelimit exceeded for 213.120.53.183:27005: 259 packets rejected.
Waited 8.8ms for SteamNetworkingSockets lock [ServiceThread]
SteamNetworkingSockets lock held for 12.4ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Waited 2.9ms for SteamNetworkingSockets lock [ServiceThread]
SteamNetworkingSockets lock held for 11.6ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Waited 138.4ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 1332ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 1113.8ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Waited 167.2ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 1381ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 1114.8ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Waited 53.6ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 243ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 23.3ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
Waited 142.5ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 1404ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
**** WARNING: Watchdog timer exceeded, aborting!
PreMinidumpCallback: updating dump comment
SteamNetworkingSockets lock held for 2200.0ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
crash_20211130172927_1.dmp[29678]: Uploading dump (out-of-process)
/tmp/dumps/crash_20211130172927_1.dmp
crash_20211130172927_1.dmp[29678]: Finished uploading minidump (out-of-process): success = no
crash_20211130172927_1.dmp[29678]: error: libcurl.so: cannot open shared object file: No such file or directory
crash_20211130172927_1.dmp[29678]: file ''/tmp/dumps/crash_20211130172927_1.dmp'', upload no: ''libcurl.so: cannot open shared object file: No such file or directory''
Killed
Add "-debug" to the /home/xx/xx/srcds_run command line to generate a debug.log to help with solving this problem
Tue Nov 30 17:35:31 UTC 2021: Server restart in 10 seconds

[ThePwnt]

What is your start-command line?

What do you use to run? do you have tried to add

-nowatchdog

and see about it still crashing?

[Bacardi]

...next you could try simple, not load MM:S and SM and look do server still crash.

By removing metamod.vdf and metamod_x64.vdf file out from ...addons/ -folder and reboot server.

Move files back when you want load addons again, reboot.

[staaat]

Thank you both for feedback.

I don't have addons installed on the server at the moment, so I couldn't follow your kind suggestions Bacardi.

I did attempt to add

 -nowatchdog

but this only made the situation worse as the performance warning was continuing to repeat itself.

See my start command below:

/home/<user_name>/csgo-ds/srcds_run -game csgo -debug +exec autoexec -usercon -tickrate 128 +tv_enable 1 +tv_snapshotrate 128 -addhltv1 +tv_enable1 1 +tv_snapshotrate1 128 +tv_advertise_watchable 1 +game_type 0 +game_mode 1 +mapgroup mg_active +map de_dust2 +sv_setsteamaccount XXXXXXXXXXXXXXXXXXXXXX

--edit:
I realised that -debug was not working and I needed to install gdb (sudo apt install gdb), never used this before so I did nothing with this after the install.

I then proceeded to re-run the server using the above command line and I was able to reproduce the issue, but received a slightly different error this time:

Waited 687.0ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 981ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 33.0ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
**** WARNING: Watchdog timer exceeded, aborting!
PreMinidumpCallback: updating dump comment
Waited 254.7ms for SteamNetworkingSockets lock [ServiceThread]
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp(1852): SteamnetworkingSockets service thread waited 3729ms for lock!  This directly adds to network latency!  It could be a bug, but it's usually caused by general performance problem such as thread starvation or a debug output handler taking too long.
SteamNetworkingSockets lock held for 2380.8ms.  (Performance warning.)  ServiceThread
This is usually a symptom of a general performance problem such as thread starvation.
crash_20211201001257_1.dmp[4838]: Uploading dump (out-of-process)
/tmp/dumps/crash_20211201001257_1.dmp
crash_20211201001257_1.dmp[4838]: Finished uploading minidump (out-of-process): success = no
crash_20211201001257_1.dmp[4838]: error: Couldn't resolve host name
crash_20211201001257_1.dmp[4838]: file ''/tmp/dumps/crash_20211201001257_1.dmp'', upload no: ''Couldn't resolve host name''
Aborted (core dumped)
cat: hlds.4799.pid: No such file or directory
email debug.log to [email protected]
Wed Dec  1 00:17:22 UTC 2021: Server restart in 10 seconds

I looked into cat: hlds.4799.pid: No such file or directory' and found this post: https://forums.alliedmods.net/showthread.php?t=300118
I then proceeded to run the below command:

sudo dpkg --add-architecture i386
sudo apt-get install lib32gcc1 libstdc++6 libstdc++6:i386 libcurl4-gnutls-dev:i386

Unfortunately I'm still reproducing the issue, but I'm wondering if I'm missing something from the i386 install.

Have to park this for today as it's late in my time zone, if anyone has any suggestions based on the latest details that would be great, but I'll continue to look into this tomorrow.

[raj kaul]

you are getting performance warning in console thats the reason your server is crashing , means your instance not able to run the server also you are using 2 GOTV and with 2 relays its normal for server crash , please disable one gotv and try again

[staaat]

Hey Raj Kaul - I need to run two GOTV2 instances on one host.

For reference, I'm not using any relays here - what makes you think this?

Your right on the performance being a problem - it's what I figured the issue was at the beginning of this thread but was waiting for someone else's opinion from here. Perhaps the only option is to upgrade my hardware.

The main reason for this post was to see if anyone is aware of any cvars that limit/control the memory/cpu usage of the server? and/or if they knew what the error meant exactly, and lastly if anyone had any advice on debugging server crashes.

I have upgraded the hardware from:
CPU: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Memory: 4GB

To:
CPU: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Memory: 8GB

I appear to get a different error message:

***** OUT OF MEMORY! attempted allocation size: 13556 ****

PreMinidumpCallback: updating dump comment
Segmentation fault (core dumped)
cat: hlds.2476.pid: No such file or directory
email debug.log to [email protected]
Mon Dec  6 23:14:21 UTC 2021: Server restart in 10 seconds

Which obviously points to not having enough memory... so I'm going to look at seeing if this is an option, otherwise anyone with any suggestions or feedback on the above would be appreciated.

[Maxximou5]

If you're recording with GOTV, you will gobble up that RAM; however, if you are not recording, you probably have a memory leak somewhere.

More information here: https://forums.alliedmods.net/showthread.php?t=327724

Is this while running only one server or two?

[staaat]

Ok so as a test, I've removed the bots, joined the server and monitored server performance. Server didn't crash, ran fine no issues, memory consumption hit about 17.5% usage of 8gb

Added the 9 bots back, and what I've noticed here is that the memory usage is continuously creeping up after the player (me) joined the server, and then once the srcds_linux process hits 4GB of memory usage or requires more, it crashes.

Suggesting there is a memory leak? I've no idea, other than it is continuously creeping up - any suggestions why this is happening?

Does GOTV store information on memory? can I change this? Why would bots cause memory usage to increase? Is having 9 bots + 1 player compared to 10 players on the server an unfair test?

edit:
Maxximou5 - I'm only running one server, and I'm not recording GOTV.