Raised This Month: $68 Target: $400
 17% 

How to use Bash script to automate server hang repair.


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
DJEarthQuake
Senior Member
Join Date: Jan 2014
Location: Midwest
Old 03-15-2019 , 00:49   How to use Bash script to automate server hang repair.
Reply With Quote #1

https://forums.alliedmods.net/showpo...9&postcount=25

Quote:
Originally Posted by DJEarthQuake View Post
Make sure logging is on modded server with:
Code:
log on
What this does is waits for file size to be zero. The last 30 or so minutes of the server's history log is found and saved. Once it is 0 we know the server is hung and needs to be killed so the server restart process can resume. I use a much simpler nmap the port approach for that matter. If port 27015 is closed the hlds or srcds process (your server) begins again no matter what with optional page/e-mail options.

I had a ball using PHP beforehand since there is no such thing as a UDP ping tester. PHP made me regret not learning it 20 years ago. I had a PHP script checking UDP latency. If over spec amount of so many thousand from momentary map change, it retested again to make sure it was just a map change. PHP was the only book not on my shelf. Scary what it can sniff on a single UDP packet. Paul's Home Page. Thanks Paul. Long forgotten author of PHP.


This is my busy or lazy admin script developed 5 years ago. Finding the bad plugin, which is almost always 3rd party bots, is the best method. IE| cs and cz. CZ has bots native. CS does not.

was added and removed in this example I am sharing to eliminate false positives. Then there is a separate process killer if port is not open I used with nmap every min or however you want. HLDSWatch in your case. There is nothing like being in a perfectly good game then, JANK, your own game, from your own script. Keep the cron times long so this does not happen.

PHP Code:

cat sizer
.sh


#!/bin/bash
sleep 7
myfilesize
=`stat -c %s ~/log_GTM.log`
if [ 
$myfilesize ]
then
#exec ~/retry.sh
else
echo 
"The file size is NOT zero it's" $myfilesize 
fi
exit 0



cat logger
.sh


#!/bin/bash
MOD="gearbox"
find ~/Steam/steamapps/common/Half-Life/$MOD/logs/* -mmin -30 >~/log_GTM.log && cat log_GTM.log && ls -la log_GTM.log 
I changed this as I pasted and initially grabbed the wrong script. This 2 different files. Name as you wish .sh. chmod +x thefile.sh of course. Enjoy no longer being exasperated.

There is a much noticeable load difference not having to run and maintain server with PHP to use the latency script which would also work but cost 5% or more on the core. That is literally money as it uses more electricity.

So cstrike users replace gearbox with cstrike and so on for other mods. Make the logger every minute to 7. The sizer @hourly. Anything less than 40 minutes without running a second script to do a double-check it will lead to disappointments

Oh to the fellow with the IP problem. inet addr: xxx.xxx.xxx.x P-t-P:168.95.98.254. I think you already know 168 is your PPPoE peer, as you put x on yours, mate.



This funny thing is, this wasn't even the post I was seeking, still haven't found it, found 1 similar and posted. Now I post here!

Best of luck everybody keeping the servers running well.
__________________
"It's not the actual programming that's interesting. But it's what you can accomplish with the end results that are important." -Dennis Ritchie
"Mathematics, rightly viewed, possesses not only truth, but supreme beauty a beauty cold and austere, like that of sculpture..." -Bertrand Russell

Last edited by DJEarthQuake; 03-20-2019 at 20:21. Reason: more descriptive title
DJEarthQuake is offline
DJEarthQuake
Senior Member
Join Date: Jan 2014
Location: Midwest
Old 03-17-2019 , 09:29   Re: BASH server wedges away
Reply With Quote #2

Cleaning up VPS today for new hardware. I found the script I used before this and before running own server hosting PHP script instead of getting it from game-monitor. They shutdown and that helped me do more than just Drupalize.

PHP Code:
#!/bin/sh
SERVER=NAMEYOURSERVER
IP
=TYPEYOURIP
PORT
=27015
BAD
="999"


#NBR=`GET http://module.game-monitor.com/YOURIP:27016/data/server.js |
#grep -w "query_time" | rev | cut -c 5-8 | rev`

NBR=`GET http://goldsrc.live/servers/info/$IP:$PORT |grep ms|cut -d "<" -f 2|cut -d ">" -f2|cut -d " " -f1`

if [ 
$BAD \> $NBR ]; then
    
echo "$SERVER IS RUNNING...latency is calculating as"
    
echo $NBR
else
    echo 
"$SERVER WEDGED...latency is calculating as"
    
echo $NBR
    
~/crashb.sh
fi 
__________________
"It's not the actual programming that's interesting. But it's what you can accomplish with the end results that are important." -Dennis Ritchie
"Mathematics, rightly viewed, possesses not only truth, but supreme beauty a beauty cold and austere, like that of sculpture..." -Bertrand Russell

Last edited by DJEarthQuake; 03-29-2019 at 12:33. Reason: made NBR 2019 current
DJEarthQuake is offline
DJEarthQuake
Senior Member
Join Date: Jan 2014
Location: Midwest
Old 03-29-2019 , 12:37   Re: How to use Bash script to automate server hang repair.
Reply With Quote #3

Updated the last script so it works in 2019!
Code:
NBR=`GET http://goldsrc.live/servers/info/$IP:$PORT |grep ms|cut -d "<" -f 2|cut -d ">" -f2|cut -d " " -f1`

This then can be used to activate a pkill script on a seized up server.
__________________
"It's not the actual programming that's interesting. But it's what you can accomplish with the end results that are important." -Dennis Ritchie
"Mathematics, rightly viewed, possesses not only truth, but supreme beauty a beauty cold and austere, like that of sculpture..." -Bertrand Russell
DJEarthQuake is offline
xkonsole
Member
Join Date: Mar 2018
Old 03-30-2019 , 08:42   Re: How to use Bash script to automate server hang repair.
Reply With Quote #4

Whats the point of using third-party stuff in your scripts? That would stop working pretty insta after service changes its behavior.

Last edited by xkonsole; 03-30-2019 at 08:42.
xkonsole is offline
DJEarthQuake
Senior Member
Join Date: Jan 2014
Location: Midwest
Old 03-31-2019 , 12:06   Re: How to use Bash script to automate server hang repair.
Reply With Quote #5

Scroll to OP please. What you brought up is great. While the 3rd party feed approach is nicer than manually fixing a hung server. That was scrapped almost a decade ago with local services doing same feed also to favor the Zero-log method. When it got better? Luck. Simply installing Amxx 1.9.0 helped me find bad plugin and not need any script of the sort ever again. The 3rd party feed latency measuring needed a second check to assure it wasn't due to a map change and BAD needed to be 3000-5000 to be effective. Topple over I had at one point used 3rd party or any of a few of mine before performing pkill and long before switching to log file check. The local PHP was the fastest and the log file approach was the most reliable. Those same variances can happen on the LAN feed too. It amused me to see some sites are still running like that and it still works. We hope sites like that have more than 1 server 1 core budgets. Worse cases for more critical data are neither here nor there have happened.
__________________
"It's not the actual programming that's interesting. But it's what you can accomplish with the end results that are important." -Dennis Ritchie
"Mathematics, rightly viewed, possesses not only truth, but supreme beauty a beauty cold and austere, like that of sculpture..." -Bertrand Russell
DJEarthQuake is offline
DJEarthQuake
Senior Member
Join Date: Jan 2014
Location: Midwest
Old 09-20-2019 , 14:02   Re: How to use Bash script to automate server hang repair.
Reply With Quote #6

Wish I would have done this years ago! No more hangs!!

Code:
#!/bin/sh dpkg -l | grep curl dpkg --add-architecture i386 apt install libcurl3-gnutls:i386 -y && apt update -y && apt upgrade -y dpkg -l | grep curl | grep 386
__________________
"It's not the actual programming that's interesting. But it's what you can accomplish with the end results that are important." -Dennis Ritchie
"Mathematics, rightly viewed, possesses not only truth, but supreme beauty a beauty cold and austere, like that of sculpture..." -Bertrand Russell
DJEarthQuake is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 21:55.


Powered by vBulletin®
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Theme made by Freecode