[Astridax]

Hey guys, I've been battling with this problem on and off over two different sets of machines and a couple of years now and I have finally reached the end of my tether in knowing what to do with regards to solving this problem. There seem to be so many variables that I can't quite pin what could be causing this down.

For anyone who wants to test this out, the IP address is:
tf2.mylittlemercs.co.uk

In brief, we are having lots of lag issues with our servers. This can sometimes be coupled with high choke, even after I spent all this time reading:

http://whisper.ausgamers.com/wiki/index.php/Tickrate

Now I realise the tickrate command was removed and running at 100 tickrate is mostly redundant.

However when visiting a group of servers that were reported to run near flawlessly, lag free I noticed they were running at 33 tick compared to the stock 66 which led me to the tickrate enabler plugin. Hence I have been experimenting as a last ditch attempt to solve my extrapolation problems.

Some pictures to illustrate the net graph I have been looking at:

An example of high choke:

Spoiler

An example of both the server not keeping up with the set tickrate (here it is the default 66) and also the two bizarre extrapolation peaks we keep getting:

I ought to point out I have never seen the server fps drop below its correct value.

Spoiler

I also get a slightly smaller single peak often. These peaks are most definitely NOT random, however they can happen at random intervals. Often when there are greater than 8 players on the server.

I have tried disabling plugins one by one and nothing could be determined conclusively that helped. I've sometimes had to restart the server. I've noticed various pieces of console spam before such as:

datatable warning: player: out-of-range value (346.000000) in sendpropfloat 'm_angeyeangles[0]', clamping.

However a map change usually makes them disappear.

I then went through disabling each extension, eventually restarting the server with only metamod loaded and no meta plugins. This did prove to be stable, but at the time I wasn't looking at choke and the server didn't fill up again to test properly.

My server is set to log everything to file and also to the console so that I can watch for any anomalies. I am not sure whether this in itself could cause the lag.

To make this clear, this problem has happened even on a stock server, however it can be so difficult to test since getting a server instance full while you are messing around with it can be difficult and people will only take so much.

I am running meta version:

-> meta version
Metamod:Source version 1.10.0
Build ID: 860:a58a1912f602
Loaded As: Valve Server Plugin
Compiled on: Aug 25 2013
Plugin interface version: 15:14
SourceHook version: 5:5
http://www.metamodsource.net/

and sourcemod version:
SourceMod Version Information:
SourceMod Version: 1.5.2
SourcePawn Engine: SourcePawn 1.1, jit-x86 (build 1.5.2)
SourcePawn API: v1 = 4, v2 = 4
Compiled on: Oct 29 2013 1748
Build ID: 3946:238c41726d73
http://www.sourcemod.net/


My plugin list is as follows:

01 "Basic Comm Control" (1.5.2) by AlliedModders LLC
02 "[UMC] Prefix Exclusion" (3.4.5) by Steell
03 "[UMC] Map Commands" (3.4.5) by Steell
04 "[UMC] Ultimate Mapchooser Core" (3.4.5) by Steell
05 "[UMC] Echo Nextmap" (3.4.5) by Steell
06 "[UMC] Post-Played Exclusion" (3.4.5) by Sazpaimon and Steell
07 "[UMC] Player Limits" (3.4.5) by Steell
08 "[UMC] Player Count Monitor" (3.4.5) by Steell
09 "[UMC] Time Limits" (3.4.5) by Steell
10 "[UMC] Nominations" (3.4.5) by Steell
11 "[UMC] Vote Command" (3.4.5) by Steell
12 "[UMC] Random Cycle" (3.4.5) by Steell
13 "[UMC] Map Weight" (3.4.5) by Steell
14 "[UMC] Admin Menu" (3.4.5) by Steell
15 "[UMC] End of Map Vote" (3.4.5) by Steell
16 "[UMC] End of Map Vote Warnings" (3.4.5) by Steell
17 "[UMC] Map Rate Reweight" (3.4.5) by Steell
18 "[UMC] Rock The Vote" (3.4.5) by Steell
19 "[TF2] gScramble (Redux)" (3.0.16) by David
20 "Basic Votes" (1.5.2) by AlliedModders LLC
21 "Reserved Slots" (1.5.2) by AlliedModders LLC
22 "Admin File Reader" (1.5.2) by AlliedModders LLC
23 "Fun Commands" (1.5.2) by AlliedModders LLC
24 "Admin Menu" (1.5.2) by AlliedModders LLC
25 "Anti-Flood" (1.5.2) by AlliedModders LLC
26 "Basic Info Triggers" (1.5.2) by AlliedModders LLC
27 "Basic Commands" (1.5.2) by AlliedModders LLC
28 "TF2 Class Restrictions" (0.6) by Tsunami
29 "Basic Chat" (1.5.2) by AlliedModders LLC
30 "SMAC Command Monitor" (0.8.5.0) by SMAC Development Team
31 "SourceMod Anti-Cheat" (0.8.5.0) by SMAC Development Team
32 "SMAC Anti-Speedhack" (0.8.5.0) by SMAC Development Team
33 "SMAC Eye Angle Test" (0.8.5.0) by SMAC Development Team
34 "SMAC Rcon Locker" (0.8.5.0) by SMAC Development Team
35 "SMAC Aimbot Detector" (0.8.5.0) by SMAC Development Team
36 "SMAC ConVar Checker" (0.8.5.0) by SMAC Development Team
37 "Advertisements" (0.6) by Tsunami
38 "Player Commands" (1.5.2) by AlliedModders LLC
39 "Client Preferences" (1.5.2) by AlliedModders LLC
40 "Updater" (1.1.5) by GoD-Tony
41 "Sound Commands" (1.5.2) by AlliedModders LLC
42 "Be the Robot" (1.3) by MasterOfTheXP
43 "Spray Tracer" (5.8a) by Nican132, CptMoore, Lebson506th
44 "Admin Help" (1.5.2) by AlliedModders LLC
45 "Fun Votes" (1.5.2) by AlliedModders LLC

I have also disabled (after the latter two screenshots but before the first) any plugin that contacts a local mysql server, these plugins being:

MapRate
Notices
ParanoiaIP
Sourcebans

sm exts list:
[01] Automatic Updater (1.5.2): Updates SourceMod gamedata files
[02] Webternet (1.5.2): Extension for interacting with URLs
[03] SMRCon (1.2.1): Tools for SM plugins to interact with rcon auth and requests
[04] BinTools (1.5.2): Low-level C/C++ Calling API
[05] TF2Items (1.5.3): TF2 Item Modifier
[06] TF2 Tools (1.5.2): TF2 extended functionality
[07] SDK Hooks (1.5.2): Source SDK Hooks
[08] SDK Tools (1.5.2): Source SDK Tools
[09] Top Menus (1.5.2): Creates sorted nested menus
[10] Regex (1.5.2): Provides regex natives for plugins
[11] Client Preferences (1.5.2): Saves client preference settings
[12] SQLite (1.5.2): SQLite Driver
[13] file "steamtools.ext.so": /home/server/tf2/s1/tf/addons/sourcemod/extensions/steamtools.ext.so: cannot open shared object file: No such file or directory
[14] file "curl.ext.so": /home/server/tf2/s1/tf/addons/sourcemod/extensions/curl.ext.so: cannot open shared object file: No such file or directory
[15] Socket (3.0.1): Socket extension for SourceMod
[16] MySQL-DBI (1.5.2): MySQL driver implementation for DBI

I ought to point out the steamtools and curl extensions are used by updater and I wasn't using the plugin until recently. According to the updater plugin page:

Your server must be running at least one of the following extensions:
cURL
Socket
SteamTools (0.8.1+)

And hence I am running only socket. I didn't want to add more variables to an already complex problem.

As for meta plugins:

-> meta list
Listing 6 plugins:
[01] SourceMod (1.5.2) by AlliedModders LLC
[02] SMRCon (1.2.1) by psychonic
[03] TF2Items (1.5.3) by Asherkin & AzuiSleet & Damizean
[04] TF2 Tools (1.5.2) by AlliedModders LLC
[05] SDK Hooks (1.5.2) by AlliedModders LLC
[06] SDK Tools (1.5.2) by AlliedModders LLC

I have even gone as far as run a few srcds profiles and recorded the time of the spikes in a hope to correlate them back to actual plugin calls.

Here is the profiler XML. The profile was flushed at 9: 39 pm and hence a new profile (the one I have linked was started).

At 9: 39:46 I had a small spike followed by a large one at 9:44:08 and I stopped the profiler at 9:44:48.

https://dl.dropboxusercontent.com/u/...1384465510.xml

I think the calls are recorded in order with the time taken, so I was thinking I could make a simple program to parse through the XML and tell me what happened a specified number of seconds from the start of the profile.

Now for the obvious. Rate settings:

// Frame rate limiter
fps_max 600 (irrelevant due to updates, the server fps is limited to the tickrate)

// Max bandwidth rate allowed on server, 0 == unlimited
sv_maxrate 60000 (I read that increasing the maxrate helps with choke, however depending on the map I still get high 50+ choke and I am sure it isn't my internet connection).

// Maximum updates per second that the server will allow
sv_maxupdaterate 100 (I was running my server with 100 tick at the time, I have changed this before to reflect the -tickrate setting at run time) Either way, when running the server at the default 66, this was set to 66 as well.

Last by not least server specs:
Processor: AMD Opteron 3250
Ram: 8GiB
OS: Debian 7

I'm sorry for the wall of text, however I am fed up and felt like I should give as much information as I can think of from the offset.

Any help would be most appreciated!

[Dr_Knuckles]

What kind of connection you on?

[Impact123]

This looks like a clientside problem to me, but i might be a little bit rusty and also haven't read the whole message.
I can see that your fps drops below 66 fps in one of the pictures, note than you can't receive more packages than you have fps.
Maybe it helps if you post your clients netsettings too.

[Bacardi]

Originally Posted by Dr_Knuckles What kind of connection you on?

This and server maxplayers.

Join your server and type these in your own game console

clear

sv_minrate;sv_maxrate;sv_minupdaterate;sv_maxupdaterate;sv_mincmdrate;sv_maxcmdrate

sv_client_cmdrate_difference;sv_client_min_interp_ratio;sv_client_max_interp_ratio

rate;cl_updterate;cl_cmdrate;cl_interp_ratio

status

[Astridax]

Dr Knuckles and Impact123
I'm on a 20Mb/s down 1-2Mb/s up fibre connection.

My machine was more than powerful enough to run TF2 in years previous. However with all the recent updates, I probably should lower some things like AA.

An i7 920 with GTX 275 gpu and 6GB DDR3 ram.

Bacardi and others

To start off, the server is a 24 slot server with one slot reserved (an according to the console, one implicit slot for replay). It is started with -maxplayers 25 and -nohltv

For client settings

"rate" = "30000" (I actually thought rate was deprecated, but that may be server side only, I should probably raise it.)
- Max bytes/sec the host can receive data

"cl_updaterate" = "100" ( def. "20" )
- Number of packets per second of updates you are requesting from the server

"cl_cmdrate" = "100" ( def. "30" ) min. 10.000000 max. 100.000000 (I forgot to change this when switching to using tickrate of 100)
- Max number of command packets sent to server per second

"cl_interp_ratio" = "2" ( def. "2.0" )
client notconnected
- Sets the interpolation amount (final amount is cl_interp_ratio / cl_updaterate).

Server settings


sv_minrate;sv_maxrate;sv_minupdaterate;sv_max updaterate;sv_mincmdrate;sv_maxcmdrate
"sv_minrate" = "3500"
replicated

- Min bandwidth rate allowed on server, 0 == unlimited
"sv_maxrate" = "60000" ( def. "0" )
replicated

- Max bandwidth rate allowed on server, 0 == unlimited
"sv_minupdaterate" = "10"
replicated

- Minimum updates per second that the server will allow
"sv_maxupdaterate" = "100" ( def. "66" )
replicated

- Maximum updates per second that the server will allow
"sv_mincmdrate" = "10"
replicated

- This sets the minimum value for cl_cmdrate. 0 == unlimited.
"sv_maxcmdrate" = "66" (I again forgot to change this for 100 tick, however even at 66 tick this problem happened.)
replicated
- (If sv_mincmdrate is > 0), this sets the maximum value for cl_cmdrate. (Might explain why out packets are never 100/s)

sv_client_cmdrate_difference;sv_client_min_in terp_ratio;sv_client_max_interp_ratio
"sv_client_cmdrate_difference" = "20" - Never heard of this variable before today.
replicated
- cl_cmdrate is moved to within sv_client_cmdrate_difference units of cl_updaterate before it is clamped between sv_mincmdrate and sv_maxcmdrate.

"sv_client_min_interp_ratio" = "1" - At a point a few years ago I used to set this to -1.
replicated

- This can be used to limit the value of cl_interp_ratio for connected clients (only while they are connected).
-1 = let clients set cl_interp_ratio to anything

any other value = set minimum value for cl_interp_ratio

"sv_client_max_interp_ratio" = "5"
replicated

- This can be used to limit the value of cl_interp_ratio for connected clients (only while they are connected). If sv_client_min_interp_ratio is -1, then this cvar has no effect.

status (the server is empty at the moment)
hostname: [MLM] - ||24/7 KOTH|| - Much King, So Capture||
version : 1998941/24 1998941 secure
udp/ip : 89.16.161.202:27015 (public ip: 89.16.161.202)
account : logged in
map : koth_badlands at: 0 x, 0 y, 0 z
replay : recording
players : 2 (26 max)

# userid name uniqueid connected ping loss state adr
# 2 "Replay" BOT active
# 483 "[MLM-MP] - Astridax" STEAM_0:0:20335300 09:04 28 0 active **.**.**.222:27005

One thing I was wondering is if anyone knew what stringcmds or string tables were on the net_graph. Usually during spikes these sky rocket and I get a few gaps in the graph, however I've been unable to find any info.

You are more than welcome to add me on steam if you want to help out testing some of this, thanks in advance!

[Dr_Knuckles]

Upload is your problem, or youll need to adjust rates to match that slow speed, which is what I figured.

[Bacardi]

if you look net_graph in: value and peak numbers top of diagram, it tells how much action happen on your server.

Let's say if average value is 500 (depend how many players and are shooting), multiply it updaterate 66.
500*66 = 33000 B/s
What is 33 KB/s (Kilo Bytes per second) per player.

Your upload 2Mbps (Mega bits per second), what is in bytes:
2000 Kbps / 8 = 250 KB/s

Now if you share your upload bandwidth 250 KB/s by 33 KB/s = 7 players max.

-----------------------------------------

If set updaterate max to 20 (what is lame )
500*20 = 10000 B/s ~ 10 KB/s
250 KB/s / 10 KB/s = 25 players max

[Astridax]

Oh guys, you get me wrong here. I'm not hosting this on my machine! I'm on a dedicated box with a 100Mb/s symmetric link, so both up and download speeds. I've had lag issues before and previously I was on a 1Gb/s symmetric link with a different host.

It is my home internet connection which is 20:2! The one I use for playing .

Here is a real world speed test:

Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Bytemark Computer Consulting Ltd (89.16.161.202)...
Selecting best server based on ping...
Hosted by Vorboss Limited (London) [2.52 km]: 21.617 ms
Testing download speed........................................
Download: 75.50 Mbit/s
Testing upload speed........................................ ..........
Upload: 90.73 Mbit/s
Share results: http://www.speedtest.net/result/3103891730.png



I ought to point out that I'm not the only one to have those double extrapolation peaks. I've had friends on monitoring net_graph and they have lagged right at the point in time I did, with corresponding peaks.

[Bacardi]

Then try

sv_minrate 50000
sv_maxrate 0
sv_minupdaterate 33
sv_maxupdaterate 66
net_splitpacket_maxrate 30000


But if player's FPS drop, then they updaterate drop etc. etc. Can't do much about it.

[Astridax]

Should I stick this back on to 66 tick. Ideally I'd run it at 33 tick since it is only a pub server. Especially if it reduces my resource usage.

I'll try and report back tomorrow when the server is full again. I did wonder if I should increase minrate, however couldn't it make it harder for people on slower connections to play. Then again I suppose 50,000 B/s is only 400Kb/s upload for each client. Most have above that right ?

Another thread mentioned net_splitpacket_maxrate, which I think is set at the default 15K atm, however I couldn't find much about what it is meant to do.