[wiigor]

Introduction

A lot of times the available network and cpu resources for a gameserver are limited. Demands for these resources change continously when operating a server.
They change not only when the number of players change, also every player may have other connection properties such as an different client sides rate settings.

It may be challenging to manually find optimal settings when the resources are limited. Also in practise rates must be set to accomodate for the maximum peak usage rate of the server.
Or the maximum number of players must be set very conservative .

The conclusion is When server resources are limited static rates are almost never the most optimal

Manual vs Automatic

Therefore it would be handy to have a plugin that once installed will manage the server load automatically.
A real world analogy would beto having a having a car with automatic transmission, where you can change
gears as compared to a car with no transmission or a manual transmission that is only used once a day or
so when a admin notices the speeds are not optimal.

Examples of such resources in the server world are are:
*the server machine's processing power.
*the networkconnection upstream and/or downstream bandwidth
*a monthly data limit.

Who can benefit from such a plugin?

*everyone who wants their player to support more players with lower resouces per player
without ever having lag because of overloading their resources.

Examples are

• people who want to host a server at home for friends, which usually have very limited resources, a lot of times dont set the rates ok and the users that join lag. This plugin will just remove the headache of server rate settings and set it optimally both for the server owner as for the players.

• Another example is a game server provider. Which may give an option to clients to have a more player server with dynamic rates at a lower price, or also themselves making a profit because they can run more software servers on one machine.
• HIGH level wars cannot use it should be played on a non resource limited server, but thats only about 5% of the usage of al SRCDS servers.
• The other 95% of the non HIGH war servers could benefit from such a plugin

Purpose of this topic

This topic is for discussion about such a plugin. I will be making it but i are open for others to contribute. The more contributions there are the better the plugin will be. Cause a lot of people know more than one. Thats why im posting this here.

First, what im posting here is not fixed im open for suggestions or corrections.I have set the following priorities for what the plugin should do:

plugin priorities

1. Prevent server lag. The most important priority is to make the probability of server lag because of load exceeding the available resources very small.
2. Use the server resources as much as possible. If the server is half full use that cpu and network more instead of sticking to lower settings.
3. Fairness, by this its meaned that every player should be able to choose there own rates from between a global minimum and maximum ie: no client side variables must ever be changed. Nor should players be treated differently. They all get the same minimum and maximum rate settings.
4. Reduce the amount of setting updates to a minimum. players(and the automatic lag compensation) may adapt to a specific rate setting, changing it too much will deteriorate player performance therefore rate changes should be reduced to a certain value(which may be changed by a cvar if really wanted although the default value should suit everyone)

Lets get started:

The plugin will have to do 3 tasks:

1 measure a certain variable possible multiple recording times to get a series for the second step to do some basic statistics
2 decide, if, when and to what value to change the variable.
3 change the variable.


The tasks may look simple in theory but in practise there are many possible (wrong, good and better) ways to do them.
I want to find ways that are close to optimum, and im really open to suggestions and hints/help to find these optimal ways.

I will discuss possible options


1.1 Measurements,

What should the interval be between the measurements?

If overload is about to occur you want to be able to change the settings downward fast. But you need to measure the upcoming overload first.
But what is a measurespeed thats high enough, why not just set it to 0.1 to be sure. What is the performance impact of a measurement every
0.1 second. But why measure so often if you only want to change the variables as little as possible.

For now im sticking to a 1 second default measurement rate, as i have tested it and the resource usage seems to change slow enough to
use this rate. but i want to make the measurement rate a cvar so people can choose how often a certain load variable is measured.
but also i want to provide a good set default value (which atm i think is 1 second)

How many measurements should be made before making a decision?

Decisions made based upon a series of measurements may have a higher probability to be better than a decision based on one measurement.
for example as the usage% of a resource drops and increases using a single measurement may set the load too high.


example cpu usage:

measurements:

40% 45% 65% 50% 55% [45%] > decision?<

maximum = 65%
average = 50%
minimum = 45%
last = 55%

setting the value to the average minimum or last is obviously wrong as there are higher peaks than those values which will cause lag.

therefore i will measure at the set measure interval rate and only remember the maximum value of the current serie of measurements.
when taking a decision the maximum value is used and the maxium is reset. then measuring continues yielding a new maximum for the next decision.


How to measure bandwidth usage?
There is the handy function below in the API. It gives the current upload and download speed in bytes/s the srcds server is using:

GetServerNetStats(&Float:in, &Floatut);So getting the bandwidth usage is no problem.



How to measure CPU usage?

For CPU usage there is not an function in the api that i know of to get it.

• A task someone willing to contribute might do to really help the creation of this plugin is to find a neat way to get this value.

i know the rcon stats console command prints the cpu usage and i know that that value is probably correct for use in this plugin.
But i know no other way of getting it other than dumping it to a file and extracting it with the file read options in the api.
there should be a better and more elegant way. If you know please let us know.


2 Decisions.

Should a setting be changed based on the current measurements? To which value should it be changed and why? And when should the change take place. This is what concerns the decision part of the plugin.

Every resource may have its own decision code. because every resource may need a different decision strategy.

I really dont have a decision strategy yet but im working on it for updaterate and cmdrate:

possible choices are to only check if a setting is to low or high and just add or substract a constant value to it untill its ok. but this needs many updates. and with a small constant value the plugin will be setting a value higher and lower al the time. a too big constant will change the rate too much / too fast causing lag that can be noticed by the players

therefore its important that the decision is really good so the amount of actual changes in settings is minimal and the changes themselves are more optimal.

its also possible to work with a factor of the current rates instead of a constant value. For example everytime the load is too high multiply the setting by 0.9 (or something). But why 0.9 why not another value?

The best would be if there is a function which takes the old setting the current load and the desired load and calculated the setting that is needed to get at the desired load. For the update and commandrate / cpu this is possible i think. Im working on it. But i would appreciate help and suggestions.

custom cvars also play a role in the decisions:

These are the values i invented.9with the suggested default value)
(at = plugin abbreviation names could change im not sure of the name yet)

at_enabled = 1 //enables or disables the plugin
at_maxcpu = 90 // cpu value at which the server must immediatly lower the settings to return to a cpu value below it.
at_maxin = 512000 // maximum incoming network bytes per second value the server may use.
at_maxout = 64000 // maximum outgoing network bytes per second value the server may use.
(this setting will support 32 players without lag(hitting will be more difficult but there will be no yerkyness/hiccups prediction works very well.)
i have done extensive research and found this:

1Knowledge about rate variables:



1.1Updaterate

1.1.1sv_minupdaterate

• setting it to 0 will set not minimum updaterate the client can choose and have no effect on sv_maxupdaterate
• setting it to 1 or higher will limit the client to use at least this updaterate.
• Setting sv_minupdaterate higher than sv_maxupdaterate will cause strange behaviour as sv_maxupdaterate will not reflect the set sv_maxupdaterate value.

1.1.2sv_maxupdaterate

The following settings were set with a non-restrictive sv_minupdaterate:
·setting 0 gives the maximum server updaterate(only limited by the server tickrate)
·setting 1 gives hiccups every second as the client exceeds the default maximum prediction time, stops the objects on the screen from moving, and waits for the next server update.
·Setting 2 will not give hiccups, but cannot be set when sv_maxupdaterate was previously set to 0 If it was set to another value it will change to 2: First set it to another value then to 2.
·2-9 when shooting a barrel the barrel moving is delayed because the next update takes some time to arrive. Physics are thus a bit off. In Player-player interaction this fact is hidden by using prediction. Its gives no disturbing hiccups in player- player contact.
·Setting it to 10 or higher also gives acceptable physics for objects like barrels



1.2Commandrate

Command rate can be dynamically changed to all values without apparent hiccups and regardless of the current updaterate values. Changing sv_maxcmdrate is straightforward

1.2.1sv_mincmdrate

• setting 0 will totally disable enforcement of sv_mincmdrate and sv_maxcmdrate

the client is able to choose any value.

• Setting 1 or higher enables enforcement of sv_mincmdrate and sv_maxcmdrate

The client _cl_cmdrate value will be clamped between sv_mincmdrate and sv_maxcmdrate

1.2.2sv_maxcmdrate

• setting it to 0 causes the clients display to literally freeze regardless of sv_mincmdrate setting. The client will also time out after the time out period and drop from the server.

Because the server thinks the client has crashed as the client sends no commands. Also
The clients HL2.exe may crash with this setting. Also accessing the server by rcon is sometimes impossible. (as it will accept no incoming commands)

• setting it to 1 will not give the problems mentioned with the 0 setting. But it will cause severe hiccups in which the player command is revoked and the players is teleported back to a previous point. The player is almost unable to move.`
• setting it to 2 – 9 gives hiccups in reducing severity s rate increases, the predicted physics do not match with the physics using default rate settings. Every second there is a hiccup. These effects get smaller as the sv_maxcmdrate increases.
• Setting it to 10 or higher is acceptable

A minimum setting of 5 for updaterate and 10 for commandrate is acceptable for player-player contacts and shows no yerkyness.



3 Change

Changing variables seems straightforward. But it isnt :p
For example its not possible to change the actual updaterate from any value below 66 to 100. The svmaxupdaterate will change but the actual meassured values at net_graph 2 wont change!
This is only an example. But it is much worse:

the actual maxupdaterate will change from any value between 1 and 14 to any value between 1 and 14 directly
the actual maxupdaterate will NOT change from 14 to 15 or 15 to 14

BUT it will change from 14 to 16 or 16 to 14.

the actual maxupdaterate will change from any value between 16 and 22 to any value between 16 and 22 directly
as long as the difference between the old and the new value is at least 2...

this is a part of the equation :

rate minimum decrease to change | actual value
2 1
16 2
23 3
36 6
40 7
50 10
67 17
80 23
87 27
90 28
100 34
i plotted this in a graph and concluded there is a certain formula that valve must have used.

i came at minimum difference = (current value/16)² but it doesnt exactly match the pattern i found. Im afraid it has to do with integer division or something. Im really not able to deduce the formula valve has used for this if i would have it i could set the values to the next possible value.

i have allready tried for example when going from sv_maxupdaterate 100 to sv_maxupdaterate 90 to first issue a maxupdaterate 10 (or so) . if done immediatly it has no effect and the actual value stays 100.
if done with a delay of at least 0.5 second or 50 server ticks? then
it will change from 100 to 10 then to 90 and it stays at 90 but this method really cause a displacement of all players and a severe hiccup so its not the way to go.

If i would have the formula i would only need to set the sv_maxupdaterate to 65 then to 100 which would be a bit better. But without the formula i cannot do this. Cause there are a lot of possible rate changes so for example when i want to change rate from 49 to 53 will this work? the formula would tell the minimum difference needed. But without it i have to guess or try al values by hand and put them in a table or something. Which is not efficient.

• So if someone can deduce the formula used by valve for setting the sv_maxupdaterate it would be really great help.

OF course if someone knows a way to circumvent this quadratic limitation on changing sv_maxupdaterate it would be better. or an alternative way to change it. I just want to be able to change maxupdaterate to 75 when its 90 or from 50 to 45 etc without it dont changing.


summary

This plugin once installed will prevent lag, use resources more optimally depending on the load and make it a lote easier to manage the resources.
There are still some important design decisions to be made as well as some practical problems.

Every contribution will be taken in consideration when making the plugin.

• questions
• suggestions
• functions or other code
• equations
• suggested cvar list
• tips

regarding the plugin related cvars and resource management can all be posted in this topic

After the most important decisions are made i will post the an alpha version of the plugin in this post. then it will be a joint effort to adapt the code untill it is truly usable.

[wiigor]

I have been thinking of ways to reduce the amount of adjustments to the variables this plugin will manage. And one way is to distinguish between two cases:

• immediatly needed changes when preventing overload of a certain resource is needed.
• not so fast needed changes only to bring the variable from a less optimum to a more optimum target range.

If any measurement gives or exceeds 100% of the set value by the at_max... convar (where ... can be any resource like cpu upstream downstream) then new parameters should be calculated and set immediatly to reduce the load.

If the value is not close to critical value but also not optimal, the change can be postponed to one change in so many seconds, or when no player- player contacts occur such as when a new round begins.

This system will give the ensurance of no lag due to resource overload, a minimum amount of rate changes to achieve this. and also a convergence to an optimum value.

Below this is lined out in an image. Of course the 100% 90% 80% and 70% are an example (although i think they will make good default values) but these will be able to be changed if desired, automatically, by cvars or by console command.

[bl4nk]

AFAIK the rate commands have nothing to do with CPU usage. They're meant for controlling the amount of bandwidth the server uses. How you could control the CPU usage is beyond me at this moment.

[wiigor]

This is a good question, which i will try to answer.

Originally Posted by bl4nk AFAIK the rate commands have nothing to do with CPU usage. They're meant for controlling the amount of bandwidth the server uses. How you could control the CPU usage is beyond me at this moment.

answer:

The cpu load can be regulated by the sv_maxcmdrate and the sv_maxupdaterate in a very powerful, direct and easy way because of the linear relationship between network rate and cpu load.

During a server tick the following happens:

• the information packets received from the players is processed.
• Their commands (position, movement, menu choices, shots etc)are evaluated and validated. decisions are made (such as hit or not?) conflicting updates are resolved by the server algorithms.
• Then the world model in the server is updated to reflect the changes.
• Updates are sent at the clients when needed but every packet send to any client is custom made for that client, so it gets only the other player positions it can see (to reduce update size and prevent hacks)

tickrate is just the rate at which the server initialises such a round.
Packets can come it at any moment. not all player packets come in during the same tick, because of different network delays. also not all players may choose to send a packet every tick.

The server may well check 100 times a second for new packets but by default clients only send 30 command packets to the server every second. this is no problem at all it just doesnt evaluate new actions of that player untill it gets a packet.

• What happens is if the server receives less packets a second it will have to do proportionaly less work per second. if you only get half of the packets you normally get you only have to do half of the work.

The same accounts for the updates. If you only send half of the updates per second you only use half of the processing resources needed for that job.

A server that doesn't receive packets and sends updates will do no work only than being in a loop and sleeping

---------------example-----------

very simplified example:

while(true) {
tick++
if(packetsToProcesses()) { <- nothing? to do hmm lets sleep.
processPackets() <- less packets is less work and more packets is more work. no packets to send or receive is no work!
}
sleep (0.01)
}

• a tick doesnt cost any effort in itself. its the packet stream receiving processing and sending that does. -> maximum packet rates are controlled by sv_maxupdaterate and sv_maxcmdrate

less packets = less processing = less cpu usage. halving the rates means halve the cpu need.



The amount of work that needs to be done is (almost)linearly dependant on the amount of data that is to be received and sent.

example:

sv_maxupdaterate 10 and sv_maxcmdrate 10.

will use approximately 10 times less cpu

than

sv_maxupdaterate 100 and sv_maxcmdrate 100.

When you set them close to zero 2 (not lower because when lower other effects occur)

sv_maxupdaterate 2 and sv_maxcmdrate 2

the server will show almost 0 cpu usage.


I have tested this in practice. feel free to do so too.


-----------------------------------

This means tickrate can always be set at the maximum recommended commandline value no matter how slow the cpu is. by maximum i mean that the engine doesnt give strange effects. 66 or 100 depending on the mod.

then the load can be regulated by the sv_maxcmdrate and the sv_maxupdaterate in a very direct and easy way because of the linear relationship between network rate and cpu load.

But we still need to have an elegant way to get the cpu load value into the plugin as there is no function in the API to get it. the rcon stats value seems to be correct,
maybe someone knows or is willing to elaborate a good way to get it into the plugin(or another solution to get it), that would be a very great contribution to this project.

[BAILOPAN]

Huh?

The server processes all packets every frame no matter what. The server also ticks whether there are packets or not. The cvars you mention control how often the server dispatches datagrams back to clients.

Perhaps what you're experiencing is making the game less playable for clients, so they don't send as much back to the server as they normally would. "The less I tell you about, the less you will give back to me." I wouldn't consider that a reliable or linear relationship though. It could affect CPU usage indirectly, since the server will have less returning datagrams to process.

But clients, theoretically, can send whatever they want as often as they want. It's the tick rate that dominates everything.

A more solid relationship is the number of players in the server. The more players, the higher load you get. If the logic is, "worsen the playability when X players are in game," well then you shouldn't have X players in game.

(YMMV blah blah)

[wiigor]

The server sets the send rate of the clients by protocol.

The server instructs the client to use a certain maximum send rate of datagrams.

The sv_mincmdrate and sv_maxcmmndrate are boundaries which the client should follow, that is the protocol which the server and client netcode follow

1. The client sets cl_cmdrate a certain desired rate at which it wants to send commands to the server
2. The server will just anounce sv_mincmdrate and sv_maxcmdrate to the client.
3. The client try to send at least sv_mincmdrate command datagrams to the server and will not send more than sv_maxcmdrate
4. if the client doesnt obey the server rate limits, it gets disconnected.

Network usage is the most important factor in cpu server usage

Its true that the ticks continue at the set rate. But the amount of
work every tick does depends on the amount of datagrams received and
sent every tick. Lower send or receive rates mean less datagrams every
second decreasing cpu use.

Player number is not a good indicator for cpu and or network use

For the simple reason that every player can have other network rates set.

For example

Player 1 might want to send his commands 100 times a second
to the server and receive updates on the world state 100 times a second.

Player 2 might have the default settings of sending his commands 30 times a second and getting updates 20 times a second.

Player 3 might have custom rate settings that suit his computer and connection best like: cl_cmdrate 60 cl_updaterate 40

Player 4......


OF course the amount of players has a lot of impact on the load. But it also dependant on the amount of updates each player sends or receives every second, the size of the update.

average amount of updates per player x the average size of the update x actual number of players = bandwidth.

That is why the bandwidth is the best variable to reflect the cpu load.
There is also a relationship between bandwidth and cpu use that is not totally linear its still a bit exponential but as a rough estimate is almost linear.

example:

16 players using a maximum of commandrate 100 and updaterate 100 will use about

1600 datagrams upstream and 1600 datagrams downstream at the server.

Imagine the cpu is at 80% in this situation. And that the server bandwidth use is not the limit (for example 100mbit connection)

Asume 16 more players join. join the server. there will in the end be 32 players. The cpu load will logically increase as every player that joins not only increases the amount of datagrams that need to be sent and received also the size of the sent datagrams increases because there needs to be more info and more events because there are more players visible.

Then this is what the plugin should do.

1. measure cpu load and Check if its too high
2. If its too high the amount of processing should be lowered this can be done by reducing the amount of datagrams.
3. Given the current cpu load and desired cpu load predict the factor that should be multiplied with the current datagram settings to reduce the load to the desired cpu load.
4. This prediction can be done easily because the relationship between cpu and network use is almost linear on a small interval of the maximum cpu range.
5. if desired the predict function can also be made exponential(to reflect bandwidth use better when more players join) , but this is not needed because if the system detects the cpu is still to high it will do another prediction again lowering the cpu usage. The only penalty is more rate changes. But eventually the value converges to the target of cpu%. The use of the predict function is to minimize the rate changes by hitting the target value as close as possible (instead of lowering it by 5 or multiplaying by 0.95 untill its ok then increasing again etc.)

The server will now send and receive less updates probably about 35 for each player so the cpu load remains around 80%
The reason that it will not cut the updaterate in half because there are double the amount of players, is that it looks at the bandwidth that needs not
at the packet rate. Because more players need not only more datagrams but also larger datagrams and this is reflected in the bandwidth

Why server outgoing bandwidth used to be quadratic with number of players but now is almost linear

The reason that the outgoing bandwidth - player relationship used to be quadratic is that every player would get the same update from the server all player positions, velocities, accelerations and all events.

The outgoing bandwidth B at the server with number of players N was like this:

B = 1 + n * n-1 = 1 + n² - n



This caused besides outgoing bandwidth wasting also many cheats to be developed like radars where you could see all the enemies too, or wallhacks where you could see enemies through the wall.

The solutions were simple, only send to each player what it needs to see. Only information players see on their screen is sent to each player. This reduced the packet size a lot.

So the first equation changed to something approaching this

B = n^x

where
where x is a value greater than 1 but smaller than 2.
Im pretty confident is something like 1.1-1.2

But as i said it doesn't matter in practise on a small interval just assume it is linear, the predicted new values will be a bit off but who cares if cpu was 90% and the target value is 80% i dont care if sometimes the variable updating sets it at 85% and another time 75% .

Thats why there is a green target range if you look at the green part of image in posted the in the second post in this topic.

Any value between 90% and 70% wont cause more changes in the variables. A good prediction function gets the value as close to 80% as possible, to reduce the probability that more changes are needed. (its in the middle of the increase and reduce boundaries at 70% and 80%)
But even if its off it still doesnt cause more changes causes its still acceptable.
And even if its totally off, nothing bad will happen, then the function will be called again and it will get a second chance to converge more to the target value, at the expense of another network value change.(which really isnt such an expense).

[sawce]

[sawce]

update/cmdrate add nothing to CPU load; frames tick away regardless of whether or not there are packets that need to be processed. The only way to lower CPU load would be lowering framerate, which has an adverse affect on gameplay. update/cmd would only help with network throttling.

Now you have the problem of FINDING cpu load / network from within the plugin.

This just isn't practical.

[wiigor]

update/cmdrate do have to do with cpu load. Its easy to see,if you have a srcds server:
Use tickrate 100 and sv_maxupdaterate and sv_maxcmdrate 100.

Do this test with an arbitrary amount of players on the server:

1. Observe the actual cpu load.
2. Set the sv_maxupdaterate to 5 and sv_maxcmdrate to 10
3. Observe the actual cpu load

the new cpu load will be a lot lower! cause a lot of less work is done means a lot less cpu load. Just try it.

So updaterate and cmdrate do have a lot to do with the serverload. and can be used to change it a lot. Cutting them by half will give half the cpu amount (with the same amount of players)

Its true that these variables control the network load, but because its the network load that generates the cpu load, they also control the cpu load.
cpu load is dependant on network load. less network load = less cpu load. roughly like this 1/2 network load = 1/2 cpu load AND 1/10 network load = 1/10 cpu load.

finding network load from the plugin is easy there is already a function for the incoming and outgoing bandwidth in the api. the maximum in and out bytes/sec are filled in by cvar
I already know a way to get them automatically, set the available bandwidth cvar to INTEGER_MAX_VALUE (well however it is called here) then just wait untill lag occurs. lagg can be found by recording every second the median of the pings of all players of the last minute. If the instantaneous value of a ping is a certain factor larger than the average of all pings lag is detected and the current in and out bandwidth is noted. these values are used by the plugin.

example:

table with delays

second 1 second 2 .... second 60 second 60 + 1
player 1 5ms 17ms 8ms 5 ms <= lan player
player 2 104ms 138ms 160ms 295ms
player 3 50ms 600ms 80 ms 265ms
player 4 39ms 17ms 50ms 267ms
player 5 27ms 65ms 27ms 305ms



median: 50 ms 65 ms 80 ms [ 265 ms ] this median indicates lag because its a certain set factor (ie 2) greater than the average of the other medians

remember the average of the new median and the old average so only one variable avg median counter is needed.


This might well be a very bad method but if its bad its only because of the implementation details, a fact is that server lag can be detected and distinguished from random dispersed latency jitter of specific connected players. Server connection limit caused lag can be detected and the current in and out rates stored. making this proccess automatic



This will make the plugin in essence totally fire and forget.

• just put it in the plugins dir.
• The plugin will determine the maximum in and out bandwidth when lag occurs.
• If and only when lag has occured using the determined network maximum speeds to apply rate control preventing future lag by staying below these tested maximum network loads
• The plugin will give a close to maximum server resouce usage at all times, while making the probability of annoying lag very small
• No cvars need to be set at all to get good performance. without requiring the admin to set any network or cpu variables the plugin will hold the optimum for that situation with those resources automatically.

[L. Duke]

With all of the time you spent writing pages and pages of info on how a plugin could work, you could have learned PAWN and written your own plugin. Of course you would have found that the rate commands are not correlated with CPU usage.