[wonderous]

* EDIT 30th Aug 1400hrs: Added full result set *
This question pertains to both srcds and hlds, but I decided to post it on srcds since it's more relevant to srcds.

Introduction
Just this month I have just started trying to migrate all my Windows servers to Linux.

First thing I noticed was that the idle CPU usage on Linux (i.e. when the server is idle and empty, non-hibernating servers for srcds) is much higher than on Windows. I know the 'top' command is not a very reliable indicator for CPU usage, but at least was the thing got me investigating whether my theory was true.

Example of CPU usage of gameservers on a Linux VM (Ubuntu 16.04 x64)

Tasks: 158 total,   1 running, 157 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.5 us,  1.9 sy,  0.0 ni, 93.5 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem :  2047956 total,   244680 free,   713012 used,  1090264 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1109520 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6400 valve_ds  20   0  152196  72080  17452 S   8.0  3.5   0:03.43 hlds_linux
 6439 csgo_ds   20   0  354272 188088  24548 S   6.3  9.2   0:05.42 srcds_linux
 6404 cstrike+  20   0  134704  92652  16524 S   5.3  4.5   0:03.07 hlds_linux
 6426 czero_ds  20   0  134460  94296  16500 S   5.3  4.6   0:03.55 hlds_linux
 6418 left4de+  20   0  442356 219232  20980 S   4.7 10.7   0:04.16 srcds_linux

I'm aware the CPU time is a better measure than CPU usage for actual CPU load, but even then those idle CPU usage are pretty darn high.

Setup
I primarily work on Windows, and the Windows gameservers run directly on the Windows Host OS. Whereas, the Linux gameservers run in a Linux Virtualbox VM with 2GB on the same Windows Host OS.

Windows Host:
Baremetal instance with 16GB of RAM; i7 CPU
Windows 10 64bit
5 idle Windows gameserver instances

Linux VM on the Windows Host:
Virtualbox instance with 2GB of RAM; assigned all cores of i7 CPU
Ubuntu 16.04 x64
5 idle Linux gameserver instances

Linux VM on the Windows Host (control):
Virtualbox instance with 2GB of RAM; assigned all cores of i7 CPU
Ubuntu 16.04 x64
No gameserver instances

There were a total of 5 Gameservers: 1 czero (hlds), 1 cstrike (hlds), 1 valve (hlds), 1 left4dead2 (srcds), 1 csgo (srcds). The Linux and Windows gameserver setups were identical for each game instance: same files, same environment/cvars: For hlds, sys_ticrate was set to 512; for srcds, sm_cvar fps_max was set to 500. Both Linux and Windows servers were started with the same command line, except their binaries: For srcds, srcds.exe on Windows and srcds_linux on Linux; for hlds, hlds.exe on Windows and hlds_linux on Linux.

Results
Start Time: 2018-08-29 14:38:19
End Time: 2018-08-30 13:28:17
Elapsed: 82198 seconds

The gameservers were all started promptly at 2018-08-29 14:38:19. The end of the data collection occured at 2018-08-30 13:28:17. The total uptime of all the gameservers was 82198 seconds.

Figure 1: CPU Usage and CPU time of 1) 5 Windows idle gameservers processes, 2) Linux VM process running 5 idle gameservers, and 3) a clean Linux VM with no running gameservers (control), as seen on the Windows Host

ProcessName PID   Username Status  CPUTime(HH:mm:ss) StartTime           WorkingSet(Memory) Memory(private bytes) PeakWorkingSet(Memory) CommandLine
srcds       1820  csgo_ds  Running 00:42:21          2018-08-29 14:38:19 71,331 K           645,926 K             792,678 K              D:\ds\csgo_ds\srcds\srcds.exe -nobots -usercon -port 27115 -ip 10.0.0.1 -game csgo -steamport 28116 -condebug -console -hltv -tickrate 128 -secure +game_mode 1 +tv_enable 1 +tv_port 29116 +sv_pure 0 +game_type 0 +sv_lan 0
srcds       15448 left4de+ Running 00:19:37          2018-08-29 14:38:24 59,219 K           409,677 K             346,042 K              D:\ds\left4dead2_ds\srcds\srcds.exe -port 27015 -secure -steamport 28016 -ip 10.0.0.1 -condebug -console -game left4dead2 +map "c12m1_hilltop coop" +sv_lan 0
hlds        20736 valve_ds Running 00:03:40          2018-08-29 14:38:13 24,543 K           106,676 K             78,635 K               D:\ds\valve_ds\hlds\hlds.exe -secure -game valve -noipx -console -condebug +port 27915 +ip 10.0.0.1 +sys_ticrate 512 +maxplayers 32 +map boot_camp +sv_lan 0
hlds        11728 czero_ds Running 00:01:06          2018-08-29 14:38:07 24,932 K           107,511 K             106,835 K              D:\ds\czero_ds\hlds\hlds.exe -secure -game czero -noipx -console -condebug +sv_lan 0 +maxplayers 32 +ip 10.0.0.1 +map fy_iceworld_cz +port 27615
hlds        10616 cstrike+ Running 00:01:01          2018-08-29 14:38:02 24,260 K           104,185 K             106,045 K              D:\ds\cstrike_ds\hlds\hlds.exe -secure -game cstrike -noipx -console -condebug +sv_lan 0 +maxplayers 32 +ip 10.0.0.1 +map fy_iceworld_classic +port 27815
VirtualBox  7076  vbox_us+ Running 06:31:08          2018-08-29 14:37:51 98,660 K           106,881 K             140,349 K              60eaff78-4bdd-042d-2e72-669728efd737-suplib-3rdchild --comment ubuntuclean --startvm eb147245-03c3-482e-b94e-1dc1ad8b3481 --no-startvm-errormsgbox "--sup-hardening-log=C:\Users\vbox_user\VirtualBox VMs\ubuntuclean\Logs\VBoxHardening.log"
VirtualBox  18696 vbox_us+ Running 00:00:48          2018-08-29 14:37:49 113,782 K          108,847 K             114,663 K              60eaff78-4bdd-042d-2e72-669728efd737-suplib-3rdchild --comment "ubuntuclean Clone" --startvm 6ee567b4-44a8-48d9-ad67-d27266b51d84 --no-startvm-errormsgbox "--sup-hardening-log=C:\Users\vbox_user\VirtualBox VMs\ubuntuclean Clone\Logs\VBoxHardening.log"

Figure 2: CPU Usage and CPU time of the 5 idle running gameservers in the Linux VM, seen with the 'top' utility

top - 13:28:04 up 22:50,  1 user,  load average: 2.04, 1.63, 1.51
Tasks: 144 total,   1 running, 143 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.5 us,  2.5 sy,  0.0 ni, 89.4 id,  0.0 wa,  0.0 hi,  0.6 si,  0.0 st
KiB Mem :  2047956 total,    71904 free,   674912 used,  1301140 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1179428 avail Mem
  scroll coordinates: y = 1/144 (tasks), x = 1/12 (fields)
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1894 valve_ds  20   0  154660  71428  18400 S  11.0  3.5 151:05.77 hlds_linux
 1889 csgo_ds   20   0  354656 189516  25628 S   9.8  9.3 126:00.79 srcds_linux
 1887 cstrike+  20   0  149900  99104  17904 S   8.5  4.8 114:54.62 hlds_linux
 1884 czero_ds  20   0  158948 100660  17672 S   7.3  4.9 114:27.82 hlds_linux
 2038 left4de+  20   0  459912 257304  21004 S   6.1 12.6  90:55.09 srcds_linux

Figure 3: Comparison between the Total CPU time and the CPU time per minute of idle Windows gameserver processes and their Linux counterparts

Total CPU time of Windows gameservers: 2541 + 1177 + 220 + 66 + 61 = 4065 seconds
CPU time per minute of Windows gameservers: (4065 / 82198) * 60 = 2.967226 ≈ 3 CPU seconds per minute

Total CPU time of Linux gameservers: 23468 - 48 = 23516 seconds
CPU time per minute of Linux gameservers: (23516 / 82198) * 60 = 23516 ≈ 18 CPU seconds per minute

Ratio of Total CPU Time of Linux idle gameservers to Windows idle gameservers: 23516 / 4065 = 5.784994 ≈ 6

The total CPU time of Windows idle gameservers is the sum of the CPU times of all 5 gameserver processes running on the Windows Host (PIDs 1820, 15448, 20736, 11728, 10616), which was 4065 CPU seconds.
In contrast, the total CPU time of Linux idle gameservers is the difference between a loaded Linux VM process with the servers running (PID 7076 on the Windows Host) and a fully-idle Linux VM process with no servers running (PID 18696 on the Windows Host), which was 23516 CPU seconds.

The results reveal that idle Linux gameserver instances use approximately 6 times more CPU time than their Windows counterparts.

Discussion
The results demonstrate that idle CPU usage of Linux gameserver instances is higher than the Windows gameserver instances.

There is reason to wonder why idle Windows gameservers run with less CPU usage than the same gameservers on Linux. A possible reason Windows gameservers run with so little CPU usage is because of it's well-known tendency to run processes at a lower tickrate when the hardware acceleration is not utilized. At least, from what I know about hlds, there used to be a Booster MM Plugin specifically made for Windows hlds, to raise the tickrate to the value of the cvar 'sys_ticrate' by tricking Windows to turn on hardware acceleration; and before that plugin was released I remember some people (on this forum I think) resorted to opening Windows Media Player in the background just to force hardware acceleration on Windows. While latest builds of hlds seem to have addressed these issues on Windows - now tickrates can go up to 900 on an i7 CPU on Windows, as seen in 'stats' command - could it still be possible that the Windows hlds 'stats' command shows a much high fps than that the process is running at?

To answer the question requires measuring the true fps of the server, which can be done by measuring ping response times, assuming there is no network latency. From my testing, pings to the 3 hlds servers consistently hovered between 1 and 3ms, for hlds instances whose fps hovered between 400-700 shown by the 'stats' command, revealing that the hlds instances were accurately reporting the processes' fps. Ping tests were also performed on the 2 srcds instances: for left4dead2 which has a limit of 30 fps, the ping hovered between 1 to 38ms, as expected; for csgo which has a limit of 128 fps, the ping hovered between 1 and 8ms, again as expected. For hlds, servers appear to target the 'sys_ticrate' cvar (which was set to 500); for srcds, servers appear to target the 'max_fps' cvar (which was set to 500) while also being limited by the game's engine (for csgo, 128 ticks; for left4dead2, 30 ticks). Hence, it may be said that the 'stats' command is produces an accurate fps of a server instance.

Knowing that the 'stats' command is in fact an accurate measure of the true fps of a running gameserver instance, the only thing left do is to compare the fps of Windows gameservers as shown by the 'stats' command to their Linux counterparts. Testing reveals that there is no difference between the their fps.

Hence, the differences between OS instances is not attributable to fps.

It should be noted that the exceptioanlly high CPU usage of idle Linux gameserver processes in the 'top' command on Linux machines appears to be unrelated to the nature of their setup (e.g. in this study, a Linux VM running on top of a Windows Host.). Idle Linux gameserver processes consistently showed high CPU usage and CPU time in various other Linux setups, whether baremetal and virtualized:
1. On a baremetal Linux (Ubuntu 16.04 x64) box.
2. Cloud Linux VPS (Ubuntu 16.04 x64) which I would consider to be bare metal.
3. Linux Hyper-V VM (Ubuntu 16.04 x64), on top of a Windows Host.

Also, on a side note, it appears that in output of the 'top' command in a Linux VM, CPU usage (%) and CPU time fail to accurately reflect that on the Host OS. For instance, the sum of the CPU usage (%) in above in Figure 2 is: 42.7% (11.0% + 9.8% + 8.5% + 7.3% + 6.1%) inside the Linux VM, whereas, the CPU usage on the Windows Host hovered between 1-7%, revealing an error rate of about +610%. Also, from the same figure, the CPU time of the idle gameserver instances was (151 + 126 + 114 + 114 + 90) * 60 + (5 + 0 + 54 + 27 + 55) = 35841 CPU seconds, and after controlling for the non-gameserver VM processes, would be 35841 - 48 = 35793, whereas the CPU time of the VM process as reported on the Windows Host was 6 * 3600 + 31 * 60 + 8 = 23468, with the error rate of +53%.

Conclusion
The idle CPU usage of Linux gameserver instances is higher than the Windows gameserver instances, by about 6 times. These differences between idle OS instances is not due to a difference in their running fps.

Is there anyone out there who's an expert in hosting hlds/srcds on Linux that realized the same thing?
I want the servers to be as performant as possible, and possibly be able to run with the least hardware resources possible.

[wonderous]

EDIT: Removed this, no longer relevant

[Neuro Toxin]

Could this be the VM playing funny burgers compared to a bare metal box running the same OS with a VM?

[wonderous]

Originally Posted by Neuro Toxin Could this be the VM playing funny burgers compared to a bare metal box running the same OS with a VM?

Nope, and pretty certain of that. It appears to stay high regardless of whether the OS is baremetal or virtualized.

Even Linux gameservers show the same trend even on Cloud VPS (tried Azure, Digitalocean) - I dont know what the datacenter setups are like, but I would consider them bare metal.

Showed the same on a baremetal Linux (Ubuntu 16.04) box.

And they show the same trend also on a Hyper-V setup.

So it appears it's not a virtualization issue.

[Maxximou5]

Unless you're comparing results from the same exact baremetal instance, not sure what you're expecting.

I don't see how a VM with unidentified specs can compare to a baremetal instance with only a known processor.

[wonderous]

Originally Posted by Maxximou5 Unless you're comparing results from the same exact baremetal instance, not sure what you're expecting. I don't see how a VM with unidentified specs can compare to a baremetal instance with only a known processor.

I'm not very sure what you mean here. I think i should have been clearer in my original post. In my initial experiment (original post), I used the very same machine to run the Windows gameservers instances on baremetal, and the VM (with the Linux gameserver instances) also on the baremetal - in terms of specs, the VM had identical specs to the Host OS, except it was assigned 2GB of RAM, and was given all CPU cores of the i7 CPU.

I've added more clarity to my original post.

[Maxximou5]

Originally Posted by wonderous I'm not very sure what you mean here. I think i should have been clearer in my original post. In my initial experiment (original post), I used the very same machine to run the Windows gameservers instances on baremetal, and the VM (with the Linux gameserver instances) also on the baremetal - in terms of specs, the VM had identical specs to the Host OS, except it was assigned 2GB of RAM, and was given all CPU cores of the i7 CPU.

Thank you for clarifying, this makes a lot more sense to me now. As before, I understood it as you were comparing an unknown VPS machine to a baremetal Windows server, which would have very little in terms of being comparable.

If you're the host of the VM and you've set it up correctly, I can't see why VM would cause any slowdown. You're not sharing resources with a neighbor, unless you specified too, at which point would cause interference and issues.

Do you have any statistics or evidence of the idle usage on the Windows machine?

Also curious, why would use Windows 10 and not a Windows Server OS?

[wonderous]

Originally Posted by Maxximou5 Thank you for clarifying, this makes a lot more sense to me now. As before, I understood it as you were comparing an unknown VPS machine to a baremetal Windows server, which would have very little in terms of being comparable.

Glad that my post is understandable.

Originally Posted by Maxximou5 Do you have any statistics or evidence of the idle usage on the Windows machine?

EDIT: I added the data set in my original post.

Originally Posted by Maxximou5 Also curious, why would use Windows 10 and not a Windows Server OS?

I've been running servers on Windows 8 / 10 64bit well for quite a while with no issues. I have a Windows Home Server license, but I dont like the clutter and clunk of the extra server processes. This is unlike Windows Server 2012/2016 Datacenter that allows you to install only the features you want; it installs a lot of extra stuff that take away performance.
And Windows Datacenter for gameservers? That is overkill and way too expensive for simple gameservers.
Also with license of Window Server, you run into licensing issues with a lot of Windows freeware, which prohibits you from installing the software entirely (you can't even run the .exe or .msi installers), which means missing out on a lot of good tools.
My experience with hosting hlds / srcds on Windows has been nothing but seeing excellent performance.

[insanedude]

I've been having similar issues, as the ones you've described above.
I tested with SRCDS (CS:GO) and HLDS (CS 1.6) on a Debian 9.11 using the default kernel.
While idle, the process uses between 30% and 35% per core.

Messing around with the kernel (changing it's timer frequency to 100 or 1000, changing it's default scheduler, it's preemtation model etc, applying RT patches) did have some effect.
I noticed, that with changing the frequency to 100HZ and No Forced Preemtion (server), the CPU usage while idle drops down from 30-30% to 12-15%... However, there are 'random' spikes where it jumps up (according to htop). I do not know if this is the OS not properly reporting.. or if it could be due to the VM sharing resources with other VMs on the same host (although the host is not heavily loaded).
I've also tried Centos, but idle CPU is similar.

I have a couple of bare metal boxes with old Debian (7 & on them, without modified kernels and HLDS has ~12% stable idle CPU usage.. which is bizarre.. as they're with much older CPU's and much older OS's.
Similar servers with 1 VM on them on ESXi 6.5 & 6.7 has idle CPU usage much higher.

I haven't tried installing a newer OS on those boxes, as the servers are kinda 'live' and if i have to install Debian 7 or 8 back onto them.. it'll be a pain.