[MegaBrutal]

I know there are many threads about Fast Download already, but none of them gave me answers. I want to enable FastDL for my server, but it simply doesn't work. At all.

I have the following CVARs set:

sv_allowdownload 1
sv_allowupload 1
sv_downloadurl "http://coralie.megabrutal.com/valve/"

Now I would expect my client should download assets from the specified URL if they are not present locally. Yet assets are being downloaded in the ordinary way from HLDS. The HTTP server doesn't even get any requests, so it's not about permission problems. I can also download content directly with my web browser. It seems Half-Life totally ignores the FastDL URL specified.

Now I enabled a sufficient level of developer console and see that Half-Life does acknowledge the download URL – it just doesn't use it.

BUILD 8684 SERVER (0 CRC)
Server # 56
Serverinfo packet received.
Clearing memory
loading sprites/640hud3.spr
loading sprites/640hud7.spr
loading sprites/640hud8.spr
loading sprites/640hud9.spr
loading sprites/640hud5.spr
loading sprites/640hud6.spr
loading sprites/crosshairs.spr
loading sprites/640hud4.spr
loading sprites/640hud1.spr
loading sprites/640hud2.spr
loading sprites/iplayer.spr
loading sprites/iplayerblue.spr
loading sprites/iplayerred.spr
loading sprites/iplayerdead.spr
loading sprites/tile.spr
loading sprites/laserbeam.spr
loading sprites/camera.spr
loading sprites/voiceicon.spr
Remote host:  Coralie
Couldn't find map maps/bitrnit.bsp, server will download the map
Spooling demo header.
MP3_InitStream(25, media\Half-Life17.mp3) successful
Using http://coralie.megabrutal.com/valve/ as primary download location
Verifying and downloading resources...

After this, the download commences with the regular method through HLDS. Do I need to set another CVAR? Or do I need to enable the utilization of FastDL in my client? I'm not even sure whether it's a client or server side problem.

My other question is, how should I secure the webspace? Obviously I denied access to .cfg, .ini files, because letting anyone to download my server.cfg with the RCON password doesn't seem like a good idea. Still I'm afraid that I might have left something available that allows people to find a way to hijack my server. If I go the opposite approach and instead of denying access to files surely not need to be downloaded, I rather have a whitelist of files I allow access to, I'm afraid I would miss something and deny access to some files clients would legitimately need to download. I don't even know what happens when clients can't download a file from the FastDL server – do they just fall back to regular download or they fail to join the game ultimately?

[MegaBrutal]

Found the problem, it was trivial... Firewall blocked HTTP over IPv4 but allowed it over IPv6. Now Half-Life only tries IPv4 (shame) while my browser had no problem downloading the file over IPv6, this is why I didn't catch the error when I tested. Also somehow I forced Half-Life to finally reveal an error message saying "Failed HTTP download of http://coralie.megabrutal.com/valve/..." which would have been tremendously useful had I seen it earlier. So it's fixed. Now I have various permission issues due to the fact that HLDS and nginx run as different users and while most files are world readable, some are not; but I'll sort that out.

My other question about securing the webspace while allowing access to anything that clients legitimately need still stands. Now I see that clients fall back to regular download when HTTP fails, but I also noticed that HL seems to give up on HTTP altogether and stops trying it until restart (while it could possibly download other files).

[fysiks]

Originally Posted by MegaBrutal Half-Life only tries IPv4 (shame) while my browser had no problem downloading the file over IPv6

This game [engine] was created 25 years ago . . . It hasn't had any major updates for a very long time.

Originally Posted by MegaBrutal So it's fixed. Now I have various permission issues due to the fact that HLDS and nginx run as different users and while most files are world readable, some are not; but I'll sort that out. My other question about securing the webspace while allowing access to anything that clients legitimately need still stands

Originally Posted by MegaBrutal My other question is, how should I secure the webspace? Obviously I denied access to .cfg, .ini files, because letting anyone to download my server.cfg with the RCON password doesn't seem like a good idea. Still I'm afraid that I might have left something available that allows people to find a way to hijack my server. If I go the opposite approach and instead of denying access to files surely not need to be downloaded, I rather have a whitelist of files I allow access to, I'm afraid I would miss something and deny access to some files clients would legitimately need to download.

Your server files and your fast download web server should be entirely and completely separate. Only put files on the web server that you want to have available publicly. Only map and plugin resource files that are going to be downloaded by the client should be on your web server.

NFO Servers has a feature to sync files between the two servers which I assume has some sort of filter to determine what files should and shouldn't be sync'd (I'm not sure what that filter is). I generally dislike this method. The important part is that the web server files are still separate from the game server (likely on entirely different hardware that is better optimized for web traffic).

Originally Posted by MegaBrutal I don't even know what happens when clients can't download a file from the FastDL server – do they just fall back to regular download or they fail to join the game ultimately?

It falls back to direct download from the server. This is why you should make sure that your fast download web server is reliable.

Originally Posted by MegaBrutal Also somehow I forced Half-Life to finally reveal an error message saying "Failed HTTP download of http://coralie.megabrutal.com/valve/..." which would have been tremendously useful had I seen it earlier.

Maybe you already found it but you can enable developer mode with "developer 1" in console. This gives you much more info about what's going on in the background that you don't normally need when things are working correctly.

Originally Posted by MegaBrutal HL seems to give up on HTTP altogether and stops trying it until restart (while it could possibly download other files).

This is a very well known bug and will likely never get fixed.

[gabuch2]

I have no idea how to do it with Nginx but with Apache you can restrict certain filetypes from downloading.

[MegaBrutal]

Originally Posted by fysiks This game [engine] was created 25 years ago . . . It hasn't had any major updates for a very long time.

True, but someone on this forum mentioned that HL supports IPv6 for Fast Download and so I had that expectation. Also at one point I did implement IPv6 support for my application which originally only supported IPv4, so I know it doesn't take much to convert a software to be IP version agnostic. Agreed it might take a little more with such a complicated software like HL, but still somehow I thought they got around to do it. Wonder if Source supports IPv6.

Originally Posted by fysiks Your server files and your fast download web server should be entirely and completely separate. Only put files on the web server that you want to have available publicly. Only map and plugin resource files that are going to be downloaded by the client should be on your web server.

Why though? It made lots of sense in the old days for performance concerns, but now we have multicore servers with blazingly fast SSDs and gigabit upload speeds. Since HLDS is single-threaded (as far as I know), it may only claim 1 core at maximum. So I don't see any bottleneck here, I think one server may host HLDS and the Fast Download webserver just fine.

Originally Posted by fysiks It falls back to direct download from the server. This is why you should make sure that your fast download web server is reliable.

It wouldn't be that much of a problem if HL wouldn't give up on using FastDL altogether... But overall it's still better than if it would prevent people from connecting at all.

Originally Posted by fysiks Maybe you already found it but you can enable developer mode with "developer 1" in console. This gives you much more info about what's going on in the background that you don't normally need when things are working correctly.

Yeah, I actually set "developer 5" to ensure I see everything. I think 3 is the highest setting that actually makes sense, but I wasn't sure, so I set a number that surely includes everything.

Originally Posted by gabuch2 I have no idea how to do it with Nginx but with Apache you can restrict certain filetypes from downloading.

Yeah, I generally prefer Apache because I know it much better, that's what I've spent most of my life with. Yet I chose nginx for this purpose because it's known to be more lightweight. Anyway, here's how I blocked certain file extensions with nginx (it goes into the "server" tag within "http"):

        location ~\.(cfg|log|ini|so|dll)$ {
	    deny all;
        }

If I've known about an even more lightweight and performant webserver, I'd use that, especially if it's written in Rust. I was thinking that it would be possible to implement a webserver using existing libraries that would be aware of HLDS and cache the files needed for the next map in advance so it could serve them faster. But I'm not sure if it would improve anything, as file access is not the bottleneck here, and the OS already implements its own file system cache anyway.

[fysiks]

Originally Posted by MegaBrutal True, but someone on this forum mentioned that HL supports IPv6 for Fast Download and so I had that expectation. Also at one point I did implement IPv6 support for my application which originally only supported IPv4, so I know it doesn't take much to convert a software to be IP version agnostic. Agreed it might take a little more with such a complicated software like HL, but still somehow I thought they got around to do it. Wonder if Source supports IPv6.

It's less about "how much work is it" and more about about incentives for them to do it at all. Also, it's a fact that even the smallest change in software can create cascading effects so companies have to put most, if not all, changes through regression testing, public beta testing, etc. Time is money.

And for fun, this just popped into my head: "If you don't want it broken worse, don't fix it at all" lol. They've fixed things before only to cause other bugs to occur.

Originally Posted by MegaBrutal Why though? It made lots of sense in the old days for performance concerns, but now we have multicore servers with blazingly fast SSDs and gigabit upload speeds. Since HLDS is single-threaded (as far as I know), it may only claim 1 core at maximum. So I don't see any bottleneck here, I think one server may host HLDS and the Fast Download webserver just fine.

I guess my suggestion is mostly just that, a suggestion, not a requirement. I have two reasons. One, it minimizes the risk of exposing important files when you accidentally make a change in settings on your web server that exposes sensitive files. If your web server never has access to the sensitive files then it can't ever expose them.

Secondly, there are some modifications that can be made to maps that don't need to be implemented in your fast download version of the file. If I were to modify a map in such a way that it doesn't prevent compatibility, I can do so without causing people to download a slightly different map than what they would download from most other servers that host the same map. Granted, it's not all that big of issue functionally since those customization will be negated when they connect to a server that uses the non-customized version anyways.

Originally Posted by MegaBrutal It wouldn't be that much of a problem if HL wouldn't give up on using FastDL altogether... But overall it's still better than if it would prevent people from connecting at all.

If the web server is down, there is no other option than to download directly from the server. The bug where it permanently falls back to direct download when it fails once is a bummer and certainly isn't the intended functionality (hence we it's called a bug).