[checkster]

Originally Posted by arne1288 I think, both with this one and the statements during your post editing, that you need to read my post again: I don't assume that you go to Kiwi or REMA 1000 to pick up your soap, then go home, and then go back to Kiwi or REMA 1000 for your food, then go home and then go back to Kiwi or REMA 1000 for your toilet paper. Or do you? That would make it three trips towards your shopping chain, and why would you do that, when you can do it all in one go? In one go, it would be faster and more time consuming for you. Currently, the fastdl system (and most likely Workshop too) works like the above. It is contacting the remote sever once per file, and as such, the result is the same as if you were going to Kiwi or REMA 1000 multiple times, rather than taking all what you want in one go. Slower speeds. In the previous post, I said "If you could direct [...]": $ head -n 2 materials.txt materials/sprites/trails/psychball.vtf materials/sprites/trails/mat_rain.vmt $ cat download_individual_materials.sh #!/bin/sh for material in `cat materials.txt`; do wget "https://fastdl.example.net/TEST_20170131/$material" -oO /dev/null done $ time ./download_individual_materials.sh real 10m27.270s user 2m42.728s sys 0m28.396s $ time wget "https://fastdl.example.net/TEST_20170131/materials.tar.gz" -oO /dev/null real 0m19.042s user 0m11.828s sys 0m4.276s $ time wget "https://fastdl.example.net/TEST_20170131/materials.tar.bz2" -oO /dev/null real 0m19.421s user 0m10.744s sys 0m3.776s Server-side: $ du -sh materials materials.tar.gz materials.tar.bz2 714M materials 336M materials.tar.gz 305M materials.tar.bz2 $ Individual files (sv_downloadurl way) gives 10 minutes and roughly 30 seconds of downloading. One big file gives 19 - 19.5 seconds of downloading. And again: I said "If you could direct [...]", as things appear right now at least, you cannot make the downloads go this way. There is a huge difference between "being capped" and "feels like it is capped", in regards to your claim about the speeds being capped. I hope this makes more sense to you.

KUK But yea, can I claim drunk?

[Powerlord]

Originally Posted by arne1288 I think, both with this one and the statements during your post editing, that you need to read my post again: I don't assume that you go to Kiwi or REMA 1000 to pick up your soap, then go home, and then go back to Kiwi or REMA 1000 for your food, then go home and then go back to Kiwi or REMA 1000 for your toilet paper. Or do you? That would make it three trips towards your shopping chain, and why would you do that, when you can do it all in one go? In one go, it would be faster and more time consuming for you. Currently, the fastdl system (and most likely Workshop too) works like the above. It is contacting the remote sever once per file, and as such, the result is the same as if you were going to Kiwi or REMA 1000 multiple times, rather than taking all what you want in one go. Slower speeds. In the previous post, I said "If you could direct [...]": $ head -n 2 materials.txt materials/sprites/trails/psychball.vtf materials/sprites/trails/mat_rain.vmt $ cat download_individual_materials.sh #!/bin/sh for material in `cat materials.txt`; do wget "https://fastdl.example.net/TEST_20170131/$material" -oO /dev/null done $ time ./download_individual_materials.sh real 10m27.270s user 2m42.728s sys 0m28.396s $ time wget "https://fastdl.example.net/TEST_20170131/materials.tar.gz" -oO /dev/null real 0m19.042s user 0m11.828s sys 0m4.276s $ time wget "https://fastdl.example.net/TEST_20170131/materials.tar.bz2" -oO /dev/null real 0m19.421s user 0m10.744s sys 0m3.776s Server-side: $ du -sh materials materials.tar.gz materials.tar.bz2 714M materials 336M materials.tar.gz 305M materials.tar.bz2 $ Individual files (sv_downloadurl way) gives 10 minutes and roughly 30 seconds of downloading. One big file gives 19 - 19.5 seconds of downloading. And again: I said "If you could direct [...]", as things appear right now at least, you cannot make the downloads go this way. There is a huge difference between "being capped" and "feels like it is capped", in regards to your claim about the speeds being capped. I hope this makes more sense to you.

Do I need to point out the obvious flaw that you didn't gz or bz2 the individual materials files and that will make a time difference?

[DarkDeviL]

Originally Posted by Powerlord Do I need to point out the obvious flaw that you didn't gz or bz2 the individual materials files and that will make a time difference?

They would show the exact same result - that a full archive would take less time to download than the individual files due to many round trips.

And that was the whole point, ... but as you wish:

Everything .bz2'ed (default "bzip2 <file>"), and tar.(gz|bz2)'ed the .bz2's:

$ cat download_individual_materials.sh
#!/bin/sh
for material in `cat materials.txt`;
do
  wget "https://fastdl.example.net/TEST_20170131/$material.bz2" -oO /dev/null
done
$ time ./download_individual_materials.sh

real    9m21.653s
user    2m27.788s
sys     0m23.172s
$ time wget "https://fastdl.example.net/TEST_20170131/materials-bz2.tar.gz" -oO /dev/null

real    0m21.417s
user    0m10.340s
sys     0m3.252s
$ time wget "https://fastdl.example.net/TEST_20170131/materials-bz2.tar.bz2" -oO /dev/null

real    0m15.053s
user    0m10.552s
sys     0m3.456s

Server-side:

$ du -sh materials materials-bz2.tar.gz materials-bz2.tar.bz2
310M    materials
300M    materials-bz2.tar.gz
301M    materials-bz2.tar.bz2

The individual files saved you one minute by bz2'ing them compared to not being bz2'ed at all.

With one full archive, it still saves you around 9 - 10 minutes, whether or not that the contents of the tar.(gz|bz2) is bz2'ed inside.