Raised This Month: $12 Target: $400
 3% 

[ANY] Async - Efficient event based HTTPS client. HTTP2 and Compression supported


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-29-2014 , 22:29   [ANY] Async - Efficient event based HTTPS client. HTTP2 and Compression supported
Reply With Quote #1

Async

An HTTP(S) client focused on ease of use, performance, and stability.

We moved to Github.

Source Code: https://github.com/bottiger1/sourcemod-async
Include: https://github.com/bottiger1/sourcem...pawn/async.inc
Binary: https://github.com/bottiger1/sourcemod-async/releases

Newest Update

November 13, 2019

This is a major update. Everything has been moved to Github. Libraries have all been updated and HTTP2 and Brotli compression is now supported. You can also have the extension gzip post data outside the game thread.

PHP Code:
CurlHandle h Async_CurlNew();
// tell curl to request compressed data. the data you get will be automatically decompressed
Async_CurlSetString(hCURLOPT_ACCEPT_ENCODING"");
// to compress post data. 

// Your webserver or website must manually decompress this data. Most webservers do not automatically do this.
// compression is done with zlib
Async_CurlPostRawCompress(hurldatastrlen(data), OnRequestComplete); 
PHP Code:
// to decompress data from Async_CurlPostRawCompress in php
$post file_get_contents('php://input');
if(
$_SERVER['HTTP_CONTENT_ENCODING'] == 'gzip')
{
    
$post zlib_decode($post128000);

Why another HTTP client?

Unlike the other CURL based HTTP clients here that spawn a thread for each request, this extension spawns only 1 thread and handles all requests through an event loop.

One may think 1 thread is worse than 8 threads, but http requests are mostly limited by network speed and not cpu. By using the same thread for the requests, it cuts down on context switching overhead. This is why many benchmarks show that single threaded network loops are often faster than ones that just spawn 1 thread per request. It is also safer. If you spawn 100 threads, it will take a lot of memory.

Example

Code:
#include <async>

public OnPluginStart() {
    new CurlHandle:h = Async_CurlNew(123);
    Async_CurlGet(h, "http://www.sourcemod.net", OnRequestDone);
}

public OnRequestDone(CurlHandle:request, curlcode, httpcode, size, any:userdata) {
    decl String:buffer[size+1];
    // see http://curl.haxx.se/libcurl/c/libcurl-errors.html for curlcode numbers
    // see https://en.wikipedia.org/wiki/List_of_HTTP_status_codes for http codes
    if(curlcode == 0 && httpcode == 200) {
        Async_CurlGetData(request, buffer, size+1);
        PrintToServer("%s", buffer);
    }
    Async_Close(request);
}
Curl errors codes are located here http://curl.haxx.se/libcurl/c/libcurl-errors.html

Usage Notes
  • It is up to you to use options and set buffer sizes correctly (ie Using a number for a string option or setting buffer sizes larger than the actual size). I don't have time to idiot-proof everything. Not doing so may result in crashes.
  • Function callback options are not supported.
  • If you make an array a bit larger than 16834 (default sourcemod stack size). Sourcemod will silently fail. This is a Sourcemod problem, not a problem with the extension.
  • This extension does not use the Sourcemod handle system which means it does not count towards the 16k handle limit. But you must close handles yourself with Async_Close() and NOT with CloseHandle().
  • However the handles will be closed for you if the plugin is unloaded.
  • Sending a post larger than 1000 bytes to some webservers will cause the request to not post any data and time out. To fix this, you must add an empty Expect: header. Async_CurlAddHeader(h, "Expect: "); This is because curl sends an Expect command and the webserver just doesn't reply.
  • Windows will not be supported but the code is written to be platform-neutral, so there should be Windows support as long as someone builds a copy.

Why another HTTP client?

As I am writing this there are currently 3 options. There are problems with each one as described here.

https://forums.alliedmods.net/showthread.php?t=232396
  • Curl
    People report that it starts failing randomly. It seems to support every single CURL option and is a nightmare to maintain. Does something crazy like using select in each request thread.
  • Sockets
    You need to implement a lot of logic just for http. I've seen a few people here make the assumption that the entire response will return in one callback.
  • Steamtools/Steamworks
    The amount of reverse engineering needed to make this works scares me and it should scare you too. It also doesn't support years old features like ECDSA which means Valve probably doesn't care about maintaining the HTTP client.

How to Compile

Compiling this extension is really annoying. I have tried to write notes on how to do this in readme.txt. My releases are compiled in Ubuntu 18.04 x64.

Changelog
Code:
11/13/2019

2.0

Moved to Github. https://github.com/bottiger1/sourcem...commits/master

7/9/2015

1.6
  • Fixed Async_CurlGetData so it will only add a null byte to the end if the sourcepawn buffer is larger than data received.
5/13/2015 1.5
  • Removed boost and added back the C++0x for compiling. Libstdc++ is now statically compiled into the binary to avoid any problems with the game's libstdc++ being a different version or missing. This should not cause any problems because the extension does not use dlopen or throws any exceptions that I know of.
4/23/2015 1.4
  • You can now close requests that are being processed. They will close without firing the callback when they finish or timeout.
1.3
  • Removed unused code
1.2
  • Added debug symbols to the build.
4/22/2015 1.2
  • Improved Stability
1.1
  • Added Async_CurlGetDataString. This is the same as Async_CurlGetData except it will put a null byte at the end of your buffer for you.
  • Boost is now required instead of worrying if the c++ library your game uses is new enough.
  • Library versions: libcurl/7.42.0 OpenSSL/1.0.1f zlib/1.2.8 c-ares/1.10.0 libidn/1.28
3/29/2014 1.0
  • Initial Release
__________________

Last edited by bottiger; 12-18-2019 at 00:11. Reason: updated example
bottiger is offline
VoiDeD
AlliedModders Donor
Join Date: Mar 2009
Location: Illinois, USA
Old 03-30-2014 , 19:29   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #2

Quote:
Originally Posted by bottiger View Post
  • Steamworks
    The amount of reverse engineering needed to make this works scares me and it should scare you too.
You mean, the complete lack of reverse engineering?

The Steamworks extension is built against the SW SDK, and makes use of a version safe interface that was designed for the exact use-case it fulfills.
__________________
VoiDeD is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 03-30-2014 , 20:10   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #3

So, this is basically just a slimmed down version of the Curl extension?
__________________
Not currently working on SourceMod plugin development.
Powerlord is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 16:20   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #4

Quote:
Originally Posted by VoiDeD View Post
You mean, the complete lack of reverse engineering?

The Steamworks extension is built against the SW SDK, and makes use of a version safe interface that was designed for the exact use-case it fulfills.
Isn't Steamworks based on didrole's automated reversing tool because Steamworks isn't public? And up until 9 days ago it was not updated for 3 months causing many functions to fail.

http://osw.didrole.com/diff.php
https://github.com/SteamRE/open-stea.../master?page=2

Even if Steamworks is public now you still have this problem and extensions need to be built and re-built for games using different versions of Steamworks.

There is also the fact that the Steam HTTP doesn't support ECDSA, something that was in OpenSSL 3 years ago.

Quote:
Originally Posted by Powerlord View Post
So, this is basically just a slimmed down version of the Curl extension?
It uses asynchronous IO which is more efficient than 1 thread per request. On Ubuntu each thread is allocated 8MB by default unless you fiddle with the settings and hope it doesn't crash due to lack of stack space.

And it does not use Sourcemod handles which has a limit of 16k which is easily reached if you use plugins like tf2itemsinfo. This could have been one of the reasons why the old curl extension was failing after a long period of time.
__________________

Last edited by bottiger; 03-31-2014 at 16:23.
bottiger is offline
VoiDeD
AlliedModders Donor
Join Date: Mar 2009
Location: Illinois, USA
Old 03-31-2014 , 16:32   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #5

Quote:
Originally Posted by bottiger View Post
Isn't Steamworks based on didrole's automated reversing tool because Steamworks isn't public? And up until 9 days ago it was not updated for 3 months causing many functions to fail.

http://osw.didrole.com/diff.php
https://github.com/SteamRE/open-stea.../master?page=2

Even if Steamworks is public now you still have this problem and extensions need to be built and re-built for games using different versions of Steamworks.
The Steamworks SDK was released by Valve months ago. Regardless of whether you make use of OSW or the official SDK, extensions don't need to be rebuilt at all. The Steamtools and Steamworks extensions HTTP features will work forever.
__________________
VoiDeD is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 18:08   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #6

Quote:
Originally Posted by VoiDeD View Post
The Steamworks SDK was released by Valve months ago. Regardless of whether you make use of OSW or the official SDK, extensions don't need to be rebuilt at all. The Steamtools and Steamworks extensions HTTP features will work forever.
The HTTP interface probably won't break but it is still a non-zero chance.

"Safe" interfaces isn't a guarantee that everything will continue to work. It only means that it won't crash. Valve can still deprecate old interfaces so that they don't work. I have personally experienced this problem.

Even Kyles includes a steamclient download and tells people to install it.

https://forums.alliedmods.net/showthread.php?t=229556

I don't know about everyone else, but I would prefer not to replace a file that can get clobbered by an update or worry about the game using a new interface not included in Kyles' version.
__________________

Last edited by bottiger; 03-31-2014 at 18:09.
bottiger is offline
KyleS
SourceMod Plugin Approver
Join Date: Jul 2009
Location: Segmentation Fault.
Old 03-31-2014 , 18:13   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #7

Quote:
Originally Posted by bottiger View Post
Even Kyles includes a steamclient download and tells people to install it.

https://forums.alliedmods.net/showthread.php?t=229556

I don't know about everyone else, but I would prefer not to replace a file that can get clobbered by an update or worry about the game using a new interface not included in Kyles' version.
That's for Family Sharing as the async result struct changed. If you're not using that, any version should be fine. However, the primary purpose for the extension was that, so it has that.
KyleS is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 19:14   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #8

Quote:
Originally Posted by KyleS View Post
That's for Family Sharing as the async result struct changed. If you're not using that, any version should be fine. However, the primary purpose for the extension was that, so it has that.
Same concept applies though. Older games may not have the interface that Steamworks was compiled to use.

The HTTP interface has only changed twice but it has changed.
__________________

Last edited by bottiger; 03-31-2014 at 19:14.
bottiger is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 04-22-2015 , 22:16   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #9

Removed C++0x requirement. Download the newest version if you had troubles loading it.
__________________
bottiger is offline
KyleS
SourceMod Plugin Approver
Join Date: Jul 2009
Location: Segmentation Fault.
Old 04-22-2015 , 23:54   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #10

Quote:
Originally Posted by bottiger View Post
Removed C++0x requirement. Download the newest version if you had troubles loading it.
Lorph,

Boost is almost always a significant regression in any project.
KyleS is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 11:19.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode