Raised This Month: $110 Target: $400
 27% 

[ANY] Async - Efficient event based HTTP client


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-29-2014 , 22:29   [ANY] Async - Efficient event based HTTP client
Reply With Quote #1

Async

An HTTP(S) client focused on ease of use, performance, and stability. This is accomplished by using CURL with libuv, the async library that powers node.js.

Example

Code:
#include <async>

public OnPluginStart() {
    new CurlHandle:h = Async_CurlNew(123);
    Async_CurlGet(h, "http://www.sourcemod.net", OnRequestDone);
}

public OnRequestDone(CurlHandle:request, curlcode, httpcode, size, any:userdata) {
    decl String:buffer[size+1];
    // see http://curl.haxx.se/libcurl/c/libcurl-errors.html for curlcode numbers
    // see https://en.wikipedia.org/wiki/List_of_HTTP_status_codes for http codes
    if(curlcode == 0 && httpcode == 200) {
        Async_CurlGetData(request, buffer, size+1);
        PrintToServer("%s", buffer);
    }
    Async_Close(request);
}
Curl errors codes are located here http://curl.haxx.se/libcurl/c/libcurl-errors.html

API

Code:
funcenum Async_CurlCallback {
    public(CurlHandle:request, curlcode, httpcode, size),
    public(CurlHandle:request, curlcode, httpcode, size, any:userdata),
};

// Creating and closing requests
// Unless you are unloading the extension or your plugin gets unloaded, you must close all handles yourself
native CurlHandle:Async_CurlNew(any:userdata=0);
native Async_Close(any:handle);

// Starting requests
// Once your start a request all operations on them will fail until the callback fires
native Async_CurlGet(CurlHandle:request, const String:url[], Async_CurlCallback:f);
native Async_CurlPost(CurlHandle:request, const String:url[], const String:post[], Async_CurlCallback:f);
native Async_CurlPostRaw(CurlHandle:request, const String:url[], const String:post[], postsize, Async_CurlCallback:f);
// For use with advanced options
native Async_CurlStart(CurlHandle:request, Async_CurlCallback:f);

// Common functions
native Async_CurlGetData(CurlHandle:request, const String:buffer[], buffer_size);
// same as above except it adds a null byte for you at the end. The buffer size should be 1 larger than size.
stock Async_CurlGetDataString(CurlHandle:request, String:buffer[], buffer_size, size);
native Async_CurlAddHeader(CurlHandle:request, const String:value[]);

// Setting advanced options
// Not all options supported, using the wrong option/function may result in a crash
// http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
// http://curl.haxx.se/libcurl/c/curl_multi_setopt.html
native Async_CurlSetInt(CurlHandle:request, CURLOPT:option, value);
native Async_CurlSetString(CurlHandle:request, CURLOPT:option, const String:value[]);
native Async_CurlSetGlobalInt(CurlHandle:request, CURLMOPT:option, value);

// Getting advanced info
// Not all options supported, using the wrong option/function may result in a crash
// http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html
native Async_CurlGetInt(CurlHandle:request, CURLINFO:option, &value);
native Async_CurlGetString(CurlHandle:request, CURLINFO:option, const String:value[]);

// Debugging info
native Async_CurlErrorString(curlcode, const String:buffer[], buffer_size);
native Async_CurlVersion(const String:buffer[], buffer_size);
Usage Notes
  • It is up to you to use options and set buffer sizes correctly (ie Using a number for a string option or setting buffer sizes larger than the actual size). I don't have time to idiot-proof everything. Not doing so may result in crashes.
  • Function callback options are not supported.
  • If you make an array a bit larger than 16834 (default sourcemod stack size). Sourcemod will silently fail. This is a Sourcemod problem, not a problem with the extension.
  • This extension does not use the Sourcemod handle system which means it does not count towards the 16k handle limit. But you must close handles yourself with Async_Close().
  • However the handles will be closed for you if the plugin is unloaded.
  • Windows will not be supported but the code is written to be platform-neutral, so there should be Windows support as long as someone builds a copy.

Why another network client?

As I am writing this there are currently 3 options. There are problems with each one as described here.

https://forums.alliedmods.net/showthread.php?t=232396
  • Curl
    People report that it starts failing randomly. It seems to support every single CURL option and is a nightmare to maintain. Does something crazy like using select in each request thread.
  • Sockets
    You need to implement a lot of logic just for http. I've seen a few people here make the assumption that the entire response will return in one callback.
  • Steamtools/Steamworks
    The amount of reverse engineering needed to make this works scares me and it should scare you too. It also doesn't support years old features like ECDSA which means Valve probably doesn't care about maintaining the HTTP client.

Changelog

7/9/2015

1.6
  • Fixed Async_CurlGetData so it will only add a null byte to the end if the sourcepawn buffer is larger than data received.

5/13/2015

1.5
  • Removed boost and added back the C++0x for compiling. Libstdc++ is now statically compiled into the binary to avoid any problems with the game's libstdc++ being a different version or missing. This should not cause any problems because the extension does not use dlopen or throws any exceptions that I know of.

4/23/2015

1.4
  • You can now close requests that are being processed. They will close without firing the callback when they finish or timeout.

1.3
  • Removed unused code

1.2
  • Added debug symbols to the build.

4/22/2015

1.2
  • Improved Stability

1.1
  • Added Async_CurlGetDataString. This is the same as Async_CurlGetData except it will put a null byte at the end of your buffer for you.
  • Boost is now required instead of worrying if the c++ library your game uses is new enough.
  • Library versions: libcurl/7.42.0 OpenSSL/1.0.1f zlib/1.2.8 c-ares/1.10.0 libidn/1.28

3/29/2014

1.0
  • Initial Release
Attached Files
File Type: inc async.inc (10.6 KB, 90 views)
File Type: zip async.ext.zip (1.56 MB, 94 views)
File Type: zip async.zip (340.3 KB, 109 views)
__________________

Last edited by bottiger; 07-09-2015 at 16:48. Reason: fixed Async_CurlGetData example
bottiger is offline
VoiDeD
SourceMod Plugin Unapprover
Join Date: Mar 2009
Location: Illinois, USA
Old 03-30-2014 , 19:29   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #2

Quote:
Originally Posted by bottiger View Post
  • Steamworks
    The amount of reverse engineering needed to make this works scares me and it should scare you too.
You mean, the complete lack of reverse engineering?

The Steamworks extension is built against the SW SDK, and makes use of a version safe interface that was designed for the exact use-case it fulfills.
__________________
VoiDeD is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 03-30-2014 , 20:10   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #3

So, this is basically just a slimmed down version of the Curl extension?
__________________
Am I back? Well, we'll see.
Powerlord is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 16:20   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #4

Quote:
Originally Posted by VoiDeD View Post
You mean, the complete lack of reverse engineering?

The Steamworks extension is built against the SW SDK, and makes use of a version safe interface that was designed for the exact use-case it fulfills.
Isn't Steamworks based on didrole's automated reversing tool because Steamworks isn't public? And up until 9 days ago it was not updated for 3 months causing many functions to fail.

http://osw.didrole.com/diff.php
https://github.com/SteamRE/open-stea.../master?page=2

Even if Steamworks is public now you still have this problem and extensions need to be built and re-built for games using different versions of Steamworks.

There is also the fact that the Steam HTTP doesn't support ECDSA, something that was in OpenSSL 3 years ago.

Quote:
Originally Posted by Powerlord View Post
So, this is basically just a slimmed down version of the Curl extension?
It uses asynchronous IO which is more efficient than 1 thread per request. On Ubuntu each thread is allocated 8MB by default unless you fiddle with the settings and hope it doesn't crash due to lack of stack space.

And it does not use Sourcemod handles which has a limit of 16k which is easily reached if you use plugins like tf2itemsinfo. This could have been one of the reasons why the old curl extension was failing after a long period of time.
__________________

Last edited by bottiger; 03-31-2014 at 16:23.
bottiger is offline
VoiDeD
SourceMod Plugin Unapprover
Join Date: Mar 2009
Location: Illinois, USA
Old 03-31-2014 , 16:32   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #5

Quote:
Originally Posted by bottiger View Post
Isn't Steamworks based on didrole's automated reversing tool because Steamworks isn't public? And up until 9 days ago it was not updated for 3 months causing many functions to fail.

http://osw.didrole.com/diff.php
https://github.com/SteamRE/open-stea.../master?page=2

Even if Steamworks is public now you still have this problem and extensions need to be built and re-built for games using different versions of Steamworks.
The Steamworks SDK was released by Valve months ago. Regardless of whether you make use of OSW or the official SDK, extensions don't need to be rebuilt at all. The Steamtools and Steamworks extensions HTTP features will work forever.
__________________
VoiDeD is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 18:08   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #6

Quote:
Originally Posted by VoiDeD View Post
The Steamworks SDK was released by Valve months ago. Regardless of whether you make use of OSW or the official SDK, extensions don't need to be rebuilt at all. The Steamtools and Steamworks extensions HTTP features will work forever.
The HTTP interface probably won't break but it is still a non-zero chance.

"Safe" interfaces isn't a guarantee that everything will continue to work. It only means that it won't crash. Valve can still deprecate old interfaces so that they don't work. I have personally experienced this problem.

Even Kyles includes a steamclient download and tells people to install it.

https://forums.alliedmods.net/showthread.php?t=229556

I don't know about everyone else, but I would prefer not to replace a file that can get clobbered by an update or worry about the game using a new interface not included in Kyles' version.
__________________

Last edited by bottiger; 03-31-2014 at 18:09.
bottiger is offline
KyleS
SourceMod Plugin Approver
Join Date: Jul 2009
Location: Segmentation Fault.
Old 03-31-2014 , 18:13   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #7

Quote:
Originally Posted by bottiger View Post
Even Kyles includes a steamclient download and tells people to install it.

https://forums.alliedmods.net/showthread.php?t=229556

I don't know about everyone else, but I would prefer not to replace a file that can get clobbered by an update or worry about the game using a new interface not included in Kyles' version.
That's for Family Sharing as the async result struct changed. If you're not using that, any version should be fine. However, the primary purpose for the extension was that, so it has that.
KyleS is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 03-31-2014 , 19:14   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #8

Quote:
Originally Posted by KyleS View Post
That's for Family Sharing as the async result struct changed. If you're not using that, any version should be fine. However, the primary purpose for the extension was that, so it has that.
Same concept applies though. Older games may not have the interface that Steamworks was compiled to use.

The HTTP interface has only changed twice but it has changed.
__________________

Last edited by bottiger; 03-31-2014 at 19:14.
bottiger is offline
bottiger
AlliedModders Donor
Join Date: Dec 2010
Old 04-22-2015 , 22:16   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #9

Removed C++0x requirement. Download the newest version if you had troubles loading it.
__________________
bottiger is offline
KyleS
SourceMod Plugin Approver
Join Date: Jul 2009
Location: Segmentation Fault.
Old 04-22-2015 , 23:54   Re: [ANY] Async - Efficient event based HTTP client
Reply With Quote #10

Quote:
Originally Posted by bottiger View Post
Removed C++0x requirement. Download the newest version if you had troubles loading it.
Lorph,

Boost is almost always a significant regression in any project.
KyleS is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 03:08.


Powered by vBulletin®
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Theme made by Freecode