View Single Post
Author Message
dordnung
Veteran Member
Join Date: Apr 2010
Old 12-23-2010 , 04:40   [EXTENSION] System2 - Easy to use HTTP(S)/FTP Request API and useful system natives
Reply With Quote #1

Description:

System2 allows you to easily make HTTP, HTTPS and FTP requests and also provides some useful system natives, which missing in the Sourcemod API.

Features of System2:
  • Make HTTP(S)/FTP requests with a lot of options:
    • Set the port of the request
    • Download response to a local file (download files)
    • Define the timeout for the request
    • Get progress information (useful when downloading or uploading stuff)
    • Send body data
    • Set custom headers
    • Set user agent and basic authentication
    • Use the HTTP methods: GET, POST, PUT, PATCH, DELETE or HEAD
    • Determine whether to follow redirects or not
    • Upload files to a FTP server
    • List files/folders in a FTP directory
    • Get useful information of the response
      • The status code
      • The body data
      • The headers
      • The HTTP version
      • The last visited URL
      • The total execution time
      • The amount of downloaded / uploaded bytes
      • The average download / upload speed
  • URL encode / decode strings
  • Copy files
  • Extract and Compress files with 7-ZIP
  • Execute system commands and get the output and exit status
  • Retrieve the operating system of the server
  • Calculate the MD5 hash of a string
  • Calculate the MD5 hash of a file
  • Calculate the CRC32 hash of a string
  • Calculate the CRC32 hash of a file

Note 1: To allow the extraction and compression of files, the included data/system2/linux/amd64/7z for 64bit Linux, data/system2/linux/i386/7z for 32bit Linux or data/system2/win/7z.exe file on Windows needs execution rights (744 on linux).
Note 2: You shouldn't need any extra librarys on your machine. But if you get any error just tell me and I'll take a look at it.


Install the extension:
  • Download the extension's ZIP file
  • Extract its content to your server's addons/sourcemod folder
  • Give 7-ZIP execution rights (optional, only if a plugin uses compression):
    • 64-Bit: Give data/system2/linux/amd64/7z execution rights (744)
    • 32-Bit: Give data/system2/linux/i386/7z execution rights (744)


Code examples for Devs:
Or take a look at the system2_http.sp and system2_test.sp examples or system2.inc and request.inc include files


If you miss any other functionality, just ask for it or open an issue on Github and I'll try to implement it.


Configure the request:

Note: When you create requests you have to delete them with the delete statement when you don't need them anymore, until then they can be used more then once.
  • Make a simple HTTP GET request:
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com?action=test");
    httpRequest.GET();

    // Requests have to be deleted, until then they can be used more then once
    delete httpRequest
  • Make a simple HTTP POST request (By default content type application/x-www-form-urlencoded will be used):
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com");
    httpRequest.SetData("action=test");
    httpRequest.POST(); 
  • Save output to a file (download a file):
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com/test.txt");
    httpRequest.SetOutputFile("addons/sourcemod/data/test.txt");
    httpRequest.GET(); 
  • Use a custom port:
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com");
    httpRequest.SetPort(8080);
    httpRequest.GET(); 
  • Set request headers (and use the HTTP PUT method):
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com");
    httpRequest.SetHeader("Content-Type""application/json");
    httpRequest.SetHeader("Referer""http://google.com");
    httpRequest.SetData("{}");
    httpRequest.PUT(); 
  • Set timeout and follow redirects feature (and use the HTTP DELETE method):
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com");
    httpRequest.FollowRedirects false;
    httpRequest.Timeout 5;
    httpRequest.DELETE(); 
  • Pass any data to the response callback (and use the HTTP HEAD method):
    PHP Code:
    int anyValue 5;

    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com");
    httpRequest.Any anyValue;
    httpRequest.HEAD(); 
  • Use a progress callback:
    PHP Code:
    System2HTTPRequest httpRequest = new System2HTTPRequest(HttpResponseCallback"http://example.com/test.txt");
    httpRequest.SetProgressCallback(HttpProgressCallback);
    httpRequest.SetOutputFile("test.txt");
    httpRequest.GET();

    public 
    void HttpProgressCallback(System2HTTPRequest requestint dlTotalint dlNowint ulTotalint ulNow) {
        
    PrintToServer("Downloaded %d of %d bytes"dlNowdlTotal);

  • List a directory of a FTP server:
    PHP Code:
    System2FTPRequest ftpRequest = new System2FTPRequest(FtpResponseCallback"ftp://example.com");
    ftpRequest.ListFilenamesOnly true;
    ftpRequest.SetAuthentication("username""password");
    ftpRequest.SetPort(21);
    ftpRequest.StartRequest(); 
  • Download a file from a FTP server:
    PHP Code:
    System2FTPRequest ftpRequest = new System2FTPRequest(FtpResponseCallback"ftp://example.com/test.txt");
    ftpRequest.SetPort(21);
    ftpRequest.SetAuthentication("username""password");
    ftpRequest.SetOutputFile("test.txt");
    ftpRequest.StartRequest(); 
  • Upload a file to a FTP server:
    PHP Code:
    System2FTPRequest ftpRequest = new System2FTPRequest(FtpResponseCallback"ftp://example.com/test.txt");
    ftpRequest.AppendToFile true;
    ftpRequest.CreateMissingDirs false;
    ftpRequest.SetPort(21);
    ftpRequest.SetProgressCallback(FtpProgressCallback);
    ftpRequest.SetInputFile("test.txt");
    ftpRequest.StartRequest(); 


Use the response:

Note 1: You get the data that you have set in the request with the request parameter in the callback, which is a copy of the original request (and therefore can not be deleted, but reused).
Note 2: The response parameter is only valid in the callback and only if success is true.
  • Get the status code and the time needed for the request:
    PHP Code:
    public void HttpResponseCallback(bool success, const char[] errorSystem2HTTPRequest requestSystem2HTTPResponse responseHTTPRequestMethod method) {
        if (
    success) {
            
    char lastURL[128];
            
    response.GetLastURL(lastURLsizeof(lastURL));

            
    int statusCode response.StatusCode;
            
    float totalTime response.TotalTime;

            
    PrintToServer("Request to %s finished with status code %d in %.2f seconds"lastURLstatusCodetotalTime);
        } else {
            
    PrintToServer("Error on request: %s"error);
        }

  • Get the content of the response:
    PHP Code:
    public void HttpResponseCallback(bool success, const char[] errorSystem2HTTPRequest requestSystem2HTTPResponse responseHTTPRequestMethod method) {
        if (
    success) {
            
    char[] content = new char[response.ContentLength 1];
            
    response.GetContent(contentresponse.ContentLength 1);

            
    PrintToServer("Content of the response: %s"content);
        }

  • Retrieve the content of the response line by line:
    PHP Code:
    public void HttpResponseCallback(bool success, const char[] errorSystem2HTTPRequest requestSystem2HTTPResponse responseHTTPRequestMethod method) {
        if (
    success) {
            
    char content[128];
            for (
    int found 0found response.ContentLength;) {
                
    found += response.GetContent(contentsizeof(content), found"\n");
                
    PrintToServer("Line %d: %s"foundcontent);
            }
        }

  • Get headers of the response:
    PHP Code:
    public void HttpResponseCallback(bool success, const char[] errorSystem2HTTPRequest requestSystem2HTTPResponse responseHTTPRequestMethod method) {
        if (
    success) {
            
    PrintToServer("Found %d headers:"response.Headers);

            
    char name[128];
            
    char value[128];

            
    ArrayList headers response.GetHeaders();
            for (
    int i 0headers.Lengthi++) {
                
    headers.GetString(inamesizeof(name));
                
    response.GetHeader(namevaluesizeof(value));
                
                
    PrintToServer("\t%s: %s"namevalue);
            }

            
    // Or just a single header...
            
    if (response.GetHeader("Content-Type"valuesizeof(value))) {
                
    PrintToServer("Use content type %s..."value);
            }
        }

  • Get download/upload size/speed:
    PHP Code:
    public void FtpResponseCallback(bool success, const char[] errorSystem2FTPRequest requestSystem2FTPResponse response) {
        if (
    success) {
            
    char file[PLATFORM_MAX_PATH];
            
    request.GetInputFile(filesizeof(file));

            if (
    strlen(file) > 0) {
                
    PrintToServer("Uploaded %d bytes with %d bytes / second"response.UploadSizeresponse.UploadSpeed);
            } else {
                
    PrintToServer("Downloaded %d bytes with %d bytes / second"response.DownloadSizeresponse.DownloadSpeed);
            }
        }

  • Upload the same file to another URL in the response (Reuse the request):
    PHP Code:
    public void FtpResponseCallback(bool success, const char[] errorSystem2FTPRequest requestSystem2FTPResponse response) {
        if (
    success) {
            
    // Only change the URL and start the request again (other options will retained)
            
    request.SetURL("ftp://example.com/test2.txt");
            
    request.StartRequest();
        }



Execute commands:
  • Execute a threaded system command
    PHP Code:
    public void OnPluginStart() {
        
    System2_ExecuteThreaded(ExecuteCallback"ls /home");
    }

    public 
    void ExecuteCallback(bool success, const char[] commandSystem2ExecuteOutput outputany data) {
        if (!
    success || output.ExitStatus != 0) {
            
    PrintToServer("Couldn't execute commands %s successfully"command);
        } else {
            
    char outputString[128];
            
    output.GetOutput(outputStringsizeof(outputString));
            
    PrintToServer("Output of the command %s: %s"commandoutputString);
        }

  • Compress a file:
    PHP Code:
    if (!System2_Compress(CompressCallback"test.txt""test.zip")) {
        
    PrintToServer("7-ZIP was not found or is not executable!");

  • Extract an archive:
    PHP Code:
    if (!System2_Extract(ExtractCallback"test.zip""addons/sourcemod/data")) {
        
    PrintToServer("7-ZIP was not found or is not executable!");

  • It may be that for some users with 64-bit system the 64-bit 7-ZIP file can't be executed (for example if they have an old libstdc++).
    So it's safer to check this before and then use the 32-bit executable, if this works.
    PHP Code:
    // Boolean which holds the knowledge whether we have to force using the 32-bit executable
    force32Bit false;

    char binDir[PLATFORM_MAX_PATH];
    char binDir32Bit[PLATFORM_MAX_PATH];

    // First check if we can use the 7-ZIP executable which is suitable for the running machine
    if (!System2_Check7ZIP(binDirsizeof(binDir))) {
        
    // If not: Check if 32-bit 7-ZIP can be used
        
    if (!System2_Check7ZIP(binDir32Bitsizeof(binDir32Bit), true)) {
            
    // Print an error if both can't be executed
            
    if (StrEqual(binDirbinDir32Bit)) {
                
    LogError("ERROR: 7-ZIP was not found or is not executable at '%s'"binDir);
            } else {
                
    LogError("ERROR: 7-ZIP was not found or is not executable at '%s' or '%s'"binDirbinDir32Bit);
            }
        } else {
            
    // 64-bit does not seem to work, but 32-bit do. We have to memorize this.
            
    force32Bit true;
        }
    }

    // Now use the knowledge whether we have to force 32-bit or not.
    System2_Compress(CompressCallback"test.txt""test.zip"ARCHIVE_ZIPLEVEL_90force32Bit); 


Sourcecode on Github: https://github.com/dordnung/System2
Sourcecode of 7-ZIP: https://sourceforge.net/projects/sevenzip/files/7-Zip

Changelog:

Code:
3.3.2
- Revert "Disable certificate revocation checks to fix Schannel error"

3.3.1
- Use Debian Strech for compiling as LTS support for jessie ended
- Update OpenSSL to version 1.1.1k and libcurl to 7.76.0
- Fix out-of-memory bug when downloading large files by disable saving of output content when downloading a file
- Add errno message if PosixOpen (command execution) fails
- Disable certificate revocation checks to fix Schannel error

3.3.0
- Fixes crashing on Sourcemod 11 caused by threads
- Updated to libcurl 7.72.0

3.2.1 
- Automatically use 32-Bit 7-ZIP when 64-Bit is not executable
- Fix incomplete file when downloading files with FTP
- Added libcurl certificate file for SSL instead of using system's certificate file
- Updated to the latest libcurl version

3.2
- Added ability to use a proxy for requests (Pull Request by Kyle)
  See System2Request.SetProxy and System2Request.SetProxyAuthentication methods for more information
- Updated to the latest OpenSSL and libcurl version

3.1
- Fixed crash when creating many threads

3.0
- Almost all previous natives are now deprecated
- Implemented an enormous HTTP / FTP Request API replacing the old natives
- Made extension much more stable
- Added check if 7-ZIP is executable on Linux
- Output of an execution is now a methodmap replacing the progress approach
- Provide exit status of an execution
- Cleaned up API
- A lot more

2.6
- Fixed that the error string on the TransferUpdated callback wasn't empty when there was no error
- Only allowing one FTP connection at the same time, because RFC only allow once
- Added MD5 and CRC32 calculating for strings and files

2.5
- Complete code restructure
- A lot of bug fixes
- Added 7z for 32bit and 64bit linux
- Updated openssl and libcurl to newest versions
- Added command to command callback

2.32
- Fixed some curl bugs

2.31
- Fixed wrong any parameter on callback function

2.3
- Added parameter any:data to all callbacks and natives with callbacks

2.21
- Fixed a bug that CopyFile doesn't copied all data

2.2
- Added native System2_GetPage to get the content of a page
- Now you can retrieve the whole output from a threaded system command (Sorry i had to change the CmdCallback for this!)
- Support for SSL on Linux (Windows already have support)

2.1 
- Security fixes
- Now every progress update will call the forward

2.0
- Added prefix System2_ to natives
- Added DownloadFile
- Added DownloadFTPFile
- Added UploadFTPFile
- Added CopyFile
- Added CompressFile
- Added ExtractArchive

1.5 - Proper usage of threading. Thanks to KyleS

1.4
- RunCommand works now non threaded
- Added RunThreadCommand for threaded commands
- Callback for threaded commands gives now: executed command, output and status code
- RunCommand returns status code and gives output
- Command parameter now with arguments
- No need of Metamod -> Only one file for all games!

1.3 - RunCommand is now Threaded and have a callback function!
1.2 - Improved Extension: works now also for left4dead(2)
1.1 - Added GetOS
1.0 - Release
Attached Files
File Type: zip system2.zip (5.60 MB, 2865 views)
__________________

Last edited by dordnung; 04-16-2021 at 17:10.
dordnung is offline