[ALIK]

Originally Posted by Shooting King Yes

What kind? Examples?

[Shooting King]

Originally Posted by Nixon133 Whether also there will be a continuation of the module?

Yes

Originally Posted by Nixon133 Improvement, completion?

Yes, If it can be optimized in anyway which i overlooked, just open a pull req on Github.

[Black Rose]

You're right. It's much more appropriate to continue the discussion here.

I previously had a fixed size first packet of 2048B. But I encountered problems where sometimes the HTTP header wouldn't fit and in other cases I managed to get 2 beginnings of chunks inside one "packet".

The solution was to get the header one byte at a time until the end was found and then the rest in packets of 64KB.

Also, if the transfer is chunked (streamed with no fixed size) you have to read the size of the chunk one byte at a time, download the chunk, preferably the number of bytes previously announced by the server and then read the next chunk size byte for byte again. Repeat until the size of the next chunk is 0.

So I need to change the number of bytes to receive on-the-fly.


Other than that I feel it should be possible to include threaded sockets. Not as a requirement, just as an option. Fully automatic of course, set_native_filter() is an amazing native for these things.

Here's the parts from my code:

Header:

while ( gBufferPos < sizeof gDataBuffer && socket_recv(gInformation[gIndex][_hSocket], gDataBuffer[gBufferPos], 2) ) { // Get one byte 

Reading chunk size:

socket_recv(gInformation[gIndex][_hSocket], tempdata, 2); // Get one byte 

Chunked, Large transfers (>2GB) and regular transfers:

    static stTempLarge[16];
    new tempsize;
    
    if ( gInformation[gIndex][_Status] & STATUS_CHUNKED_TRANSFER )
        tempsize = min(gInformation[gIndex][_EndOfChunk] - gInformation[gIndex][_BytesReceived] + 1, sizeof gDataBuffer); // Get the remaining bytes of the chunk unless it's too big, then try to fill the buffer.

    else if ( gInformation[gIndex][_Status] & STATUS_LARGE_SIZE ) {
        large_add(stTempLarge, sizeof stTempLarge, gInformation[gIndex][_FilesizeLarge], sizeof gInformation[][_FilesizeLarge]);
        large_sub(stTempLarge, sizeof stTempLarge, gInformation[gIndex][_BytesReceivedLarge], sizeof gInformation[][_BytesReceivedLarge]);
        large_add(stTempLarge, sizeof stTempLarge, gOneLarge, sizeof gOneLarge);
        
        if ( large_comp(stTempLarge, sizeof stTempLarge, gBufferSizeLarge, sizeof gBufferSizeLarge) == 1 ) // Compares how much is remaining on the download and if it's bigger than the buffer, try to fill the buffer.
            tempsize = sizeof gDataBuffer;
        else
            tempsize = large_toint(stTempLarge, sizeof stTempLarge); // Otherwise try getting whatever's left.
    }
    else
        tempsize = min(gInformation[gIndex][_Filesize] - gInformation[gIndex][_BytesReceived] + 1, sizeof gDataBuffer); // Get the remaining bytes unless too much, then try to fill buffer.

    gBufferPos = 0;

    if ( ( gBufferLen = socket_recv(gInformation[gIndex][_hSocket], gDataBuffer, tempsize) ) <= 0 ) { // Execute. 

[Shooting King]

I still dont understand whats the problem with native socket_recv_t( const iThreadHandle, const CallBackHandler[], const iRecvDataLen ); ._. You can change iRecvDataLen 1B or 10B or 10GB your wish.

[Black Rose]

You call socket_recv_t 1 time.
The handler is called several times. At this point the datalen is set and cannot be changed. I need to parse the data sent by the server and react to that depending on the type of transfer. If the transfer is chunked I have to dynamically set the datalen to hit the end of the chunk perfectly.

en.wikipedia.org/wiki/Chunked_transfer_encoding#Example

[Shooting King]

Oh ! I get it now. Try v1.1.

[Black Rose]

Perfect. I love the implementation. I'll get to work as soon as possible. Hopefully this week.

[tenub]

In the download example, what happens if multiple "/test" commands are executed at the same time since the thread handle is global? I ask because I am curious how to handle sending concurrent data to a Redis server, ie. when a server changes map and player info for up to 32 players is sent. How can we handle race conditions / locking / waiting? Sorry, I am new.

[Shooting King]

If the /test is executed multiple times(i.e before completion of file download), Multiple threads will be created overriding g_iThreadHandle with the same call backs, causing improper execution of callback functions with different handles, will not work as you have expected, ofcourse its not the way the API is intended to be used.

In this Module's Threading model, there are no bottlenecks, so no race conditions/locking. One unique handle per thread, One socket per thread.

Your possible models (to be used in your plugin),
Sequential Model -> Create a thread, sendData of player1, In the callback, sendData of player2, again in callback send data of player3,....
Or
Concurrent/Parallel Model -> Create 32 Threads (if you have enough system resources), Send data of one PLayer1 from Thread1, PLayer2 on Thread2, and so on...., but don't expect the order to be same on the other endpoint, like player2 Data may be received first or player3-player1-player2 or some other combination.

In Either case, There will be no race/locking in the Module.

[tenub]

Originally Posted by Shooting King Concurrent/Parallel Model -> Create 32 Threads (if you have enough system resources), Send data of one PLayer1 from Thread1, PLayer2 on Thread2, and so on...., but don't expect the order to be same on the other endpoint, like player2 Data may be received first or player3-player1-player2 or some other combination. In Either case, There will be no race/locking in the Module.

I see now. For each possible connection I would need to create unique handles. One more question if you wouldn't mind. Consider hooking into player "say" events to deliver a payload to the same Redis server in order to broadcast in-game chat to the web. How would you go about this without throttling player chat, ie. forcing a player to only send 1 message every X seconds (even then there is no guarantee the thread will complete in time for the next message from the same player)? Consider also the worst case scenario of 32 players each sending a message every 200ms or so.