Different packet (bitstream) buffer received in the client

[0x416c69]

Hi, first I've gotta say thank you and nice job on your perfect network engine.

I have a BIG BitStream (140kB big usually) sending using Send function (Packet).
The problem is, I send a BitStream along with the packet from the server and I receive it in another form inside the client, and the funny part is this is OK and doesn't have any problem while the client is localhost! (Not actual localhost, router connected (like 192.168.1.100) or connected using remote ip address but initially locally)

My best guess about this problem not happening on locally connected clients is about their ping time and packet arrival time, it's local host, it's fast so there won't be any corrupted data along with the packet!

Here's how I record the BitStream data:

// bitStream is our bitstream and it's totally fine and working
SString strGooz;
for(uint i = 0; i < bitStream->GetNumberOfBytesUsed(); i++)
{
	BYTE bGooz = *(bitStream->GetData() + i);
	strGooz += SString("0x%02X ", bGooz);
}
SString strShitShow("Length: %iBytes, Data: %s\n", bitStream->GetNumberOfBytesUsed(), *strGooz);
SharedUtil::FileAppend("whatgoozwant.txt", strShitShow, true);
bitStream->ResetReadPointer();

The data I've recorded inside the server:
server.txt

So far so good, everything is perfect and here is the record of the received BitStream data inside the REMOTELY outsider client:
client.txt

As you can see, the lengths are equal, even some huge parts are equal but if you compare them using DiffChecker THERE ARE some misplaced parts received in the client!
Diff link: https://www.diffchecker.com/W4DgQ7o7

As I told, I have no problem receiving this packet (BitStream actually) inside locally connected clients, I've checked the output, they are equal on locally connectors.

Also the packet from the server is sent using RELIABLE_ORDERED and HIGH_PRIORITY.

So, is it a limit? is it a bug? Should I enable something or use a function?

[0x416c69]

For a better explanation, I've reproduced this issue by writing the following code.
As you can see, all the packets are verified on localhost but it's a different story when the client is remotely connected!

/// ---------------------------------
/// ---------------------------------
/// ---------------------------------
/// ---------------------------------

#define _ITERATOR_DEBUG_LEVEL 0
#include <iostream>
#include <string>
#include <thread>
#include <include/slikenet/BitStream.h>
#include <MessageIdentifiers.h>
#include <RakPeerInterface.h>
#include "picosha2.h"

using namespace std;
using namespace SLNet;

using BYTE = unsigned char;
using uint = unsigned long;

const short             PacketsToSend                    = 32;
const unsigned short    sPort                            = 7777;
RakPeerInterface*       rpinterface;
string                  strChecksums[PacketsToSend];
bool                    bIsServer                        = false;
short                   sCount                           = 0;
SystemAddress           rsaAddr;

template<typename ... Args>
string string_format(const std::string& format, Args ... args)
{
    size_t size = snprintf(nullptr, 0, format.c_str(), args ...) + 1;
    unique_ptr<char[]> buf(new char[size]);
    snprintf(buf.get(), size, format.c_str(), args ...);
    return string(buf.get(), buf.get() + size - 1);
}

unsigned char GetPacketIdentifier(Packet *p)
{
    if(p == 0) return 255;

    if((unsigned char)p->data[0] == ID_TIMESTAMP)
    {
        RakAssert(p->length > sizeof(MessageID) + sizeof(Time));
        return (unsigned char)p->data[sizeof(MessageID) + sizeof(Time)];
    }
    else return (unsigned char)p->data[0];
}

void SendBackBufferChecksums()
{
    for(short i = 0; i < PacketsToSend; i++)
    {
        BitStream bs;
        bs.WriteCasted<BYTE>(ID_USER_PACKET_ENUM);
        bs.Write(i);
        bs.Write(strChecksums[i].c_str(), 64);

        rpinterface->Send(&bs, HIGH_PRIORITY, RELIABLE_ORDERED, 0, rsaAddr, false);
    }
}

void AppendBufferToFile(BYTE* pbBuf, uint ulSize)
{
    string strBuf;
    strBuf += string_format("Length: %lu-", ulSize);
    for(uint i = 0; i < ulSize; i++) strBuf += string_format("%02X", *(pbBuf + i));
    strBuf += "\n";
    FILE* fh;
    fopen_s(&fh, (bIsServer ? "serverbuf.txt" : "clientbuf.txt"), "ab");
    fwrite(&strBuf.at(0), 1, strBuf.length(), fh);
    fclose(fh);
    std::vector<unsigned char> hash(picosha2::k_digest_size);
    picosha2::hash256(strBuf.begin(), strBuf.end(), hash.begin(), hash.end(), strBuf.length() + 64);
    strChecksums[sCount++] = picosha2::bytes_to_hex_string(hash.begin(), hash.end());

    if(!bIsServer && sCount == PacketsToSend) SendBackBufferChecksums();
}

void packetthread()
{
    while(true)
    {
        for(Packet * p = rpinterface->Receive(); p; rpinterface->DeallocatePacket(p), p = rpinterface->Receive())
        {
            switch(GetPacketIdentifier(p))
            {
                case ID_USER_PACKET_ENUM:
                {
                    BitStream bs(p->data, p->length, false);
                    bs.IgnoreBytes(sizeof(BYTE)); // Ignore the packet identifier

                    if(!bIsServer)
                    {
                        uint iBytes;
                        bs.Read(iBytes);

                        unique_ptr<BYTE[]> bBuffer(new BYTE[iBytes]);
                        bs.ReadAlignedBytes(bBuffer.get(), iBytes);

                        AppendBufferToFile(bBuffer.get(), iBytes);
                    }
                    else
                    {
                        short sID;
                        bs.Read(sID);

                        char szHash[65] = {0};
                        bs.Read(szHash, 64);

                        if(strChecksums[sID] != szHash)
                        {
                            cout << "WARNING: Checksum check for packet #" << sID << " has failed." << endl;
                            cout << "Requested: " << strChecksums[sID] << ", Received: " << szHash << endl;
                        }
                        else cout << "Packet #" << sID << " is verified." << endl;

                        if(sID == PacketsToSend-1) cout << "End of checking" << endl;
                    }

                    break;
                }

                case ID_CONNECTION_REQUEST_ACCEPTED:
                {
                    rsaAddr = p->systemAddress;
                    
                    cout << "Connection to the server is established." << endl;

                    break;
                }

                case ID_NEW_INCOMING_CONNECTION:
                {
                    rsaAddr = p->systemAddress;

                    cout << "The client is now connected." << endl;

                    for(short i = 0; i < PacketsToSend; i++)
                    {
                        // Generate random sequence of bytes
                        uint iBytes = 150000 + (rand() % 512); // 146kB at least
                        unique_ptr<BYTE[]> bBuffer(new BYTE[iBytes]);
                        for(uint j = 0; j < iBytes; j++) bBuffer[j] = (rand() % 255);

                        BitStream bs;
                        bs.WriteCasted<BYTE>(ID_USER_PACKET_ENUM);
                        bs.Write(iBytes);
                        bs.WriteAlignedBytes(bBuffer.get(), iBytes);

                        AppendBufferToFile(bBuffer.get(), iBytes);

                        rpinterface->Send(&bs, HIGH_PRIORITY, RELIABLE_ORDERED, 0, rsaAddr, false);
                    }

                    break;
                }
            }
        }
    }
}

int main()
{
    cout << "Type 'server' to start the server or ip address to run the client" << endl;
    string strOut;
    cin >> strOut;
    rpinterface = RakPeerInterface::GetInstance();
    if(strOut == "server")
    {
        bIsServer = true;

        SocketDescriptor socketDescriptor(sPort, 0);
        socketDescriptor.socketFamily = AF_INET;

        rpinterface->Startup(1, &socketDescriptor, 1);
        rpinterface->SetMaximumIncomingConnections(1);
        rpinterface->SetOccasionalPing(true);

        rpinterface->SetTimeoutTime(10000, UNASSIGNED_SYSTEM_ADDRESS);

        cout << "Server is ready to accept connections!" << endl;
    }
    else
    {
        SocketDescriptor socketDescriptor(0, 0);
        socketDescriptor.socketFamily = AF_INET;

        rpinterface->Startup(65535, &socketDescriptor, 1);
        rpinterface->SetOccasionalPing(true);

        if((rpinterface->Connect(strOut.c_str(), sPort, NULL, 0) == CONNECTION_ATTEMPT_STARTED)) cout << "Connection attempt started" << endl;
    }
    std::thread recvthread(packetthread);
    recvthread.join();
    return 0;
}

Get picosha2.h from here: https://github.com/okdshin/PicoSHA2

[0x416c69]

larku@c15dd90
This commit fixes my problem, may I merge this commit using a Pull Request?

[Luke1410]

Great catch here 0x416c69 --- it looks indeed this commit points to a bug in RakNet which hasn't been fixed in SLikeNet yet. If you like, create a pull request for SLikeNet. I'd put it on top of the task list post the current 0.1.1 release and will see the fix going into 0.2.0 then.

If it's vital for you to have it in the current SLikeNet repository, pls let me know and I'll see to make it available to you right away.

(internal case number SLNET-177)

[0x416c69]

Oh thanks but I'm not in a hurry, just stick to your schedule, I've created the pull request, I've also improved that commit, it was a little bit messy.

[Luke1410]

Thanks great --- in this case, I'll tackle it directly after the 0.1.1 release as planned.

[Luke1410]

The pull request was merged and is expected to be shipped with SLikeNet 0.2.0. Thanks once more for your contribution.
Since the pullrequest contained additional changes, we split it up internally (additional case numbers: SLNET-180, SLNET-181).

We also did some architecture and code style related tweaks to the original pull request.

Note that while we did a first initial code/design review of the issue, we'll perform a more detailed review at a later time (though still planned to be done in time for SLikeNet 0.2.0). We'll also integrate your described test case in our automated test runners.

If any additional details come out of that review/test process, we'll update this ticket here.

[0x416c69]

@Luke1410 Hello!

I was wondering if this project is officially dead now.

This is a fairly critical bug which completely renders the split packet and big packet transmission unusable and the patch is still not included in the latest released stable version.

Not to mention that it's been 4~5 years now.

[thexkey]

Have you tried reaching him on Discord?

[0x416c69]

No, but the issue has been found and it's somewhat already patched but the patch was planned to be released on 0.2.0, I thought we'd get that in a year in the worst case scenario but now each time a new version is released we have to manually apply the patch which is somewhat bothersome and we're scared to add more patches as well which would effectively make us incompatible with the changes in the branch here.

I want to know if there will be no more updates so we could just maintain our own version of SLikeNet in our repository. I could be wrong and the SVN might still be getting the updates, I haven't checked the SVN.