WIP: packet parsing

[aronwk-aaron]

idk, maybe this will be useful

[lcdr]

• please break this up into individual files with a hierarchy of general packets importing specific ones, similar to how the file format ksys import the common ksy, in general try to follow the hierarchy of lu_packets

• packets which only include "todo" do not add value and only increase the already high linecount of this PR, just remove them until they are documented

[aronwk-aaron]

I do not intend to complete this tbh.

For point one: Kaitai imports are broken in most places and tools and it's horrible to actually use to develop this, which is why it is like this currently.

For point two: I wanted to actually get this mostly completed before putting this up BUT kaitai has issues with non-byte aligned data.

It can read a bit, but then you are unable to read any data aside from more bits unaligned. i.e. with 33 bits, read a bit, and then read a float. What should happen is you get a bit, and then the float uses the remaining 32 bits
what actually happens is you read a bit, and then kaitai aligns the data to the next byte and then reads the float.

visually this is the result
you want the bottom, but you actually get the top
the 7 of course are unused and skipped
and because the amount of data you have is on the bottom, you have an overrun

Shamlessly stole most of this explanation and glorious MSPaint from @EmosewaMC

Felt bad about throwing it away, so wanted to put it out here and document some of the pitfalls with kaitia

[generalmimon]

@aronwk-aaron:

For point one: Kaitai imports are broken in most places and tools and it's horrible to actually use to develop this, which is why it is like this currently.

FYI, there have been a lot of fixes to imports recently, so it's very likely that you wouldn't consider them broken anymore. Unfortunately, these fixes haven't been released yet, they only live on the main branch of https://github.com/kaitai-io/kaitai_struct_compiler (so you would have to build the compiler from source, which isn't hard to do, but I understand most people don't want to).

As for "broken in (...) and tools", could you be more specific? I can think of kaitai-io/kaitai_struct_webide#172, which doesn't mean that you can't use imports in the Web IDE, but it's more of a "broken GUI" problem (more specifically, the fact that the "Create Folder" button is there even though it doesn't actually work). Otherwise, I think flat imports (without any slashes) should work fine at https://ide.kaitai.io/devel/.

As for https://github.com/kaitai-io/kaitai_struct_visualizer, I don't remember any problems with imports there.

i.e. with 33 bits, read a bit, and then read a float. What should happen is you get a bit, and then the float uses the remaining 32 bits

Interesting, until now I haven't heard of a format that needs to parse unaligned floats from any bit position. Thanks for mentioning it in kaitai-io/kaitai_struct#576 (comment), when the time comes to implement it, at least we'll know it has real practical use, which is always good to know.

[aronwk-aaron]

Hey @generalmimon I can explain more of the issues I had with imports
@EmosewaMC can explain some more details about the alignment roadblocks we had, basically we need unaligned anything, a true bitstream.

That's great to hear that imports have been improved! By places and tools, I mainly meant tools. Like the webIDE, which I was trying to use folders so that explains one issue.
And what I was looking for in a tool was a visualizer and editor in one, like how the webIDE works.

The vscode plugin only works with absolute imports, which impairs portability: fudgepop01/kaitai_ide_vscode#33

And I did open an issue with Hobbits awhile ago since their visualizer also has the same issue with relative imports: Mahlet-Inc/hobbits#168
And it looks like I figured out the issue with the webIDE "folder" and imports but didn't fully understand it back then.

After working around the import issues, and realizing the alignment roadblocks we have. I was a bit exhausted at the whole thing, so apologies if I blew it out of proportion lol.
When it does work, it's great. Especially for more visual learners like myself, and it helps us verify that we are reverse engineering the networking protocols correctly since we have packet captures from the live game in addition to ones that we can recreate on our own now.

[EmosewaMC]

If examples of bitstreams are needed I can provide serialize and deserialize of them as examples but for the most part its needing to read single bits at a time since bits were used for data compression in this game (lets say you had to send a 64 bit value over the wire, but you have an already established default on both ends, you can just write 1 bit (a 0) instead of the value).

I do understand bitstreams are far more uncommon but personally i was really eager to use Kaitai and we really want to use it for data in the game messages if bitstreams are implemented. If they are please give us a shout, we'd love to give it a go.