Add UEFI support

[xaionaro]

Just in case wondering: does it make sense to add support of UEFI files (using https://github.com/linuxboot/fiano)?

[wader]

Hi, i don't know much about UEFI. Is it similar to an executable format like ELF? haven't look close at the go package, for fq to be able to use it you probably have to have some kind of API that supports iterate thru strucutres or some way of accessing offsets/lengths where things are located.

[xaionaro]

Is it similar to an executable format like ELF?

It is basically a tree structure, where roughly speaking on the leafs it may contain PE32 or other files.

for fq to be able to use it you probably have to have some kind of API that supports iterate thru strucutres or some way of accessing offsets/lengths where things are located.

Yeah, this is exactly what fiano/pkg/uefi package is about (for UEFI layouts).

[wader]

Ok! Interesting, and PE32 here is a 32-bit portable executable (windows exe)?

Don't have much to look into it right now but it would be helpful for me or someone who wants to work on it to have some resources like small examples files or links to tools to create own files.

[xaionaro]

Interesting, and PE32 here is a 32-bit portable executable (windows exe)?

Yeaaaaah. I also was a bit shocked when found out they use EXE files inside UEFI images. Though it works different (from Windows) there, but the executable container is the same, yes.

Don't have much to look into it right now but it would be helpful for me or someone who wants to work on it to have some resources like small examples files or links to tools to create own files

At this stage I was just wondering if this makes sense, just to keep this opportunity in mind :)

[wader]

Feels like it could be a good fit. Also fq has infrastructure to do "subformat" decoding so once where is a PE decode (or other formats in UEFI) it could automatically do nested decoding. Im currently working on a much improved ELF decoder (https://github.com/wader/fq/tree/elf if you want to sneak peak) and i hope PE is not that much different, but i've heard it's quite legacy heavy :)

[tlehman]

I am working on adding MBR support for looking at the 512 byte section of bootable iso images. I'll look into UEFI next since that's the successor to the legacy MBR BIOS booting scheme.

[wader]

@tlehman Great to hear! Feel free to open issues or give me a shout if you have any questions

[wader]

BTW i've thought about adding x86, arm, etc ISA decoders to fq if that is interesting for MBR, UEFI etc?. Have some branch somewhere with an early prototype that add text section decode to the elf decoder and even do some symbol resolution

[tlehman]

Yes! MBR is 16bit x86 only. Since it's a special rigid format, it may even work to disassemble it and view the instructions. For my first pass, I just want to identify the regions used for the partition table, the post message and the other parts that the BIOS expects.

[wader]

Sounds like a good plan. A strategy i've used to decode formats i don't know much about it so try not decode to "deep" at first but instead try decode most size, length fields etc and add raw fields to begin with to verify and get overall feel of the format. After that start add depth and details.

I will try cleanup the prototype ISA branch so you can have a look. My idea is to have separate formats for ISAs so ex a x86_64, arm64, ... formats that can be used by other formats like elf, macho etc but also allow them to be use adhoc like fq '.path.to.interesting.bits | arm64' file. In the prototype the result was just an array of objects describing each instruction and the decoding is based on https://pkg.go.dev/golang.org/x/arch/

[wader]

@tlehman Here is a PR with a cleaned up version of my old branch #215 the symbol lookup stuff seems to be broken. I don't really remember if it ever worked. Probably need some rethinking and it's not unlikely i've misunderstood how it works.

If you want to work on it feel free to use it or totally redo it another way :)

[tlehman]

@wader I am having a hard time figuring out what's wrong in my branch, I added the format mbr.go, registered the format and started with a basic DecodeFn like raw. I am still getting "mbr: format group not found". I'm probably missing something basic, but I can't figure out what it is

[wader]

@tlehman Hi, i think what is missing is an import in format/all/all.go that is what causes init to be called so that the format gets registered.

The documentation is lacking a bit but there is some general help for adding a decoder here https://github.com/wader/fq/blob/master/doc/dev.md#implement-a-decoder and also some dev tips how to generate actual test output and documentation here https://github.com/wader/fq/blob/master/doc/dev.md#development-tips

[tlehman]

Awesome, thank you!

[wader]

👍

My usual workflow is to run/iterate with go run fq.go -d <format> dv some/file etc (-d <format> to get a result even on failure and dv to get stack trace and bit ranges). Test input files can be added to format/<format>/testdata/<name> and "fqtest" files as format/<format>/testdata/<name>.fqtest. The fqtest are kind of a transcript running fq but you only need to provide input expression(s) and then use WRITE_ACTUAL=1 go test ./format etc to overwrite the fqtest files with actual output added. If you run format tests without WRITE_ACTUAL set you will get failure and diff if the output does not match. I should probably document this better.

Also useful to do things like fq -n '[1,2,3,4] | mbr | dv' if you want to adhoc decode some binary (in this case a 4 byte binary with the bytes 1,2,3,4)

[tlehman]

Yeah I would love to see a "from 0 to 1, how to add a new binary format to fq". This comment goes most of the way, can you add this or link to it from the README?

[wader]

That is a good idea, i should come up with some format that can show most decode api concepts 🤔 maybe should be directly linked to from the readme for visibility