Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyRDP format #959

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ prores_frame,
[protobuf](doc/formats.md#protobuf),
protobuf_widevine,
pssh_playready,
[pyrdp](doc/formats.md#pyrdp),
[rtmp](doc/formats.md#rtmp),
sll2_packet,
sll_packet,
Expand Down
11 changes: 11 additions & 0 deletions doc/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@
|[`protobuf`](#protobuf) |Protobuf |<sub></sub>|
|`protobuf_widevine` |Widevine&nbsp;protobuf |<sub>`protobuf`</sub>|
|`pssh_playready` |PlayReady&nbsp;PSSH |<sub></sub>|
|[`pyrdp`](#pyrdp) |PyRDP&nbsp;Replay&nbsp;Files |<sub></sub>|
|[`rtmp`](#rtmp) |Real-Time&nbsp;Messaging&nbsp;Protocol |<sub>`amf0` `mpeg_asc`</sub>|
|`sll2_packet` |Linux&nbsp;cooked&nbsp;capture&nbsp;encapsulation&nbsp;v2 |<sub>`inet_packet`</sub>|
|`sll_packet` |Linux&nbsp;cooked&nbsp;capture&nbsp;encapsulation |<sub>`inet_packet`</sub>|
Expand Down Expand Up @@ -1195,6 +1196,16 @@ $ fq -d protobuf '.fields[6].wire_value | protobuf | d' file
### References
- https://developers.google.com/protocol-buffers/docs/encoding

## pyrdp
PyRDP Replay Files.

### Authors
- Olivier Bilodeau <[email protected]>, Maintainer
- Lisandro Ubiedo, Author

### References
- https://github.com/GoSecure/pyrdp

## rtmp
Real-Time Messaging Protocol.

Expand Down
1 change: 1 addition & 0 deletions format/all/all.fqtest
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ prores_frame Apple ProRes frame
protobuf Protobuf
protobuf_widevine Widevine protobuf
pssh_playready PlayReady PSSH
pyrdp PyRDP Replay Files
rtmp Real-Time Messaging Protocol
sll2_packet Linux cooked capture encapsulation v2
sll_packet Linux cooked capture encapsulation
Expand Down
1 change: 1 addition & 0 deletions format/all/all.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ import (
_ "github.com/wader/fq/format/postgres"
_ "github.com/wader/fq/format/prores"
_ "github.com/wader/fq/format/protobuf"
_ "github.com/wader/fq/format/pyrdp"
_ "github.com/wader/fq/format/riff"
_ "github.com/wader/fq/format/rtmp"
_ "github.com/wader/fq/format/tar"
Expand Down
1 change: 1 addition & 0 deletions format/format.go
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ var (
Protobuf = &decode.Group{Name: "protobuf"}
ProtobufWidevine = &decode.Group{Name: "protobuf_widevine"}
PSSH_Playready = &decode.Group{Name: "pssh_playready"}
PYRDP = &decode.Group{Name: "pyrdp"}
RTMP = &decode.Group{Name: "rtmp"}
SLL_Packet = &decode.Group{Name: "sll_packet"}
SLL2_Packet = &decode.Group{Name: "sll2_packet"}
Expand Down
6 changes: 5 additions & 1 deletion format/markdown/markdown.jq
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,11 @@ def _markdown_children_to_text($width):
| join("")
) as $text
| if $text == .destination then $text
else "\($text) (\(.destination))"
else
if .destination | startswith("mailto:") then
"<\(.destination[7:])>"
else "\($text) (\(.destination))"
end
end
)
elif .type == "code_block" then .literal | rtrimstr("\n") | split("\n") | " " + join("\n ")
Expand Down
109 changes: 109 additions & 0 deletions format/pyrdp/pdu/client_data.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
// Copyright (c) 2022-2023 GoSecure Inc.
// Copyright (c) 2024 Flare Systems
// Licensed under the MIT License
package pdu

import (
"github.com/wader/fq/pkg/decode"
"github.com/wader/fq/pkg/scalar"
)

const (
RDP4 = 0x80001
RDP5 = 0x80004
RDP10 = 0x80005
RDP10_1 = 0x80006
RDP10_2 = 0x80007
RDP10_3 = 0x80008
RDP10_4 = 0x80009
RDP10_5 = 0x8000a
RDP10_6 = 0x8000b
RDP10_7 = 0x8000c
RDP10_8 = 0x8000d
RDP10_9 = 0x8000e
RDP10_10 = 0x8000f
)

var RDPVersionMap = scalar.UintMapSymStr{
RDP4: "4",
RDP5: "5",
RDP10: "10",
RDP10_1: "10_1",
RDP10_2: "10_2",
RDP10_3: "10_3",
RDP10_4: "10_4",
RDP10_5: "10_5",
RDP10_6: "10_6",
RDP10_7: "10_7",
RDP10_8: "10_8",
RDP10_9: "10_9",
RDP10_10: "10_10",
}

const (
CLIENT_CORE = 0xc001
CLIENT_SECURITY = 0xc002
CLIENT_NETWORK = 0xc003
CLIENT_CLUSTER = 0xc004
)

var clientDataMap = scalar.UintMapSymStr{
CLIENT_CORE: "core",
CLIENT_SECURITY: "security",
CLIENT_NETWORK: "network",
CLIENT_CLUSTER: "cluster",
}

func parseClientData(d *decode.D, length int64) {
d.FieldStruct("client_data", func(d *decode.D) {
header := d.FieldU16("header", clientDataMap)
dataLen := int64(d.FieldU16("length") - 4)

switch header {
case CLIENT_CORE:
ParseClientDataCore(d, dataLen)
case CLIENT_SECURITY:
ParseClientDataSecurity(d, dataLen)
case CLIENT_NETWORK:
ParseClientDataNetwork(d, dataLen)
case CLIENT_CLUSTER:
ParseClientDataCluster(d, dataLen)
default:
// Assert() once all functions are implemented and tested.
d.FieldRawLen("data", dataLen*8)
return
}
})
}

func ParseClientDataCore(d *decode.D, length int64) {
d.FieldU32("version", RDPVersionMap)
d.FieldU16("desktop_width")
d.FieldU16("desktop_height")
d.FieldU16("color_depth")
d.FieldU16("sas_sequence")
d.FieldU32("keyboard_layout")
d.FieldU32("client_build")
d.FieldUTF16LE("client_name", 32, scalar.StrActualTrim("\x00"))
d.FieldU32("keyboard_type")
d.FieldU32("keyboard_sub_type")
d.FieldU32("keyboard_function_key")
d.FieldRawLen("ime_file_name", 64*8)
d.FieldRawLen("code_data", 98*8)
}

func ParseClientDataSecurity(d *decode.D, length int64) {
d.FieldU32("encryption_methods")
d.FieldU32("ext_encryption_methods")
}

func ParseClientDataNetwork(d *decode.D, length int64) {
d.FieldU32("channel_count")
length -= 4
d.FieldRawLen("channel_def_array", length*8)
}

func ParseClientDataCluster(d *decode.D, length int64) {
d.FieldU32("flags")
d.FieldU32("redirected_session_id")
}
117 changes: 117 additions & 0 deletions format/pyrdp/pdu/client_info.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// Copyright (c) 2022-2023 GoSecure Inc.
// Copyright (c) 2024 Flare Systems
// Licensed under the MIT License
package pdu

import (
"github.com/wader/fq/pkg/decode"
"github.com/wader/fq/pkg/scalar"
)

func parseClientInfo(d *decode.D, length int64) {
d.FieldStruct("client_info", func(d *decode.D) {
pos := d.Pos()
var (
isUnicode bool
hasNull bool
nullN uint64 = 0
unicodeN uint64 = 0
)
codePage := d.FieldU32("code_page")
flags := d.U32()
d.SeekRel(-4 * 8)
d.FieldStruct("flags", decodeFlagsFn)

isUnicode = ((flags & INFO_UNICODE) != 0)
hasNull = (codePage == 1252 || isUnicode)

if hasNull {
nullN = 1
}
if isUnicode {
unicodeN = 2
}

domainLength := int(d.FieldU16("domain_length") + nullN*unicodeN)
usernameLength := int(d.FieldU16("username_length") + nullN*unicodeN)
passwordLength := int(d.FieldU16("password_length") + nullN*unicodeN)
alternateShellLength := int(d.FieldU16("alternate_shell_length") + nullN*unicodeN)
workingDirLength := int(d.FieldU16("working_dir_length") + nullN*unicodeN)

d.FieldUTF16LE("domain", domainLength, scalar.StrActualTrim("\x00"))
d.FieldUTF16LE("username", usernameLength, scalar.StrActualTrim("\x00"))
d.FieldUTF16LE("password", passwordLength, scalar.StrActualTrim("\x00"))
d.FieldUTF16LE("alternate_shell", alternateShellLength, scalar.StrActualTrim("\x00"))
d.FieldUTF16LE("working_dir", workingDirLength, scalar.StrActualTrim("\x00"))

extraLength := length - ((d.Pos() - pos) / 8)
if extraLength > 0 {
d.FieldStruct("extra_info", func(d *decode.D) {
d.FieldU16("address_family", scalar.UintHex)
addressLength := int(d.FieldU16("address_length"))
d.FieldUTF16LE("address", addressLength, scalar.StrActualTrim("\x00"))
clientDirLength := int(d.FieldU16("client_dir_length"))
d.FieldUTF16LE("client_dir", clientDirLength, scalar.StrActualTrim("\x00"))
// TS_TIME_ZONE_INFORMATION structure
// https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpbcgr/526ed635-d7a9-4d3c-bbe1-4e3fb17585f4
d.FieldU32("timezone_bias")
d.FieldUTF16LE("timezone_standardname", 64, scalar.StrActualTrim("\x00"))
})

// XXX: there's more extra info but here's everything we need from the
// client (other than UTC info)
}
})
}

const (
// flags
INFO_MOUSE = 0x00000001
INFO_DISABLECTRLALTDEL = 0x00000002
INFO_AUTOLOGON = 0x00000008
INFO_UNICODE = 0x00000010
INFO_MAXIMIZESHELL = 0x00000020
INFO_LOGONNOTIFY = 0x00000040
INFO_COMPRESSION = 0x00000080
INFO_ENABLEWINDOWSKEY = 0x00000100
INFO_REMOTECONSOLEAUDIO = 0x00002000
INFO_FORCE_ENCRYPTED_CS_PDU = 0x00004000
INFO_RAIL = 0x00008000
INFO_LOGONERRORS = 0x00010000
INFO_MOUSE_HAS_WHEEL = 0x00020000
INFO_PASSWORD_IS_SC_PIN = 0x00040000
INFO_NOAUDIOPLAYBACK = 0x00080000
INFO_USING_SAVED_CREDS = 0x00100000
INFO_AUDIOCAPTURE = 0x00200000
INFO_VIDEO_DISABLE = 0x00400000
INFO_RESERVED1 = 0x00800000
INFO_RESERVED2 = 0x01000000
INFO_HIDEF_RAIL_SUPPORTED = 0x02000000
)

func decodeFlagsFn(d *decode.D) {
d.FieldBool("mouse")
d.FieldBool("disabledctrlaltdel")
d.FieldRawLen("unused0", 1)
d.FieldBool("autologon")
d.FieldBool("unicode")
d.FieldBool("maximizeshell")
d.FieldBool("logonnotify")
d.FieldBool("compression")
d.FieldBool("enablewindowskey")
d.FieldRawLen("unused1", 4)
d.FieldBool("remoteconsoleaudio")
d.FieldBool("force_encrypted_cs_pdu")
d.FieldBool("rail")
d.FieldBool("logonerrors")
d.FieldBool("mouse_has_wheel")
d.FieldBool("password_is_sc_pin")
d.FieldBool("noaudioplayback")
d.FieldBool("using_saved_creds")
d.FieldBool("audiocapture")
d.FieldBool("video_disable")
d.FieldBool("reserved1")
d.FieldBool("reserved2")
d.FieldBool("hidef_rail_supported")
d.FieldRawLen("unused2", 6)
}
80 changes: 80 additions & 0 deletions format/pyrdp/pdu/clipboard_data.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
// Copyright (c) 2022-2023 GoSecure Inc.
// Copyright (c) 2024 Flare Systems
// Licensed under the MIT License
package pdu

import (
"github.com/wader/fq/pkg/decode"
"github.com/wader/fq/pkg/scalar"
)

const (
// Message types.
CB_TYPE_MONITOR_READY = 0x0001
CB_TYPE_FORMAT_LIST = 0x0002
CB_TYPE_FORMAT_LIST_RESPONSE = 0x0003
CB_TYPE_FORMAT_DATA_REQUEST = 0x0004
CB_TYPE_FORMAT_DATA_RESPONSE = 0x0005
CB_TYPE_TEMP_DIRECTORY = 0x0006
CB_TYPE_CLIP_CAPS = 0x0007
CB_TYPE_FILECONTENTS_REQUEST = 0x0008
CB_TYPE_FILECONTENTS_RESPONSE = 0x0009
CB_TYPE_LOCK_CLIPDATA = 0x000a
CB_TYPE_UNLOCK_CLIPDATA = 0x000b
)

var cbTypesMap = scalar.UintMapSymStr{
CB_TYPE_MONITOR_READY: "monitor_ready",
CB_TYPE_FORMAT_LIST: "format_list",
CB_TYPE_FORMAT_LIST_RESPONSE: "format_list_response",
CB_TYPE_FORMAT_DATA_REQUEST: "format_data_request",
CB_TYPE_FORMAT_DATA_RESPONSE: "format_data_response",
CB_TYPE_TEMP_DIRECTORY: "temp_directory",
CB_TYPE_CLIP_CAPS: "clip_caps",
CB_TYPE_FILECONTENTS_REQUEST: "filecontents_request",
CB_TYPE_FILECONTENTS_RESPONSE: "filecontents_response",
CB_TYPE_LOCK_CLIPDATA: "lock_clipdata",
CB_TYPE_UNLOCK_CLIPDATA: "unlock_clipdata",
}

const (
// Message flags.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could multiple bit be set? cbFlagsMap will atm only map if one bit is set

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe multiple bits could be set, yes.

It's vague and I understand the original mistake since two out of three messages are OK or FAIL which don't seem compatible.

I'll fix it. I think I have a good idea for an approach.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So my initial approach to achieve this was to replicate what you did in 69ec44a. Naively I wrote this:

func parseClipboardData(d *decode.D, length int64) {
	d.FieldStruct("clipboard_data", func(d *decode.D) {
		msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
		d.FieldStruct("msg_flags", func(d *decode.D) {
			d.FieldBool("cb_response_ok")
			d.FieldBool("cb_response_fail")
			d.FieldBool("cb_ascii_names")
		})
		dataLength := d.FieldU32("data_len")

This is exactly how the protocol is documented and the UintMapSymStr we had before.

image

image

Enhancing the protocols' parsers would be straightforward. However, I realized that FieldStruct doesn't respect byte order and when I was wondering why, I realized well it can't since it doesn't know the "type" we are about to read...

So to work the parser now looks like this:

func parseClipboardData(d *decode.D, length int64) {
	d.FieldStruct("clipboard_data", func(d *decode.D) {
		msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
		d.FieldStruct("msg_flags", func(d *decode.D) {
			d.FieldRawLen("unused0", 5)
			d.FieldBool("cb_ascii_names")
			d.FieldBool("cb_response_fail")
			d.FieldBool("cb_response_ok")

			d.FieldRawLen("unused1", 8)
		})
		dataLength := d.FieldU32("data_len")

In addition to being harder to write, it also has a UX impact with the unnecessary unused fields being exposed to the user:

image

This is not how the protocol is described on the wire.

Now, trying to improve this, is there a way we could make FieldStruct or a variant of it byte order aware?

func parseClipboardData(d *decode.D, length int64) {
	d.FieldStruct("clipboard_data", func(d *decode.D) {
		msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
		d.FieldU16Bitfield("msg_flags", func(d *decode.D) {
			d.FieldBool("cb_response_ok")
			d.FieldBool("cb_response_fail")
			d.FieldBool("cb_ascii_names")
		})
		dataLength := d.FieldU32("data_len")

Using FieldU16Bitfield would under the hood parse the value as a uint16 in little-endian byte order and automatically jump after the 13 unused bits.

If this is too complicated to implement, I think the previous way (const list + UintMapSymStr) was better then. Both easier to write and easier to analyze as an analyst (no unnecessary unused fields). However, the bitfield instead of a map problem remains.

Otherwise, is there something else in the decode API I don't see or understand that would be better?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to work the parser now looks like this:

func parseClipboardData(d *decode.D, length int64) {
	d.FieldStruct("clipboard_data", func(d *decode.D) {
		msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
		d.FieldStruct("msg_flags", func(d *decode.D) {
			d.FieldRawLen("unused0", 5)
			d.FieldBool("cb_ascii_names")
			d.FieldBool("cb_response_fail")
			d.FieldBool("cb_response_ok")

			d.FieldRawLen("unused1", 8)
		})
		dataLength := d.FieldU32("data_len")

In addition to being harder to write, it also has a UX impact with the unnecessary unused fields being exposed to the user:

image

Yeap this gets a bit messy at times with how fq was designed to work, that is to not hide anything and also give "access" to all bits somehow even unused and unknown stuff.

So in general i've usually opted to let format decoders decode things as detailed as possible to have as few assumption what a end user would like to query and look at, e.g. maybe someone whats to extract or query the unused bits?

This is not how the protocol is described on the wire.

Now, trying to improve this, is there a way we could make FieldStruct or a variant of it byte order aware?

func parseClipboardData(d *decode.D, length int64) {
	d.FieldStruct("clipboard_data", func(d *decode.D) {
		msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
		d.FieldU16Bitfield("msg_flags", func(d *decode.D) {
			d.FieldBool("cb_response_ok")
			d.FieldBool("cb_response_fail")
			d.FieldBool("cb_ascii_names")
		})
		dataLength := d.FieldU32("data_len")

Using FieldU16Bitfield would under the hood parse the value as a uint16 in little-endian byte order and automatically jump after the 13 unused bits.

If this is too complicated to implement, I think the previous way (const list + UintMapSymStr) was better then. Both easier to write and easier to analyze as an analyst (no unnecessary unused fields). However, the bitfield instead of a map problem remains.

Otherwise, is there something else in the decode API I don't see or understand that would be better?

All decoding in fq currently is on purpose on bit level made to "hide" byte boundaries, this is to allow decoding bitstream formats, common for media codecs etc, without much fuzz. And as you have noticed this put some burden on some format decoders to byte align and maybe have to deal with padding, unknown or unused bits that a byte-oriented decoder would not care about, fq even automatically adds "gap" fields for bit ranges a decode skips or for some reason can't know about (trailing bits in a format with explicit ending), this is so that no bits are hidden/unreachable.

So in general if there is a clear bit field it's probably best to decode it they way you did above: a struct with bool/raw fields for each bit even unused or unknown ones. Will be a bit verbose but now all of them are accessible via a jq query. Btw if there bit flag combinations etc that have some special meaning a decode can add "synthetic" fields of any type that will not be "backed" by a bit range but will be visible and queryable (ex samples count in the mp3 decoder https://github.com/wader/fq/blob/master/format/mpeg/mp3_frame.go#L195)

But yeap all this becomes a bit messy and uninvited when decoding little endian bit fields that are > 1 byte :( currently there is no helpers for dealing with it but i have tried to come up with some way of specifying things in the same order as in a spec etc and the let fq figure things out. Actually in the kaitai prototype there is some support for what kaitai calls "bit-endian" that does more or less this. Sadly things get even messier when decoding some bit range inside a little endian integer as now a range a field can "span" two or more bytes and be non-continuous in the bitstream :(. Not sure how to visualise that in the "dump" tree, atm i think i would just opt to let the fiend be the range of the first and the last bit backing the field.

Sorry for wall of text! and hope i managed to explain myself properly and all this is a bit of an unsolved issue with fq so i'm glad someone has input on it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have made this definitely clearer. Having access to bit-sequence unused at the time the parser was written but that could be used later is a strong argument too. User-friendly before plugin programmer-friendly.

CB_FLAG_NONE = 0
CB_FLAG_RESPONSE_OK = 0x0001
CB_FLAG_RESPONSE_FAIL = 0x0002
CB_FLAG_ASCII_NAMES = 0x0004
)

var cbFlagsMap = scalar.UintMapSymStr{
CB_FLAG_NONE: "none",
CB_FLAG_RESPONSE_OK: "response_ok",
CB_FLAG_RESPONSE_FAIL: "response_fail",
CB_FLAG_ASCII_NAMES: "ascii_names",
}

var cbParseFnMap = map[uint16]interface{}{
CB_TYPE_FORMAT_DATA_RESPONSE: parseCbFormatDataResponse,
}

func parseClipboardData(d *decode.D, length int64) {
d.FieldStruct("clipboard_data", func(d *decode.D) {
msgType := uint16(d.FieldU16("msg_type", cbTypesMap))
d.FieldU16("msg_flags", cbFlagsMap)
dataLength := d.FieldU32("data_len")

cbParser, ok := cbParseFnMap[msgType]
if ok {
parseFn, ok := cbParser.(func(d *decode.D, length uint64))
if ok {
parseFn(d, dataLength)
return
}
}
// Assert() once all functions are implemented.
d.FieldRawLen("data", int64(dataLength*8))
})
}

func parseCbFormatDataResponse(d *decode.D, length uint64) {
d.FieldRawLen("data", int64(length*8))
}
Loading
Loading