-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PyRDP format #959
Open
obilodeau
wants to merge
18
commits into
wader:master
Choose a base branch
from
obilodeau:format-pyrdp
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
PyRDP format #959
Changes from 17 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
5d35eb6
Sending PyRDP parser upstream
obilodeau 392fabe
Clarified MIT license usage alongside copyright info
obilodeau 87633e6
refactor: Changed all FieldStrFn callers to FieldUTF16LE
obilodeau 3904c45
refactoring: UintMap to UintMapSymStr
obilodeau 051493d
refactor: package name aligned with directory name
obilodeau ec03f87
removed dead code
obilodeau fd85952
doc: added format to README
obilodeau dab2c6e
Resolved all linter errors
obilodeau 6ae10da
pyrdp: Add doc and move init func to top
wader a79f64f
help: Render mailto:addr as <addr>
wader 67bf55e
pyrdp: Use field description for formatted timestamp
wader a6b2468
pyrdp: camelCase vars and lowercase number literals
wader c59e7cc
pyrdp: Less symbol stuttering
wader 3986f20
pyrdp: Add unused fields to replace gap fields
wader 36d5cd4
pyrdp: Simplify always true condition
wader 86f28b6
pyrdp: Move pdu types into pdu package and some cleeanup
wader 92ad156
pyrdp: Reorganize pdu consts a bit
wader 69ec44a
pyrdp: Decode client info flags correctly
wader File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -110,6 +110,7 @@ | |
|[`protobuf`](#protobuf) |Protobuf |<sub></sub>| | ||
|`protobuf_widevine` |Widevine protobuf |<sub>`protobuf`</sub>| | ||
|`pssh_playready` |PlayReady PSSH |<sub></sub>| | ||
|[`pyrdp`](#pyrdp) |PyRDP Replay Files |<sub></sub>| | ||
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |<sub>`amf0` `mpeg_asc`</sub>| | ||
|`sll2_packet` |Linux cooked capture encapsulation v2 |<sub>`inet_packet`</sub>| | ||
|`sll_packet` |Linux cooked capture encapsulation |<sub>`inet_packet`</sub>| | ||
|
@@ -1195,6 +1196,16 @@ $ fq -d protobuf '.fields[6].wire_value | protobuf | d' file | |
### References | ||
- https://developers.google.com/protocol-buffers/docs/encoding | ||
|
||
## pyrdp | ||
PyRDP Replay Files. | ||
|
||
### Authors | ||
- Olivier Bilodeau <[email protected]>, Maintainer | ||
- Lisandro Ubiedo, Author | ||
|
||
### References | ||
- https://github.com/GoSecure/pyrdp | ||
|
||
## rtmp | ||
Real-Time Messaging Protocol. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
// Copyright (c) 2022-2023 GoSecure Inc. | ||
// Copyright (c) 2024 Flare Systems | ||
// Licensed under the MIT License | ||
package pdu | ||
|
||
import ( | ||
"github.com/wader/fq/pkg/decode" | ||
"github.com/wader/fq/pkg/scalar" | ||
) | ||
|
||
const ( | ||
RDP4 = 0x80001 | ||
RDP5 = 0x80004 | ||
RDP10 = 0x80005 | ||
RDP10_1 = 0x80006 | ||
RDP10_2 = 0x80007 | ||
RDP10_3 = 0x80008 | ||
RDP10_4 = 0x80009 | ||
RDP10_5 = 0x8000a | ||
RDP10_6 = 0x8000b | ||
RDP10_7 = 0x8000c | ||
RDP10_8 = 0x8000d | ||
RDP10_9 = 0x8000e | ||
RDP10_10 = 0x8000f | ||
) | ||
|
||
var RDPVersionMap = scalar.UintMapSymStr{ | ||
RDP4: "4", | ||
RDP5: "5", | ||
RDP10: "10", | ||
RDP10_1: "10_1", | ||
RDP10_2: "10_2", | ||
RDP10_3: "10_3", | ||
RDP10_4: "10_4", | ||
RDP10_5: "10_5", | ||
RDP10_6: "10_6", | ||
RDP10_7: "10_7", | ||
RDP10_8: "10_8", | ||
RDP10_9: "10_9", | ||
RDP10_10: "10_10", | ||
} | ||
|
||
const ( | ||
CLIENT_CORE = 0xc001 | ||
CLIENT_SECURITY = 0xc002 | ||
CLIENT_NETWORK = 0xc003 | ||
CLIENT_CLUSTER = 0xc004 | ||
) | ||
|
||
var clientDataMap = scalar.UintMapSymStr{ | ||
CLIENT_CORE: "core", | ||
CLIENT_SECURITY: "security", | ||
CLIENT_NETWORK: "network", | ||
CLIENT_CLUSTER: "cluster", | ||
} | ||
|
||
func parseClientData(d *decode.D, length int64) { | ||
d.FieldStruct("client_data", func(d *decode.D) { | ||
header := d.FieldU16("header", clientDataMap) | ||
dataLen := int64(d.FieldU16("length") - 4) | ||
|
||
switch header { | ||
case CLIENT_CORE: | ||
ParseClientDataCore(d, dataLen) | ||
case CLIENT_SECURITY: | ||
ParseClientDataSecurity(d, dataLen) | ||
case CLIENT_NETWORK: | ||
ParseClientDataNetwork(d, dataLen) | ||
case CLIENT_CLUSTER: | ||
ParseClientDataCluster(d, dataLen) | ||
default: | ||
// Assert() once all functions are implemented and tested. | ||
d.FieldRawLen("data", dataLen*8) | ||
return | ||
} | ||
}) | ||
} | ||
|
||
func ParseClientDataCore(d *decode.D, length int64) { | ||
d.FieldU32("version", RDPVersionMap) | ||
d.FieldU16("desktop_width") | ||
d.FieldU16("desktop_height") | ||
d.FieldU16("color_depth") | ||
d.FieldU16("sas_sequence") | ||
d.FieldU32("keyboard_layout") | ||
d.FieldU32("client_build") | ||
d.FieldUTF16LE("client_name", 32, scalar.StrActualTrim("\x00")) | ||
d.FieldU32("keyboard_type") | ||
d.FieldU32("keyboard_sub_type") | ||
d.FieldU32("keyboard_function_key") | ||
d.FieldRawLen("ime_file_name", 64*8) | ||
d.FieldRawLen("code_data", 98*8) | ||
} | ||
|
||
func ParseClientDataSecurity(d *decode.D, length int64) { | ||
d.FieldU32("encryption_methods") | ||
d.FieldU32("ext_encryption_methods") | ||
} | ||
|
||
func ParseClientDataNetwork(d *decode.D, length int64) { | ||
d.FieldU32("channel_count") | ||
length -= 4 | ||
d.FieldRawLen("channel_def_array", length*8) | ||
} | ||
|
||
func ParseClientDataCluster(d *decode.D, length int64) { | ||
d.FieldU32("flags") | ||
d.FieldU32("redirected_session_id") | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
// Copyright (c) 2022-2023 GoSecure Inc. | ||
// Copyright (c) 2024 Flare Systems | ||
// Licensed under the MIT License | ||
package pdu | ||
|
||
import ( | ||
"github.com/wader/fq/pkg/decode" | ||
"github.com/wader/fq/pkg/scalar" | ||
) | ||
|
||
func parseClientInfo(d *decode.D, length int64) { | ||
d.FieldStruct("client_info", func(d *decode.D) { | ||
pos := d.Pos() | ||
var ( | ||
isUnicode bool | ||
hasNull bool | ||
nullN uint64 = 0 | ||
unicodeN uint64 = 0 | ||
) | ||
codePage := d.FieldU32("code_page") | ||
flags := d.U32() | ||
d.SeekRel(-4 * 8) | ||
d.FieldStruct("flags", decodeFlagsFn) | ||
|
||
isUnicode = ((flags & INFO_UNICODE) != 0) | ||
hasNull = (codePage == 1252 || isUnicode) | ||
|
||
if hasNull { | ||
nullN = 1 | ||
} | ||
if isUnicode { | ||
unicodeN = 2 | ||
} | ||
|
||
domainLength := int(d.FieldU16("domain_length") + nullN*unicodeN) | ||
usernameLength := int(d.FieldU16("username_length") + nullN*unicodeN) | ||
passwordLength := int(d.FieldU16("password_length") + nullN*unicodeN) | ||
alternateShellLength := int(d.FieldU16("alternate_shell_length") + nullN*unicodeN) | ||
workingDirLength := int(d.FieldU16("working_dir_length") + nullN*unicodeN) | ||
|
||
d.FieldUTF16LE("domain", domainLength, scalar.StrActualTrim("\x00")) | ||
d.FieldUTF16LE("username", usernameLength, scalar.StrActualTrim("\x00")) | ||
d.FieldUTF16LE("password", passwordLength, scalar.StrActualTrim("\x00")) | ||
d.FieldUTF16LE("alternate_shell", alternateShellLength, scalar.StrActualTrim("\x00")) | ||
d.FieldUTF16LE("working_dir", workingDirLength, scalar.StrActualTrim("\x00")) | ||
|
||
extraLength := length - ((d.Pos() - pos) / 8) | ||
if extraLength > 0 { | ||
d.FieldStruct("extra_info", func(d *decode.D) { | ||
d.FieldU16("address_family", scalar.UintHex) | ||
addressLength := int(d.FieldU16("address_length")) | ||
d.FieldUTF16LE("address", addressLength, scalar.StrActualTrim("\x00")) | ||
clientDirLength := int(d.FieldU16("client_dir_length")) | ||
d.FieldUTF16LE("client_dir", clientDirLength, scalar.StrActualTrim("\x00")) | ||
// TS_TIME_ZONE_INFORMATION structure | ||
// https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpbcgr/526ed635-d7a9-4d3c-bbe1-4e3fb17585f4 | ||
d.FieldU32("timezone_bias") | ||
d.FieldUTF16LE("timezone_standardname", 64, scalar.StrActualTrim("\x00")) | ||
}) | ||
|
||
// XXX: there's more extra info but here's everything we need from the | ||
// client (other than UTC info) | ||
} | ||
}) | ||
} | ||
|
||
const ( | ||
// flags | ||
INFO_MOUSE = 0x00000001 | ||
INFO_DISABLECTRLALTDEL = 0x00000002 | ||
INFO_AUTOLOGON = 0x00000008 | ||
INFO_UNICODE = 0x00000010 | ||
INFO_MAXIMIZESHELL = 0x00000020 | ||
INFO_LOGONNOTIFY = 0x00000040 | ||
INFO_COMPRESSION = 0x00000080 | ||
INFO_ENABLEWINDOWSKEY = 0x00000100 | ||
INFO_REMOTECONSOLEAUDIO = 0x00002000 | ||
INFO_FORCE_ENCRYPTED_CS_PDU = 0x00004000 | ||
INFO_RAIL = 0x00008000 | ||
INFO_LOGONERRORS = 0x00010000 | ||
INFO_MOUSE_HAS_WHEEL = 0x00020000 | ||
INFO_PASSWORD_IS_SC_PIN = 0x00040000 | ||
INFO_NOAUDIOPLAYBACK = 0x00080000 | ||
INFO_USING_SAVED_CREDS = 0x00100000 | ||
INFO_AUDIOCAPTURE = 0x00200000 | ||
INFO_VIDEO_DISABLE = 0x00400000 | ||
INFO_RESERVED1 = 0x00800000 | ||
INFO_RESERVED2 = 0x01000000 | ||
INFO_HIDEF_RAIL_SUPPORTED = 0x02000000 | ||
) | ||
|
||
func decodeFlagsFn(d *decode.D) { | ||
d.FieldBool("mouse") | ||
d.FieldBool("disabledctrlaltdel") | ||
d.FieldRawLen("unused0", 1) | ||
d.FieldBool("autologon") | ||
d.FieldBool("unicode") | ||
d.FieldBool("maximizeshell") | ||
d.FieldBool("logonnotify") | ||
d.FieldBool("compression") | ||
d.FieldBool("enablewindowskey") | ||
d.FieldRawLen("unused1", 4) | ||
d.FieldBool("remoteconsoleaudio") | ||
d.FieldBool("force_encrypted_cs_pdu") | ||
d.FieldBool("rail") | ||
d.FieldBool("logonerrors") | ||
d.FieldBool("mouse_has_wheel") | ||
d.FieldBool("password_is_sc_pin") | ||
d.FieldBool("noaudioplayback") | ||
d.FieldBool("using_saved_creds") | ||
d.FieldBool("audiocapture") | ||
d.FieldBool("video_disable") | ||
d.FieldBool("reserved1") | ||
d.FieldBool("reserved2") | ||
d.FieldBool("hidef_rail_supported") | ||
d.FieldRawLen("unused2", 6) | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
// Copyright (c) 2022-2023 GoSecure Inc. | ||
// Copyright (c) 2024 Flare Systems | ||
// Licensed under the MIT License | ||
package pdu | ||
|
||
import ( | ||
"github.com/wader/fq/pkg/decode" | ||
"github.com/wader/fq/pkg/scalar" | ||
) | ||
|
||
const ( | ||
// Message types. | ||
CB_TYPE_MONITOR_READY = 0x0001 | ||
CB_TYPE_FORMAT_LIST = 0x0002 | ||
CB_TYPE_FORMAT_LIST_RESPONSE = 0x0003 | ||
CB_TYPE_FORMAT_DATA_REQUEST = 0x0004 | ||
CB_TYPE_FORMAT_DATA_RESPONSE = 0x0005 | ||
CB_TYPE_TEMP_DIRECTORY = 0x0006 | ||
CB_TYPE_CLIP_CAPS = 0x0007 | ||
CB_TYPE_FILECONTENTS_REQUEST = 0x0008 | ||
CB_TYPE_FILECONTENTS_RESPONSE = 0x0009 | ||
CB_TYPE_LOCK_CLIPDATA = 0x000a | ||
CB_TYPE_UNLOCK_CLIPDATA = 0x000b | ||
) | ||
|
||
var cbTypesMap = scalar.UintMapSymStr{ | ||
CB_TYPE_MONITOR_READY: "monitor_ready", | ||
CB_TYPE_FORMAT_LIST: "format_list", | ||
CB_TYPE_FORMAT_LIST_RESPONSE: "format_list_response", | ||
CB_TYPE_FORMAT_DATA_REQUEST: "format_data_request", | ||
CB_TYPE_FORMAT_DATA_RESPONSE: "format_data_response", | ||
CB_TYPE_TEMP_DIRECTORY: "temp_directory", | ||
CB_TYPE_CLIP_CAPS: "clip_caps", | ||
CB_TYPE_FILECONTENTS_REQUEST: "filecontents_request", | ||
CB_TYPE_FILECONTENTS_RESPONSE: "filecontents_response", | ||
CB_TYPE_LOCK_CLIPDATA: "lock_clipdata", | ||
CB_TYPE_UNLOCK_CLIPDATA: "unlock_clipdata", | ||
} | ||
|
||
const ( | ||
// Message flags. | ||
CB_FLAG_NONE = 0 | ||
CB_FLAG_RESPONSE_OK = 0x0001 | ||
CB_FLAG_RESPONSE_FAIL = 0x0002 | ||
CB_FLAG_ASCII_NAMES = 0x0004 | ||
) | ||
|
||
var cbFlagsMap = scalar.UintMapSymStr{ | ||
CB_FLAG_NONE: "none", | ||
CB_FLAG_RESPONSE_OK: "response_ok", | ||
CB_FLAG_RESPONSE_FAIL: "response_fail", | ||
CB_FLAG_ASCII_NAMES: "ascii_names", | ||
} | ||
|
||
var cbParseFnMap = map[uint16]interface{}{ | ||
CB_TYPE_FORMAT_DATA_RESPONSE: parseCbFormatDataResponse, | ||
} | ||
|
||
func parseClipboardData(d *decode.D, length int64) { | ||
d.FieldStruct("clipboard_data", func(d *decode.D) { | ||
msgType := uint16(d.FieldU16("msg_type", cbTypesMap)) | ||
d.FieldU16("msg_flags", cbFlagsMap) | ||
dataLength := d.FieldU32("data_len") | ||
|
||
cbParser, ok := cbParseFnMap[msgType] | ||
if ok { | ||
parseFn, ok := cbParser.(func(d *decode.D, length uint64)) | ||
if ok { | ||
parseFn(d, dataLength) | ||
return | ||
} | ||
} | ||
// Assert() once all functions are implemented. | ||
d.FieldRawLen("data", int64(dataLength*8)) | ||
}) | ||
} | ||
|
||
func parseCbFormatDataResponse(d *decode.D, length uint64) { | ||
d.FieldRawLen("data", int64(length*8)) | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could multiple bit be set? cbFlagsMap will atm only map if one bit is set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe multiple bits could be set, yes.
It's vague and I understand the original mistake since two out of three messages are OK or FAIL which don't seem compatible.
I'll fix it. I think I have a good idea for an approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So my initial approach to achieve this was to replicate what you did in 69ec44a. Naively I wrote this:
This is exactly how the protocol is documented and the
UintMapSymStr
we had before.Enhancing the protocols' parsers would be straightforward. However, I realized that
FieldStruct
doesn't respect byte order and when I was wondering why, I realized well it can't since it doesn't know the "type" we are about to read...So to work the parser now looks like this:
In addition to being harder to write, it also has a UX impact with the unnecessary unused fields being exposed to the user:
This is not how the protocol is described on the wire.
Now, trying to improve this, is there a way we could make FieldStruct or a variant of it byte order aware?
Using
FieldU16Bitfield
would under the hood parse the value as a uint16 in little-endian byte order and automatically jump after the 13 unused bits.If this is too complicated to implement, I think the previous way (const list +
UintMapSymStr
) was better then. Both easier to write and easier to analyze as an analyst (no unnecessary unused fields). However, the bitfield instead of a map problem remains.Otherwise, is there something else in the decode API I don't see or understand that would be better?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeap this gets a bit messy at times with how fq was designed to work, that is to not hide anything and also give "access" to all bits somehow even unused and unknown stuff.
So in general i've usually opted to let format decoders decode things as detailed as possible to have as few assumption what a end user would like to query and look at, e.g. maybe someone whats to extract or query the unused bits?
All decoding in fq currently is on purpose on bit level made to "hide" byte boundaries, this is to allow decoding bitstream formats, common for media codecs etc, without much fuzz. And as you have noticed this put some burden on some format decoders to byte align and maybe have to deal with padding, unknown or unused bits that a byte-oriented decoder would not care about, fq even automatically adds "gap" fields for bit ranges a decode skips or for some reason can't know about (trailing bits in a format with explicit ending), this is so that no bits are hidden/unreachable.
So in general if there is a clear bit field it's probably best to decode it they way you did above: a struct with bool/raw fields for each bit even unused or unknown ones. Will be a bit verbose but now all of them are accessible via a jq query. Btw if there bit flag combinations etc that have some special meaning a decode can add "synthetic" fields of any type that will not be "backed" by a bit range but will be visible and queryable (ex samples count in the mp3 decoder https://github.com/wader/fq/blob/master/format/mpeg/mp3_frame.go#L195)
But yeap all this becomes a bit messy and uninvited when decoding little endian bit fields that are > 1 byte :( currently there is no helpers for dealing with it but i have tried to come up with some way of specifying things in the same order as in a spec etc and the let fq figure things out. Actually in the kaitai prototype there is some support for what kaitai calls "bit-endian" that does more or less this. Sadly things get even messier when decoding some bit range inside a little endian integer as now a range a field can "span" two or more bytes and be non-continuous in the bitstream :(. Not sure how to visualise that in the "dump" tree, atm i think i would just opt to let the fiend be the range of the first and the last bit backing the field.
Sorry for wall of text! and hope i managed to explain myself properly and all this is a bit of an unsolved issue with fq so i'm glad someone has input on it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have made this definitely clearer. Having access to bit-sequence unused at the time the parser was written but that could be used later is a strong argument too. User-friendly before plugin programmer-friendly.