Repository navigation
feat(parser): add the EvtxECmd map engine #541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
87 commits
Select commit
Hold shift + click to select a range
8dd5c9f
feat(parser): add the EvtxECmd map engine
adamgell 23dca7b
fix(parser): join repeated elements, matching verified EvtxECmd behav…
adamgell 4fe3ab7
feat(event-log): load EvtxECmd .map files from disk
adamgell 6f819fd
fix(parser): render map templates in a single pass
adamgell 9ed4e3e
feat(parser): build XPath queries so the service does the filtering
adamgell 2976833
perf(event-log): filter in the service and query channels concurrently
adamgell c91d4fc
feat(event-log): give the live view a server-side time window
adamgell f0f2e94
feat(event-log): convert rendered event XML into the map engine's nod…
adamgell 5867298
fix(event-log): take a slice in the XML close helper
adamgell 80a57c2
feat(event-log): surface the System block fields every event carries
adamgell 462c976
feat(event-log): export the events on screen to CSV, TSV, JSON, or ev…
adamgell 4ec7066
feat(event-log): apply loaded maps to events and surface the columns
adamgell 99e810b
fix(parser): emit queries the Event Log service actually accepts
adamgell 9e448d0
fix(event-log): gate the new commands behind the event-log feature
adamgell 50fb050
fix(parser): quote by delimiter choice and budget by expressions, not…
adamgell ff7020c
test(event-log): exercise the live query path against a real service
adamgell cc36318
feat(parser): render event descriptions from captured provider metadata
adamgell 43b9bab
feat(event-log): read captured provider metadata from a provider data…
adamgell a233344
test(event-log): read a real provider database, not just synthetic fi…
adamgell 11f34b1
fix(parser): accept the signed integers the provider metadata actuall…
adamgell bd031f8
feat(event-log): render provider descriptions for events read from files
adamgell 41f6407
test(event-log): prove the description chain against a real database
adamgell 30f30e0
feat(event-log): group the event list by level, provider, channel, id…
adamgell 16d9ea3
feat(event-log): wire grouping into the list and add a group-by control
adamgell f1e82bd
feat(event-log): add a saved filter library
adamgell 212463e
feat(event-log): persist the filter library and apply saved filters f…
adamgell 5830712
feat(event-log): add a column model with visibility, order, and widths
adamgell 84a78bd
feat(event-log): render the list from the column config and add a cho…
adamgell d3dec44
feat(parser): merge Windows events and text logs into one timeline
adamgell c4805ea
feat(event-log): place events on the unified timeline and expose it o…
adamgell 3309070
feat(event-log): frontend model for the unified timeline
adamgell 8de9b26
feat(event-log): render the unified timeline
adamgell adfeb11
feat(event-log): decode .etl payloads and fix the dead file-open path
adamgell 70db7bd
test(event-log): cover the .evtx file path against a real capture
adamgell 4784b3f
fix(parser): give every split query node an Id and pin the structured…
adamgell f655a9e
fix(event-log): stop the map loader hiding gaps and depending on read…
adamgell 7ebe1f3
docs(eventmap): a format conversion cannot be byte-faithful
adamgell 1338145
fix(event-log): import the shared timestamp parser in the live path
adamgell 174767e
fix(event-log): explain what is missing from a parse instead of count…
adamgell f7bb7b0
feat(event-log): show what is missing from the loaded events
adamgell ee3ff9c
fix(event-log): show event times in one stated zone instead of three …
adamgell a35403b
feat(event-log): let map-derived values be columns, not just detail-p…
adamgell eac274e
fix(event-log): stop dropping fields from events that carry no Data e…
adamgell c9961b1
fix(parser): cost the time window correctly and record the real expre…
adamgell 9ce3e90
fix(event-log): stop misattributing values in rendered descriptions
adamgell 55b0294
refactor(event-log): own the map and provider registries in AppState
adamgell 033e8f5
fix(event-log): pass the registry in the live service tests
adamgell d6ffbbd
fix(event-log): remove a duplicated argument in two service tests
adamgell 8b3d950
fix(event-log): stop an oversized exclusion filter silently returning…
adamgell 722e79c
fix(parser): bound the terms that repeat in every node, not just the …
adamgell d44afcb
test(event-log): drive coverage gaps through the store, not just the …
adamgell 21a0c05
fix(event-log): label positional fields by slot, and use the one filt…
adamgell 9dae0fc
fix(event-log): send event origins over the wire with the keys the UI…
adamgell 79de6e2
test(event-log): give the timeline fixture distinguishable ids, and d…
adamgell cb2b13c
fix(event-log): apply the time window on refresh, and stop emitting o…
adamgell 48dce30
fix(event-log): close four saved-filter findings I had misread as alr…
adamgell a57ec6a
fix(event-log): stop a wide id range hanging the tab, and name classi…
adamgell 9133042
fix(event-log): export the columns the grid showed, and stop reopenin…
adamgell 6f988c6
fix(event-log): make saving and reordering work for macOS and keyboar…
adamgell 3cbfe51
fix(event-log): gate the event-log-only dependencies, and tighten wea…
adamgell 2965fbf
fix(parser): make provider metadata able to read its own output back
adamgell 270b557
perf(event-log): stop shipping every record's XML for a delimited export
adamgell 91273d8
fix(event-log): verify the reader's reply at the boundary, once
adamgell ba55802
fix(event-log): keep the empty live region in the accessibility tree
adamgell 20c01be
fix(event-log): refuse malformed value paths, and declare the grouped…
adamgell 6c546d5
fix(event-log): clear a selection collapse hides, and stop two tests …
adamgell d3d18dc
perf(eventmap): parse each map's paths once instead of once per record
adamgell 306a251
Merge remote-tracking branch 'origin/main' into feat/eventmap-engine
adamgell 01e1d3d
fix(clipboard): let the mirror tests wait for the write they assert
adamgell baaafc6
Merge remote-tracking branch 'origin/main' into feat/eventmap-engine
adamgell 4ad735d
fix(eventmap): make the memoized cache impossible to outlive its source
adamgell d72096c
docs(eventmap): say what compiled() actually does
adamgell 282b90c
fix(event-log): read the live path's fields from the parsed tree
adamgell 05ae283
test(event-log): add a timing harness for the live scan
adamgell 38f66be
fix(event-log): query channels one at a time, not all in one request
adamgell 8964b2d
docs(event-log): stop claiming a default event cap that does not exist
adamgell 5086b7e
test(event-log): account for where a scan's memory goes
adamgell fd6fdbf
fix(event-log): stop reporting a truncated channel as a complete one
adamgell 3c9cca1
refactor(event-log): move the fetch-failure decision where CI can tes…
adamgell 41b8ec6
feat(event-log): deliver a channel's records as they are read
adamgell f008ac9
fix(event-log): count render failures and slice previews by character
adamgell 6a47c0f
fix(event-log): build the file path's summary the way the live path does
adamgell 9b7c5a4
fix(event-log): guard channel processing and bound the sequence scan
adamgell 2de996e
fix(event-log): reject a --days value that overflows the time window
adamgell ee5af28
fix(event-log): report a channel whose refresh failed as a gap
adamgell 1a631b6
fix(event-log): count channels with gaps separately from gap entries
adamgell 4b2bbdb
fix(event-log): clear loadError on refresh and pin the failure state
adamgell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,337 @@ | ||
| //! Applying a map to an event. | ||
| //! | ||
| //! Resolution is deliberately non-fatal. Maps are written against a provider's superset of | ||
| //! fields, so an individual event legitimately omits some of them. A missing field is a coverage | ||
| //! state, reported on [`MappedValue::unresolved`], never an error and never silently blank. | ||
|
|
||
| use super::model::{EventMap, MapEntry, MapProperty}; | ||
| use super::node::EventNode; | ||
| use super::path::ValuePath; | ||
|
|
||
| /// One normalized column produced from an event. | ||
| #[derive(Debug, Clone, PartialEq, Eq)] | ||
| pub struct MappedValue { | ||
| /// The column this fills. | ||
| pub property: MapProperty, | ||
| /// The rendered text, with every resolved placeholder substituted. | ||
| pub text: String, | ||
| /// Variables whose paths matched nothing, in template order. | ||
| /// | ||
| /// Non-empty means `text` still contains their raw `%Name%` placeholders. Callers decide | ||
| /// whether to show the column, hide it, or flag it; the engine refuses to guess by blanking | ||
| /// the placeholder, which would present a partial value as a complete one. | ||
| pub unresolved: Vec<String>, | ||
| } | ||
|
|
||
| impl MappedValue { | ||
| /// True when every placeholder in the template resolved. | ||
| pub fn is_complete(&self) -> bool { | ||
| self.unresolved.is_empty() | ||
| } | ||
| } | ||
|
|
||
| /// The full result of applying a map to one event. | ||
| #[derive(Debug, Clone, PartialEq, Eq, Default)] | ||
| pub struct MappedEvent { | ||
| /// Columns in map order. | ||
| pub values: Vec<MappedValue>, | ||
| /// Path expressions that failed to parse, as `(variable, expression)`. | ||
| /// | ||
| /// A malformed expression is a defect in the map file rather than in the event, so it is | ||
| /// surfaced separately instead of being reported as a missing field. | ||
| pub invalid_paths: Vec<(String, String)>, | ||
| } | ||
|
|
||
| impl MappedEvent { | ||
| /// Returns the text written to `property`, if the map produced it. | ||
| pub fn value_for(&self, property: &MapProperty) -> Option<&str> { | ||
| self.values | ||
| .iter() | ||
| .find(|value| &value.property == property) | ||
| .map(|value| value.text.as_str()) | ||
| } | ||
| } | ||
|
|
||
| /// Applies every entry of `map` to `event`, which must be the `Event` element. | ||
| pub fn apply_map(map: &EventMap, event: &EventNode) -> MappedEvent { | ||
| let mut result = MappedEvent::default(); | ||
| for entry in &map.maps { | ||
| result | ||
| .values | ||
| .push(apply_entry(map, entry, event, &mut result.invalid_paths)); | ||
| } | ||
| result | ||
| } | ||
|
|
||
| fn apply_entry( | ||
| map: &EventMap, | ||
| entry: &MapEntry, | ||
| event: &EventNode, | ||
| invalid_paths: &mut Vec<(String, String)>, | ||
| ) -> MappedValue { | ||
| let mut text = entry.property_value.clone(); | ||
| let mut unresolved = Vec::new(); | ||
|
|
||
| for binding in &entry.values { | ||
| let placeholder = format!("%{}%", binding.name); | ||
| if !text.contains(&placeholder) { | ||
| continue; | ||
| } | ||
|
|
||
| let resolved = match ValuePath::parse(&binding.value) { | ||
| Ok(path) => path.evaluate(event).map(|value| value.into_owned()), | ||
| Err(_) => { | ||
| let defect = (binding.name.clone(), binding.value.clone()); | ||
| if !invalid_paths.contains(&defect) { | ||
| invalid_paths.push(defect); | ||
| } | ||
| None | ||
| } | ||
| }; | ||
|
|
||
| let translated = resolved.and_then(|raw| match map.lookup_for(&binding.name) { | ||
| Some(lookup) => lookup.translate(&raw), | ||
| None => Some(raw), | ||
| }); | ||
|
|
||
| match translated { | ||
| Some(value) => text = text.replace(&placeholder, &value), | ||
| None => unresolved.push(binding.name.clone()), | ||
| } | ||
| } | ||
|
|
||
| // A template may reference a variable the map never binds. That is still an unresolved | ||
| // placeholder from the reader's point of view, so report it the same way. | ||
| for name in remaining_placeholders(&text) { | ||
| if !unresolved.contains(&name) { | ||
| unresolved.push(name); | ||
| } | ||
| } | ||
|
|
||
| MappedValue { | ||
| property: entry.property.clone(), | ||
| text, | ||
| unresolved, | ||
| } | ||
| } | ||
|
|
||
| fn remaining_placeholders(text: &str) -> Vec<String> { | ||
| let mut names = Vec::new(); | ||
| let mut rest = text; | ||
| while let Some(start) = rest.find('%') { | ||
| let after = &rest[start + 1..]; | ||
| let Some(end) = after.find('%') else { break }; | ||
| let name = &after[..end]; | ||
| if !name.is_empty() && !name.contains(char::is_whitespace) { | ||
| names.push(name.to_string()); | ||
| } | ||
| rest = &after[end + 1..]; | ||
| } | ||
| names | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
| use crate::eventmap::model::{Lookup, MapEntry, ValueBinding}; | ||
| use std::collections::BTreeMap; | ||
|
|
||
| fn event() -> EventNode { | ||
| EventNode::new("Event").with_child( | ||
| EventNode::new("EventData") | ||
| .with_child( | ||
| EventNode::new("Data") | ||
| .with_attribute("Name", "SubjectUserName") | ||
| .with_text("adam"), | ||
| ) | ||
| .with_child( | ||
| EventNode::new("Data") | ||
| .with_attribute("Name", "SubjectDomainName") | ||
| .with_text("TEST"), | ||
| ) | ||
| .with_child( | ||
| EventNode::new("Data") | ||
| .with_attribute("Name", "BusType") | ||
| .with_text("7"), | ||
| ), | ||
| ) | ||
| } | ||
|
|
||
| fn binding(name: &str, field: &str) -> ValueBinding { | ||
| ValueBinding { | ||
| name: name.to_string(), | ||
| value: format!(r#"/Event/EventData/Data[@Name="{field}"]"#), | ||
| } | ||
| } | ||
|
|
||
| fn map_with(entries: Vec<MapEntry>, lookups: Vec<Lookup>) -> EventMap { | ||
| EventMap { | ||
| author: None, | ||
| description: None, | ||
| event_id: 4624, | ||
| channel: "Security".to_string(), | ||
| provider: "Microsoft-Windows-Security-Auditing".to_string(), | ||
| maps: entries, | ||
| lookups, | ||
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn substitutes_multiple_placeholders_in_one_template() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::UserName, | ||
| property_value: "%domain%\\%user%".to_string(), | ||
| values: vec![ | ||
| binding("domain", "SubjectDomainName"), | ||
| binding("user", "SubjectUserName"), | ||
| ], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!(mapped.value_for(&MapProperty::UserName), Some("TEST\\adam")); | ||
| assert!(mapped.values[0].is_complete()); | ||
| } | ||
|
|
||
| #[test] | ||
| fn missing_field_is_reported_and_leaves_the_placeholder_visible() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "LogonType %LogonType%".to_string(), | ||
| values: vec![binding("LogonType", "LogonType")], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| let value = &mapped.values[0]; | ||
| assert_eq!(value.unresolved, vec!["LogonType".to_string()]); | ||
| assert!(!value.is_complete()); | ||
| assert_eq!(value.text, "LogonType %LogonType%"); | ||
| } | ||
|
|
||
| #[test] | ||
| fn lookup_translates_a_bound_variable() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "Bus: %BusType%".to_string(), | ||
| values: vec![binding("BusType", "BusType")], | ||
| }], | ||
| vec![Lookup { | ||
| name: "BusType".to_string(), | ||
| default: Some("Unknown code".to_string()), | ||
| values: BTreeMap::from([("7".to_string(), "USB".to_string())]), | ||
| }], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!( | ||
| mapped.value_for(&MapProperty::PayloadData(1)), | ||
| Some("Bus: USB") | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn lookup_default_applies_to_an_unknown_code() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "Bus: %BusType%".to_string(), | ||
| values: vec![binding("BusType", "BusType")], | ||
| }], | ||
| vec![Lookup { | ||
| name: "BusType".to_string(), | ||
| default: Some("Unknown code".to_string()), | ||
| values: BTreeMap::new(), | ||
| }], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!( | ||
| mapped.value_for(&MapProperty::PayloadData(1)), | ||
| Some("Bus: Unknown code") | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn a_template_with_no_placeholders_is_emitted_verbatim() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "Screen saver invoked".to_string(), | ||
| values: vec![], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!( | ||
| mapped.value_for(&MapProperty::PayloadData(1)), | ||
| Some("Screen saver invoked") | ||
| ); | ||
| assert!(mapped.values[0].is_complete()); | ||
| } | ||
|
|
||
| #[test] | ||
| fn a_placeholder_with_no_binding_is_reported_unresolved() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "Value %NeverBound%".to_string(), | ||
| values: vec![], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!(mapped.values[0].unresolved, vec!["NeverBound".to_string()]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn a_malformed_path_is_a_map_defect_not_a_missing_field() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "%broken%".to_string(), | ||
| values: vec![ValueBinding { | ||
| name: "broken".to_string(), | ||
| value: "EventData/Data".to_string(), | ||
| }], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!( | ||
| mapped.invalid_paths, | ||
| vec![("broken".to_string(), "EventData/Data".to_string())] | ||
| ); | ||
| assert_eq!(mapped.values[0].unresolved, vec!["broken".to_string()]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn an_unused_binding_does_not_affect_the_result() { | ||
| let map = map_with( | ||
| vec![MapEntry { | ||
| property: MapProperty::PayloadData(1), | ||
| property_value: "User %user%".to_string(), | ||
| values: vec![ | ||
| binding("user", "SubjectUserName"), | ||
| binding("unused", "DoesNotExist"), | ||
| ], | ||
| }], | ||
| vec![], | ||
| ); | ||
|
|
||
| let mapped = apply_map(&map, &event()); | ||
| assert_eq!( | ||
| mapped.value_for(&MapProperty::PayloadData(1)), | ||
| Some("User adam") | ||
| ); | ||
| assert!(mapped.values[0].is_complete()); | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.