Repository navigation
Add filtered binary log replay - #961
KirillOsenkov merged 3 commits into
Conversation
Allow callers to filter replayed event records using lightweight record metadata. Modern length-framed logs skip rejected event payloads while preserving auxiliary records and required culture initialization, and accepted events can be reserialized into compact standalone binlogs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 31bd9671-1d4c-4cd4-bd37-105fea073c45
Always complete and join the threaded replay queue when an event filter throws, and retain the CurrentUICulture preamble when filtering legacy pre-v18 binlogs. Add regressions for worker cleanup and legacy culture handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 31bd9671-1d4c-4cd4-bd37-105fea073c45
JanKrivanek
left a comment
There was a problem hiding this comment.
Looks good and helpful. I added couple questions
If BuildEventArgsReader.cs is still in sync with dotnet/msbuild (and it should) - we should reflect the change there as well
- document why SkipBytes cannot use Stream.Seek and reuse the shared StreamExtensions.SkipBytes helper - track the pre/post deserialization filters as non-null locals so the null-forgiving operator is no longer needed - surface BinaryLogEventFilterException instead of unwrapping it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724
|
@JanKrivanek re: keeping A few things differ there, driven by what msbuild's reader actually has:
|
|
I'm guessing there's no perf overhead when no filter is specified? and we don't care much about perf when the filter is specified? would be good to see some numbers for large-ish binlogs. |
Adds an opt-in BinaryLogEventFilter that decides which events a binlog replay deserializes and dispatches, so consumers that need only a subset of a large binary log no longer pay for deserializing every event. - BuildEventArgsReader.Read(BinaryLogEventFilter) reads the common fields, offers them to the filter as BinaryLogEventMetadata and skips the type specific payload of rejected records in length framed (v18+) logs - accepted records reuse the already read common fields instead of rereading them - legacy formats and TargetSkipped (whose original context lives in the type specific payload) fall back to filtering after deserialization - BinaryLogReplayEventSource.EventFilter forwards the filter and forces structured reading, since raw record passthrough would bypass the filter - filter callback exceptions are wrapped in BinaryLogEventFilterException so they abort the replay instead of being treated as recoverable read errors Mirrors KirillOsenkov/MSBuildStructuredLog#961 to keep BuildEventArgsReader.cs in sync between the two repos. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724
I measured the dotnet/msbuild port (#14693) by replaying binlog: call Replay() => subscribe to AnyEventRaised -> count events, and wrap the replay in GC.GetTotalAllocatedBytes(precise: true). There's a warmup pass first, and a forced GC before every iteration.
more info is here dotnet/msbuild#14693 |
Adds an opt-in BinaryLogEventFilter that decides which events a binlog replay deserializes and dispatches, so consumers that need only a subset of a large binary log no longer pay for deserializing every event. - BuildEventArgsReader.Read(BinaryLogEventFilter) reads the common fields, offers them to the filter as BinaryLogEventMetadata and skips the type specific payload of rejected records in length framed (v18+) logs - accepted records reuse the already read common fields instead of rereading them - legacy formats and TargetSkipped (whose original context lives in the type specific payload) fall back to filtering after deserialization - BinaryLogReplayEventSource.EventFilter forwards the filter and forces structured reading, since raw record passthrough would bypass the filter - filter callback exceptions are wrapped in BinaryLogEventFilterException so they abort the replay instead of being treated as recoverable read errors Mirrors KirillOsenkov/MSBuildStructuredLog#961 to keep BuildEventArgsReader.cs in sync between the two repos. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724
Adds an opt-in BinaryLogEventFilter that decides which events a binlog replay deserializes and dispatches, so consumers that need only a subset of a large binary log no longer pay for deserializing every event. - BuildEventArgsReader.Read(BinaryLogEventFilter) reads the common fields, offers them to the filter as BinaryLogEventMetadata and skips the type specific payload of rejected records in length framed (v18+) logs - accepted records reuse the already read common fields instead of rereading them - legacy formats and TargetSkipped (whose original context lives in the type specific payload) fall back to filtering after deserialization - BinaryLogReplayEventSource.EventFilter forwards the filter and forces structured reading, since raw record passthrough would bypass the filter - filter callback exceptions are wrapped in BinaryLogEventFilterException so they abort the replay instead of being treated as recoverable read errors Mirrors KirillOsenkov/MSBuildStructuredLog#961 to keep BuildEventArgsReader.cs in sync between the two repos. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724
Adds an opt-in BinaryLogEventFilter that decides which events a binlog replay deserializes and dispatches, so consumers that need only a subset of a large binary log no longer pay for deserializing every event. - BuildEventArgsReader.Read(BinaryLogEventFilter) reads the common fields, offers them to the filter as BinaryLogEventMetadata and skips the type specific payload of rejected records in length framed (v18+) logs - accepted records reuse the already read common fields instead of rereading them - legacy formats and TargetSkipped (whose original context lives in the type specific payload) fall back to filtering after deserialization - BinaryLogReplayEventSource.EventFilter forwards the filter and forces structured reading, since raw record passthrough would bypass the filter - filter callback exceptions are wrapped in BinaryLogEventFilterException so they abort the replay instead of being treated as recoverable read errors Mirrors KirillOsenkov/MSBuildStructuredLog#961 to keep BuildEventArgsReader.cs in sync between the two repos. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724
## Summary Adds opt-in `Exclude=kind[,kind...]` binary logger parameters for both live builds and replay, using the same pre-serialization filtering implementation. - **Logger-scoped filtering:** each `-bl` can have its own exclusions. Console output, file loggers, other binary loggers, and build success/failure are unaffected. Selection matches exact record kinds; lifecycle records are protected, and evaluation start/finish events must be excluded together. Retained BuildCheck diagnostic record kinds are preserved across rewrites, including in unfiltered companion outputs, so subsequent filtering keeps the same distinctions. - **Replay API:** `BinaryLogReplayEventSource.EventFilter` remains available for custom predicates based on record kind and build context. Unlike logger parameters, it filters all replay subscribers and can skip rejected payloads before deserialization in supported length-framed records. Legacy logs and `TargetSkipped` filter after deserialization. Callback failures abort replay; API forward-compatible recovery remains intact. Legacy exception serialization is supported only on .NET Framework for AppDomain compatibility. - **Replay hot paths:** exclusions use frozen sets and direct record-kind lookups. Cancellation is polled before the first read and at most every 100 event-record reads, including rejected events; CLI filtered replay also checks cancellation before publication. - **Safe CLI rewriting:** filtered replay requires new binary log destinations and supported input formats. Outputs are staged until replay and logger finalization succeed, with no-overwrite publication per file, not an all-output transaction. Ordinary builds retain normal binary logger overwrite behavior; direct `BinaryLogger` use does not provide the CLI's publication safeguards. - **Not redaction:** embedded project/import files are independent of event selection. Use `ProjectImports=None` to omit them. `ProjectImports=ZipFile` is supported for live builds, but not for the staged filtered-replay operation. ## Examples Filter a binlog while the original build runs: ```powershell dotnet build App.csproj "-bl:build.binlog;Exclude=Message,Warning;ProjectImports=None" ``` Apply the same logger parameters when rewriting an existing binlog: ```powershell dotnet msbuild input.binlog "-bl:filtered.binlog;Exclude=Message,Warning;ProjectImports=None" -noAutoResponse ``` ## Local validation Full Windows repository build and bootstrap smoke checks succeeded. Binary-logging and serialization tests passed on `net11.0` and `net472`, with 3 platform-specific skips. The CLI/filtering selection passed **290 tests** across both target frameworks. Coverage includes immutable exclusions, exact record-kind selection, preserved BuildCheck kinds and repeated filtering, current/legacy filtered replay, bounded cancellation and resume, .NET Framework exception round-tripping, independent output archives, evaluated/dynamic embedded files, and publication safeguards. Hosted CI results are tracked separately in the PR checks. Related: KirillOsenkov/MSBuildStructuredLog#961. See the [binary-log guide](https://github.com/YuliiaKovalova/msbuild/blob/dev/ykovalova/binlog-event-filter/documentation/wiki/Binary-Log.md#filtering-events-during-builds-and-replay) for supported exclusions, API examples, and replay restrictions. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a410b70-1a9f-4315-b4bf-30e0d7391724 Copilot-Session: e15e4366-f954-4905-9dac-b7c01e75682f Copilot-Session: fa109eec-c957-4fe5-9656-2290d48b7c30
Summary
BinLogReader.EventFiltercallback with record kind and build event context metadataTargetSkippedrecordsMotivation
Consumers that need only a subset of a large binary log currently deserialize every event. This API lets them reject records after reading lightweight common metadata, then connect accepted replay events to
BinaryLoggerto produce a compact, standalone filtered binlog.Validation
main