Skip to content

Add a test harness, caching pins, and a baseline benchmark for the V2 generator (S1) - #8525

Merged
Aaronontheweb merged 1 commit into
devfrom
feature/serialization-v2-generator-harness
Sep 9, 2026
Merged

Aaronontheweb merged 1 commit into
devfrom
feature/serialization-v2-generator-harness

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Sep 9, 2026 •

Copy link
Copy Markdown
Member

Stack, bottom to top: S1 #8525 -> S2 #8526 -> S3 #8527 -> object elements #8528 -> S4 #8530 -> S5 #8532 -> S6 #8533 -> F1 #8534 -> F2 #8536 -> F3 #8537 -> S7 #8538. Each PR is one commit on top of the one below it. This PR is S1. It reopens #8521 with the same commit, now with every branch in this repository so GitHub can show the stack.

What changes

Nothing in what the generator emits. This PR adds the tools the later steps are measured with. The only edits to the generator project make twelve model classes internal instead of private and add a 48-line entry point that runs the existing extraction code, so tests can call it without a compiler driver.

What it adds

  • A shared test harness. Tests/Harness/GeneratorTestHarness.cs builds the set of referenced assemblies once per test process and offers three calls: run the generator, compile a helper assembly to reference, and run the generator twice across an edit. It reports which pipeline steps ran and why. The diagnostics and cross-assembly test files moved onto it without changing any test name or assertion. The diagnostics tests went from about 10 seconds to about 2.
  • A model snapshot. A test parses all 18 serializable types in the golden corpus and saves the extracted models with Verify. An extraction bug now shows up as a model diff, not only as changed generated text.
  • Five caching scenarios, pinned to today's behavior. Each one edits a file and records which generator steps ran again and which files were regenerated. Each carries a comment naming the later PR expected to improve it. A guard test fails if the number of tracked steps changes without this file changing too.
  • Two architecture guards. No model type may hold a Roslyn symbol, compilation, syntax node, or location, checked by reflection over every model type. Every model type must compare by content, not by reference. The first guard caught a symbol field planted on purpose during a negative test.
  • A baseline benchmark. Akka.Benchmarks/Serialization/SourceGeneratorBenchmarks.cs builds a synthetic project of 5 serializers with 100 messages each and measures a cold run plus three kinds of edit on a warm generator, with the memory diagnoser on.

Baseline numbers

Benchmark, 5 x 100 messages, short run Mean Allocated
Fresh driver, full corpus 75.3 ms 23.2 MB
Warm driver, one field renamed 114.8 ms 23.2 MB
Warm driver, comment edit inside a message 60.4 ms 7.8 MB
Warm driver, unrelated file edited 70.1 ms 7.8 MB

Two things stand out. Renaming one field costs more than a cold run, because the generator re-extracts every type and then regenerates every file as well. An edit that changes nothing still costs most of a cold run, because Roslyn re-runs the per-type step on any change and only the later collect step reuses a cached result.

Pinned scenarios, today

Scenario Per-type steps Collect step Files whose text changed
Unrelated file edited Unchanged Cached none
One field renamed 1 Modified, rest Unchanged Modified the owning serializer only
Comment added inside a message Unchanged Cached none
Keystroke in an unrelated file Unchanged Cached none
Metadata reference added Unchanged Cached none

S3 targets the rename row. S5 targets the no-change rows. S6 targets locations.

How it was checked

278 tests pass. Generator builds with warnings as errors. Akka.Benchmarks builds. No golden or wire snapshot changed.

@Aaronontheweb Aaronontheweb added serialization akka.net v1.6 Akka.NET v1.6-related issues tests labels Sep 9, 2026
@Aaronontheweb
Aaronontheweb added this pull request to stack #8531 September 9, 2026 01:53
@Aaronontheweb
Aaronontheweb force-pushed the feature/serialization-v2-generator-harness branch from 18f1728 to 217e16f Compare September 9, 2026 13:31
@Aaronontheweb Aaronontheweb changed the title Serialization.V2: generator test harness, internal models, incrementality scenarios, baseline benchmark (S1) Add a test harness, caching pins, and a baseline benchmark for the V2 generator (S1) Sep 9, 2026
@Aaronontheweb
Aaronontheweb force-pushed the feature/serialization-v2-generator-harness branch from 217e16f to f51e528 Compare September 9, 2026 17:14
…lity scenarios, baseline benchmark

Sets up the test infrastructure for the generator architecture pass: promotes the
pipeline's cached models to internal, adds a shared GeneratorTestHarness that builds
the metadata reference set once per process instead of once per test, migrates the
diagnostics and cross-assembly specs onto it, adds a ParseMessageForTests entry point
plus a Verify snapshot of the golden corpus's extracted models, pins today's
incrementality behavior across five edit scenarios, adds reflection-based structural
guards over the model types, and adds a baseline BenchmarkDotNet suite for the
generator's incremental driver.
@Aaronontheweb
Aaronontheweb force-pushed the feature/serialization-v2-generator-harness branch from f51e528 to 28e6987 Compare September 9, 2026 18:08
@Aaronontheweb
Aaronontheweb merged commit 846029a into dev Sep 9, 2026
16 checks passed
@Aaronontheweb
Aaronontheweb deleted the feature/serialization-v2-generator-harness branch September 9, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

akka.net v1.6 Akka.NET v1.6-related issues serialization tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant