-
Notifications
You must be signed in to change notification settings - Fork 1
backlog: file B-0182 — filter formal verification to Linux only (Aaron 2026-05-03) #1405
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
AceHack
merged 3 commits into
main
from
backlog/b0182-linux-only-formal-verification-tests-aaron-2026-05-03
May 3, 2026
Merged
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
104 changes: 104 additions & 0 deletions
104
...x-only-formal-verification-tests-pure-math-no-os-difference-aaron-2026-05-03.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
| id: B-0182 | ||
| priority: P2 | ||
| status: open | ||
| title: Filter formal-verification tests to standard Linux only — TLC + Alloy + Lean are pure-math computation; no OS-specific behavior | ||
| tier: ci-architecture | ||
| effort: M | ||
| ask: Aaron 2026-05-03 — *"we don't have to run formal verifical on all the oses, we can run that just on linux it does not have differents between OS like scirpt and code that touch the enviroment. just the standard linux too don't need it to run on slim"* | ||
| created: 2026-05-03 | ||
| last_updated: 2026-05-03 | ||
| depends_on: [] | ||
| composes_with: [docs/research/2026-05-03-math-proofs-honest-assessment.md, B-0017] | ||
| tags: [ci, formal-verification, tlc, alloy, lean, cost-optimization, linux-only, tests] | ||
| --- | ||
|
|
||
| # Filter formal-verification tests to standard Linux only | ||
|
|
||
| ## Problem | ||
|
|
||
| Currently the F# test suite runs the same formal-verification tests on: | ||
|
|
||
| - **gate.yml**: `macos-26`, `ubuntu-24.04`, `ubuntu-24.04-arm` (every PR) | ||
| - **low-memory.yml**: `ubuntu-slim` (every push to main + nightly) | ||
|
|
||
| But formal verification (TLC, Alloy, Lean) is pure-math computation: | ||
| no OS-specific behavior, no environment touching, no script differences. | ||
| Running TLC on macOS produces the same result as Linux. Running Alloy | ||
| on ARM produces the same result as x64. The duplication wastes CI | ||
| time + cost. | ||
|
|
||
| ## Aaron 2026-05-03 verbatim | ||
|
|
||
| > *"we don't have to run formal verifical on all the oses, we can run | ||
| > that just on linux it does not have differents between OS like scirpt | ||
| > and code that touch the enviroment. just the standard linux too | ||
| > don't need it to run on slim"* | ||
|
|
||
| ## Scope | ||
|
|
||
| Filter to standard `ubuntu-24.04` only. Skip on: | ||
|
|
||
| - `macos-26` (gate.yml) | ||
| - `ubuntu-24.04-arm` (gate.yml) | ||
| - `windows-2025` / `windows-11-arm` (gate.yml post-merge) | ||
| - `ubuntu-slim` (low-memory.yml) | ||
|
|
||
| Affected test classes: | ||
|
|
||
| - `tests/Tests.FSharp/Formal/Tlc.Runner.Tests.fs` — 9 [Fact] entries | ||
| - `tests/Tests.FSharp/Formal/Alloy.Runner.Tests.fs` — 3 [Fact] entries | ||
| - (Future) Lean integration tests if any are added at the F# layer | ||
|
|
||
| ## Implementation options | ||
|
|
||
| ### Option A — Runtime OS check in `toolchainReady()` | ||
|
|
||
| Cheapest. Add `Environment.OSVersion.Platform` check to the existing | ||
| `toolchainReady()` helpers; return false on non-Linux so tests | ||
| silent-no-op (matches existing skip-on-missing-toolchain pattern). | ||
|
|
||
| Pros: minimal code change; consistent with existing skip discipline | ||
| (no java → silent skip). | ||
| Cons: macOS dev local runs would silent-skip even though TLC works | ||
| locally (regression vs current behavior; devs lose local validation). | ||
|
|
||
| Mitigation: gate the OS check on `CI=true` env var so dev laptops | ||
| still run them. | ||
|
AceHack marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Option B — Custom `[<LinuxOnly>]` xunit attribute | ||
|
|
||
| More structured. Custom xunit `Theory`/`Fact` decorator that skips | ||
| the test on non-Linux runners. | ||
|
|
||
| Pros: explicit intent at each test; easy to grep for skipped tests. | ||
| Cons: more code; needs xunit collection-attribute machinery. | ||
|
|
||
| ### Option C — Separate test project `Tests.FSharp.Formal` | ||
|
|
||
| Most invasive. Move TLC + Alloy tests to a dedicated project that | ||
| the build matrix only includes on Linux. | ||
|
|
||
| Pros: cleanest separation; doesn't run formal-verification build | ||
| on macOS/ARM at all (saves more time than just skipping). | ||
| Cons: substantial refactor; affects build matrix. | ||
|
|
||
| **Recommendation**: Option A with `CI=true` env-gate. Smallest change; | ||
| preserves dev-local validation. | ||
|
|
||
| ## Composes with | ||
|
|
||
| - `docs/research/2026-05-03-math-proofs-honest-assessment.md` | ||
| (the assessment doc; CI cost is implicit context for "runs in CI" | ||
| A-grade gating) | ||
| - B-0017 Operational Resonance Dashboard (when CI cost becomes a | ||
| visible dashboard metric) | ||
|
|
||
| ## Effort estimate | ||
|
|
||
| M (1-2 days): | ||
| - Half day: pick Option, implement | ||
| - Half day: verify CI matrix shape unchanged (dotnet test still runs; | ||
| just specific tests skip); confirm dev local still validates | ||
| - Doc updates: math-proofs assessment B1/A4 sections + low-memory.yml | ||
| + gate.yml comments | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.