Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
475 changes: 404 additions & 71 deletions TODO/TODO_credential_lifecycle.md

Large diffs are not rendered by default.

62 changes: 62 additions & 0 deletions TODO/TODO_hacks.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,65 @@ Consolidated 4 copies of `build_cloud_credential_graph_for_runtime` into

Removed `upsert_and_run` and unused imports (`CliToolError`, `Value`,
`execute_cli_tool_op`, `HashMap`) from `lib/tools/clippy/src/ops.rs`.

---

## 10. DAG typing is structural at edge-level, but node I/O is still dynamic

**Where**:
- `core/ir/src/builder.rs` (strong edge/type/cardinality checks)
- `core/ir/src/types.rs` (`TypeId(pub String)`)
- `core/exec/src/lib.rs` (`Executable::execute(HashMap<String, Value>)`)
- `core/exec/src/execute.rs` (`execute_single_node` and input mock injection)
- `core/codegen/src/testgen/codegen.rs` (mock type checks skip `input_mocks`)

**What happens**:
- We get strong DAG build-time guarantees for edge wiring:
- type compatibility
- cardinality compatibility
- fan-in rejection for scalar ports
- cycle prevention
- But node execution boundaries are still `HashMap<String, Value>`, so wrong
value types can still be injected at runtime (especially entrypoint ports).
- `MockSpec` type mismatch checks currently validate `transport_mocks` and
`boundary_mocks`, but not `input_mocks`.

**Why it's a hack**:
- The model appears strongly typed end-to-end, but there is still a dynamic
escape hatch at node call boundaries.
- This is where regressions like optional bool/string coercion drift or
semantic placeholder values can bypass structural guarantees.

**Supporting examples (current repo)**:
1. `parse_impersonate` expects a REST payload with `accessToken`, but the type
system only knows `TransportResponse`; shape-valid placeholders can still be
behavior-invalid (`lib/gcp-ops/src/ops.rs`).
2. `compare_*_content.check_mode` wrong-type tests (`Value::Str("<WRONG>")`)
are meaningful because node entrypoint inputs are runtime maps, not typed
structs (`gunbc-dag/src/*/generated_tests.rs`).
3. `BlobOps::CompareContent` relies on strict input extraction
(`optional_bool_strict`) to reject wrong-typed inputs at runtime
(`lib/blob/src/lib.rs`).
4. CLI parsing still has permissive coercion (`Int` uses parse-or-0) which is
runtime behavior, not structural typing (`core/cli/src/lib.rs`).

**Suggested fix (incremental, DAG-first)**:
1. Add `input_mocks` type validation in testgen coverage checks (same level as
existing boundary/transport mock compatibility checks).
2. Generate typed node input/output wrappers from DAG signatures (e.g.
`ParseImpersonateIn`, `ParseImpersonateOut`) and use them in generated tests
+ helper APIs.
3. Add typed entrypoint injection APIs (`set_input_typed`) to avoid ad-hoc
`Value` maps for common call paths.
4. Introduce refined semantic types for carrier payloads where needed:
- `ImpersonationResponse` (validated schema)
- `AccessToken`
- `ScopeSet`
5. Keep transport/world effects as runtime checks; push everything else to
DAG build-time or generated type wrappers.

**Boundary of guarantee (explicit)**:
- Compile/build time can guarantee: DAG structure, port compatibility,
cardinality, typed wrappers, and mock shape/type correctness.
- Runtime must still validate: external provider payloads, auth scopes,
permissions, and freshness/availability of real resources.
100 changes: 100 additions & 0 deletions TODO/TODO_testgen_seed_policy_postmortem.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Testgen Seed Policy Post-Mortem (Auth Regression)

Status date: 2026-02-12
Owner: codegen/testgen + auth modeling

## Incident

`make gist-recent` failed in real mode with:

- `missing accessToken in impersonation response`

Generated optional-input tests were seeding required semantic auth inputs with
shape-valid placeholders (for example shell `<MOCK>` response forms), which do
not satisfy parser semantics.

## Root Cause

The generator treated two different properties as equivalent:

- structural/type validity ("this value has the right outer type")
- semantic validity ("this value is meaningful for this operation")

For auth and transport carrier types, those are not equivalent.

## Missing Pattern

The missing abstraction is a **seed policy matrix** keyed by:

- type class (what kind of value this is)
- test mode/context (how the node is being executed)

Without that matrix, testgen defaults drift toward local heuristics and spot
fixes.

## Rule We Added (Current Slice)

For `Real` single-node optional-input tests:

- required semantic-carrier inputs must be explicitly seeded from authored
data, not synthesized placeholders.
- accepted explicit seed sources:
- `MockSpec::input_mock`
- `MockSpec::node_example`
- `Node::with_example`
- if missing, generation hard-fails with a clear panic.

Current semantic-carrier class includes:

- `TransportRequest`
- `TransportResponse`
- `Credential`
- `Secret`
- `FilesystemHandle`
- `NetworkHandle`
- `ToolHandle`

## General Pattern (Target, All Types/Modes)

Policy must be centralized and deterministic:

1. Classify type into seed class:
- `StructuralGeneratable`
- `SemanticCarrier`
2. Classify test context:
- `RealSingleNodeRequiredInput`
- `DryRunBoundaryMock`
- `LiveFlowInput`
3. Apply matrix:
- if class/context requires explicit seed: hard-fail on missing explicit seed
- otherwise allow witness/synthetic generation

Seed provenance must be tracked and validated in priority order:

- explicit (authored mock/example)
- witness (contract/type-derived)
- synthetic fallback (last resort)

The key invariant:

- semantic-carrier inputs are never silently satisfied by synthetic fallback in
contexts where behavior correctness is being asserted.

## Why This Avoids Spot Fixes

Failures become policy-driven, not node-driven:

- new nodes automatically inherit rules by type class + mode
- missing authored seeds are caught at generation time
- no parser-local hacks (for example special-casing placeholder strings)

## Follow-Up Work

1. Move seed-class classification to a shared IR-level module so codegen/testgen
and future generators consume one source of truth.
2. Extend matrix enforcement beyond current slice:
- scenario generation contexts
- live-flow generation contexts
3. Add tests that assert unknown semantic carrier types fail closed unless
explicitly classified.
4. Keep parser behavior strict; no placeholder-specific parsing branches.
1 change: 1 addition & 0 deletions TODO/consolidation.md
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,7 @@ compiler version is now captured directly, regardless of environment setup.
- [x] Builder strings compile-time validation (`#[tool_target]`) — §6.1
- [x] Remove static CODEGEN_SOURCES path list — §6.3
- [ ] Design hermeticity annotation for `Shell` transport (see §8 design problem)
- [ ] Design DAG typing hardening plan (typed node I/O wrappers + input_mock type validation + semantic carrier refinements) — see `TODO/TODO_hacks.md` §10

### Remaining (extension features — from architecture-debt.md §16)

Expand Down
Loading
Loading