Cut the fused format: commit_closure_json_v2, and a closure that can name what it lacks

dag/gunbc/scm/commit_closure.dag

@@ -0,0 +1,252 @@
+module gunbc.scm.commit_closure
+
+// WHAT A COMMIT CLOSURE IS, INDEPENDENT OF HOW IT IS WRITTEN DOWN.
+//
+// A closure is a set of content-addressed objects plus the root the commit names. That is the whole
+// logical content. Positions, JSON, member names and format tags are all facts about ONE WAY of
+// writing it down, and none of them belong here -- this module is what every realization must agree
+// about, and a realization is chosen beside it, never inside it.
+//
+// THE PROPERTY THIS MODULE EXISTS TO MAKE REPRESENTABLE, and it is the reason the format it replaces
+// had to be recut rather than extended:
+//
+//   A closure may be INCOMPLETE, and an incomplete closure must be able to NAME WHAT IS MISSING.
+//
+// The predecessor format could not express that, and not by oversight. It recorded structure only --
+// kind, edge labels, and which EARLIER POSITION each edge pointed at -- with every identity re-derived
+// on load. A backward position cannot name an object the document does not contain, so an unresolved
+// reference named nothing at all and "supply the missing object" was unperformable. The disposition
+// was absent BY CONSTRUCTION, not by low population.
+//
+// WHY THAT FORMAT WAS RIGHT TO DO IT, because the recut gives something up and must say so. Omitting
+// identities was a construction wall, not an accident: a decoder is a SECOND PRODUCER, reading
+// identities from a file it does not control, and a wire that carries an identity is a wire a decoder
+// can lie in. With no such field, forging an identity is not caught -- it is UNWRITABLE. The store
+// would otherwise answer for a locator whose payload was never hashed to it, and checkout would
+// reconstruct a program the commit never named.
+//
+// SO THE TWO PROPERTIES ARE ONE DECISION SEEN FROM TWO SIDES:
+//
+//   unforgeable identity   =>  no identity on the wire
+//   nameable absent object =>  an identity on the wire
+//
+// The recut keeps both by asking what each reference NEEDS to name. A reference to an object the
+// document CONTAINS needs no identity -- the object is right there, and its identity is derived from
+// its own bytes. Only a reference to an object the document LACKS needs one, and that is the single
+// place a digest cannot do the damage the predecessor feared: there is NO PAYLOAD IN THE DOCUMENT for
+// a forged digest to be mis-bound to. So the forgeable surface is exactly the set of objects the
+// document does not contain -- the set for which forgery has nothing to bind.
+//
+// THE TRUST ROOT MOVES, AND THIS IS THE HONEST STATEMENT OF WHAT IS WEAKER.
+//
+// Deriving a supplied object's identity from its own bytes and comparing proves "I got the object THIS
+// DOCUMENT NAMED". It does NOT prove "I got the object the COMMIT named" -- a forged uncontained-arm digest
+// names an attacker-chosen object, and the supplied impostor matches the forgery exactly because that
+// is what was forged. The closure then reconstructs with the impostor in place of the real object: a
+// program the commit never named, reached through the very arm this recut adds.
+//
+// That is closed by ONE CONDITION, stated here as a condition rather than assumed -- and the
+// condition is about the LOGICAL ROOT IDENTITY, not about the bytes:
+//
+//   The root identity DERIVED from contained structure plus uncontained child identities MUST be
+//   compared against an expected commit identity the consumer obtains from OUTSIDE the document.
+//
+// This needs no new machinery, because object_store already derives a record's identity from its
+// kind and its children's target identities. An uncontained child contributes its stated digest to
+// its parent's derived identity and transitively to the root's, so substituting Hc with Hx changes
+// the derived root. A forged document is therefore a document that does not reconstruct to the
+// commit you asked for, and the comparison catches it.
+//
+// WHY NOT HASH THE WIRE BYTES, which was this clause's first form and was wrong. Byte identity makes
+// representation detail semantic: the same logical closure emitted with a different object-table
+// order, or under a different encoding entirely, would become a DIFFERENT commit. That undoes the
+// separation this whole recut exists to make -- the logical closure is the concept, a serialization
+// is one way of writing it down. Two identities, kept apart:
+//
+//   commit identity      = the derived Merkle root over canonical logical structure
+//   publication identity = an optional digest of exact publication bytes, for corruption detection
+//                          or envelope authentication, and NEVER the semantic commit identity
+//
+// But note precisely what changed: the predecessor's guarantee was UNCONDITIONAL -- the format had no
+// field to lie in -- and this one is CONDITIONAL on an expected commit identity being obtained
+// independently. A reader of this format must not inherit the old file's unconditional claim. It is
+// strictly weaker, it is the price of being able to name what is missing, and it is paid knowingly.
+//
+// AND THE RUNG OF THE DIGEST ITSELF IS NOT AUTHENTICATION. ObjectId is fnv1a64 structural. It
+// establishes structural consistency and refuses collisions at the current local rung; it is not an
+// adversarial or cross-tenant cryptographic commitment, and calling this "Merkle" must not import
+// that stronger claim. dissolve-on: a computing cryptographic digest reachable from .dag.
+
+import std.types { Bool, Int, List, list_length }
+import gunbc.scm.object_store {
+  ObjectId,
+  ObjectRecord,
+  ObjectStore,
+  StoredEdge,
+  object_id_eq,
+  store_contains,
+  store_records_in_dependency_order,
+}
+
+// WHY AN ADMISSION IS A VALUE AND NOT A FLAG, and this is the difference between moving a wall and
+// keeping one.
+//
+// Before the recut, an incomplete closure COULD NOT BE WRITTEN: the encoder computed which references
+// were unresolvable and refused. After the recut it CAN be written -- that is the entire point -- and
+// the danger is that the only thing left stopping an accidental partial write is a caller who
+// remembers to check. That is validation, and validation of exactly the shape that decays: the
+// encoder accepts a closure plus a caller's assurance, and one day the assurance is wrong.
+//
+// So the uncontained arm is constructible ONLY from an evidenced decision. The arm CARRIES the admission,
+// the admission is `sole_constructor` and minted here, and an encoder therefore cannot fabricate one
+// to get itself out of trouble -- it has to be HANDED one by whoever decided a partial closure was
+// publishable. An undeclared partial closure has no constructor, so it is not refused, it is
+// unwritable, which is where the predecessor's guarantee lived too.
+//
+// THE REVIEW TELL, stated so it can be checked against a later diff: if a consumer of this module
+// grows a check that the encoder used to perform, the wall was MOVED rather than kept, and this
+// paragraph is the thing that was violated.
+// THE RUNG THIS ACTUALLY OCCUPIES, AND IT IS LOWER THAN TWO REVIEWERS READ IT AS.
+//
+// `sole_constructor` stops a caller writing the RECORD LITERAL. It does not stop a caller calling
+// the public mint below, because .dag has no module privacy -- so the token is unforgeable in shape
+// and freely OBTAINABLE in practice. Worse, it is not bound to any subject:
+//
+//   admit closure A  ->  token T
+//   encode closure B with T  ->  ACCEPTED
+//
+// The encoder checks that a token is PRESENT, never that it is ABOUT the closure being encoded. That
+// is authority substitution in the exact form this repository has already named: a true admission
+// about one subject answering for another because no relation binds them. The existing mutation
+// (delete the check) proves the check is READ; it proves nothing about what the token is about.
+//
+// So the honest rung is MITIGATABLE, not structural: an accidental partial write is caught, a
+// mis-attributed one is not. This paragraph exists because the reported rung must equal the rung
+// executed evidence establishes, and review 54944 read the wall as "structural" -- which is the
+// inflation section 4b calls worse than sitting low, since an inflated class never ranks for
+// climbing.
+//
+// NEXT-RUNG TRIGGER, and it is a shape change rather than a check: the admitted carrier binds the
+// closure and its unresolved population together --
+//
+//   AdmittedPartialCommitClosure sole_constructor { closure, unresolved, reason }
+//
+// minted by a function that DERIVES `unresolved` from the closure it is given, with the encoder
+// taking the admitted carrier instead of a closure plus a free-floating token. Then "a token for A
+// used on B" has no spelling at all. Discriminator the repair owes: an admission produced for A,
+// attempted against B, refuses or cannot be constructed.
+type PartialClosureAdmission sole_constructor {
+  reason: PartialClosureReason
+}
+
+// WHY A CLOSED VOCABULARY RATHER THAN A STRING. A reason nobody can enumerate is a reason nobody can
+// refuse, and the two admitted causes have genuinely different owners: one is a transfer decision,
+// the other is a graft that object_store deliberately supports.
+type PartialClosureReason
+  = ClosureTrimmedForTransfer
+  | ClosureGraftedWithoutChildren
+
+fn admit_partial_closure(reason: PartialClosureReason) -> PartialClosureAdmission {
+  PartialClosureAdmission { reason: reason }
+}
+
+fn admission_reason(admission: PartialClosureAdmission) -> PartialClosureReason {
+  admission.reason
+}
+
+// A closure is the store plus the root the commit names. Nothing about how it is written down.
+type CommitClosure {
+  store: ObjectStore
+  root: ObjectId
+}
+
+// WHY THE WIRE ARM IS "UNCONTAINED" AND THIS FUNCTION SAYS "UNRESOLVED", which is not two names for
+// one thing (DESIGN section 3) but two different facts one join apart.
+//
+// CONTAINED / UNCONTAINED is INTRINSIC to the document: it states whether this document carries the
+// object's payload, and that is true of the bytes forever, wherever they are read. UNRESOLVED is
+// RELATIVE: it is what you get by joining an uncontained identity against the store you actually
+// have. An object this document does not carry may already sit in the local store, in another
+// generation, in a cache, or at a remote -- so it is uncontained here and not unresolved for you.
+//
+// Naming the wire arm "absent" would have frozen a relative availability observation into immutable
+// vocabulary, which is exactly the state-space conflation that makes a reader draw "missing" from a
+// document that only ever said "not carried here".
+//
+//   uncontained identities  +  the store you have  ->  unresolved identities
+//
+// THE ACCEPTANCE TEST OF THE WHOLE RECUT, in one function.
+//
+// This is what the predecessor could not have at any price: the set of object identities the closure
+// REFERS TO and DOES NOT CONTAIN. Each is a content address, so each names exactly what to fetch.
+// A backward position could never appear in this list, because a position that does not resolve
+// inside the document denotes nothing outside it.
+//
+// An EMPTY list is the complete closure. That is the ordinary case and it is not a special arm --
+// completeness is "nothing is missing", derived, never a stored flag that could disagree with the
+// objects beside it.
+fn unresolved_identities(closure: CommitClosure) -> List<ObjectId> {
+  concat(uncontained_targets(store: closure.store), unresolved_root(closure: closure))
+}
+
+// TABLE COMPLETENESS WITHOUT A ROOT, because two different consumers ask two different questions and
+// only one of them has a root to ask about.
+//
+// A COMMIT closure asks "is anything this program reaches missing", which includes its own root. A
+// REPOSITORY object table asks only "does any edge point at a record this table does not hold" --
+// it carries many commits over one population, so there is no single root and the roots it does
+// have are checked separately against the commit list. Folding the root into this would have forced
+// the repository to invent one.
+// UNCONTAINED TARGETS ACCUMULATE BY PREPEND AND ARE REVERSED ONCE, the same idiom this stack applies
+// in commit_closure_json_v2 encode_step and in repository_envelope's commit fold. It is restated at
+// the third site because the first two were not enough: this module documented the rule and then
+// reintroduced the shape it forbids, which is the §6 point exactly -- fixing a cost shape where a
+// review names it is not the same as not having the defect, and the sweep has to cover every
+// accumulator a change introduces.
+//
+// BOTH LEVELS WERE QUADRATIC, and the inner one is why the per-record helper takes the caller's
+// accumulator instead of returning a fresh list. Building a list per record and joining it at the
+// top copies the whole accumulator once per RECORD; prepending each child onto the single threaded
+// accumulator copies nothing at either level, and one linear reverse at the top restores authored
+// order exactly.
+//
+// THE REVERSE IS LOAD-BEARING, AND THAT SENTENCE COST A MEASUREMENT RATHER THAN AN ASSERTION. An
+// earlier revision of this comment claimed order was already observed and named two existing claims
+// as the evidence. It was false: names_exactly is a MEMBERSHIP fold and every closure fixture
+// asserted exactly ONE missing identity, and a one-element list reversed is itself. Dropping the
+// reverse was measured to leave every claim in the file green. The claim that holds it up is
+// two_uncontained_children_are_named_in_order, written for this and verified both ways -- with the
+// reverse exit=0, without it exit=1.
+//
+// The realized n is not the argument. A delta encoder folds this over the whole store, and DESIGN §6
+// rejects "n is small here" as a non-time-stable fact: reuse is what changes n.
+fn uncontained_targets(store: ObjectStore) -> List<ObjectId> {
+  reverse(
+    fold(
+      store_records_in_dependency_order(store: store),
+      init: [],
+      f: fn(acc, record) { uncontained_in_record(store: store, acc: acc, record: record) }
+    )
+  )
+}
+
+fn uncontained_in_record(store: ObjectStore, acc: List<ObjectId>, record: ObjectRecord) -> List<ObjectId> {
+  fold(
+    record.children,
+    init: acc,
+    f: fn(inner, child) {
+      if store_contains(store: store, identity: child.target) { inner } else { concat([child.target], inner) }
+    }
+  )
+}
+
+// THE ROOT IS A REFERENCE LIKE ANY OTHER, and a closure missing its own root is the sharpest case of
+// incompleteness there is -- a document that names a commit whose top object it does not carry.
+// The predecessor encoded the root positionally too, so this case was as unnameable as the rest.
+fn unresolved_root(closure: CommitClosure) -> List<ObjectId> {
+  if store_contains(store: closure.store, identity: closure.root) { [] } else { [closure.root] }
+}
+
+fn closure_is_complete(closure: CommitClosure) -> Bool {
+  list_length(items: unresolved_identities(closure: closure)) == 0
+}