diff --git a/docs/doctoring/release-publication-file-stability.md b/docs/doctoring/release-publication-file-stability.md new file mode 100644 index 000000000..1f7594cc1 --- /dev/null +++ b/docs/doctoring/release-publication-file-stability.md @@ -0,0 +1,47 @@ +# Release publication file stability + +## Scope + +Noema's release materialization and immutable-release publication receipt are buyer-facing supply-chain evidence. Before this change, release materialization used pathname-based reads for the source archive and SBOM, and the receipt validated a pathname before reopening it for JSON parsing or SHA-256 hashing. Those split check/use boundaries allowed local pathname replacement or in-place mutation to substitute different bytes without changing the operator-visible argument. The publication receipt was also copied into an isolated artifact as one file even though it imported a sibling module, so the first real publication would not have had a closed runtime dependency set. + +The protected publication workflow already constrains the release bundle to a fixed artifact handoff and rejects symbolic links before publication. The scripts still have to authenticate the exact local bytes they parse and hash, and the executable copied across the isolated handoff must remain runnable without relying on files that are not shipped with it. + +## Implemented boundary + +`scripts/lib/stable-file-evidence.mjs` provides the release-materialization step with a bounded no-follow descriptor reader. The isolated `scripts/release-publication-receipt.mjs` carries the same small descriptor-read contract inline so the existing one-file handoff remains self-contained rather than silently depending on repository files that are absent in the publication job. The descriptor contract: + +1. requires a positive reviewed byte ceiling and an available `O_NOFOLLOW`/read-only open contract; +2. rejects a pathname that is a symlink, non-regular file, empty file, or declared oversize before opening; +3. opens the exact pathname with `O_NOFOLLOW` and compares pathname metadata with the opened descriptor; +4. reads only through that descriptor, with a streaming `maximum + 1` ceiling instead of trusting the initial size alone; +5. revalidates descriptor device, inode, mode, size, modification time, change time, and observed byte count after the read; +6. re-resolves the pathname after the read and requires it still to identify the same regular-file device/inode/mode/size; and +7. closes the descriptor on success and failure. + +`release-evidence.mjs` now hashes the exact accepted source-archive and SBOM bytes instead of reopening their pathnames. Its manifest and `SHA256SUMS` outputs use an unpredictable owner-only temporary file followed by atomic rename. The output-directory real-directory check remains in place; this change does not claim immunity to a hostile replacement of an ancestor directory outside the reviewed GitHub-hosted-runner threat boundary. + +The publication receipt retains one accepted snapshot of `release-evidence.json` for both semantic validation and release-asset hashing. It requires `--release-evidence` to identify the exact `release-evidence.json` inside the supplied release asset directory, preventing a separately parsed manifest from being combined with a different hashed asset. `SHA256SUMS` is parsed from the same retained bytes that were hashed into the local asset map. The receipt output likewise uses an unpredictable temporary file plus atomic rename. + +The publication workflow deliberately copies only `release-publication-receipt.mjs` into the sterile handoff. The receipt therefore contains no relative module imports; its duplicate-decoded-JSON-key scanner, stable descriptor reader, and atomic output helper are intentionally self-contained boundary code. This avoids an acquisition-path defect in which tests run the script from the repository successfully but the isolated publication job cannot resolve an unshipped sibling module. + +The descriptor reader fails closed when a runtime cannot provide the no-follow flag. Noema's publication workflow currently executes on Ubuntu GitHub-hosted runners; this control does not claim equivalent filesystem semantics on runtimes where Node does not expose the required flag. + +## Verification strategy + +`test/stable-release-file-evidence.test.ts` includes a real temporary-file/symlink case plus deterministic filesystem-adapter cases for path-to-descriptor replacement, in-place descriptor mutation, same-byte pathname replacement after reading, short reads, streamed oversize, unsupported open flags, non-files, empty files, and descriptor closure after failure. `test/release-evidence-file-stability.test.ts` binds release materialization to the stable reader and atomic evidence writer. `test/release-publication-output-atomicity.test.ts` requires temporary exclusive creation plus rename for the final receipt, and `test/release-publication-runtime-handoff.test.ts` verifies that the isolated one-file executable has no relative runtime dependency while the workflow authenticates that exact file across its handoffs. Existing immutable-release publication tests continue to exercise the complete receipt CLI, exact asset set, digest checks, malformed UTF-8 handling, duplicate decoded JSON keys, immutable-policy checks, and publication evidence contract. + +This change does **not** prove that a GitHub Release exists, that Cloudflare deployment succeeded, that production KPIs are healthy, or that legal/IP transfer rights are complete. It strengthens only the local evidence-consumption and publication boundary used when those external facts are eventually captured. + +## Standards and implementation basis + +POSIX.1-2024 specifies that `open()` with `O_NOFOLLOW` fails when the final pathname component is a symbolic link. Node.js exposes the corresponding filesystem constant and documents that it causes open to fail for a symbolic-link path. Those primitives remove the final-component symlink-follow step from the open operation; the additional descriptor/path identity checks are Noema's application-level control for replacement and mutation across the full read window. + +NIST SSDF 1.1 recommends defining, implementing, and verifying software security requirements throughout the development lifecycle. Noema treats release evidence byte identity and executable handoff closure as such requirements because publication receipts are later consumed as acquisition and supply-chain evidence. + +## References + +Node.js contributors. (2026). *File system: Node.js v25.9.0 documentation*. Node.js. https://nodejs.org/download/release/v25.9.0/docs/api/fs.html + +Souppaya, M., Scarfone, K., & Dodson, D. (2022). *Secure Software Development Framework (SSDF) version 1.1: Recommendations for mitigating the risk of software vulnerabilities* (NIST Special Publication 800-218). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218 + +The Open Group. (2024). *open, openat — open a file*. In *The Open Group Base Specifications Issue 8, IEEE Std 1003.1-2024*. https://pubs.opengroup.org/onlinepubs/9799919799/functions/open.html diff --git a/scripts/lib/stable-file-evidence.mjs b/scripts/lib/stable-file-evidence.mjs new file mode 100644 index 000000000..9de318f13 --- /dev/null +++ b/scripts/lib/stable-file-evidence.mjs @@ -0,0 +1,158 @@ +import { + closeSync, + constants, + fstatSync, + lstatSync, + openSync, + readSync, +} from "node:fs"; + +const MAXIMUM_SIGNED_OPEN_FLAG = 0x7fff_ffff; +const defaultFileSystem = Object.freeze({ + closeSync, + constants, + fstatSync, + lstatSync, + openSync, + readSync, +}); + +function fail(label, detail) { + throw new Error(`${label} ${detail}`); +} + +function safeOpenFlag(value, { allowZero }) { + return Number.isSafeInteger(value) + && value >= 0 + && value <= MAXIMUM_SIGNED_OPEN_FLAG + && (allowZero || value !== 0); +} + +function requireRegularMetadata(metadata, label, maximumBytes) { + if (!metadata || typeof metadata !== "object" || typeof metadata.isFile !== "function") { + fail(label, "metadata is unavailable"); + } + if (typeof metadata.isSymbolicLink === "function" && metadata.isSymbolicLink()) { + fail(label, "must not be a symbolic link"); + } + if (!metadata.isFile()) { + fail(label, "must be a regular file"); + } + if (!Number.isSafeInteger(metadata.size) || metadata.size < 0) { + fail(label, "has an invalid byte size"); + } + if (metadata.size === 0) { + fail(label, "must not be empty"); + } + if (metadata.size > maximumBytes) { + fail(label, `exceeds the ${maximumBytes}-byte ceiling`); + } + return metadata; +} + +function sameIdentity(left, right) { + return left.dev === right.dev + && left.ino === right.ino + && left.mode === right.mode + && left.size === right.size; +} + +function sameStableDescriptor(left, right) { + return sameIdentity(left, right) + && left.mtimeMs === right.mtimeMs + && left.ctimeMs === right.ctimeMs; +} + +/** + * Read one bounded regular file through a no-follow descriptor and accept the + * bytes only while both descriptor state and the pathname-to-inode mapping stay + * stable for the complete read. + * + * @param {string} path filesystem path to read + * @param {string} label bounded diagnostic label that never contains file bytes + * @param {number} maximumBytes positive safe byte ceiling + * @param {object} fileSystem injectable Node-compatible filesystem adapter for deterministic race tests + * @returns {Buffer} exact accepted bytes + */ +export function readStableRegularFile( + path, + label, + maximumBytes, + fileSystem = defaultFileSystem, +) { + if (typeof path !== "string" || path.length === 0) { + fail("stable file", "path must be a non-empty string"); + } + if (typeof label !== "string" || label.length === 0) { + fail("stable file", "label must be a non-empty string"); + } + if (!Number.isSafeInteger(maximumBytes) || maximumBytes <= 0) { + fail(label, "requires a positive safe byte ceiling"); + } + + const noFollow = fileSystem.constants?.O_NOFOLLOW; + const readOnly = fileSystem.constants?.O_RDONLY; + if (!safeOpenFlag(noFollow, { allowZero: false })) { + fail(label, "requires a supported no-follow open flag"); + } + if (!safeOpenFlag(readOnly, { allowZero: true })) { + fail(label, "requires a supported read-only open flag"); + } + + const pathMetadata = requireRegularMetadata( + fileSystem.lstatSync(path), + label, + maximumBytes, + ); + const descriptor = fileSystem.openSync(path, readOnly | noFollow); + try { + const openedMetadata = requireRegularMetadata( + fileSystem.fstatSync(descriptor), + label, + maximumBytes, + ); + if (!sameIdentity(pathMetadata, openedMetadata)) { + fail(label, "changed before read"); + } + + const chunks = []; + let totalBytes = 0; + while (totalBytes <= maximumBytes) { + const remaining = maximumBytes + 1 - totalBytes; + const target = Buffer.allocUnsafe(Math.min(64 * 1024, remaining)); + const bytesRead = fileSystem.readSync(descriptor, target, 0, target.length, null); + if (bytesRead === 0) { + break; + } + chunks.push(target.subarray(0, bytesRead)); + totalBytes += bytesRead; + } + if (totalBytes > maximumBytes) { + fail(label, `exceeded the ${maximumBytes}-byte ceiling while reading`); + } + + const finalMetadata = requireRegularMetadata( + fileSystem.fstatSync(descriptor), + label, + maximumBytes, + ); + if (!sameStableDescriptor(openedMetadata, finalMetadata)) { + fail(label, "changed while being read"); + } + if (totalBytes !== openedMetadata.size) { + fail(label, "byte count differs from the opened descriptor size"); + } + + const finalPathMetadata = requireRegularMetadata( + fileSystem.lstatSync(path), + label, + maximumBytes, + ); + if (!sameIdentity(openedMetadata, finalPathMetadata)) { + fail(label, "pathname changed while being read"); + } + return Buffer.concat(chunks, totalBytes); + } finally { + fileSystem.closeSync(descriptor); + } +} diff --git a/scripts/release-evidence.mjs b/scripts/release-evidence.mjs index 2f2b20747..d3fdeccaa 100644 --- a/scripts/release-evidence.mjs +++ b/scripts/release-evidence.mjs @@ -4,12 +4,14 @@ import { existsSync, lstatSync, mkdirSync, - readFileSync, statSync, - writeFileSync, } from "node:fs"; import { basename, resolve } from "node:path"; -import { hasDuplicateJsonObjectKeys } from "./normalize-commercial-readiness-evidence.mjs"; +import { readStableRegularFile } from "./lib/stable-file-evidence.mjs"; +import { + hasDuplicateJsonObjectKeys, + writeAtomically, +} from "./normalize-commercial-readiness-evidence.mjs"; const EXPECTED_REPOSITORY = "ContextualWisdomLab/noema"; const EXPECTED_SBOM_NAME = "noema.cdx.json"; @@ -52,31 +54,6 @@ function parseArguments(argv) { }; } -function requireRegularFile(path, label, maxBytes) { - if (!existsSync(path)) { - fail(`${label} does not exist: ${path}`); - } - const linkStatus = lstatSync(path); - if (linkStatus.isSymbolicLink()) { - fail(`${label} must not be a symbolic link`); - } - const status = statSync(path); - if (!status.isFile()) { - fail(`${label} must be a regular file`); - } - if (status.size <= 0) { - fail(`${label} must not be empty`); - } - if (status.size > maxBytes) { - fail(`${label} exceeds the ${maxBytes}-byte limit`); - } - return status; -} - -function sha256(path) { - return createHash("sha256").update(readFileSync(path)).digest("hex"); -} - function requireString(value, label) { if (typeof value !== "string" || value.trim().length === 0) { fail(`${label} must be a non-empty string`); @@ -84,6 +61,18 @@ function requireString(value, label) { return value.trim(); } +function readStableBytes(path, label, maximumBytes) { + try { + return readStableRegularFile(path, label, maximumBytes); + } catch (error) { + fail(`${label} could not be read safely: ${error instanceof Error ? error.message : String(error)}`); + } +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + function validateReleaseIdentity() { const repository = requireString(process.env.GITHUB_REPOSITORY, "GITHUB_REPOSITORY"); const commitSha = requireString( @@ -183,12 +172,6 @@ function validateSbom(sbom, version) { }; } -function requireSafeOutputPath(path, label) { - if (existsSync(path) && lstatSync(path).isSymbolicLink()) { - fail(`${label} must not be a symbolic link`); - } -} - function run() { const { sourcePath, sbomPath, outputDir } = parseArguments(process.argv.slice(2)); const identity = validateReleaseIdentity(); @@ -203,11 +186,11 @@ function run() { fail("source archive and SBOM paths must be different files"); } - const sourceStatus = requireRegularFile(sourcePath, "source archive", MAX_SOURCE_BYTES); - const sbomStatus = requireRegularFile(sbomPath, "SBOM", MAX_SBOM_BYTES); + const sourceBytes = readStableBytes(sourcePath, "source archive", MAX_SOURCE_BYTES); + const sbomBytes = readStableBytes(sbomPath, "SBOM", MAX_SBOM_BYTES); let sbomText; try { - sbomText = new TextDecoder("utf-8", { fatal: true }).decode(readFileSync(sbomPath)); + sbomText = new TextDecoder("utf-8", { fatal: true }).decode(sbomBytes); } catch (error) { fail(`SBOM is not valid UTF-8: ${error instanceof Error ? error.message : String(error)}`); } @@ -235,11 +218,8 @@ function run() { const manifestPath = resolve(outputDir, "release-evidence.json"); const checksumsPath = resolve(outputDir, "SHA256SUMS"); - requireSafeOutputPath(manifestPath, "release evidence manifest"); - requireSafeOutputPath(checksumsPath, "checksum manifest"); - - const sourceDigest = sha256(sourcePath); - const sbomDigest = sha256(sbomPath); + const sourceDigest = sha256(sourceBytes); + const sbomDigest = sha256(sbomBytes); const manifest = { schemaVersion: 1, generatedAt: identity.generatedAt, @@ -252,28 +232,26 @@ function run() { subject: { name: basename(sourcePath), sha256: sourceDigest, - bytes: sourceStatus.size, + bytes: sourceBytes.byteLength, mediaType: "application/gzip", }, sbom: { name: basename(sbomPath), sha256: sbomDigest, - bytes: sbomStatus.size, + bytes: sbomBytes.byteLength, ...sbomSummary, }, }; - writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, { - encoding: "utf8", - mode: 0o644, - }); - const manifestDigest = sha256(manifestPath); + const manifestText = `${JSON.stringify(manifest, null, 2)}\n`; + const manifestDigest = sha256(Buffer.from(manifestText, "utf8")); + writeAtomically(manifestPath, manifestText); const checksums = [ `${sourceDigest} ${basename(sourcePath)}`, `${sbomDigest} ${basename(sbomPath)}`, `${manifestDigest} ${basename(manifestPath)}`, ].join("\n"); - writeFileSync(checksumsPath, `${checksums}\n`, { encoding: "utf8", mode: 0o644 }); + writeAtomically(checksumsPath, `${checksums}\n`); console.log( `release-evidence: PASS repository=${identity.repository} version=${identity.version} head=${identity.commitSha}`, @@ -286,4 +264,4 @@ try { const message = error instanceof Error ? error.message : String(error); console.error(`release-evidence: FAIL: ${message.slice(0, 1000)}`); process.exitCode = 1; -} \ No newline at end of file +} diff --git a/scripts/release-publication-receipt.mjs b/scripts/release-publication-receipt.mjs index 1cd16aa2a..830d8cf10 100644 --- a/scripts/release-publication-receipt.mjs +++ b/scripts/release-publication-receipt.mjs @@ -1,29 +1,332 @@ #!/usr/bin/env node import { createHash } from "node:crypto"; import { - existsSync, + closeSync, + constants, + fstatSync, lstatSync, mkdirSync, - readFileSync, - statSync, + mkdtempSync, + openSync, + readSync, + renameSync, + rmSync, writeFileSync, } from "node:fs"; -import { basename, dirname, resolve } from "node:path"; -import { hasDuplicateJsonObjectKeys } from "./normalize-commercial-readiness-evidence.mjs"; +import { basename, dirname, join, resolve } from "node:path"; const EXPECTED_REPOSITORY = "ContextualWisdomLab/noema"; const MAX_JSON_BYTES = 16 * 1024 * 1024; const MAX_ASSET_BYTES = 512 * 1024 * 1024; +const MAX_JSON_NESTING_DEPTH = 256; +const MAXIMUM_SIGNED_OPEN_FLAG = 0x7fff_ffff; const SHA_PATTERN = /^[0-9a-f]{40}$/i; const DIGEST_PATTERN = /^sha256:([0-9a-f]{64})$/i; const SEMVER_PATTERN = /^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/; const CANONICAL_UTC_TIMESTAMP_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; +const JSON_PRIMITIVE_PATTERN = + /(?:-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?|true|false|null)/y; +const defaultFileSystem = Object.freeze({ + closeSync, + constants, + fstatSync, + lstatSync, + openSync, + readSync, +}); function fail(message) { throw new Error(message); } +function fileFail(label, detail) { + fail(`${label} ${detail}`); +} + +function safeOpenFlag(value, { allowZero }) { + return Number.isSafeInteger(value) + && value >= 0 + && value <= MAXIMUM_SIGNED_OPEN_FLAG + && (allowZero || value !== 0); +} + +function requireRegularMetadata(metadata, label, maximumBytes) { + if (!metadata || typeof metadata !== "object" || typeof metadata.isFile !== "function") { + fileFail(label, "metadata is unavailable"); + } + if (typeof metadata.isSymbolicLink === "function" && metadata.isSymbolicLink()) { + fileFail(label, "must not be a symbolic link"); + } + if (!metadata.isFile()) { + fileFail(label, "must be a regular file"); + } + if (!Number.isSafeInteger(metadata.size) || metadata.size < 0) { + fileFail(label, "has an invalid byte size"); + } + if (metadata.size === 0) { + fileFail(label, "must not be empty"); + } + if (metadata.size > maximumBytes) { + fileFail(label, `exceeds the ${maximumBytes}-byte ceiling`); + } + return metadata; +} + +function sameIdentity(left, right) { + return left.dev === right.dev + && left.ino === right.ino + && left.mode === right.mode + && left.size === right.size; +} + +function sameStableDescriptor(left, right) { + return sameIdentity(left, right) + && left.mtimeMs === right.mtimeMs + && left.ctimeMs === right.ctimeMs; +} + +/** + * Read one bounded regular file through a no-follow descriptor and accept the + * bytes only while descriptor state and pathname identity stay stable. + */ +function readStableRegularFile( + path, + label, + maximumBytes, + fileSystem = defaultFileSystem, +) { + if (typeof path !== "string" || path.length === 0) { + fileFail("stable file", "path must be a non-empty string"); + } + if (typeof label !== "string" || label.length === 0) { + fileFail("stable file", "label must be a non-empty string"); + } + if (!Number.isSafeInteger(maximumBytes) || maximumBytes <= 0) { + fileFail(label, "requires a positive safe byte ceiling"); + } + + const noFollow = fileSystem.constants?.O_NOFOLLOW; + const readOnly = fileSystem.constants?.O_RDONLY; + if (!safeOpenFlag(noFollow, { allowZero: false })) { + fileFail(label, "requires a supported no-follow open flag"); + } + if (!safeOpenFlag(readOnly, { allowZero: true })) { + fileFail(label, "requires a supported read-only open flag"); + } + + const pathMetadata = requireRegularMetadata( + fileSystem.lstatSync(path), + label, + maximumBytes, + ); + const descriptor = fileSystem.openSync(path, readOnly | noFollow); + try { + const openedMetadata = requireRegularMetadata( + fileSystem.fstatSync(descriptor), + label, + maximumBytes, + ); + if (!sameIdentity(pathMetadata, openedMetadata)) { + fileFail(label, "changed before read"); + } + + const chunks = []; + let totalBytes = 0; + while (totalBytes <= maximumBytes) { + const remaining = maximumBytes + 1 - totalBytes; + const target = Buffer.allocUnsafe(Math.min(64 * 1024, remaining)); + const bytesRead = fileSystem.readSync(descriptor, target, 0, target.length, null); + if (bytesRead === 0) { + break; + } + chunks.push(target.subarray(0, bytesRead)); + totalBytes += bytesRead; + } + if (totalBytes > maximumBytes) { + fileFail(label, `exceeded the ${maximumBytes}-byte ceiling while reading`); + } + + const finalMetadata = requireRegularMetadata( + fileSystem.fstatSync(descriptor), + label, + maximumBytes, + ); + if (!sameStableDescriptor(openedMetadata, finalMetadata)) { + fileFail(label, "changed while being read"); + } + if (totalBytes !== openedMetadata.size) { + fileFail(label, "byte count differs from the opened descriptor size"); + } + + const finalPathMetadata = requireRegularMetadata( + fileSystem.lstatSync(path), + label, + maximumBytes, + ); + if (!sameIdentity(openedMetadata, finalPathMetadata)) { + fileFail(label, "pathname changed while being read"); + } + return Buffer.concat(chunks, totalBytes); + } finally { + fileSystem.closeSync(descriptor); + } +} + +function skipJsonWhitespace(text, state) { + while (state.index < text.length) { + const character = text[state.index]; + if (character !== " " && character !== "\t" && character !== "\n" && character !== "\r") { + return; + } + state.index += 1; + } +} + +function parseJsonStringToken(text, state) { + const start = state.index; + state.index += 1; + let escaped = false; + while (state.index < text.length) { + const character = text[state.index]; + const code = text.charCodeAt(state.index); + if (code < 0x20) { + throw new SyntaxError("JSON strings cannot contain unescaped control characters."); + } + state.index += 1; + if (escaped) { + escaped = false; + } else if (character === "\\") { + escaped = true; + } else if (character === '"') { + return JSON.parse(text.slice(start, state.index)); + } + } + throw new SyntaxError("JSON string was not terminated."); +} + +function parseJsonPrimitive(text, state) { + JSON_PRIMITIVE_PATTERN.lastIndex = state.index; + const match = JSON_PRIMITIVE_PATTERN.exec(text); + if (!match) { + throw new SyntaxError(`Unexpected JSON token at character ${state.index}.`); + } + state.index += match[0].length; + return false; +} + +function parseJsonArray(text, state, depth) { + state.index += 1; + skipJsonWhitespace(text, state); + if (text[state.index] === "]") { + state.index += 1; + return false; + } + let duplicate = false; + while (true) { + duplicate = parseJsonValue(text, state, depth) || duplicate; + skipJsonWhitespace(text, state); + if (text[state.index] === "]") { + state.index += 1; + return duplicate; + } + if (text[state.index] !== ",") { + throw new SyntaxError(`Expected an array comma at character ${state.index}.`); + } + state.index += 1; + skipJsonWhitespace(text, state); + } +} + +function parseJsonObject(text, state, depth) { + state.index += 1; + skipJsonWhitespace(text, state); + if (text[state.index] === "}") { + state.index += 1; + return false; + } + const keys = new Set(); + let duplicate = false; + while (true) { + if (text[state.index] !== '"') { + throw new SyntaxError(`Expected an object key at character ${state.index}.`); + } + const key = parseJsonStringToken(text, state); + if (keys.has(key)) { + duplicate = true; + } + keys.add(key); + skipJsonWhitespace(text, state); + if (text[state.index] !== ":") { + throw new SyntaxError(`Expected an object colon at character ${state.index}.`); + } + state.index += 1; + skipJsonWhitespace(text, state); + duplicate = parseJsonValue(text, state, depth) || duplicate; + skipJsonWhitespace(text, state); + if (text[state.index] === "}") { + state.index += 1; + return duplicate; + } + if (text[state.index] !== ",") { + throw new SyntaxError(`Expected an object comma at character ${state.index}.`); + } + state.index += 1; + skipJsonWhitespace(text, state); + } +} + +function parseJsonValue(text, state, depth) { + if (depth > MAX_JSON_NESTING_DEPTH) { + throw new RangeError("JSON evidence nesting exceeds the reviewed limit."); + } + skipJsonWhitespace(text, state); + const character = text[state.index]; + if (character === "{") { + return parseJsonObject(text, state, depth + 1); + } + if (character === "[") { + return parseJsonArray(text, state, depth + 1); + } + if (character === '"') { + parseJsonStringToken(text, state); + return false; + } + return parseJsonPrimitive(text, state); +} + +function hasDuplicateJsonObjectKeys(text) { + if (typeof text !== "string") { + throw new TypeError("JSON evidence must be supplied as text."); + } + const state = { index: 0 }; + skipJsonWhitespace(text, state); + const duplicate = parseJsonValue(text, state, 0); + skipJsonWhitespace(text, state); + if (state.index !== text.length) { + throw new SyntaxError(`Unexpected trailing JSON content at character ${state.index}.`); + } + return duplicate; +} + +/** Replace one receipt atomically without opening a predictable output file. */ +function writeAtomically(path, content) { + const parentDirectory = dirname(path); + mkdirSync(parentDirectory, { recursive: true }); + const temporaryDirectory = mkdtempSync(join(parentDirectory, ".noema-release-receipt-")); + const temporaryPath = join(temporaryDirectory, "receipt.json"); + try { + writeFileSync(temporaryPath, content, { + encoding: "utf8", + flag: "wx", + mode: 0o600, + }); + renameSync(temporaryPath, path); + } finally { + rmSync(temporaryDirectory, { force: true, recursive: true }); + } +} + function parseArguments(argv) { const accepted = new Set([ "--policy", @@ -88,35 +391,15 @@ function requireCanonicalUtcTimestamp(value, label) { return timestamp; } -function requireRegularFile(path, label, maxBytes = MAX_JSON_BYTES) { - if (!existsSync(path)) { - fail(`${label} does not exist: ${path}`); - } - const linkStatus = lstatSync(path); - if (linkStatus.isSymbolicLink()) { - fail(`${label} must not be a symbolic link`); - } - const status = statSync(path); - if (!status.isFile()) { - fail(`${label} must be a regular file`); - } - if (status.size <= 0) { - fail(`${label} must not be empty`); - } - if (status.size > maxBytes) { - fail(`${label} exceeds the ${maxBytes}-byte limit`); - } - return status; -} - -function readJson(path, label) { - requireRegularFile(path, label); - let bytes; +function readStableBytes(path, label, maximumBytes) { try { - bytes = readFileSync(path); + return readStableRegularFile(path, label, maximumBytes); } catch (error) { - fail(`${label} could not be read: ${error instanceof Error ? error.message : String(error)}`); + fail(`${label} could not be read safely: ${error instanceof Error ? error.message : String(error)}`); } +} + +function parseJsonBytes(bytes, label) { let text; try { text = new TextDecoder("utf-8", { fatal: true }).decode(bytes); @@ -140,8 +423,13 @@ function readJson(path, label) { } } -function sha256(path) { - return createHash("sha256").update(readFileSync(path)).digest("hex"); +function readJson(path, label) { + const bytes = readStableBytes(path, label, MAX_JSON_BYTES); + return { bytes, value: parseJsonBytes(bytes, label) }; +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); } function validateIdentity() { @@ -194,8 +482,14 @@ function requireExactNames(actual, expected, label) { } } -function validateChecksums(checksumsPath, assetsByName) { - const lines = readFileSync(checksumsPath, "utf8") +function validateChecksums(checksumsBytes, assetsByName) { + let text; + try { + text = new TextDecoder("utf-8", { fatal: true }).decode(checksumsBytes); + } catch (error) { + fail(`SHA256SUMS is not valid UTF-8: ${error instanceof Error ? error.message : String(error)}`); + } + const lines = text .split(/\r?\n/) .map((line) => line.trim()) .filter(Boolean); @@ -308,7 +602,12 @@ function validateReleaseIdentity(view, api, identity, resolvedTagCommitSha) { function run() { const args = parseArguments(process.argv.slice(2)); const identity = validateIdentity(); - const evidence = readJson(args.releaseEvidencePath, "release evidence manifest"); + const canonicalReleaseEvidencePath = resolve(args.assetDir, "release-evidence.json"); + if (args.releaseEvidencePath !== canonicalReleaseEvidencePath) { + fail("release evidence manifest path must identify the exact release asset"); + } + const releaseEvidence = readJson(args.releaseEvidencePath, "release evidence manifest"); + const evidence = releaseEvidence.value; if ( evidence.schemaVersion !== 1 || evidence.source?.repository !== identity.repository @@ -330,11 +629,15 @@ function run() { const expectedNames = sortedAssetNames(assetPaths); const assetsByName = new Map(); for (const path of assetPaths) { - const status = requireRegularFile(path, `release asset ${basename(path)}`, MAX_ASSET_BYTES); + const label = `release asset ${basename(path)}`; + const bytes = path === canonicalReleaseEvidencePath + ? releaseEvidence.bytes + : readStableBytes(path, label, MAX_ASSET_BYTES); assetsByName.set(basename(path), { name: basename(path), - bytes: status.size, - sha256: sha256(path), + bytes: bytes.byteLength, + sha256: sha256(bytes), + retainedBytes: bytes, }); } if (assetsByName.get(sourceName)?.sha256 !== evidence.subject.sha256) { @@ -343,13 +646,17 @@ function run() { if (assetsByName.get("noema.cdx.json")?.sha256 !== evidence.sbom.sha256) { fail("release evidence SBOM digest mismatch"); } - validateChecksums(resolve(args.assetDir, "SHA256SUMS"), assetsByName); + const checksumAsset = assetsByName.get("SHA256SUMS"); + if (!checksumAsset) { + fail("SHA256SUMS release asset is missing"); + } + validateChecksums(checksumAsset.retainedBytes, assetsByName); - const policy = validatePolicy(readJson(args.policyPath, "immutable release policy response")); - const releaseView = readJson(args.releaseViewPath, "release view response"); - const releaseApi = readJson(args.releaseApiPath, "release API response"); + const policy = validatePolicy(readJson(args.policyPath, "immutable release policy response").value); + const releaseView = readJson(args.releaseViewPath, "release view response").value; + const releaseApi = readJson(args.releaseApiPath, "release API response").value; const verification = validateVerification( - readJson(args.verificationPath, "release verification response"), + readJson(args.verificationPath, "release verification response").value, expectedNames, identity, ); @@ -395,10 +702,6 @@ function run() { }; }); - if (existsSync(args.outputPath) && lstatSync(args.outputPath).isSymbolicLink()) { - fail("release publication receipt output must not be a symbolic link"); - } - mkdirSync(dirname(args.outputPath), { recursive: true, mode: 0o755 }); const receipt = { schemaVersion: 1, generatedAt: identity.generatedAt, @@ -413,10 +716,7 @@ function run() { verification, assets, }; - writeFileSync(args.outputPath, `${JSON.stringify(receipt, null, 2)}\n`, { - encoding: "utf8", - mode: 0o644, - }); + writeAtomically(args.outputPath, `${JSON.stringify(receipt, null, 2)}\n`); console.log( `release-publication-receipt: PASS repository=${identity.repository} tag=${identity.tag} head=${identity.commitSha}`, ); diff --git a/test/release-evidence-file-stability.test.ts b/test/release-evidence-file-stability.test.ts new file mode 100644 index 000000000..2fd4101f1 --- /dev/null +++ b/test/release-evidence-file-stability.test.ts @@ -0,0 +1,23 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +describe("release evidence materialization file integrity", () => { + it("consumes source and SBOM bytes through the stable descriptor reader", () => { + const source = readFileSync("scripts/release-evidence.mjs", "utf8"); + + expect(source).toContain('from "./lib/stable-file-evidence.mjs"'); + expect(source).toContain("readStableRegularFile"); + expect(source).not.toContain("readFileSync"); + expect(source).not.toContain("sha256(sourcePath)"); + expect(source).not.toContain("sha256(sbomPath)"); + }); + + it("publishes both retained evidence files through the reviewed atomic writer", () => { + const source = readFileSync("scripts/release-evidence.mjs", "utf8"); + + expect(source).toContain("writeAtomically(manifestPath"); + expect(source).toContain("writeAtomically(checksumsPath"); + expect(source).not.toContain("writeFileSync"); + expect(source).not.toContain("requireSafeOutputPath"); + }); +}); diff --git a/test/release-publication-output-atomicity.test.ts b/test/release-publication-output-atomicity.test.ts new file mode 100644 index 000000000..01f415f5e --- /dev/null +++ b/test/release-publication-output-atomicity.test.ts @@ -0,0 +1,17 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +describe("release publication receipt output", () => { + it("publishes through an unpredictable temporary file plus atomic rename", () => { + const source = readFileSync("scripts/release-publication-receipt.mjs", "utf8"); + + expect(source).toContain("function writeAtomically"); + expect(source).toContain("mkdtempSync"); + expect(source).toContain('flag: "wx"'); + expect(source).toContain("renameSync(temporaryPath, path)"); + expect(source).toContain("writeAtomically(args.outputPath"); + expect(source).not.toContain("writeFileSync(args.outputPath"); + expect(source).not.toContain("existsSync(args.outputPath)"); + expect(source).not.toContain("lstatSync(args.outputPath)"); + }); +}); diff --git a/test/release-publication-runtime-handoff.test.ts b/test/release-publication-runtime-handoff.test.ts new file mode 100644 index 000000000..8d17d3293 --- /dev/null +++ b/test/release-publication-runtime-handoff.test.ts @@ -0,0 +1,27 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +describe("release publication receipt runtime handoff", () => { + it("keeps the isolated handoff executable self-contained", () => { + const receipt = readFileSync("scripts/release-publication-receipt.mjs", "utf8"); + const workflow = readFileSync(".github/workflows/release-evidence.yml", "utf8"); + + expect(receipt).not.toMatch(/from\s+["']\.\//); + expect(workflow).toContain( + "install -m 0644 scripts/release-publication-receipt.mjs release-publication-receipt.mjs", + ); + expect(workflow).toContain( + 'node "$BUNDLE_DIR/release-publication-receipt.mjs"', + ); + }); + + it("authenticates the exact self-contained executable in both artifact handoffs", () => { + const workflow = readFileSync(".github/workflows/release-evidence.yml", "utf8"); + const receiptMentions = workflow.match(/release-publication-receipt\.mjs/g) ?? []; + + expect(receiptMentions.length).toBeGreaterThanOrEqual(5); + expect(workflow).toContain("release-publication-receipt.mjs \\"); + expect(workflow).toContain("sha256sum --check verification-handoff.sha256"); + expect(workflow).toContain("sha256sum --check release-bundle.sha256"); + }); +}); diff --git a/test/stable-release-file-evidence.test.ts b/test/stable-release-file-evidence.test.ts new file mode 100644 index 000000000..ac0d3f825 --- /dev/null +++ b/test/stable-release-file-evidence.test.ts @@ -0,0 +1,191 @@ +import { mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { readStableRegularFile } from "../scripts/lib/stable-file-evidence.mjs"; + +type Metadata = { + dev: number; + ino: number; + mode: number; + size: number; + mtimeMs: number; + ctimeMs: number; + isFile: () => boolean; + isSymbolicLink: () => boolean; +}; + +function metadata(overrides: Partial = {}): Metadata { + return { + dev: 1, + ino: 2, + mode: 0o100600, + size: 3, + mtimeMs: 10, + ctimeMs: 11, + isFile: () => true, + isSymbolicLink: () => false, + ...overrides, + }; +} + +function fakeFileSystem({ + pathMetadata = metadata(), + openedMetadata = metadata(), + finalMetadata = metadata(), + finalPathMetadata = metadata(), + chunks = [Buffer.from("abc")], + constants = { O_RDONLY: 0, O_NOFOLLOW: 0x20000 }, +}: { + pathMetadata?: Metadata; + openedMetadata?: Metadata; + finalMetadata?: Metadata; + finalPathMetadata?: Metadata; + chunks?: Buffer[]; + constants?: { O_RDONLY?: number; O_NOFOLLOW?: number }; +} = {}) { + let statCalls = 0; + let chunkIndex = 0; + let closed = false; + const fileSystem = { + constants, + lstatSync: () => (statCalls++ === 0 ? pathMetadata : finalPathMetadata), + openSync: () => 7, + fstatSync: () => (statCalls++ === 1 ? openedMetadata : finalMetadata), + readSync: (_fd: number, target: Buffer, offset: number, length: number) => { + const chunk = chunks[chunkIndex++]; + if (!chunk) return 0; + const bounded = chunk.subarray(0, length); + bounded.copy(target, offset); + return bounded.length; + }, + closeSync: () => { + closed = true; + }, + }; + return { fileSystem, wasClosed: () => closed }; +} + +describe("stable release file evidence", () => { + it("reads exact bytes from a bounded regular file and rejects a real symlink", () => { + const temp = mkdtempSync(join(tmpdir(), "noema-stable-release-file-")); + try { + const target = join(temp, "target.json"); + const link = join(temp, "link.json"); + writeFileSync(target, "abc", "utf8"); + symlinkSync(target, link); + + expect(readStableRegularFile(target, "release input", 16)).toEqual(Buffer.from("abc")); + expect(() => readStableRegularFile(link, "release input", 16)).toThrow(/symbolic link|no-follow/i); + } finally { + rmSync(temp, { recursive: true, force: true }); + } + }); + + it("fails closed when no-follow or read-only flags are unavailable", () => { + const missingNoFollow = fakeFileSystem({ constants: { O_RDONLY: 0 } }); + expect(() => + readStableRegularFile("evidence", "release input", 16, missingNoFollow.fileSystem), + ).toThrow(/no-follow/i); + + const invalidReadOnly = fakeFileSystem({ constants: { O_RDONLY: -1, O_NOFOLLOW: 1 } }); + expect(() => + readStableRegularFile("evidence", "release input", 16, invalidReadOnly.fileSystem), + ).toThrow(/read-only/i); + }); + + it("rejects invalid arguments, malformed metadata, symlinks, non-files, and unsafe sizes", () => { + expect(() => readStableRegularFile("", "release input", 16)).toThrow(/path/i); + expect(() => readStableRegularFile("evidence", "", 16)).toThrow(/label/i); + expect(() => readStableRegularFile("evidence", "release input", 0)).toThrow(/byte ceiling/i); + + for (const [pathMetadata, expected] of [ + [null as unknown as Metadata, /metadata is unavailable/i], + [{ ...metadata(), isFile: undefined } as unknown as Metadata, /metadata is unavailable/i], + [metadata({ size: Number.NaN }), /invalid byte size/i], + [metadata({ size: -1 }), /invalid byte size/i], + [metadata({ isSymbolicLink: () => true }), /symbolic link/i], + [metadata({ isFile: () => false }), /regular file/i], + [metadata({ size: 0 }), /empty/i], + [metadata({ size: 17 }), /byte ceiling/i], + ] as const) { + const fake = fakeFileSystem({ pathMetadata }); + expect(() => readStableRegularFile("evidence", "release input", 16, fake.fileSystem)).toThrow( + expected, + ); + } + }); + + it("rejects path-to-descriptor identity drift and always closes the descriptor", () => { + for (const openedMetadata of [ + metadata({ dev: 9 }), + metadata({ ino: 9 }), + metadata({ size: 2 }), + metadata({ isFile: () => false }), + ]) { + const fake = fakeFileSystem({ openedMetadata }); + expect(() => readStableRegularFile("evidence", "release input", 16, fake.fileSystem)).toThrow( + /changed before read|regular file/i, + ); + expect(fake.wasClosed()).toBe(true); + } + }); + + it("rejects streamed oversize and descriptor mutation while bytes are consumed", () => { + const oversized = fakeFileSystem({ + pathMetadata: metadata({ size: 3 }), + openedMetadata: metadata({ size: 3 }), + chunks: [Buffer.from("abcd")], + }); + expect(() => readStableRegularFile("evidence", "release input", 3, oversized.fileSystem)).toThrow( + /exceeded.*byte ceiling/i, + ); + expect(oversized.wasClosed()).toBe(true); + + for (const finalMetadata of [ + metadata({ dev: 9 }), + metadata({ ino: 9 }), + metadata({ mode: 0o100644 }), + metadata({ size: 4 }), + metadata({ mtimeMs: 12 }), + metadata({ ctimeMs: 13 }), + metadata({ isFile: () => false }), + ]) { + const fake = fakeFileSystem({ finalMetadata }); + expect(() => readStableRegularFile("evidence", "release input", 16, fake.fileSystem)).toThrow( + /changed while being read|regular file/i, + ); + expect(fake.wasClosed()).toBe(true); + } + }); + + it("rejects pathname replacement after descriptor read even when accepted bytes are unchanged", () => { + for (const finalPathMetadata of [ + metadata({ dev: 8 }), + metadata({ ino: 8 }), + metadata({ size: 4 }), + metadata({ isSymbolicLink: () => true }), + metadata({ isFile: () => false }), + ]) { + const fake = fakeFileSystem({ finalPathMetadata }); + expect(() => readStableRegularFile("evidence", "release input", 16, fake.fileSystem)).toThrow( + /pathname changed|symbolic link|regular file/i, + ); + expect(fake.wasClosed()).toBe(true); + } + }); + + it("rejects short reads instead of hashing or parsing a partial file", () => { + const fake = fakeFileSystem({ + pathMetadata: metadata({ size: 4 }), + openedMetadata: metadata({ size: 4 }), + finalMetadata: metadata({ size: 4 }), + finalPathMetadata: metadata({ size: 4 }), + chunks: [Buffer.from("abc")], + }); + expect(() => readStableRegularFile("evidence", "release input", 16, fake.fileSystem)).toThrow( + /byte count/i, + ); + expect(fake.wasClosed()).toBe(true); + }); +}); diff --git a/vitest.config.ts b/vitest.config.ts index 8ebfba454..179707cb5 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -15,6 +15,7 @@ export default defineConfig({ "scripts/workflow-registry-disable-plan.mjs", "scripts/production-environment-governance-audit.mjs", "scripts/lib/external-scheduler-evidence-audit.mjs", + "scripts/lib/stable-file-evidence.mjs", "scripts/lib/strict-json-evidence.mjs", "scripts/external-scheduler-evidence-audit.mjs", ],