diff --git a/blueprints/us-equities/hosting/README.md b/blueprints/us-equities/hosting/README.md index e54dccee5..6fa87e51b 100644 --- a/blueprints/us-equities/hosting/README.md +++ b/blueprints/us-equities/hosting/README.md @@ -1,135 +1,505 @@ -# Native research workflow hosting - -**Latest:** [Cited research runtime](../research-runtime/README.md) adds a fresh -two-step native Dagu packet/Parquet acceptance and a standalone Claude Opus 5 -report with reconciled telemetry. The paired Astra → Claude recipe is validated -but allowance-blocked. [Native restic acceptance](backup/README.md) demonstrates -encrypted backup and byte-identical restore of 22 selected public reference files. -Neither adds a scheduled broker service or off-host durability. - -Dagu 2.16.6 now hosts local research run history on loopback port 18525. Its -upstream CLI completed the three-step [workflow](equity-research-evidence.yaml): DuckDB -summarized six LEAN simulated events into three orders, then both evidence -validators passed. The native history retained successful, failed and cancelled -runs after a service restart. This is local research hosting, not broker recovery -or a hosted autonomous trading system. - -The pinned Linux amd64 release archive matched the publisher's SHA-256: -`06c3ed951fb58408313b1db25bc9f90ff2f427cbdbe68aaff55cd5465c167717`. -The [receipt](receipt.json) retains the initial schema/environment failures and -the corrected run. Dagu step IDs require underscores; execution requires explicit -environment passthrough in this version. No model inference was needed. - -## Native commands - -Set `DAGU`, `RESEARCH_HOME`, `STACK_REPO`, `SDK_ENV`, `LEAN_EVENTS` and a fresh -`RESEARCH_OUTPUT` directory to your own installed paths. The environment names are -explicitly allowed in [config.yaml.example](config.yaml.example). Install that -configuration privately with mode 0600. Its upstream `auth.mode: none` serves a -passwordless dashboard on `127.0.0.1` only, with DAG write/run permissions disabled. -Do not expose this local mode through a network listener or tunnel. The release contains the binary, -license and current workflow schema. No Docker or cloud account is required. +# Unattended US-equities research and journal recovery + +The templates run the named `nyse-post-close-evidence` research job with Dagu +**2.16.6** and recover selected research SQLite journals with restic **0.19.1**. +This serves US-equities research and historical simulation: the DAG summarizes +an operator-supplied local LEAN simulation and validates its evidence. The research +operator's simulation job owns refreshing `LEAN_EVENTS` after that session's +actual close and **by 16:25 America/New_York**, before the 16:30 attempt. It neither +acquires market data nor executes broker orders. + +The [decision](../../../docs/decisions/2026-10-05-trading-unattended-hosting-recovery.md) +records the alternatives, scheduler comparison and remaining gates. These are +portable examples. PR-4 installed no unit, started no scheduler and used no +existing host service. Operating-host and alert selection is **user decision 2**. + +## One Dagu process and an explicit job environment + +[dagu-equities.service.example](dagu-equities.service.example) runs `dagu start-all` +with `Restart=on-failure`, combining the web UI and scheduler in one process. +[config.yaml.example](config.yaml.example) disables the optional coordinator in +that file; it otherwise defaults to enabled. Sources: +`dagucloud/dagu@v2.16.6:internal/cmd/startall.go:32-45` and +`internal/cmn/config/loader.go:1311-1319` +([start-all](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/cmd/startall.go#L32), +[default](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/cmn/config/loader.go#L1311)). + +Replace every `/path/to/...` in a **private copy** of the unit and DAG. Put the +four job variables `STACK_REPO`, `SDK_ENV`, `LEAN_EVENTS` and `RESEARCH_OUTPUT` +in the DAG's `env:` block. This keeps the job complete when invoked by either +the scheduler or native CLI. `SDK_ENV` is the installed SDK built from +[`adoption/sdk/requirements-linux-x86_64-py313.lock:241`](../../../adoption/sdk/requirements-linux-x86_64-py313.lock), +which locks exchange-calendars 4.13.2. Use that maintained native environment. +The scheduled job adds no second runtime or lock; the restart drill alone uses +the isolated pinned PyYAML overlay described below. +An `Environment=` line in the unit cannot supply them: `env -i` discards that +inherited environment. Sources: `coreutils/coreutils@v9.4:src/env.c:824-838` +([implementation](https://github.com/coreutils/coreutils/blob/v9.4/src/env.c#L824)) and +`dagucloud/dagu@v2.16.6:internal/spec/dag.go:1663-1673` +([DAG env](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/spec/dag.go#L1663)). +`env_passthrough` is an **allowlist**. The four scheduled-job entries and all seven +entries used by [research-runtime](../research-runtime/README.md) remain in the +config: `LEAN_RESULTS`, `RESEARCH_WORKSPACE`, `NATIVE_CODEX_HOME`, +`NATIVE_CODEX_BIN`, `NATIVE_CLAUDE_BIN`, `NATIVE_RUNTIME_PATH` and +`SDK_OBSERVATION_DIR`. Those recipes must also supply their own environment to +their native CLI calls; the scheduled DAG obtains its four values from its own +block. Sources: `dagucloud/dagu@v2.16.6:internal/cmn/config/loader.go:344-347` +and `internal/cmn/config/env.go:77-95`. + +Deploy only the private research DAG to the chosen Dagu DAG directory, with its +local input and output paths adapted. Each run writes +`order-events-${DAG_RUN_ID}.parquet`, preserving earlier outputs and avoiding the +summarizer's overwrite refusal on the following session. Dagu supplies this +identifier (`dagucloud/dagu@v2.16.6:internal/runctx/context_env.go:28`, +`internal/cmn/runenv/keys.go:12-13` +[source](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/cmn/runenv/keys.go#L12)). + +The UI listens on loopback and keeps `permissions.run_dags: false` and +`permissions.write_dags: false`. On 2.16.6 the run permission is an API guard; +the scheduler's local dispatch path starts its subprocess independently. This +is a **source finding**, awaiting observation in the operating-host drill. +Sources: `dagucloud/dagu@v2.16.6:internal/service/frontend/api/v1/dagruns.go:163-167` +([API guard](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/service/frontend/api/v1/dagruns.go#L163)) +and `internal/service/scheduler/dag_executor.go:280-321` +([local dispatch](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/service/scheduler/dag_executor.go#L280)). +Anonymous loopback observation requires a trusted local user; these permissions +do not cover all administration APIs. DAG artifact storage stays off. + +## Native installation and operating-host service boundary + +These commands are **for the operator later**, after user decision 2 selects the +operating host. Use a clean upstream release on a new Linux/WSL host, verified +against its publisher's checksum; reuse a previously verified installation at the +same pin. For Linux amd64 the historical verified archive SHA-256 is +`06c3ed951fb58408313b1db25bc9f90ff2f427cbdbe68aaff55cd5465c167717` +([historical receipt](receipt.json)); still verify the downloaded archive against +the tagged release's `checksums.txt`: ```sh gh release download v2.16.6 --repo dagucloud/dagu \ --pattern dagu_2.16.6_linux_amd64.tar.gz --pattern checksums.txt -sha256sum dagu_2.16.6_linux_amd64.tar.gz awk '$2 == "dagu_2.16.6_linux_amd64.tar.gz"' checksums.txt | sha256sum --check --strict +tar -xzf dagu_2.16.6_linux_amd64.tar.gz "$DAGU" version -mkdir -p "$RESEARCH_OUTPUT" -env -i HOME="$HOME" PATH=/usr/bin:/bin \ - STACK_REPO="$STACK_REPO" SDK_ENV="$SDK_ENV" \ - LEAN_EVENTS="$LEAN_EVENTS" RESEARCH_OUTPUT="$RESEARCH_OUTPUT" \ - "$DAGU" start --context local --dagu-home "$RESEARCH_HOME" \ - --run-id "$RUN_ID" "$STACK_REPO/blueprints/us-equities/hosting/equity-research-evidence.yaml" -"$DAGU" history --context local --dagu-home "$RESEARCH_HOME" --format json +install -m 0600 config.yaml.example "$RESEARCH_HOME/config.yaml" ``` -For a new host, adapt the binary/home paths in the example user unit, install it -as `~/.config/systemd/user/dagu-equities.service`, and then run: +Set `DAGU`, `RESEARCH_HOME`, `STACK_REPO`, `SDK_ENV`, `LEAN_EVENTS` and +`RESEARCH_OUTPUT` to private operator paths, adapt the private DAG and unit, and +keep **config and private DAG mode 0600**. Download/extraction must target a +private installation directory, with `DAGU` pointing to its extracted executable. +The supported upstream release is +[dagucloud/dagu@v2.16.6](https://github.com/dagucloud/dagu/releases/tag/v2.16.6); +no rebuilt Dagu or restic binary belongs to this recipe. + +`auth.mode: none` permits anonymous reads and some administration on loopback; +**do not expose it with a network listener or tunnel**. The historical Basic +configuration returned 401 and repeatedly challenged the browser. When changing +to `none`, remove the `auth.basic` subsection: the native config validator refuses +a Basic block under that mode. The [dated auth observation](../../../evidence/artifacts/gap-wave2-20260923/us-equities__data-quality-orchestration/2-dashboard-auth-mode.json) +retains that finding, distinct from this round's offline probes. Use upstream +authentication in a separately reviewed network deployment. `run_dags: false` +and `write_dags: false` do not form a global Viewer role. Artifact storage is +enabled by default; this DAG disables it explicitly. Native scratch acceptance +on **2.16.6 and 2.17.2** found that `base.yaml` does not propagate that key. +**2.17.2 adds anonymous cross-run `GET /api/v1/artifacts` under `none`**; its +historical boundary remains relevant to any later pin change. See the +[2026-09-25 source/observation record](../../../docs/decisions/2026-09-25-workstation-sota-refresh.md). +This round keeps 2.16.6 and runs neither version's server. + +After adapting and placing the private unit under the operating user's +`~/.config/systemd/user/dagu-equities.service`, the operator's lifecycle commands +are: ```sh +loginctl enable-linger systemctl --user daemon-reload systemctl --user enable --now dagu-equities.service systemctl --user restart dagu-equities.service +systemctl --user status dagu-equities.service "$DAGU" history --context local --dagu-home "$RESEARCH_HOME" --format json +# Later removal preserves the evidence directories: +systemctl --user disable --now dagu-equities.service +``` + +`loginctl enable-linger` without a name selects the caller: it keeps the user +manager alive after logout and permits boot activation +(`systemd/systemd@v255:man/loginctl.xml:186-195` +[source](https://github.com/systemd/systemd/blob/v255/man/loginctl.xml#L186)). +It **cannot keep a WSL VM or Windows host running**. Windows/WSL shutdown, VM +lifetime and host reboot interrupt this user unit; unattended availability still +needs the later host receipt. `start-all` replaces the previous manual-history +`server` example and owns the scheduler within the same supervised process. +`Restart=on-failure` provides process restart, with no claim of resuming in-flight +steps or catching up a missed schedule. The builder executed none of these +service/linger commands. + +## NYSE session schedule + +[equity-research-evidence.yaml](equity-research-evidence.yaml) names its schedule +contract **nyse-post-close-evidence** in the description and tag. Its cron is +`CRON_TZ=America/New_York 30 16 * * 1-5`: one attempt at 16:30 New York time on +each weekday, with daylight-saving changes handled by the explicit time zone. +This pin has no separate schedule-name field; its cron object permits an +expression and optional runtime profile. Sources: +`dagucloud/dagu@v2.16.6:internal/cmn/schema/dag.schema.json:855-876`, +`internal/ir/schedule.go:281-305`, `go.mod:65`, and the locked parser +`robfig/cron@v3.0.1:parser.go:93-103` +([timezone parser](https://github.com/robfig/cron/blob/v3.0.1/parser.go#L93)). + +Cron cannot skip exchange holidays. The normal `calendar_check` step runs +[session_day.py](session_day.py) in `SDK_ENV`, checks exchange-calendars **4.13.2**, +and queries XNYS for today's New York date and actual close. It constructs bounds +from **December 1 of the previous year through January 31 of the following year**, +bracketing New Year's Day and observed January holidays before the first session. +The native calendar's session boundaries determine coverage +(`gerrymanoim/exchange_calendars@4.13.2:exchange_calendars/exchange_calendar.py:1257-1279`). +A truly uncovered date remains a hard error. A covered non-session prints +`non_session`; the `nyse_session` precondition skips that step and its dependents. +Import, package-version or calendar failures fail the normal check step instead +of being mistaken for a holiday. A manual run before close prints `before_close` +and skips the research chain. + +The SDK lock cited above is on this branch's base. Calendar sources: +`gerrymanoim/exchange_calendars@4.13.2:exchange_calendars/exchange_calendar.py:1012-1016,1263-1279` +and `exchange_calendars/exchange_calendar_xnys.py:157-165` +([session API](https://github.com/gerrymanoim/exchange_calendars/blob/4.13.2/exchange_calendars/exchange_calendar.py#L1263), +[close times](https://github.com/gerrymanoim/exchange_calendars/blob/4.13.2/exchange_calendars/exchange_calendar_xnys.py#L157)). +**Early-close days still run at 16:30**, after their earlier close, with no +additional cron or run. Step skip and dependency propagation come from +`dagucloud/dagu@v2.16.6:internal/runtime/runner.go:1633-1648,1267-1281` +([precondition result](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/runtime/runner.go#L1633)). + +On a session after close, `--events "${LEAN_EVENTS}"` also requires the events +file's mtime to be between that session's actual close and the check instant, +and no later than **16:25 America/New_York** on that date. +`missing_input`, `stale_input`, `future_input` or `late_input` are visible normal-step output +tokens with **exit 1**, so Dagu fails the run and blocks the dependent evidence +chain. They appear in `dagu history --status failed`; only `non_session` and +`before_close` retain a successful guard and a skipped research chain. +Calendar/import/IO errors remain failures. The producer must complete and atomically publish that session's +local LEAN simulation output by 16:25; mtime is an availability contract, not +proof of the market dates or quality inside a file. No upstream acquisition job +is introduced here. + +## Manual run, cancellation and replay + +On the chosen operating host, use the same private config and DAG as the service. +Start a manual check with a fresh, operator-chosen run ID: + +```sh +"$DAGU_BIN" start --context local --dagu-home "$PRIVATE_DAGU_HOME" \ + --config "$PRIVATE_CONFIG" --run-id "$NEW_RUN_ID" "$PRIVATE_DAG" +``` + +In another terminal, cancel that exact active run and inspect its native history: + +```sh +"$DAGU_BIN" stop --context local --dagu-home "$PRIVATE_DAGU_HOME" \ + --config "$PRIVATE_CONFIG" --run-id "$NEW_RUN_ID" equity-research-evidence +"$DAGU_BIN" history --context local --dagu-home "$PRIVATE_DAGU_HOME" \ + --config "$PRIVATE_CONFIG" --run-id "$NEW_RUN_ID" \ + --status succeeded,failed,aborted --limit 1000 --format json equity-research-evidence +``` + +Inspect an aborted or failed run before deciding whether recovery is possible. +A missing, stale or late input failure **cannot be replayed for that session**: +correcting the input after the scheduled run would miss the **16:25 New York** +publication cutoff. After a later-step failure, replay with another fresh run ID +using the start command **on the same New York date, after that session's close**, +with the **unchanged, on-time input**. Its mtime must remain after the close and +no later than both the check time and that date's 16:25 cutoff. +`DAG_RUN_ID` keeps earlier output intact. A later replay +evaluates its current New York date: before close or on a non-session day it +records a succeeded run with skipped research steps; after another session's +close it processes that other session, subject to the freshness guard. It does +not recover the original session. Before counting a replay as recovery, inspect: + +```sh +"$DAGU_BIN" status --context local --dagu-home "$PRIVATE_DAGU_HOME" \ + --config "$PRIVATE_CONFIG" --run-id "$NEW_RUN_ID" equity-research-evidence +``` + +Every research node (`calendar_check`, `nyse_session`, `prepare_output`, +`summarize`, `baseline_evidence`, `catalog_evidence`) must show succeeded; +a green run with skipped nodes does not count as recovery. A manual run never +proves scheduler dispatch or catch-up. These +commands are operator procedures, not commands run by this builder. Sources: +`dagucloud/dagu@v2.16.6:internal/cmd/start.go:45-67; internal/cmd/stop.go:20-38,53-91; +internal/cmd/history.go:160-345; internal/cmd/flags.go:463-481; internal/cmd/status.go:18-47`, corroborated by +the installed 2.16.6 `start`, `stop` and `history --help` (all exit 0). + +## Journal snapshot and restic procedure + +[journal_recovery.py](journal_recovery.py) stays in `hosting/` alongside the job +and operator drills. It owns recovery of selected research files without coupling +to a broker adapter or changing an order-state writer. It is stdlib integration +glue around CPython's published Online Backup example and native restic commands. + +Select **every journal required by the research consumer**, with explicit required +tables. The procedure opens each source read-only and uses +`sqlite3.Connection.backup(..., pages=-1)` within a read transaction to a new staging directory, then closes the destination +in DELETE journal mode. Never copy an active SQLite, WAL or SHM file directly. +The source must already use **WAL**; the worker checks `PRAGMA journal_mode` +and refuses other modes by default without modifying the source. Explicit +`--allow-non-wal` on `snapshot` or `cycle` accepts writer blocking: the source +read transaction holds a SHARED lock throughout the bounded copy, so a +rollback-journal writer can receive `SQLITE_BUSY` while trying to commit. +Prefer quiescing those writers before opting in. WAL permits concurrent writers. +Source: `sqlite/sqlite@version-3.53.1:src/backup.c:346-354,383-389`. +Sources: `python/cpython@v3.12.3:Doc/library/sqlite3.rst:1107-1155` +([concurrent backup and example](https://github.com/python/cpython/blob/v3.12.3/Doc/library/sqlite3.rst#L1107)) +and `Modules/_sqlite/connection.c:2013,2067-2102`. One native step avoids restarting +an incremental copy on each WAL commit. An isolated stdlib subprocess bounds each +copy at **120 seconds**, including worker startup; timeout kills and waits for +the worker and prevents inventory publication. The maximum logical journal size +is **256 MiB** per source, checked in the read transaction. Operators can set +`--backup-timeout` and `--max-journal-bytes` after measuring the required sizes and +copy times; neither limit silently drops a journal. These are explicit integration +policy limits, not upstream defaults. Source: +`python/cpython@v3.12.3:Doc/library/subprocess.rst:62-68` (native timeout). +The timeout bounds the spawned worker; the OS's initial process creation itself +cannot always be interrupted, as the same source explains. +Each journal is independently +consistent; this does not create an atomic transaction across journals. +Consumers needing one must quiesce their writers before taking the snapshots. + +The frozen inventory records every staged file's relative path, SHA-256, bytes, +`PRAGMA integrity_check` and required tables' row counts. It is written inside +staging and as a separate immutable **external oracle**. Retain that oracle +independently from the restored copy. Staging stays frozen through backup; a +failed snapshot operation never becomes an accepted inventory. + +Every nonzero native restic exit fails the procedure, including **exit 3**, when +restic creates a snapshot but cannot read all source files. Snapshot existence +is insufficient for acceptance. Source: +`restic/restic@v0.19.1:doc/040_backup.rst:787-806` +([exit codes](https://github.com/restic/restic/blob/v0.19.1/doc/040_backup.rst#L787)). + +Run the cycle **daily**, including weekends and holidays, +using one private rotation-state file per repository. Default checks rotate +`1/7` through `7/7` in seven daily invocations. The cursor +advances only after successful native checking. The **whole 1..7 rotation**, from +its cycle-start timestamp through successful completion and the next rollover, +has an **eight-day elapsed-time bound in UTC**. The one-day slack covers the +25-hour fall-back day plus 60 seconds of daily start drift: the rollover is +seven days and one hour later across fall-back, or seven days and seven minutes +later with that drift. There is no strict per-step 24-hour cutoff. A +cycle origin older than eight days, even when subset 7 completed and the cursor +has returned to 1, a gap since the last success over eight days, reversed +clock or pre-v2 state requires `--full-check`; only successful full reading can +restart the cursor. Check the previous origin before resetting it for subset 1. +The script captures the invocation time before snapshot/backup and checks completion +time again before publishing state. This is an operator execution requirement; no backup timer +was enabled by PR-4. Every cycle also restores and compares its snapshot. +Source: `restic/restic@v0.19.1:doc/045_working_with_repos.rst:482-521` +([deterministic partitions](https://github.com/restic/restic/blob/v0.19.1/doc/045_working_with_repos.rst#L482)). + +Restore the exact returned snapshot ID's staging subtree to a **new** directory, +with `--verify --overwrite never`. Compare the exact restored file set, inventory +bytes and every file's SHA-256 and byte count against the external oracle; then +check journal integrity and required row counts. **integrity_check and row counts +alone do not prove completeness**: altered records can preserve both. +Source: `restic/restic@v0.19.1:doc/050_restore.rst:55-73` +([subfolder restore](https://github.com/restic/restic/blob/v0.19.1/doc/050_restore.rst#L55)). + +For an already initialized **local** repository and its operator-owned password +file, the template command is: + +```sh +env TMPDIR="$PRIVATE_TMP" nice -n 19 python3 \ + blueprints/us-equities/hosting/journal_recovery.py cycle \ + --restic-bin "$RESTIC_BIN" --repository "$LOCAL_REPOSITORY" \ + --password-file "$RESTIC_PASSWORD_FILE" --work "$NEW_PRIVATE_WORK" \ + --rotation-state "$PRIVATE_ROTATION_STATE" \ + --backup-timeout 120 --max-journal-bytes 268435456 \ + --journal "research=$RESEARCH_JOURNAL" --required-table research=events \ + --journal "evidence=$EVIDENCE_JOURNAL" --required-table evidence=runs +``` + +Adapt names and tables to the consumer's contract; `events` and `runs` are the +fixture tables. The work directory must be new. The password file must be outside +staging; its contents are never read or printed by the Python script. Keep the +repository, raw output, state and oracle private. Native `fcntl.flock` owns the +rotation lock; the `.lock` inode remains but process exit releases the lock. +A stale lock file is safe to reuse; a live holder causes visible refusal. Never +unlink it to bypass a holder. Native temporary-file plus `os.replace` publication +avoids stale `.new` collisions; preserve failed artifacts without resetting the +cursor. Source: `python/cpython@v3.12.3:Doc/library/fcntl.rst:139-149`. + +Backup uses **`--group-by host,tags`**, with the stable host +`equity-research-recovery` and tag `journal-recovery`, so new staging paths share +a parent group. Any later `forget --keep-*` policy must use **the same grouping**, +filtered to that host/tag; first inspect `forget --dry-run --group-by host,tags` +with the chosen retention counts. Retention counts and deletion remain unaccepted; +this procedure performs neither forget nor prune. Sources: +`restic/restic@v0.19.1:doc/040_backup.rst:197-206` and +`doc/060_forget.rst:225-233`. +Controls are for **disposable repositories only**. Their backups use the distinct +host `equity-research-recovery-control` and tag `journal-recovery-control` with +the same `host,tags` grouping. An exit-3 control snapshot therefore cannot become +a production backup parent or count in production retention. The local drill +creates its disposable repository; never point a control at the operating repository. + +Remote repository URLs are outside this local procedure. A second-host consumer +can run `verify --restored "$RESTORE" --inventory "$FROZEN_INVENTORY"` against +its independently retained oracle after the separately approved destination +restore. Key recovery belongs to user decision 3. + +## Same-host operator drills and proof + +Run [drill_process_restart.py](drill_process_restart.py) **later on the operating +host**, shortly before the next eligible 16:30 slot, using an isolated uv overlay +over its locked trading Python. This drill uses upstream **PyYAML 6.0.3** to parse the +deployed configuration, rather than treating a comment as an active permission. +Run only the drill in that overlay, leaving the hash-locked `SDK_ENV` unchanged. +Set the following private paths and the next due slot on the operating host: + +```sh +env TMPDIR="$PRIVATE_TMPDIR" nice -n 19 \ + uv run --no-project --python "$SDK_ENV/bin/python" --with PyYAML==6.0.3 \ + python "$STACK_REPO/blueprints/us-equities/hosting/drill_process_restart.py" \ + --dagu-bin "$DAGU_BIN" --dagu-home "$PRIVATE_DAGU_HOME" \ + --config "$PRIVATE_CONFIG" --dag-history "$PRIVATE_DAG_HISTORY" \ + --due-at "$NEXT_DUE_SLOT" ``` -Choose a new `RUN_ID` and output directory for each run. The summarizer refuses -to overwrite an existing Parquet. Native `dagu stop --run-id ... ` -cancelled an intentionally long local step; `history` reported `aborted` even -though that cancelled CLI process returned zero. Consumers must inspect native -status, not only process exit codes. A separate exit-23 fixture recorded `failed` -and aborted its dependent step. - -**DAG name (2026-09-25).** The workflow file was renamed from -`research-evidence.yaml` to `equity-research-evidence.yaml`, and its top-level -`name:` was removed. Dagu now derives the DAG name, `equity-research-evidence`, -from the filename. That is the same name the 2026-09-19 runs used. - -The old file failed `dagu validate` under its own name on both 2.16.6 and -2.17.2, with "entrypoint document must not define name". Only a copy whose -filename matched `name:` passed. - -The renamed file validates on both versions. `dagu dry` reports -`equity-research-evidence` on both. The [receipt](receipt.json) keeps the old -path and its sha256 as historical evidence. - -## Service boundary - -The installed [user service](dagu-equities.service.example) runs `dagu server`, -not the scheduler, with a minimal inherited environment and no provider or broker -credentials. It is enabled for future user-service sessions. The current local -dashboard uses upstream `auth.mode: none`: anonymous API reads return 200 without -a Basic-auth challenge. The previous Basic mode returned 401 and repeatedly -prompted the browser. Remove its `auth.basic` subsection when changing modes: -upstream config validation rejects an `auth.basic` block under `none`, and the -server refuses to start (re-observed on 2.16.6 on 2026-09-23). Basic headers on -requests are a separate matter: under `none` they are ignored. On 2026-09-23 the -running service served `authMode: "none"` and answered 200 to both an anonymous -and a dummy-Basic GET of `/api/v1/dags`, while loopback control instances of the -same binary returned 401 anonymous under `basic`; that -[dated receipt](../../../evidence/artifacts/gap-wave2-20260923/us-equities__data-quality-orchestration/2-dashboard-auth-mode.json) -supersedes the 2026-09-19 basic-auth status codes in [receipt.json](receipt.json). - -DAG execution and DAG/wiki writes remain disabled by `run_dags: false` and -`write_dags: false`. These are not blanket read-only controls: base configuration, -views and managed-secret administration use separate role checks and remain -available to trusted local users without authentication. Keep this configuration -on loopback. Use upstream builtin/OIDC authentication for access beyond this PC. -Manual native CLI commands remain the workflow execution lane. Private -configuration and history are stored beneath the user's local application data. - -**Anonymous reads under `auth.mode: none` (2026-09-25).** Every read route is -anonymous under `none`, not only the DAG list and status. On 2.16.6 that -includes the per-run artifact file tree, preview and download. - -Two things keep observation to DAG list and status today: -- the loopback-only listener; -- no deployed DAG writing artifacts. - -Dagu does not enforce it. Artifact storage is on by default. On both 2.16.6 and -2.17.2, native scratch runs showed: -- a DAG with no `artifacts` key stored the file its step wrote; -- a DAG-level `artifacts: {enabled: false}` stored nothing; -- `base.yaml` did not pass that key on, although its `env` did reach the step. - -The research workflow therefore disables artifacts itself. - -Dagu 2.17.2 also adds an anonymous cross-run `GET /api/v1/artifacts`, which -cannot be disabled under `none`. The host stays on 2.16.6 until a need for -2.17.2 justifies that surface. The first-run example DAGs Dagu creates are -never run here (`run_dags: false`); `example-05` writes artifacts if anything -runs it. - -`systemctl --user disable --now dagu-equities.service` stops and disables it -without deleting evidence. The service restarts on process failure; the accepted -restart check establishes history persistence, not resumption of in-flight work. -Windows/WSL shutdown stops this host. No availability guarantee, remote access, -automatic schedule, cloud deployment, recurring model dispatch or broker order -writer is implied. Same-user host commands are not a security sandbox. - -Upstream: [release](https://github.com/dagucloud/dagu/releases/tag/v2.16.6), -[pinned schema](https://github.com/dagucloud/dagu/blob/v2.16.6/README_SCHEMA.md), -[native CLI](https://docs.dagu.sh/getting-started/cli), -[systemd deployment](https://docs.dagu.sh/server-admin/deployment/systemd). +`--no-project` avoids project synchronization; uv layers the parser separately +while retaining the locked calendar's site packages. Sources: +`astral-sh/uv@0.12.17:crates/uv/src/commands/project/run.rs:903-920,1040-1105` +([overlay implementation](https://github.com/astral-sh/uv/blob/0.12.17/crates/uv/src/commands/project/run.rs#L1040)); +the isolated pinned dependency precedent is +`tools/runtime-worker-evidence/checks.json:40-43`. +The parser is `yaml/pyyaml@6.0.3:lib/yaml/__init__.py:117-125` +([safe_load](https://github.com/yaml/pyyaml/blob/6.0.3/lib/yaml/__init__.py#L117)); +Dagu's mapping/default and legacy override are in +`dagucloud/dagu@v2.16.6:internal/cmn/config/loader.go:586-611`. +Supply `--dagu-bin`, `--dagu-home`, `--config`, this one DAG's +`--dag-history` directory and the exact aware `--due-at` timestamp. The script +verifies an existing unit's MainPID and executable, then uses native +`systemctl --user kill --kill-whom=main --signal=SIGKILL ` to address that +unit's current main process. The window must reserve **more than 150 seconds** +before due (90 seconds restart allowance plus 60 seconds margin) and 180 seconds +after due for completion; it is checked again just before fault delivery. +A restart observed at or after due fails with a distinct missed-slot reason. +The drill observes automatic restart and waits for the next successful due run. +Native history plus original `status.jsonl` must attest the exact `scheduleTime` +and scheduler trigger, with every required research step succeeded, while the +deployed UI has `run_dags: false`. A skipped evidence chain does not count. A manual run +does not count. The script installs, enables and starts no unit. SIGKILL induces +failure; under `Restart=on-failure`, clean SIGTERM need not trigger restart +(`systemd/systemd@v255:man/systemd.service.xml:818-836` +[source](https://github.com/systemd/systemd/blob/v255/man/systemd.service.xml#L818)). +Unit-addressed fault delivery is provided by +`systemd/systemd@v255:man/systemctl.xml:541-549,2377-2388`. +Dagu's supported REST run-details API also exposes `scheduleTime` and +`triggerType` (`dagucloud/dagu@v2.16.6:internal/service/frontend/api/v1/dagruns.go:2066-2096; +transformer.go:362,371`). This later acceptance drill retains original +`status.jsonl` to hash and corroborate the native bytes without an HTTP/auth +dependency. That internal format requires revalidation on a Dagu pin move. +The omitted `catchup_window` does not replay missed slots +(`dagucloud/dagu@v2.16.6:internal/cmn/schema/dag.schema.json:151-153`). + +**A process-restart drill is not reboot, missed-run or independent-alert acceptance.** +It observes one process failure and the following slot. Host boot, downtime across +slots, catch-up, duplicate runs and independent notification need separate host +receipts and the chosen alert path. + +Run the disposable local backup-check-restore drill with: + +```sh +env TMPDIR="$PRIVATE_TMP" nice -n 19 python3 \ + blueprints/us-equities/hosting/drill_local_recovery.py \ + --restic-bin "$RESTIC_BIN" --output "$NEW_SANITIZED_REPORT" +``` + +It generates a 0600 throwaway unlock file for restic and later deletes it, +reporting the cleanup boolean as `throwaway_unlock_file_deleted`. It creates two +synthetic WAL journals and commits to each while its Online Backup copy is active. It backs +up, reads subset **1/1** (every pack in this small proof), restores and independently +compares bytes, hashes and required state. Stdlib unittest separately exercises +the default seven-part rotation. It also runs these controls: + +- `--control snapshot` makes one staged file unreadable: native backup must exit + **3** and the procedure must exit nonzero before checking or restoring. +- `--control restore` alters the restored SQLite `user_version` header while + integrity, required table counts **and file bytes remain equal**. The SHA-256 + comparison must make the procedure exit nonzero, proving that those secondary + checks alone would accept incomplete recovery of the original state. + +The [round 2 sanitized offline proof](evidence/offline-proof-r2.json) and +[local integration receipt](../../../evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json) +retain actual native output and command exits with separate observations. +The fixture is our integration check, using the unchanged restic 0.19.1 executable. +Neither operator drill was run on an operating service. + +The scripts require **Python 3.11 or later**: `hashlib.file_digest` was added in +3.11 (`python/cpython@v3.12.3:Doc/library/hashlib.rst:267-302`). The builder checked +that API and executed on Python 3.13.15; it did not run a separate 3.11 interpreter. +The receipt notes the builder's RTK fallback; portable recipes require only plain +`env`, `nice`, the locked SDK where needed, and the upstream binaries. The retained +[round 1 proof](evidence/offline-proof.json) remains historical evidence. +The [native Dagu probes](evidence/native-dagu-probes-r2.json) observed both skip +and success chains. The [pinned cron Next probe](evidence/cron-next-probe-r2.json) +was prepared from v3.0.1 source but returned 127 because no Go compiler was found; +its DST expectations are recorded, not measured. No compiler was installed. + +The [round 3 native DAG proof](evidence/native-dagu-probes-r3.json) executes all +six deployed steps in a private scratch copy with the locked SDK, synthetic +historical LEAN input and a fixed post-close guard clock. All six steps and the +run succeed; native history with `--context local --from --status succeeded +--limit 1000` returns that manual run. Separate real-guard missing/stale/future +probes each exit 1 and record failed run/guard status 2 with no dependent marker. +The [round 3 recovery proof](evidence/offline-proof-r3.json) observes concurrent +WAL writes during both copies and verifies that both failing controls use their +separate host/tag group. These are local integration fixtures; no scheduler or +operating-host restart ran. The [round 3 receipt](../../../evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json) +pins tested source hashes and records the red/green regressions and native output. + +## Why this glue exists + +The [pinned native review](evidence/upstream-glue-review-r3.json) records installed +help, tagged release notes/source and the alternatives checked before repair. +Its parent link preserves the unchanged [round 2 review](evidence/upstream-glue-review-r2.json) +for the earlier CLI and release observations. +Native `ecal` **does exist** in exchange_calendars 4.13.2. It renders calendars; +it provides neither the post-close eligibility token nor this input-freshness +contract. Native Dagu already owns preconditions, dependency skips, retries and +handlers; none is reimplemented here. No retry/handler policy is enabled by this +job, and independent alerts still need user decision 2. + +| Kept integration | One-line gap justification at the pin | +| --- | --- | +| `session_day.py` | `gerrymanoim/exchange_calendars@4.13.2:pyproject.toml:67-68; exchange_calendars/ecal.py:100-149` ships a calendar renderer; its `exchange_calendar.py:1012-1016,1257-1279` APIs supply session/close queries, leaving our eligibility/freshness token as job policy. | +| `journal_recovery.py` | `restic/restic@v0.19.1:doc/040_backup.rst:679-703; doc/045_working_with_repos.rst:482-521; doc/050_restore.rst:55-73` supplies stream backup, partition checks and restore, leaving the external frozen oracle, required table contract, bounded rotation and failure controls to the consumer. | +| `drill_process_restart.py` | `yaml/pyyaml@6.0.3:lib/yaml/__init__.py:117-125` supplies YAML parsing; `systemd/systemd@v255:man/systemctl.xml:541-549,2377-2388` supplies unit-addressed kill and `man/systemd.service.xml:818-836` supplies restart; Dagu `@v2.16.6:internal/service/frontend/api/v1/transformer.go:362,371` supplies slot/trigger fields, leaving identity/time/holiday guards and exact-slot correlation as acceptance policy. | +| `drill_local_recovery.py` | `python/cpython@v3.12.3:Modules/_sqlite/connection.c:2067-2102` supplies Online Backup and restic `@v0.19.1:doc/050_restore.rst:55-73` supplies restore, leaving the concurrent-write fixture, two deliberately failing controls and path-free receipt as our local integration evidence. | + +Specifically checked: `restic backup --stdin-from-command -- sqlite3 "$DB" +".backup ..."` captures **stdout**, while SQLite's `.backup` writes a destination +file (`sqlite/sqlite@version-3.53.1:src/shell.c.in:9074-9099`); it would save an +empty stream instead of that DB. `.dump` emits SQL (`src/shell.c.in:3759`), which +could form a logical restore procedure but does not preserve the already-frozen +binary file inventory. Keep the **native Online Backup API**, not a rebuilt SQLite +backup implementation. CPython's single native step has no wall-time argument +(`python/cpython@v3.12.3:Modules/_sqlite/connection.c:2011-2016,2075-2102`), so +stdlib subprocess timeout bounds it. Native `fcntl.flock` replaces our previous +exclusive-create lock/sentinel, including automatic release after a crash. + +The maintained upstream **sqlite3_rsync** utility also copies live databases +(`sqlite/sqlite@version-3.53.1:tool/sqlite3_rsync.c:13-40`); it is a replication +protocol rather than this encrypted restic repository plus consumer-oracle +procedure. The final T **off-host-recovery row, line 102**, retains restic and the +stdlib Online Backup API; the prior hosting comparison deferred Litestream. This +repair keeps that decision and does not reopen it. The final row does not name +Litestream; the earlier comparison is recorded separately in the pinned review. + +**The same-host drill cannot close the off-host-recovery gate.** Closure needs +**user decision 3**: an independent destination, key recovery and a restore by a +consumer on a second host, checked against the independent oracle and required +state. Its recovery-time objective must still be selected and measured. The +scratch proof establishes no off-host durability, retention deletion or alert. + +Historical [Dagu receipt](receipt.json) and +[static-file backup acceptance](backup/README.md) retain their original scopes; +neither attests execution of the new scheduled templates. diff --git a/blueprints/us-equities/hosting/config.yaml.example b/blueprints/us-equities/hosting/config.yaml.example index 4c4cc00ed..4ffcf0b1e 100644 --- a/blueprints/us-equities/hosting/config.yaml.example +++ b/blueprints/us-equities/hosting/config.yaml.example @@ -2,11 +2,17 @@ host: 127.0.0.1 port: 18525 check_updates: false metrics: private +# dagucloud/dagu@v2.16.6:internal/cmd/startall.go:32-45 starts both services; +# internal/cmn/config/loader.go:1311-1319 defaults this optional coordinator on. +coordinator: + enabled: false env_passthrough: - STACK_REPO - SDK_ENV - LEAN_EVENTS - RESEARCH_OUTPUT + # dagucloud/dagu@v2.16.6:internal/cmn/config/loader.go:344-347; + # internal/cmn/config/env.go:77-95: passthrough is an allowlist. - LEAN_RESULTS - RESEARCH_WORKSPACE - NATIVE_CODEX_HOME diff --git a/blueprints/us-equities/hosting/dagu-equities.service.example b/blueprints/us-equities/hosting/dagu-equities.service.example index 872481f69..887b3226c 100644 --- a/blueprints/us-equities/hosting/dagu-equities.service.example +++ b/blueprints/us-equities/hosting/dagu-equities.service.example @@ -1,13 +1,19 @@ [Unit] -Description=Local research workflow history (Dagu) +Description=NYSE post-close research evidence (Dagu 2.16.6) After=network.target [Service] Type=simple -# Install the pinned binary and private config first; see README.md. -ExecStart=/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin DAGU_HOME=%h/.local/share/codex-ecosystem/dagu-equities %h/.local/share/codex-ecosystem/tools/dagu-2.16.6/dagu server +# Adapt these placeholder paths on the operating host (user decision 2). +# dagucloud/dagu@v2.16.6:internal/cmd/startall.go:32-45 +# coordinator.enabled: false belongs in config.yaml, not Environment=. +# coreutils/coreutils@v9.4:src/env.c:824-838 discards inherited Environment=. +# The four job variables belong in the private DAG's env: block; see README.md. +ExecStart=/usr/bin/env -i HOME=/path/to/research-home PATH=/usr/bin:/bin TMPDIR=/path/to/private-tmp DAGU_HOME=/path/to/dagu-home /path/to/dagu-2.16.6 start-all --config /path/to/dagu-home/config.yaml --dags /path/to/dagu-home/dags +# systemd/systemd@v255:man/systemd.service.xml:830-836 Restart=on-failure RestartSec=5 +Nice=19 UMask=0077 NoNewPrivileges=true diff --git a/blueprints/us-equities/hosting/drill_local_recovery.py b/blueprints/us-equities/hosting/drill_local_recovery.py new file mode 100644 index 000000000..c520fb744 --- /dev/null +++ b/blueprints/us-equities/hosting/drill_local_recovery.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python3 +"""Operator-run local backup/check/restore drill using disposable research fixtures. + +This local integration fixture composes the pinned primitives cited in +journal_recovery.py. It is not an upstream restic test or off-host acceptance. +No scheduler, service, broker or network destination is used. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import json +import os +from pathlib import Path +import secrets +import sqlite3 +import subprocess +import sys +import tempfile +import threading +import time +import re + +import journal_recovery as recovery + + +def scratch_directory(scratch_root: Path) -> tuple[Path, Path]: + """Keep both lexical and canonical names; operate only on the resolved root.""" + lexical = Path(tempfile.mkdtemp(prefix="equity-recovery-", dir=scratch_root)).absolute() + return lexical.resolve(), lexical + + +def assert_sanitized(value) -> None: + """Refuse publication if any Unix/Windows absolute path remains in a string.""" + if isinstance(value, str): + if re.search(r'(?.])/(?!/)[^\s,\"\']+|(?", str(lexical_root): "", + str(binary.absolute()): "", str(binary.resolve()): ""} + + def scrub(item): + if isinstance(item, str): + for path in sorted(replacements, key=len, reverse=True): + item = item.replace(path, replacements[path]) + return item + if isinstance(item, list): + return [scrub(child) for child in item] + if isinstance(item, dict): + return {scrub(key): scrub(child) for key, child in item.items()} + return item + + clean = scrub(value) + assert_sanitized(clean) + return clean + + +def drill(binary: Path, scratch_root: Path) -> dict: + os.umask(0o077) + root, lexical_root = scratch_directory(scratch_root) + unlock_file = root / "throwaway-unlock-file" + unlock_file.write_text(secrets.token_urlsafe(48) + "\n") + unlock_file.chmod(0o600) + writers = {} + stops, threads, commits, writer_errors = {}, {}, {}, [] + journals = {} + report = {"schema_version": 1, "evidence_class": "local_integration", + "fixture_class": "synthetic SQLite WAL journals", "started_at_utc": + datetime.now(timezone.utc).isoformat(), "restic_pin": "0.19.1"} + + try: + for name, table in (("research", "events"), ("evidence", "runs")): + path = root / f"{name}.sqlite3" + db = sqlite3.connect(path, isolation_level=None) + db.execute("PRAGMA journal_mode=WAL") + db.execute("PRAGMA synchronous=FULL") + db.execute("PRAGMA wal_autocheckpoint=0") + db.execute(f"CREATE TABLE {table}(id INTEGER PRIMARY KEY, payload TEXT NOT NULL)") + # About 8 MiB each: long enough to observe overlapping commits during + # a single native pages=-1 step, without delaying/reimplementing it. + db.executemany(f"INSERT INTO {table}(payload) VALUES (?)", [("fixture" * 12000,)] * 96) + writers[name] = db + journals[name] = (path, [table]) + observed = [] + + def write_continuously(name): + # Independent WAL connection/thread; CPython releases the GIL during + # sqlite3_backup_step (@v3.12.3:Modules/_sqlite/connection.c:2075-2078). + db = sqlite3.connect(journals[name][0], isolation_level=None) + try: + db.execute("PRAGMA synchronous=FULL") + db.execute("PRAGMA wal_autocheckpoint=0") + while not stops[name].is_set(): + db.execute(f"INSERT INTO {journals[name][1][0]}(payload) VALUES (?)", ("concurrent WAL commit",)) + commits[name].append(time.monotonic_ns()) + stops[name].wait(0.001) + except Exception as error: + writer_errors.append(type(error).__name__) + finally: + db.close() + + def observe_copy(name, native): + stops[name].set() + threads[name].join(timeout=10) + count = sum(native["backup_started_ns"] <= instant <= native["backup_finished_ns"] + for instant in commits[name]) + if threads[name].is_alive() or writer_errors or count < 1: + raise recovery.RecoveryError("each journal needs an observed commit during its native Online Backup call") + observed.append({"journal": name, "pages": native["pages"], "committed_writes_during_native_call": count, + "native_copy_duration_s": (native["backup_finished_ns"] - native["backup_started_ns"]) / 1e9}) + + work = root / "accepted" + work.mkdir(mode=0o700) + runner = recovery.Restic(binary, root / "repository", unlock_file, work) + runner.run("init", "--json") + for name in journals: + stops[name], commits[name] = threading.Event(), [] + threads[name] = threading.Thread(target=write_continuously, args=(name,)) + threads[name].start() + # One partition (1/1) deliberately reads every pack in this small proof. + # The operator procedure defaults to seven partitions; unittest covers its rotation. + result = recovery.cycle(runner, journals, root / "rotation-one.json", parts=1, + observation=observe_copy) + if {entry["journal"] for entry in observed} != set(journals): + raise recovery.RecoveryError("both journals must be written during their Online Backup") + inventory = json.loads((work / "frozen-inventory.json").read_text()) + independent = [] + # Independent byte/state observation, separate from recovery.verify's result. + expected_paths = {"inventory.json", *(record["path"] for record in inventory["files"])} + if {p.relative_to(work / "restored").as_posix() for p in (work / "restored").rglob("*") + if p.is_file()} != expected_paths: + raise recovery.RecoveryError("independent restore file-set comparison failed") + for record in inventory["files"]: + staged = work / "stage" / record["path"] + restored = work / "restored" / record["path"] + data = restored.read_bytes() + if data != staged.read_bytes() or hashlib.sha256(data).hexdigest() != record["sha256"]: + raise recovery.RecoveryError("independent byte/hash comparison failed") + db = sqlite3.connect(restored.as_uri() + "?mode=ro", uri=True) + try: + if db.execute("PRAGMA integrity_check").fetchall() != [("ok",)]: + raise recovery.RecoveryError("independent integrity observation failed") + counts = {t: db.execute(f'SELECT COUNT(*) FROM "{t}"').fetchone()[0] + for t in record["required_tables"]} + finally: + db.close() + if counts != record["required_tables"] or min(counts.values()) < 96: + raise recovery.RecoveryError("independent snapshot row-count comparison failed") + independent.append({"path": record["path"], "sha256": record["sha256"], + "bytes": len(data), "integrity_check": "ok", "row_counts": counts}) + controls = {} + for control in ("snapshot", "restore"): + control_work = root / ("control-" + control) + args = [sys.executable, str(Path(recovery.__file__)), "cycle", "--restic-bin", str(binary), + "--repository", str(root / "repository"), "--password-file", str(unlock_file), + "--work", str(control_work), "--rotation-state", str(root / "rotation-one.json"), + "--check-parts", "1", "--control", control] + for name, (path, tables) in journals.items(): + args.extend(["--journal", f"{name}={path}"]) + for table in tables: + args.extend(["--required-table", f"{name}={table}"]) + completed = subprocess.run(args, env=runner.env, cwd=root, text=True, + capture_output=True, timeout=900, check=False) + returned = json.loads(completed.stdout) + native = [json.loads(p.read_text()) for p in sorted(control_work.glob("restic-*.result.json"))] + controls[control] = {"exit": completed.returncode, "returned": returned, + "native_commands": native} + if control == "restore": + controls[control]["mutation_observation"] = json.loads((control_work / "restore-control.json").read_text()) + if completed.returncode == 0: + raise recovery.RecoveryError("a failing control incorrectly passed") + if control == "snapshot" and not any(c["exit"] == 3 for c in native): + raise recovery.RecoveryError("snapshot control must observe native restic exit 3") + if control == "restore" and (not any(c["args"][0] == "restore" and c["exit"] == 0 for c in native) + or "sha256/bytes mismatch" not in returned.get("reason", "")): + raise recovery.RecoveryError("restore control must fail the comparison after native restore") + report.update(result="passed", copy_observations=observed, frozen_inventory=inventory, + native_commands=runner.commands, comparison=result, + independent_observation=independent, controls=controls, + limitations=["Same-host disposable repository and key; no off-host recovery.", + "No deployed scheduler, process restart, reboot, missed-run or independent alert drill.", + "Two per-journal snapshots; no atomic transaction spanning both journals.", + "Subset 1/1 only; seven-part rotation is tested separately with stdlib fixtures."]) + finally: + for stop in stops.values(): + stop.set() + for thread in threads.values(): + thread.join(timeout=10) + for db in writers.values(): + db.close() + unlock_file.unlink(missing_ok=True) + report["throwaway_unlock_file_deleted"] = not unlock_file.exists() + report["completed_at_utc"] = datetime.now(timezone.utc).isoformat() + return sanitize(report, root=root, lexical_root=lexical_root, binary=binary) + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--restic-bin", type=Path, required=True) + parser.add_argument("--scratch-root", type=Path, default=os.environ.get("TMPDIR")) + parser.add_argument("--output", type=Path) + args = parser.parse_args(argv) + if args.scratch_root is None or not args.scratch_root.is_dir(): + parser.error("set TMPDIR or --scratch-root to an existing private scratch directory") + try: + report = drill(args.restic_bin, args.scratch_root) + if args.output: + assert_sanitized(report) + recovery.write_new(args.output, recovery.json_bytes(report)) + print(json.dumps({"result": report["result"], "matched_journals": 2, + "checked_subset": report["comparison"]["checked_subset"], + "snapshot_control_exit": report["controls"]["snapshot"]["exit"], + "restore_control_exit": report["controls"]["restore"]["exit"], + "throwaway_unlock_file_deleted": report["throwaway_unlock_file_deleted"]}, sort_keys=True)) + return 0 + except (ValueError, OSError, sqlite3.Error, subprocess.TimeoutExpired) as error: + print(json.dumps({"result": "failed", "reason": type(error).__name__})) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/blueprints/us-equities/hosting/drill_process_restart.py b/blueprints/us-equities/hosting/drill_process_restart.py new file mode 100644 index 000000000..8238deed4 --- /dev/null +++ b/blueprints/us-equities/hosting/drill_process_restart.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Operator-only drill: kill an existing start-all, observe restart and next due run. + +Run later on the chosen operating host (user decision 2), with an isolated uv +PyYAML 6.0.3 overlay over its locked trading Python; see README. SDK_ENV stays +unchanged. This script never installs, enables or starts a unit. The operator must +have deployed the private templates and have supplied fresh local research input. + +Sources: systemd/systemd@v255:man/systemd.service.xml:830-836; +dagucloud/dagu@v2.16.6:internal/cmd/startall.go:32-45; +internal/cmd/history.go:568-602; internal/persis/file/dagrun/dagrun.go:55-59; +internal/ir/run_status.go:156-179; internal/ir/status.go:10-18,133-149. +YAML parsing: yaml/pyyaml@6.0.3:lib/yaml/__init__.py:117-125 (safe_load). +Process restart is not reboot, missed-run or independent-alert acceptance. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timedelta, timezone +import hashlib +import json +import os +from pathlib import Path +import subprocess +import time + +import session_day + +RESTART_ALLOWANCE_SECONDS = 90 +RESTART_MARGIN_SECONDS = 60 + + +def unit_state(unit: str) -> dict: + completed = subprocess.run( + ["systemctl", "--user", "show", unit, "--property=MainPID,NRestarts,ActiveState,Restart,ExecStart"], + capture_output=True, text=True, timeout=15, check=False) + if completed.returncode: + raise ValueError(f"systemctl show failed (exit {completed.returncode})") + return dict(line.split("=", 1) for line in completed.stdout.splitlines() if "=" in line) + + +def due_run(status: dict, due: datetime, run_ids: set[str]) -> bool: + """A successful native record must attest the exact slot and scheduler trigger.""" + if not (status.get("name") == "equity-research-evidence" and + status.get("dagRunId") in run_ids and status.get("triggerType") == 1 and + status.get("status") == 4): + return False + # Dagu marks a run succeeded even when a precondition skips the chain. + # Accept the research job only when every required step actually succeeded. + required = {"calendar_check", "nyse_session", "prepare_output", "summarize", + "baseline_evidence", "catalog_evidence"} + nodes = {node["step"]["id"]: node.get("status") for node in status.get("nodes", [])} + if any(nodes.get(step) != 4 for step in required): + return False + slot = status.get("scheduleTime") + return bool(slot and datetime.fromisoformat(slot) == due) + + +def last_status(source: Path | bytes) -> dict: + # Upstream status.jsonl is appended while running, then compacted after finish. + # Ignore an incomplete trailing write and retain the last complete native record. + last = {} + data = source.read_bytes() if isinstance(source, Path) else source + for line in data.splitlines(): + try: + last = json.loads(line) + except json.JSONDecodeError: + continue + return last + + +def validate_window(due: datetime, now: datetime, wait_seconds: int) -> None: + if due.utcoffset() is None or now.utcoffset() is None or not 1 <= wait_seconds <= 3600: + raise ValueError("use an aware due time and a wait bound of 1..3600 seconds") + market = due.astimezone(session_day.MARKET_ZONE) + if (market.hour, market.minute, market.second, market.microsecond) != (16, 30, 0, 0): + raise ValueError("--due-at must be the named schedule's 16:30 New York slot") + if not now + timedelta(seconds=RESTART_ALLOWANCE_SECONDS + RESTART_MARGIN_SECONDS) < due < now + timedelta(seconds=wait_seconds - 180): + raise ValueError("reserve more than 150 seconds before the due slot and 180 seconds for completion") + + +def require_session(due: datetime) -> None: + checked = subprocess.run([os.sys.executable, str(Path(session_day.__file__)), "--at", due.isoformat()], + capture_output=True, text=True, timeout=60, check=False) + if checked.returncode or checked.stdout.strip() != "session": + raise ValueError(f"locked session check refused the due slot (exit {checked.returncode})") + + +def validate_process(pid: int, binary: Path, proc_root: Path = Path("/proc")) -> None: + """Identity guard only; restart/supervision belong to upstream systemd.""" + if pid <= 1: + raise ValueError("MainPID is not the chosen start-all executable") + process = proc_root / str(pid) + command = (process / "cmdline").read_bytes().split(b"\0") + if b"start-all" not in command or (process / "exe").resolve() != binary.resolve(): + raise ValueError("MainPID is not the chosen start-all executable") + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--unit", default="dagu-equities.service") + parser.add_argument("--dagu-bin", type=Path, required=True) + parser.add_argument("--dagu-home", type=Path, required=True) + parser.add_argument("--config", type=Path, required=True) + parser.add_argument("--dag-history", type=Path, required=True, + help="the chosen host's native history directory for this one DAG") + parser.add_argument("--due-at", required=True, help="the next 16:30 New York session slot, with offset") + parser.add_argument("--wait-seconds", type=int, default=900) + args = parser.parse_args(argv) + try: + # systemd/systemd@v255:man/systemctl.xml:1826-1833: PATTERN accepts globs. + if any(char in args.unit for char in "*?[]"): + raise ValueError("--unit must be a literal unit name; glob patterns are refused") + due = datetime.fromisoformat(args.due_at) + now = datetime.now(timezone.utc) + validate_window(due, now, args.wait_seconds) + # Normal helper command validates the package pin; do not kill anything on a holiday. + require_session(due) + try: + import yaml + except ImportError as error: + raise ValueError("run the restart drill with the isolated uv PyYAML 6.0.3 overlay; see README") from error + try: + config = yaml.safe_load(args.config.read_text()) + except yaml.YAMLError as error: + raise ValueError("use a valid YAML read-only UI config") from error + permissions = config.get("permissions", {}) if isinstance(config, dict) else {} + run_dags = permissions.get("run_dags", True) if isinstance(permissions, dict) else True + # Dagu@v2.16.6:internal/cmn/config/loader.go:586-611: legacy value overrides the mapping. + if isinstance(config, dict): + run_dags = config.get("permissionRunDags", run_dags) + if run_dags is not False or not args.dag_history.is_dir(): + raise ValueError("use the deployed read-only UI config and this DAG's history directory") + before = unit_state(args.unit) + if (before.get("ActiveState") != "active" or before.get("Restart") != "on-failure" or + "start-all" not in before.get("ExecStart", "") or str(args.config) not in before["ExecStart"]): + raise ValueError("the chosen existing unit must run start-all with this config and on-failure restart") + pid = int(before["MainPID"]) + # Linux operating-host drill: verify identity before signalling this one process. + validate_process(pid, args.dagu_bin) + validate_window(due, datetime.now(timezone.utc), args.wait_seconds) + deadline = time.monotonic() + args.wait_seconds + # systemd/systemd@v255:man/systemctl.xml:541-549,2377-2388: + # Address the unit's current main process, avoiding PID reuse after the guard. + fault = subprocess.run(["systemctl", "--user", "kill", "--kill-whom=main", + "--signal=SIGKILL", args.unit], + capture_output=True, text=True, timeout=15, check=False) + if fault.returncode: + raise ValueError(f"systemctl kill failed (exit {fault.returncode})") + restart_deadline = min(deadline, time.monotonic() + RESTART_ALLOWANCE_SECONDS) + while True: + after = unit_state(args.unit) + if (after.get("ActiveState") == "active" and int(after.get("MainPID", "0")) not in (0, pid) and + int(after.get("NRestarts", "0")) > int(before["NRestarts"])): + validate_process(int(after["MainPID"]), args.dagu_bin) + if datetime.now(timezone.utc) >= due: + raise ValueError("start-all restarted at or after the due slot; schedule was missed") + break + if time.monotonic() >= restart_deadline: + raise ValueError("start-all did not restart within 90 seconds") + time.sleep(2) + history_command = [str(args.dagu_bin), "history", "--context", "local", "--dagu-home", + str(args.dagu_home), "--config", str(args.config), "--format", "json", + "--from", due.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "--limit", "1000", "--status", "succeeded", "equity-research-evidence"] + while time.monotonic() < deadline: + history = subprocess.run(history_command, capture_output=True, text=True, timeout=30, check=False) + if history.returncode: + raise ValueError(f"dagu history failed (exit {history.returncode})") + ids = {row["dagRunId"] for row in json.loads(history.stdout)} + # The public history rendering omits triggerType/scheduleTime. Corroborate in + # the original native status records instead of accepting an unrelated manual run. + for path in args.dag_history.rglob("status.jsonl"): + try: + data = path.read_bytes() + except FileNotFoundError: + continue # Compaction can replace the entry after directory enumeration. + status = last_status(data) + if due_run(status, due, ids): + print(json.dumps({"result": "passed", "pid_changed": True, + "automatic_restart_observed": True, "next_due_run": "succeeded", + "scheduler_trigger_observed_with_run_dags_false": True, + "due_at": due.isoformat(), "native_status_sha256": + hashlib.sha256(data).hexdigest(), + "gate": "same-host process restart only"}, sort_keys=True)) + return 0 + time.sleep(5) + raise ValueError("the next exact due slot has no successful scheduler-triggered run") + except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired) as error: + reason = str(error) if type(error) is ValueError else type(error).__name__ + print(json.dumps({"result": "failed", "reason": reason}, sort_keys=True)) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/blueprints/us-equities/hosting/equity-research-evidence.yaml b/blueprints/us-equities/hosting/equity-research-evidence.yaml index 79433ac80..0487858fc 100644 --- a/blueprints/us-equities/hosting/equity-research-evidence.yaml +++ b/blueprints/us-equities/hosting/equity-research-evidence.yaml @@ -1,5 +1,19 @@ -description: Summarize a local LEAN simulation and validate its evidence catalog. +description: 'nyse-post-close-evidence: summarize local simulation evidence once per NYSE session after close.' type: graph +tags: [nyse-post-close-evidence] +# Schedule contract name: nyse-post-close-evidence (the DAG name is its filename). +# dagucloud/dagu@v2.16.6:internal/ir/schedule.go:281-305; +# go.mod:65 pins robfig/cron@v3.0.1:parser.go:93-103 (CRON_TZ). +# 16:30 ET follows ordinary and early closes. The bracketed native calendar query +# returns non_session on holidays (including New Year); uncovered dates fail. +schedule: 'CRON_TZ=America/New_York 30 16 * * 1-5' +# dagucloud/dagu@v2.16.6:internal/spec/dag.go:1663-1673. +# Replace these placeholders in the private deployed DAG. env -i needs no inheritance. +env: + STACK_REPO: /path/to/native-agent-stack + SDK_ENV: /path/to/trading-runtime/.venv + LEAN_EVENTS: /path/to/research-input/order-events.json + RESEARCH_OUTPUT: /path/to/research-output working_dir: ${STACK_REPO} timeout_sec: 180 max_active_runs: 1 @@ -8,8 +22,29 @@ max_active_runs: 1 artifacts: enabled: false steps: + # A normal command surfaces import/version/calendar failures instead of silently skipping. + - id: calendar_check + # LEAN_EVENTS producer: research operator's simulation job publishes this + # session's input after its actual close and by 16:25 ET. Missing/stale/future + # input prints a visible token and exits nonzero, failing the native run. + run: '"${SDK_ENV}/bin/python" "${STACK_REPO}/blueprints/us-equities/hosting/session_day.py" --events "${LEAN_EVENTS}"' + output: SESSION_DAY + # dagucloud/dagu@v2.16.6:internal/runtime/runner.go:1633-1648, 1267-1281: + # a false step precondition skips this step and its dependents, without failure. + - id: nyse_session + preconditions: + - condition: '${SESSION_DAY}' + expected: session + run: 'true' + depends: [calendar_check] + - id: prepare_output + run: 'mkdir -p "${RESEARCH_OUTPUT}"' + depends: [nyse_session] - id: summarize - run: '"${SDK_ENV}/bin/python" "${STACK_REPO}/blueprints/us-equities/data/summarize_backtest.py" "${LEAN_EVENTS}" "${RESEARCH_OUTPUT}/order-events.parquet"' + # dagucloud/dagu@v2.16.6:internal/runctx/context_env.go:28 and + # internal/cmn/runenv/keys.go:12-13: isolate each run; never overwrite earlier evidence. + run: '"${SDK_ENV}/bin/python" "${STACK_REPO}/blueprints/us-equities/data/summarize_backtest.py" "${LEAN_EVENTS}" "${RESEARCH_OUTPUT}/order-events-${DAG_RUN_ID}.parquet"' + depends: [prepare_output] - id: baseline_evidence run: python3 scripts/validate.py depends: [summarize] diff --git a/blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json b/blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json new file mode 100644 index 000000000..df03519f4 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json @@ -0,0 +1,34 @@ +{ + "schema_version": 1, + "round": "t-pr4-r2", + "evidence_class": "local_integration", + "pin": "robfig/cron@v3.0.1", + "command": "env GOPROXY=off GOSUMDB=off GOTOOLCHAIN=local go run main.go", + "exit": 127, + "result": "not_run", + "returned_output": "nice: \u2018go\u2019: No such file or directory\n", + "reason": "No go compiler found in PATH, the task ecosystem directory or checked standard toolchain locations. Network use is limited to pinned source reads; no compiler installation performed.", + "schedule": "CRON_TZ=America/New_York 30 16 * * 1-5", + "expected_cases": [ + { + "from": "2026-10-30T21:00:00Z", + "expected_next": "2026-11-02T21:30:00Z" + }, + { + "from": "2027-03-12T21:00:00Z", + "expected_next": "2027-03-15T20:30:00Z" + } + ], + "source_program": "package main\nimport (\n \"fmt\"\n \"time\"\n \"github.com/robfig/cron/v3\"\n)\nfunc main() {\n schedule, err := cron.NewParser(cron.Minute|cron.Hour|cron.Dom|cron.Month|cron.Dow).Parse(\"CRON_TZ=America/New_York 30 16 * * 1-5\")\n if err != nil { panic(err) }\n cases := [][2]string{{\"2026-10-30T21:00:00Z\",\"2026-11-02T21:30:00Z\"},{\"2027-03-12T21:00:00Z\",\"2027-03-15T20:30:00Z\"}}\n for _, c := range cases {\n from, err := time.Parse(time.RFC3339,c[0]); if err != nil { panic(err) }\n next := schedule.Next(from).UTC().Format(time.RFC3339)\n fmt.Printf(\"%s -> %s expected %s\\n\",c[0],next,c[1])\n if next != c[1] { panic(\"unexpected Next time\") }\n }\n}\n", + "go_mod": "module tpr4-cron-probe\n\ngo 1.18\n\nrequire github.com/robfig/cron/v3 v3.0.1\nreplace github.com/robfig/cron/v3 => ./cron-3.0.1\n", + "module_source_archive_sha256": "ef97328622b5eac7adfb1aa47ddab7b3f68271b9a9b76e5bc07bf2ad65bb051a", + "source_locators": [ + "dagucloud/dagu@v2.16.6:go.mod:65", + "robfig/cron@v3.0.1:parser.go:93-103", + "robfig/cron@v3.0.1:spec.go:58-175" + ], + "limitations": [ + "The expected Next values are reviewer expectations, not measured results.", + "The record-why-it-cannot-run alternative is used; Next execution remains unavailable." + ] +} diff --git a/blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json b/blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json new file mode 100644 index 000000000..5849c4ae4 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json @@ -0,0 +1,295 @@ +{ + "schema_version": 1, + "evidence_class": "source_review", + "pins": [ + "dagucloud/dagu@v2.16.6", + "dagucloud/dagu@v2.18.2" + ], + "pin_changed": false, + "release_notes": [ + { + "pin": "dagucloud/dagu@v2.16.6", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.16.6", + "published_at": "2026-09-14T14:45:57Z", + "body_sha256": "a00e5099faf6b190893c04c3a5c56bca956d214b75e573b845a0efbc39664f9b", + "selected_body_lines": [ + { + "line": 7, + "text": "- fix(scheduler): reload inherited base defaults (#2770) @yohamta0" + } + ] + }, + { + "pin": "dagucloud/dagu@v2.17.0", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.17.0", + "published_at": "2026-09-21T13:31:48Z", + "body_sha256": "5ef6e38dd9a8f3a7998404e7f67bafb55f57141b481a4783e4aa20b258adaedd", + "selected_body_lines": [ + { + "line": 7, + "text": "- feat: move default suspend flags directory under data_dir (#2784) @LouisDeconinck" + }, + { + "line": 8, + "text": "- feat: add retry --bypass-preconditions flag (#2785) @LouisDeconinck" + }, + { + "line": 30, + "text": "- Pace every run with the DAG's queue, and stop dropping queued retries (#2775) @yohamta0" + } + ] + }, + { + "pin": "dagucloud/dagu@v2.17.2", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.17.2", + "published_at": "2026-09-25T06:05:53Z", + "body_sha256": "408945419fe63f4d3789a0f98aaaf63bb0aa74f6cbe067809e545234fc3bfa65", + "selected_body_lines": [] + }, + { + "pin": "dagucloud/dagu@v2.18.0", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.18.0", + "published_at": "2026-09-29T09:26:53Z", + "body_sha256": "166dc38184cfa8a7c0845ce5b94acd138946e89dce8c7139f2969191ae29fc70", + "selected_body_lines": [ + { + "line": 5, + "text": "- feat(scheduler): support cron descriptors like @daily (#2876) @LouisDeconinck" + }, + { + "line": 7, + "text": "- feat: expose step precondition bypass in retry API and UI (#2880) @LouisDeconinck" + }, + { + "line": 32, + "text": "- fix(coordinator): preserve queue identity for distributed starts (#2895) @yoyooyooo" + }, + { + "line": 43, + "text": "- fix: load dotenv for scheduler-triggered runs (#2937) @yohamta0" + }, + { + "line": 44, + "text": "- fix: clear worker ownership when re-queuing a retry (#2940) @yohamta0" + } + ] + }, + { + "pin": "dagucloud/dagu@v2.18.1", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.18.1", + "published_at": "2026-09-29T15:24:51Z", + "body_sha256": "ce0d480bcfe9337d36b070e895fa60c91edbc9c22edd98d589dd783c689f233b", + "selected_body_lines": [] + }, + { + "pin": "dagucloud/dagu@v2.18.2", + "url": "https://github.com/dagucloud/dagu/releases/tag/v2.18.2", + "published_at": "2026-10-04T03:54:02Z", + "body_sha256": "27ee9708436cff45baf7e71a1d8f35a1abd97460f3f7f9773e236586d907d3e3", + "selected_body_lines": [ + { + "line": 5, + "text": "- feat: optional signal propagation to running DAG processes (#2885) @LouisDeconinck" + }, + { + "line": 32, + "text": "- fix: keep SIGTERM handled during supervisor shutdown (#2966) @yohamta0" + }, + { + "line": 35, + "text": "- fix: run coordinator shutdown on SIGINT and SIGTERM (#2972) @ReguiguiMohamed" + }, + { + "line": 37, + "text": "- fix(scheduler): log scheduled slots skipped while the DAG is busy (#2980) @ReguiguiMohamed" + } + ] + } + ], + "missing_release_metadata": [ + { + "tag": "v2.17.1", + "http_status": 404 + } + ], + "scheduler_changed_files": [ + { + "path": "internal/service/scheduler/dag_executor.go", + "test": false, + "v2166_sha256": "a441c4dd1f2ba49d44fa883a3b239d54ee60ce837758ba8b57478690465c0816", + "v2182_sha256": "c7efc1c87ecbc7f634b68db0d020a2d2e21af0e03b432a3598ec109aaed32ad3" + }, + { + "path": "internal/service/scheduler/dag_executor_internal_test.go", + "test": true, + "v2166_sha256": "cf877c6483f1ee4c85e976157c69bbb8b29a1ae6542623079271f20cb59bc0f1", + "v2182_sha256": "72a2b089b2f718f6be8602672db4267921fd9a86153412f0ab99d42ff3c81d08" + }, + { + "path": "internal/service/scheduler/dag_executor_test.go", + "test": true, + "v2166_sha256": "260466a640e62a742e12d025c8fd349fbe049dc771e2458bfbe7e2694c20a4ec", + "v2182_sha256": "12fa8280e7ed8bfce0cfdfad7793fe88c9a8597c2d13e0ac669c00c26244c45d" + }, + { + "path": "internal/service/scheduler/queue_conditions_external_test.go", + "test": true, + "v2166_sha256": "95acd34fc8d4022b1cd74e2d903e25b07ec45248b6d75c46eb2ddfcfce7b30e2", + "v2182_sha256": "12cf897c239ee90172f1cc963041137e4f7f0d6d4a29adb6824e722670d25fd9" + }, + { + "path": "internal/service/scheduler/queue_dispatcher.go", + "test": false, + "v2166_sha256": "82139da472ed271fb78a54aaf2fbcfbbe9b3714894afb903baf063b9ff6d6639", + "v2182_sha256": "9fdd8b4d0a1c7b5f01e1fdebd69c20ff2d9904a40426cc62526fb50225092e28" + }, + { + "path": "internal/service/scheduler/queue_processor_test.go", + "test": true, + "v2166_sha256": "2251d6a3714443669b5eb549bd381ac7caefb131a81914e6c93d071a76d79e1a", + "v2182_sha256": "4363e2924922a4355bedeb1c9f115d4fb4b1a180ab2a3893a2735c0b2d1daa41" + }, + { + "path": "internal/service/scheduler/retry_scanner.go", + "test": false, + "v2166_sha256": "fcfee9d8e6afc073bb45655c6ceacb80199785af5bece7d1614c43f5bfbe73da", + "v2182_sha256": "115a1a4007ed4dbf5d04c5c987027b9d7b8c3d9c89a8452ee91dc8570a5586cc" + }, + { + "path": "internal/service/scheduler/retry_scanner_test.go", + "test": true, + "v2166_sha256": "309b01f2393dd1606ed002926d5a2a0d62ec6d0ebc6083eb929234583e320065", + "v2182_sha256": "07bbd1df690fe990652d9d8f1673fb562b13c67276152501251f81e8833b26a0" + }, + { + "path": "internal/service/scheduler/scheduler.go", + "test": false, + "v2166_sha256": "5168e578098e1b946c62d5e1b94f9ccf8d5d1f97a9b4d0f027fed3d773dea589", + "v2182_sha256": "d3e563afe194ff02a686a1d0a817384868ebd0af69318f10a60714d7a684c392" + }, + { + "path": "internal/service/scheduler/scheduler_test.go", + "test": true, + "v2166_sha256": "8488172e46bff82ffcbebc263f6f9c4e088b35a01623757f3e179d6c86f57c6b", + "v2182_sha256": "13ba7dcbea603aecd88df29cf7c8506a809b7904a6803beaa4b0e0d30d655753" + }, + { + "path": "internal/service/scheduler/signal_unix_test.go", + "test": true, + "v2166_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "v2182_sha256": "f189c93c8153c6f9befa1e872fa66cf48121800f3265dd147a2c3b83c3ddb79d" + }, + { + "path": "internal/service/scheduler/tick_planner.go", + "test": false, + "v2166_sha256": "29fe525164af68e474b9439348b936030dc5d8714fd710f0d6c5dfac333b1496", + "v2182_sha256": "19a05a2e22cdf1a8f5d327213055c0b75cc2a64cb892dd8bb482e3f1ea304e95" + }, + { + "path": "internal/service/scheduler/tick_planner_test.go", + "test": true, + "v2166_sha256": "c1684a1a4eebda4969985643ce0fa6370f76d5911421751897329c73dee28c1f", + "v2182_sha256": "ba4b2c7c0e4da129afdb756b2f351e58a0b3a9320df3b2bbaca2ee9d937c7f40" + }, + { + "path": "internal/service/scheduler/wait_for_tick_test.go", + "test": true, + "v2166_sha256": "ba1be905f04f67e493ecd4a21278567f581701644a3652b38c6a8d30812b040e", + "v2182_sha256": "fbba473ee320532d9b09c87a907e1bd8e2a3e5fef8e1635cfbb2dd4ec2d93104" + }, + { + "path": "internal/service/scheduler/zombie_detector_test.go", + "test": true, + "v2166_sha256": "e1cfd1e59dc3a166cc7ac9efe3f8b7e02d3a8660236cfaa57940962484f7a47a", + "v2182_sha256": "e4a90d8e26ec2c1bbc15fc10a7185e4de8f8d7a0039ba9fc420890139fd86a36" + } + ], + "production_changed_files": 5, + "additional_compared_files": [ + { + "path": "internal/ir/schedule.go", + "changed": true, + "v2166_sha256": "7cfd64bc483ae2017a00a93cd79f92b28be10d7f0fe852fea430c091cc327929", + "v2182_sha256": "7f100ed084941929b5a2f866585823e7fc9546cfd20afb56d47a57d49af4986b" + }, + { + "path": "internal/runtime/condition.go", + "changed": true, + "v2166_sha256": "91fc4dc27767578461b77d1c9cd1fa6f8406e4d9440596a5205fd5da3f826713", + "v2182_sha256": "858ae6e3f58d6fcaffcdf2887f33f6d85cac786e03a4b3f56f999e39194f9f40" + }, + { + "path": "internal/runtime/runner.go", + "changed": true, + "v2166_sha256": "95ec2ba0e869bc1fc3d2d25feb5b1af0371602d0af7073d4852381faaf5d7948", + "v2182_sha256": "b1a208bebd86fabb25bbc0144bbb41e65851fe6e0ef0f20688b9bb1bddccc848" + }, + { + "path": "internal/service/scheduler/catchup.go", + "changed": false, + "v2166_sha256": "02ee33c9ff6e61a291ae5a3e3854c7bad38ef798fdcebfe60b951b33b305891e", + "v2182_sha256": "02ee33c9ff6e61a291ae5a3e3854c7bad38ef798fdcebfe60b951b33b305891e" + }, + { + "path": "internal/service/scheduler/schedule_state.go", + "changed": false, + "v2166_sha256": "c3a037ff59d1bec14acaef048204c972475532929781711bc39711d206e47e89", + "v2182_sha256": "c3a037ff59d1bec14acaef048204c972475532929781711bc39711d206e47e89" + }, + { + "path": "internal/runctx/context_env.go", + "changed": false, + "v2166_sha256": "6d0c3049cc3c0b1d3400a19c041f859354789e3ba88e0f3e1f73e4f166038a4f", + "v2182_sha256": "6d0c3049cc3c0b1d3400a19c041f859354789e3ba88e0f3e1f73e4f166038a4f" + } + ], + "findings": [ + { + "what": "global queue pacing / retry admission", + "locator": "dagucloud/dagu@v2.18.2:internal/service/scheduler/tick_planner.go:1560-1568,1658-1661", + "depends": false, + "reason": "No configured global queue and no retry policy in this research job." + }, + { + "what": "scheduler dotenv fix", + "locator": "dagucloud/dagu@v2.18.2:internal/service/scheduler/dag_executor.go:468-475", + "depends": false, + "reason": "Four job variables are literal DAG env; no dotenv." + }, + { + "what": "busy scheduled-slot logging", + "locator": "dagucloud/dagu@v2.18.2:internal/service/scheduler/tick_planner.go:758-784", + "depends": false, + "reason": "Logging added around unchanged busy skip guards." + }, + { + "what": "suspension migration and signal propagation", + "locator": "dagucloud/dagu@v2.18.2:internal/service/scheduler/scheduler.go:607-614,842-855", + "depends": false, + "reason": "No suspension migration requirement; process-restart drill uses SIGKILL, with graceful propagation unaccepted." + }, + { + "what": "retry liveness and queued-state failures", + "locator": "dagucloud/dagu@v2.18.2:internal/service/scheduler/retry_scanner.go:165-181;internal/service/scheduler/queue_dispatcher.go:929-950", + "depends": false, + "reason": "No DAG retry or global/distributed queue in this job." + }, + { + "what": "cron descriptors and timezone-prefixed descriptor normalization", + "locator": "dagucloud/dagu@v2.18.2:internal/ir/schedule.go:17-19,281-345", + "depends": false, + "reason": "Explicit five-field CRON_TZ cron, not an @ descriptor or @every interval." + }, + { + "what": "precondition abort/timeout and evaluation error precedence", + "locator": "dagucloud/dagu@v2.18.2:internal/runtime/condition.go:19-82", + "depends": false, + "reason": "One scalar token comparison; calendar runs as a normal command so missing package and calendar errors fail there. No interrupted-precondition acceptance claimed." + } + ], + "result": "No compared fix is required by the defined research schedule; retain 2.16.6 provisionally. Source comparison only; operating-host slot/restart acceptance remains open.", + "corrections": [ + "Initial internal/scheduler lookup was empty because the original path is internal/service/scheduler; it was not an empty-diff verdict.", + "Calendar tag v4.13.2 returned HTTP 404; corrected to 4.13.2 against the original source and calendar receipt." + ] +} diff --git a/blueprints/us-equities/hosting/evidence/dagu-validation.json b/blueprints/us-equities/hosting/evidence/dagu-validation.json new file mode 100644 index 000000000..b013d9e36 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/dagu-validation.json @@ -0,0 +1,5 @@ +{ + "command": "dagu validate --context local --dagu-home --config /config.yaml /blueprints/us-equities/hosting/equity-research-evidence.yaml", + "exit": 0, + "output": "time=2026-10-04T23:56:06.918-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-04T23:56:07.022-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\n" +} diff --git a/blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json b/blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json new file mode 100644 index 000000000..95bc54132 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json @@ -0,0 +1,361 @@ +{ + "schema_version": 1, + "round": "t-pr4-r2", + "evidence_class": "local_integration", + "observed_at_utc": "2026-10-05T05:30:30.573203+00:00", + "recipe_provenance": "Review verdict contains nv_summary, without an nv[2] recipe. This minimal output/scalar-precondition/dependency probe follows dagucloud/dagu@v2.16.6:internal/runtime/runner.go:1267-1281,1633-1648.", + "dagu_pin": "2.16.6", + "probes": [ + { + "token": "non_session", + "version_exit": 0, + "version_output": "2.16.6\n", + "command": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml", + "exit": 0, + "output": "Succeeded - 2026-10-05T01:30:29-04:00\n\ndag: native-chain (0s)\n\u251c\u2500log: /logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/dag-run_20261005.013029.340.034aTckD.log\n\u2502\n\u251c\u2500calendar_check (0s) [succeeded]\n\u2502 \u251c\u2500printf non_session\n\u2502 \u2502\n\u2502 \u2514\u2500stdout: /logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/run_20261005_053029Z_9aa2e0/calendar_check.20261005.013029.410.034aTckD.out\n\u2502 non_session\n\u2502 \n\u251c\u2500nyse_session [skipped]\n\u2502 \n\u2514\u2500evidence [skipped]\n\nResult: Succeeded\n\ntime=2026-10-05T01:30:29.241-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-05T01:30:29.302-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\ntime=2026-10-05T01:30:29.330-04:00 level=INFO msg=\"Executing root dag-run\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk params=\"\"\ntime=2026-10-05T01:30:29.401-04:00 level=INFO msg=\"DAG run started\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 params=[]\ntime=2026-10-05T01:30:29.407-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 step=calendar_check\ntime=2026-10-05T01:30:29.436-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 step=nyse_session\ntime=2026-10-05T01:30:29.450-04:00 level=INFO msg=\"Checking preconditions for \\\"nyse_session\\\"\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 step=nyse_session\ntime=2026-10-05T01:30:29.454-04:00 level=INFO msg=\"Preconditions failed for \\\"nyse_session\\\"\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 step=nyse_session\ntime=2026-10-05T01:30:29.486-04:00 level=INFO msg=\"DAG run finished\" dag=native-chain run-id=034aTckDp9B9yc0zMwKFQk worker-id=local attempt-id=9aa2e0 status=succeeded started-at=2026-10-05T01:30:29-04:00 finished-at=2026-10-05T01:30:29-04:00\n", + "native_status": { + "root": { + "name": "native-chain", + "id": "034aTckDp9B9yc0zMwKFQk" + }, + "name": "native-chain", + "dagRunId": "034aTckDp9B9yc0zMwKFQk", + "attemptId": "9aa2e0", + "attemptKey": "f56fc74a6e15e6d7:9aa2e0", + "status": 4, + "triggerType": 2, + "workerId": "local", + "pid": 97, + "pidStartedAt": 1791178228390, + "nodes": [ + { + "step": { + "id": "calendar_check", + "name": "calendar_check", + "commands": [ + { + "command": "printf", + "args": [ + "non_session" + ], + "cmdWithArgs": "printf non_session" + } + ], + "output": "SESSION_DAY" + }, + "stdout": "/logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/run_20261005_053029Z_9aa2e0/calendar_check.20261005.013029.410.034aTckD.out", + "stderr": "/logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/run_20261005_053029Z_9aa2e0/calendar_check.20261005.013029.410.034aTckD.err", + "workingDir": "/data/dag-run-work/native-chain/2xjnjbgxpblf3krsa6c2f6ffsy/root", + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "status": 4, + "doneCount": 1, + "outputVariables": { + "SESSION_DAY": "SESSION_DAY=non_session" + }, + "outputValue": "non_session" + }, + { + "step": { + "id": "nyse_session", + "name": "nyse_session", + "commands": [ + { + "command": "true", + "cmdWithArgs": "true" + } + ], + "depends": [ + "calendar_check" + ], + "preconditions": [ + { + "condition": "${SESSION_DAY}", + "expected": "session", + "error": "condition was not met: expected \"session\", got \"non_session\"" + } + ] + }, + "stdout": "/logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/run_20261005_053029Z_9aa2e0/nyse_session.20261005.013029.442.034aTckD.out", + "stderr": "/logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/run_20261005_053029Z_9aa2e0/nyse_session.20261005.013029.442.034aTckD.err", + "workingDir": "/data/dag-run-work/native-chain/2xjnjbgxpblf3krsa6c2f6ffsy/root", + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "status": 5 + }, + { + "step": { + "id": "evidence", + "name": "evidence", + "commands": [ + { + "command": "touch", + "args": [ + "/ran" + ], + "cmdWithArgs": "touch \"/ran\"" + } + ], + "depends": [ + "nyse_session" + ] + }, + "stdout": "", + "stderr": "", + "startedAt": "", + "finishedAt": "", + "status": 5, + "error": "upstream skipped" + } + ], + "createdAt": 1791178229486, + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "procGroup": "native-chain", + "definitionId": "native-chain", + "log": "/logs/native-chain/dag-run_20261005_053029Z_034aTckDp9B9yc0zMwKFQk/dag-run_20261005.013029.340.034aTckD.log", + "workingDir": "/data/dag-run-work/native-chain/2xjnjbgxpblf3krsa6c2f6ffsy/root", + "pendingStepRetries": null + }, + "marker_created": false + }, + { + "token": "session", + "version_exit": 0, + "version_output": "2.16.6\n", + "command": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml", + "exit": 0, + "output": "Succeeded - 2026-10-05T01:30:29-04:00\n\ndag: native-chain (0s)\n\u251c\u2500log: /logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/dag-run_20261005.013029.735.034aTckq.log\n\u2502\n\u251c\u2500calendar_check (0s) [succeeded]\n\u2502 \u251c\u2500printf session\n\u2502 \u2502\n\u2502 \u2514\u2500stdout: /logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/calendar_check.20261005.013029.808.034aTckq.out\n\u2502 session\n\u2502 \n\u251c\u2500nyse_session (0s) [succeeded]\n\u2502 \u2514\u2500true\n\u2502 \n\u2514\u2500evidence (0s) [succeeded]\n \u2514\u2500touch \"/ran\"\n\nResult: Succeeded\n\ntime=2026-10-05T01:30:29.637-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-05T01:30:29.698-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\ntime=2026-10-05T01:30:29.723-04:00 level=INFO msg=\"Executing root dag-run\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h params=\"\"\ntime=2026-10-05T01:30:29.795-04:00 level=INFO msg=\"DAG run started\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 params=[]\ntime=2026-10-05T01:30:29.802-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 step=calendar_check\ntime=2026-10-05T01:30:29.832-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 step=nyse_session\ntime=2026-10-05T01:30:29.852-04:00 level=INFO msg=\"Checking preconditions for \\\"nyse_session\\\"\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 step=nyse_session\ntime=2026-10-05T01:30:29.868-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 step=evidence\ntime=2026-10-05T01:30:29.941-04:00 level=INFO msg=\"DAG run finished\" dag=native-chain run-id=034aTckqEWNZwBAqHVQ26h worker-id=local attempt-id=bedb59 status=succeeded started-at=2026-10-05T01:30:29-04:00 finished-at=2026-10-05T01:30:29-04:00\n", + "native_status": { + "root": { + "name": "native-chain", + "id": "034aTckqEWNZwBAqHVQ26h" + }, + "name": "native-chain", + "dagRunId": "034aTckqEWNZwBAqHVQ26h", + "attemptId": "bedb59", + "attemptKey": "4cb0f50e9b0e2ec7:bedb59", + "status": 4, + "triggerType": 2, + "workerId": "local", + "pid": 145, + "pidStartedAt": 1791178228780, + "nodes": [ + { + "step": { + "id": "calendar_check", + "name": "calendar_check", + "commands": [ + { + "command": "printf", + "args": [ + "session" + ], + "cmdWithArgs": "printf session" + } + ], + "output": "SESSION_DAY" + }, + "stdout": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/calendar_check.20261005.013029.808.034aTckq.out", + "stderr": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/calendar_check.20261005.013029.808.034aTckq.err", + "workingDir": "/data/dag-run-work/native-chain/dc5dgjdqqucvq7om65ydxbylbu/root", + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "status": 4, + "doneCount": 1, + "outputVariables": { + "SESSION_DAY": "SESSION_DAY=session" + }, + "outputValue": "session" + }, + { + "step": { + "id": "nyse_session", + "name": "nyse_session", + "commands": [ + { + "command": "true", + "cmdWithArgs": "true" + } + ], + "depends": [ + "calendar_check" + ], + "preconditions": [ + { + "condition": "${SESSION_DAY}", + "expected": "session" + } + ] + }, + "stdout": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/nyse_session.20261005.013029.839.034aTckq.out", + "stderr": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/nyse_session.20261005.013029.839.034aTckq.err", + "workingDir": "/data/dag-run-work/native-chain/dc5dgjdqqucvq7om65ydxbylbu/root", + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "status": 4, + "doneCount": 1 + }, + { + "step": { + "id": "evidence", + "name": "evidence", + "commands": [ + { + "command": "touch", + "args": [ + "/ran" + ], + "cmdWithArgs": "touch \"/ran\"" + } + ], + "depends": [ + "nyse_session" + ] + }, + "stdout": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/evidence.20261005.013029.873.034aTckq.out", + "stderr": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/run_20261005_053029Z_bedb59/evidence.20261005.013029.873.034aTckq.err", + "workingDir": "/data/dag-run-work/native-chain/dc5dgjdqqucvq7om65ydxbylbu/root", + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "status": 4, + "doneCount": 1 + } + ], + "createdAt": 1791178229941, + "startedAt": "2026-10-05T01:30:29-04:00", + "finishedAt": "2026-10-05T01:30:29-04:00", + "procGroup": "native-chain", + "definitionId": "native-chain", + "log": "/logs/native-chain/dag-run_20261005_053029Z_034aTckqEWNZwBAqHVQ26h/dag-run_20261005.013029.735.034aTckq.log", + "workingDir": "/data/dag-run-work/native-chain/dc5dgjdqqucvq7om65ydxbylbu/root", + "pendingStepRetries": null + }, + "marker_created": true + }, + { + "token": "stale_input", + "version_exit": 0, + "version_output": "2.16.6\n", + "command": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml", + "exit": 0, + "output": "Succeeded - 2026-10-05T01:30:30-04:00\n\ndag: native-chain (0s)\n\u251c\u2500log: /logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/dag-run_20261005.013030.191.034aTclY.log\n\u2502\n\u251c\u2500calendar_check (0s) [succeeded]\n\u2502 \u251c\u2500printf stale_input\n\u2502 \u2502\n\u2502 \u2514\u2500stdout: /logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/run_20261005_053030Z_b370aa/calendar_check.20261005.013030.266.034aTclY.out\n\u2502 stale_input\n\u2502 \n\u251c\u2500nyse_session [skipped]\n\u2502 \n\u2514\u2500evidence [skipped]\n\nResult: Succeeded\n\ntime=2026-10-05T01:30:30.094-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-05T01:30:30.157-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\ntime=2026-10-05T01:30:30.180-04:00 level=INFO msg=\"Executing root dag-run\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 params=\"\"\ntime=2026-10-05T01:30:30.252-04:00 level=INFO msg=\"DAG run started\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa params=[]\ntime=2026-10-05T01:30:30.259-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa step=calendar_check\ntime=2026-10-05T01:30:30.286-04:00 level=INFO msg=\"Step started\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa step=nyse_session\ntime=2026-10-05T01:30:30.305-04:00 level=INFO msg=\"Checking preconditions for \\\"nyse_session\\\"\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa step=nyse_session\ntime=2026-10-05T01:30:30.311-04:00 level=INFO msg=\"Preconditions failed for \\\"nyse_session\\\"\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa step=nyse_session\ntime=2026-10-05T01:30:30.369-04:00 level=INFO msg=\"DAG run finished\" dag=native-chain run-id=034aTclYmS8noRKPBBkoB7 worker-id=local attempt-id=b370aa status=succeeded started-at=2026-10-05T01:30:30-04:00 finished-at=2026-10-05T01:30:30-04:00\n", + "native_status": { + "root": { + "name": "native-chain", + "id": "034aTclYmS8noRKPBBkoB7" + }, + "name": "native-chain", + "dagRunId": "034aTclYmS8noRKPBBkoB7", + "attemptId": "b370aa", + "attemptKey": "1e72afb7a9cc666f:b370aa", + "status": 4, + "triggerType": 2, + "workerId": "local", + "pid": 199, + "pidStartedAt": 1791178229240, + "nodes": [ + { + "step": { + "id": "calendar_check", + "name": "calendar_check", + "commands": [ + { + "command": "printf", + "args": [ + "stale_input" + ], + "cmdWithArgs": "printf stale_input" + } + ], + "output": "SESSION_DAY" + }, + "stdout": "/logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/run_20261005_053030Z_b370aa/calendar_check.20261005.013030.266.034aTclY.out", + "stderr": "/logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/run_20261005_053030Z_b370aa/calendar_check.20261005.013030.266.034aTclY.err", + "workingDir": "/data/dag-run-work/native-chain/gzynpjveef5psmljgqq6l623wq/root", + "startedAt": "2026-10-05T01:30:30-04:00", + "finishedAt": "2026-10-05T01:30:30-04:00", + "status": 4, + "doneCount": 1, + "outputVariables": { + "SESSION_DAY": "SESSION_DAY=stale_input" + }, + "outputValue": "stale_input" + }, + { + "step": { + "id": "nyse_session", + "name": "nyse_session", + "commands": [ + { + "command": "true", + "cmdWithArgs": "true" + } + ], + "depends": [ + "calendar_check" + ], + "preconditions": [ + { + "condition": "${SESSION_DAY}", + "expected": "session", + "error": "condition was not met: expected \"session\", got \"stale_input\"" + } + ] + }, + "stdout": "/logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/run_20261005_053030Z_b370aa/nyse_session.20261005.013030.293.034aTclY.out", + "stderr": "/logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/run_20261005_053030Z_b370aa/nyse_session.20261005.013030.293.034aTclY.err", + "workingDir": "/data/dag-run-work/native-chain/gzynpjveef5psmljgqq6l623wq/root", + "startedAt": "2026-10-05T01:30:30-04:00", + "finishedAt": "2026-10-05T01:30:30-04:00", + "status": 5 + }, + { + "step": { + "id": "evidence", + "name": "evidence", + "commands": [ + { + "command": "touch", + "args": [ + "/ran" + ], + "cmdWithArgs": "touch \"/ran\"" + } + ], + "depends": [ + "nyse_session" + ] + }, + "stdout": "", + "stderr": "", + "startedAt": "", + "finishedAt": "", + "status": 5, + "error": "upstream skipped" + } + ], + "createdAt": 1791178230369, + "startedAt": "2026-10-05T01:30:30-04:00", + "finishedAt": "2026-10-05T01:30:30-04:00", + "procGroup": "native-chain", + "definitionId": "native-chain", + "log": "/logs/native-chain/dag-run_20261005_053030Z_034aTclYmS8noRKPBBkoB7/dag-run_20261005.013030.191.034aTclY.log", + "workingDir": "/data/dag-run-work/native-chain/gzynpjveef5psmljgqq6l623wq/root", + "pendingStepRetries": null + }, + "marker_created": false + } + ], + "structural_validation": { + "command": "dagu validate --context local --dagu-home --config /config.yaml /blueprints/us-equities/hosting/equity-research-evidence.yaml", + "exit": 0, + "returned_output": "time=2026-10-05T01:30:30.475-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-05T01:30:30.535-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\n", + "dag_sha256": "267fe130b5e8d10676cc998e690617bc6b119c89a6ec5fbbe8fe8771e93107b9" + }, + "limitations": [ + "Native start executes short manual fixture DAGs; no start-all, scheduler, systemd, operating host or broker.", + "run_dags:false permits these manual CLI jobs; scheduler-triggered behavior still requires the later operating-host drill.", + "A native DAG can be succeeded with skipped dependents; the restart drill now requires all six research steps succeeded." + ] +} diff --git a/blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json b/blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json new file mode 100644 index 000000000..d983af24c --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json @@ -0,0 +1,107 @@ +{ + "schema_version": 1, + "round": "t-pr4-r3", + "evidence_class": "local_integration", + "dagu_pin": "2.16.6", + "calendar_pin": "4.13.2", + "clock_fixture": "2026-11-27T21:30:00Z", + "events_mtime_fixture": "2026-11-27T19:00:00Z", + "scope": "Manual native run of all six deployed steps; synthetic historical LEAN input and post-close clock; no scheduler.", + "dag_template_sha256": "6107eebb77ae8904ff6422b7e86f4c551dde36d35aa5d6e87b8f5d72b0c9397f", + "private_copy_sha256": "e71ebf99aa29a86bf0eb18ab61a054ec24bdfcb67fed5538f2be22c4662884f9", + "commands": [ + { + "verb": "validate", + "exit": 0, + "duration_s": 0.210249, + "raw_output_sha256": "e542d658b185d65bc180fbfffc22127516c034643fd63cc45c4045a6f29bdf2c" + }, + { + "verb": "start", + "exit": 0, + "duration_s": 37.141351, + "raw_output_sha256": "63b019e05930b07cc64d36140722c96bca5fab3bf913f6506fd768523cc3908d" + } + ], + "native_run_status": 4, + "native_node_statuses": { + "calendar_check": 4, + "nyse_session": 4, + "prepare_output": 4, + "summarize": 4, + "baseline_evidence": 4, + "catalog_evidence": 4 + }, + "trigger_type": 2, + "timeout_sec": 180, + "history": { + "exit": 0, + "context": "local", + "from_utc": "2026-10-05T00:00:00Z", + "status_filter": "succeeded", + "limit": 1000, + "matched_run": true, + "row_count": 1 + }, + "parquet_count": 1, + "parquet_sha256": [ + "bcb285fabdedd0129b0d54dd7097688369695f5ed806fd3a89b26db9d33c65a8" + ], + "guard_output": "session", + "unavailable_input_probes": [ + { + "token": "missing_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "0b6641cc99565df6e909406e9fc0c1c7aa9000cdb7caed04e00c610efbbc4efb" + }, + { + "token": "stale_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "7fdf89042f9cfc5c0aea90cc9ac85b22f1c4c4d1909b72c2da3d5cb482ca8969" + }, + { + "token": "future_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "452353a4995a5eabdd19bb7a6ec57cbab08ff3b0695a5f36a47d490647ba09a2" + } + ], + "completed_at_utc": "2026-10-05T07:50:43.143067+00:00", + "summarizer_returned_json": { + "scope": "LEAN bundled historical simulation, not broker fills", + "events": 1, + "distinct_orders": 1, + "statuses": { + "Filled": 1 + }, + "calendar": "XNYS", + "sample_session": "2013-10-07", + "session_open_utc": "2013-10-07T13:30:00+00:00", + "session_close_utc": "2013-10-07T20:00:00+00:00", + "source_sha256": "5d39b6390bf54295d3552923a5bc0ca7848271d300740702182a4c9aa53cad41", + "parquet_sha256": "bcb285fabdedd0129b0d54dd7097688369695f5ed806fd3a89b26db9d33c65a8" + } +} diff --git a/blueprints/us-equities/hosting/evidence/offline-proof-r2.json b/blueprints/us-equities/hosting/evidence/offline-proof-r2.json new file mode 100644 index 000000000..cb9e2a904 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/offline-proof-r2.json @@ -0,0 +1,351 @@ +{ + "comparison": { + "checked_subset": "1/1", + "inventory_sha256": "6e5bb952d7fd0261828613e6b4b27821dd6d42fd671ac1f14b36d671e794f7ed", + "matched_bytes": 16531456, + "matched_files": 2, + "required_state": "matched", + "snapshot_id": "e173073f1aea68433d4502f2f7406cdde7a741f2fc8308a24ed9f49eb3d75e8b" + }, + "completed_at_utc": "2026-10-05T05:32:33.774862+00:00", + "controls": { + "restore": { + "exit": 1, + "mutation_observation": { + "bytes_unchanged": true, + "mutation": "SQLite user_version header", + "required_state_unchanged": true, + "sha256_changed": true + }, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.012312, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-restore/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.83599, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 3, + "dirs_changed": 5, + "dirs_unmodified": 0, + "data_blobs": 1, + "tree_blobs": 9, + "data_added": 440024, + "data_added_packed": 4408, + "total_files_processed": 3, + "total_bytes_processed": 16532001, + "total_duration": 0.800409374, + "backup_start": "2026-10-05T01:32:31.299715116-04:00", + "backup_end": "2026-10-05T01:32:32.10012451-04:00", + "snapshot_id": "0f8689841dd2a17de838454f93f004cd439c69f8e9f685a1b76f0ced3945f2b0" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.780648, + "exit": 0, + "output": "using temporary cache in /control-restore/cache/restic-check-cache-3829689771\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n[0:00] 100.00% 3 / 3 snapshots\nread group #1 of 6 data packs (out of total 6 packs in 1 groups\n[0:00] 100.00% 6 / 6 packs\nno errors were found\n" + }, + { + "args": [ + "restore", + "0f8689841dd2a17de838454f93f004cd439c69f8e9f685a1b76f0ced3945f2b0:/control-restore/stage", + "--target", + "/control-restore/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.766378, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 16532001, + "bytes_restored": 16532001 + } + ] + } + } + ], + "returned": { + "reason": "file sha256/bytes mismatch", + "result": "refused" + } + }, + "snapshot": { + "exit": 1, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.015723, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-snapshot/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.786509, + "exit": 3, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 2, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 3, + "dirs_changed": 5, + "dirs_unmodified": 0, + "data_blobs": 2, + "tree_blobs": 9, + "data_added": 439323, + "data_added_packed": 4764, + "total_files_processed": 2, + "total_bytes_processed": 8266273, + "total_duration": 0.759458587, + "backup_start": "2026-10-05T01:32:30.20588227-04:00", + "backup_end": "2026-10-05T01:32:30.965340857-04:00", + "snapshot_id": "32eac2d2e6d8e8a4b83723f540cf10a456cfc73011ef7ef7bf6882c65eb502fc" + }, + { + "message_type": "error", + "error": { + "message": "open /control-snapshot/stage/journals/evidence.sqlite3: permission denied" + }, + "during": "archival", + "item": "/control-snapshot/stage/journals/evidence.sqlite3" + }, + { + "message_type": "exit_error", + "code": 3, + "message": "Warning: at least one source file could not be read" + } + ] + } + } + ], + "returned": { + "reason": "restic backup refused (exit 3)", + "result": "refused" + } + } + }, + "copy_observations": [ + { + "committed_writes_during_native_call": 6, + "journal": "evidence", + "native_copy_duration_s": 0.049027458, + "pages": -1 + }, + { + "committed_writes_during_native_call": 9, + "journal": "research", + "native_copy_duration_s": 0.052331414, + "pages": -1 + } + ], + "evidence_class": "local_integration", + "fixture_class": "synthetic SQLite WAL journals", + "frozen_inventory": { + "files": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "required_tables": { + "runs": 103 + }, + "sha256": "2672f026b8cf69cbf716b07037e263be7d5787171e4ed8599e89f0759a935334" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "required_tables": { + "events": 126 + }, + "sha256": "4809d2613be676b45b97bae9704f8e8c61b7a63b1450a009fcb6d623c3eb1f4f" + } + ], + "schema_version": 1 + }, + "independent_observation": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 103 + }, + "sha256": "2672f026b8cf69cbf716b07037e263be7d5787171e4ed8599e89f0759a935334" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 126 + }, + "sha256": "4809d2613be676b45b97bae9704f8e8c61b7a63b1450a009fcb6d623c3eb1f4f" + } + ], + "limitations": [ + "Same-host disposable repository and key; no off-host recovery.", + "No deployed scheduler, process restart, reboot, missed-run or independent alert drill.", + "Two per-journal snapshots; no atomic transaction spanning both journals.", + "Subset 1/1 only; seven-part rotation is tested separately with stdlib fixtures." + ], + "native_commands": [ + { + "args": [ + "init", + "--json" + ], + "duration_s": 3.024287, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "initialized", + "id": "ae4d596bfa0825e01e55666740ec5e48b4908d963ed797f29c7766c789e81b93", + "repository": "/repository" + } + ] + } + }, + { + "args": [ + "version" + ], + "duration_s": 0.015919, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/accepted/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.816045, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 17, + "tree_blobs": 9, + "data_added": 9312577, + "data_added_packed": 22498, + "total_files_processed": 3, + "total_bytes_processed": 16532001, + "total_duration": 0.77918784, + "backup_start": "2026-10-05T01:32:27.526759927-04:00", + "backup_end": "2026-10-05T01:32:28.305947767-04:00", + "snapshot_id": "e173073f1aea68433d4502f2f7406cdde7a741f2fc8308a24ed9f49eb3d75e8b" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.765284, + "exit": 0, + "output": "using temporary cache in /accepted/cache/restic-check-cache-653056309\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n[0:00] 100.00% 1 / 1 snapshots\nread group #1 of 2 data packs (out of total 2 packs in 1 groups\n[0:00] 100.00% 2 / 2 packs\nno errors were found\n" + }, + { + "args": [ + "restore", + "e173073f1aea68433d4502f2f7406cdde7a741f2fc8308a24ed9f49eb3d75e8b:/accepted/stage", + "--target", + "/accepted/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.738478, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 16532001, + "bytes_restored": 16532001 + } + ] + } + } + ], + "restic_pin": "0.19.1", + "result": "passed", + "schema_version": 1, + "started_at_utc": "2026-10-05T05:32:23.230897+00:00", + "throwaway_password_deleted": true +} diff --git a/blueprints/us-equities/hosting/evidence/offline-proof-r3.json b/blueprints/us-equities/hosting/evidence/offline-proof-r3.json new file mode 100644 index 000000000..46873f7af --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/offline-proof-r3.json @@ -0,0 +1,392 @@ +{ + "comparison": { + "checked_subset": "1/1", + "inventory_sha256": "be26fa25fd7b277b5994b1e713c289583e36f7524d6b31a515db2170dc58d5f2", + "matched_bytes": 16531456, + "matched_files": 2, + "required_state": "matched", + "snapshot_id": "73e7fa420736766c6c1fab6fe200dbedde193c04f377532081586098046dc8f1" + }, + "completed_at_utc": "2026-10-05T07:48:59.580496+00:00", + "controls": { + "restore": { + "exit": 1, + "mutation_observation": { + "bytes_unchanged": true, + "mutation": "SQLite user_version header", + "required_state_unchanged": true, + "sha256_changed": true + }, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.013199, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + "restic 0.19.1 compiled with go1.26.4 on linux/amd64" + ] + } + }, + { + "args": [ + "backup", + "/control-restore/stage", + "--host", + "equity-research-recovery-control", + "--tag", + "journal-recovery-control", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.867282, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 3, + "dirs_changed": 5, + "dirs_unmodified": 0, + "data_blobs": 1, + "tree_blobs": 9, + "data_added": 440030, + "data_added_packed": 4427, + "total_files_processed": 3, + "total_bytes_processed": 16532001, + "total_duration": 0.827284223, + "backup_start": "2026-10-05T03:48:56.891888143-04:00", + "backup_end": "2026-10-05T03:48:57.719172366-04:00", + "snapshot_id": "124aee8f7e4be24a458b5bbcb749053c51591c25ca4af716fa8de8299056dec5" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.801755, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + "using temporary cache in /control-restore/cache/restic-check-cache-1047705691", + "create exclusive lock for repository", + "load indexes", + "check all packs", + "check snapshots, trees and blobs", + "[0:00] 100.00% 3 / 3 snapshots", + "read group #1 of 6 data packs (out of total 6 packs in 1 groups", + "[0:00] 100.00% 6 / 6 packs", + "no errors were found" + ] + } + }, + { + "args": [ + "restore", + "124aee8f7e4be24a458b5bbcb749053c51591c25ca4af716fa8de8299056dec5:/control-restore/stage", + "--target", + "/control-restore/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.869595, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 16532001, + "bytes_restored": 16532001 + } + ] + } + } + ], + "returned": { + "reason": "file sha256/bytes mismatch", + "result": "refused" + } + }, + "snapshot": { + "exit": 1, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.016722, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + "restic 0.19.1 compiled with go1.26.4 on linux/amd64" + ] + } + }, + { + "args": [ + "backup", + "/control-snapshot/stage", + "--host", + "equity-research-recovery-control", + "--tag", + "journal-recovery-control", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.818058, + "exit": 3, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 2, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 2, + "tree_blobs": 9, + "data_added": 439317, + "data_added_packed": 4758, + "total_files_processed": 2, + "total_bytes_processed": 8266273, + "total_duration": 0.782357246, + "backup_start": "2026-10-05T03:48:55.777301122-04:00", + "backup_end": "2026-10-05T03:48:56.559658378-04:00", + "snapshot_id": "67b6b2892389e784d7c00992bc2351c7836f62e7a9b0a947f0ef0f55dd02833d" + }, + { + "message_type": "error", + "error": { + "message": "open /control-snapshot/stage/journals/evidence.sqlite3: permission denied" + }, + "during": "archival", + "item": "/control-snapshot/stage/journals/evidence.sqlite3" + }, + { + "message_type": "exit_error", + "code": 3, + "message": "Warning: at least one source file could not be read" + } + ] + } + } + ], + "returned": { + "reason": "restic backup refused (exit 3)", + "result": "refused" + } + } + }, + "copy_observations": [ + { + "committed_writes_during_native_call": 8, + "journal": "evidence", + "native_copy_duration_s": 0.049189465, + "pages": -1 + }, + { + "committed_writes_during_native_call": 7, + "journal": "research", + "native_copy_duration_s": 0.05044335, + "pages": -1 + } + ], + "evidence_class": "local_integration", + "fixture_class": "synthetic SQLite WAL journals", + "frozen_inventory": { + "files": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "required_tables": { + "runs": 105 + }, + "sha256": "c49d676b4b63b431f9be9a7bd3aad252856c1ae1ca4b31dd294bd38671bd65a8" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "required_tables": { + "events": 127 + }, + "sha256": "01a99c624a13ea86cba2546191347b9e3b1dd121aee5cbd4e14766d044718f54" + } + ], + "schema_version": 1 + }, + "independent_observation": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 105 + }, + "sha256": "c49d676b4b63b431f9be9a7bd3aad252856c1ae1ca4b31dd294bd38671bd65a8" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 127 + }, + "sha256": "01a99c624a13ea86cba2546191347b9e3b1dd121aee5cbd4e14766d044718f54" + } + ], + "limitations": [ + "Same-host disposable repository and key; no off-host recovery.", + "No deployed scheduler, process restart, reboot, missed-run or independent alert drill.", + "Two per-journal snapshots; no atomic transaction spanning both journals.", + "Subset 1/1 only; seven-part rotation is tested separately with stdlib fixtures." + ], + "native_commands": [ + { + "args": [ + "init", + "--json" + ], + "duration_s": 3.091143, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "initialized", + "id": "563f7aaab2c3391223eeb02cccba57e325234ed0637921e8bb1d6283bc68b0c9", + "repository": "/repository" + } + ] + } + }, + { + "args": [ + "version" + ], + "duration_s": 0.012186, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + "restic 0.19.1 compiled with go1.26.4 on linux/amd64" + ] + } + }, + { + "args": [ + "backup", + "/accepted/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.825084, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 17, + "tree_blobs": 9, + "data_added": 9312576, + "data_added_packed": 22516, + "total_files_processed": 3, + "total_bytes_processed": 16532001, + "total_duration": 0.786685142, + "backup_start": "2026-10-05T03:48:53.020052402-04:00", + "backup_end": "2026-10-05T03:48:53.806737544-04:00", + "snapshot_id": "73e7fa420736766c6c1fab6fe200dbedde193c04f377532081586098046dc8f1" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.773764, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + "using temporary cache in /accepted/cache/restic-check-cache-253562719", + "create exclusive lock for repository", + "load indexes", + "check all packs", + "check snapshots, trees and blobs", + "[0:00] 100.00% 1 / 1 snapshots", + "read group #1 of 2 data packs (out of total 2 packs in 1 groups", + "[0:00] 100.00% 2 / 2 packs", + "no errors were found" + ] + } + }, + { + "args": [ + "restore", + "73e7fa420736766c6c1fab6fe200dbedde193c04f377532081586098046dc8f1:/accepted/stage", + "--target", + "/accepted/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.77066, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 16532001, + "bytes_restored": 16532001 + } + ] + } + } + ], + "restic_pin": "0.19.1", + "result": "passed", + "schema_version": 1, + "started_at_utc": "2026-10-05T07:48:48.730408+00:00", + "throwaway_password_deleted": true +} diff --git a/blueprints/us-equities/hosting/evidence/offline-proof.json b/blueprints/us-equities/hosting/evidence/offline-proof.json new file mode 100644 index 000000000..8b8c93493 --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/offline-proof.json @@ -0,0 +1,339 @@ +{ + "comparison": { + "checked_subset": "1/1", + "inventory_sha256": "b0f2c4c239b57ca25c2e51e85a6967acf206514d45c9b0272ab7c473ec6686e3", + "matched_bytes": 802816, + "matched_files": 2, + "required_state": "matched", + "snapshot_id": "80b228b13dc6d2639c9f513a9f56dfe2ff28e61f5c77c51a2e92dba183aaa0f7" + }, + "completed_at_utc": "2026-10-05T03:51:24.642607+00:00", + "controls": { + "restore": { + "exit": 1, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.018846, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-restore/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--json" + ], + "duration_s": 0.782804, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 0, + "tree_blobs": 9, + "data_added": 4177, + "data_added_packed": 2897, + "total_files_processed": 3, + "total_bytes_processed": 803357, + "total_duration": 0.753974481, + "backup_start": "2026-10-04T23:51:22.369551584-04:00", + "backup_end": "2026-10-04T23:51:23.123526055-04:00", + "snapshot_id": "7b3061d7df5a8d6a18d93c18ce281aea94829368c380b64adf12ff7631c0982f" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.743623, + "exit": 0, + "output": "using temporary cache in /control-restore/cache/restic-check-cache-3201434515\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n[0:00] 100.00% 3 / 3 snapshots\nread group #1 of 4 data packs (out of total 4 packs in 1 groups\n[0:00] 100.00% 4 / 4 packs\nno errors were found\n" + }, + { + "args": [ + "restore", + "7b3061d7df5a8d6a18d93c18ce281aea94829368c380b64adf12ff7631c0982f:/control-restore/stage", + "--target", + "/control-restore/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.732437, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 803357, + "bytes_restored": 803357 + } + ] + } + } + ], + "returned": { + "reason": "file inventory mismatch (missing or extra files)", + "result": "refused" + } + }, + "snapshot": { + "exit": 1, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.012226, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-snapshot/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--json" + ], + "duration_s": 0.781126, + "exit": 3, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 2, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 0, + "tree_blobs": 9, + "data_added": 3803, + "data_added_packed": 2810, + "total_files_processed": 2, + "total_bytes_processed": 401949, + "total_duration": 0.752681412, + "backup_start": "2026-10-04T23:51:21.467918984-04:00", + "backup_end": "2026-10-04T23:51:22.220600406-04:00", + "snapshot_id": "11f4c26858dd056eebd1f931fbee775e954071f4aea7e99e206ad4d301669002" + }, + { + "message_type": "error", + "error": { + "message": "open /control-snapshot/stage/journals/evidence.sqlite3: permission denied" + }, + "during": "archival", + "item": "/control-snapshot/stage/journals/evidence.sqlite3" + }, + { + "message_type": "exit_error", + "code": 3, + "message": "Warning: at least one source file could not be read" + } + ] + } + } + ], + "returned": { + "reason": "restic backup refused (exit 3)", + "result": "refused" + } + } + }, + "copy_observations": [ + { + "committed_writes": 1, + "journal": "evidence", + "remaining_pages": 82, + "total_pages": 98 + }, + { + "committed_writes": 1, + "journal": "research", + "remaining_pages": 82, + "total_pages": 98 + } + ], + "evidence_class": "local_integration", + "fixture_class": "synthetic SQLite WAL journals", + "frozen_inventory": { + "files": [ + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "required_tables": { + "runs": 97 + }, + "sha256": "d3da8b7140827e3247d8a3bc647c9e9828d6b587fcf4bfcea7a170837bcc45f3" + }, + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "required_tables": { + "events": 97 + }, + "sha256": "7712822fae54114f0124caf9fb0d8604fcc9dddfda05e511aad364174df61159" + } + ], + "schema_version": 1 + }, + "independent_observation": [ + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 97 + }, + "sha256": "d3da8b7140827e3247d8a3bc647c9e9828d6b587fcf4bfcea7a170837bcc45f3" + }, + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 97 + }, + "sha256": "7712822fae54114f0124caf9fb0d8604fcc9dddfda05e511aad364174df61159" + } + ], + "limitations": [ + "Same-host disposable repository and key; no off-host recovery.", + "No deployed scheduler, process restart, reboot, missed-run or independent alert drill.", + "Two per-journal snapshots; no atomic transaction spanning both journals.", + "Subset 1/1 only; seven-part rotation is tested separately with stdlib fixtures." + ], + "native_commands": [ + { + "args": [ + "init", + "--json" + ], + "duration_s": 3.0445, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "initialized", + "id": "9f1afa363ee375e75b84d9e01098da594c84d694793a84e5e7fd13f2e8a16279", + "repository": "/repository" + } + ] + } + }, + { + "args": [ + "version" + ], + "duration_s": 0.011279, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/accepted/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--json" + ], + "duration_s": 0.78972, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 8, + "dirs_changed": 0, + "dirs_unmodified": 0, + "data_blobs": 3, + "tree_blobs": 9, + "data_added": 807530, + "data_added_packed": 5688, + "total_files_processed": 3, + "total_bytes_processed": 803357, + "total_duration": 0.759379716, + "backup_start": "2026-10-04T23:51:19.056499386-04:00", + "backup_end": "2026-10-04T23:51:19.815879112-04:00", + "snapshot_id": "80b228b13dc6d2639c9f513a9f56dfe2ff28e61f5c77c51a2e92dba183aaa0f7" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.749554, + "exit": 0, + "output": "using temporary cache in /accepted/cache/restic-check-cache-3312925664\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n[0:00] 100.00% 1 / 1 snapshots\nread group #1 of 2 data packs (out of total 2 packs in 1 groups\n[0:00] 100.00% 2 / 2 packs\nno errors were found\n" + }, + { + "args": [ + "restore", + "80b228b13dc6d2639c9f513a9f56dfe2ff28e61f5c77c51a2e92dba183aaa0f7:/accepted/stage", + "--target", + "/accepted/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.760115, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 803357, + "bytes_restored": 803357 + } + ] + } + } + ], + "restic_pin": "0.19.1", + "result": "passed", + "schema_version": 1, + "started_at_utc": "2026-10-05T03:51:15.065593+00:00", + "throwaway_password_deleted": true +} diff --git a/blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json b/blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json new file mode 100644 index 000000000..ff2c72f5c --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json @@ -0,0 +1,252 @@ +{ + "schema_version": 1, + "round": "t-pr4-r2", + "observed_at_utc": "2026-10-05T05:25:40.188600+00:00", + "scope": "Native functionality at locked pins; no new orchestrator, replication service, install, broker or pin selection.", + "cli_checks": [ + { + "command": "restic backup --help", + "exit": 0, + "returned_output": "EXIT STATUS\nExit status is 0 if the command was successful.\nExit status is 1 if there was a fatal error (no snapshot created).\nExit status is 3 if some source data could not be read (incomplete snapshot created).\nExit status is 10 if the repository does not exist.\nExit status is 11 if the repository is already locked.\nExit status is 12 if the password is incorrect.\n -g, --group-by group group snapshots by host, paths and/or tags, separated by comma (disable grouping with '') (default host,paths)\n --parent snapshot use this parent snapshot (default: latest snapshot in the group determined by --group-by and not newer than the timestamp determined by --time)\n --stdin-from-command interpret arguments as command to execute and store its stdout\n", + "representation": "selected help lines" + }, + { + "command": "restic check --help", + "exit": 0, + "returned_output": "backed-up data, use the --read-data or --read-data-subset flags.\n --read-data-subset subset read a subset of data packs, specified as 'n/t' for specific part, or either 'x%' or 'x.y%' or a size in bytes with suffixes k/K, m/M, g/G, t/T for a random subset\n", + "representation": "selected help lines" + }, + { + "command": "restic restore --help", + "exit": 0, + "returned_output": " --delete delete files from target directory if they do not exist in snapshot. Use '--dry-run -vv' to check what would be deleted\n --overwrite behavior overwrite behavior, one of (always|if-changed|if-newer|never) (default always)\n -t, --target string directory to extract data to\n --verify verify restored files content\n --no-extra-verify skip additional verification of data before upload (see documentation)\n --pack-size size set target pack size in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)\n", + "representation": "selected help lines" + }, + { + "command": "restic forget --help", + "exit": 0, + "returned_output": "first divided into groups according to \"--group-by\", and after that the policy\n -g, --group-by group group snapshots by host, paths and/or tags, separated by comma (disable grouping with '') (default host,paths)\n -n, --dry-run do not delete anything, just print what would be done\n", + "representation": "selected help lines" + }, + { + "command": "dagu start-all --help", + "exit": 0, + "returned_output": "Simultaneously start the web UI server, scheduler, and optionally coordinator in a single command.\nThis convenience command combines the functionality of 'dagu server', 'dagu scheduler',\nand optionally 'dagu coordinator' into a single process. The web UI provides the management\ninterface, the scheduler handles automated DAG-run execution based on defined schedules,\nand the coordinator (when enabled) manages distributed task execution across workers.\nThe coordinator is enabled by default and can be disabled by setting coordinator.enabled=false\nin the config file or DAGU_COORDINATOR_ENABLED=false as an environment variable.\n --coordinator.host string Host address to bind the coordinator gRPC server to (default: 127.0.0.1)\n --coordinator.advertise string Address to advertise in service registry (default: auto-detected hostname)\n --coordinator.port int Port number for the coordinator gRPC server (default: 50055)\n # Default mode (coordinator enabled)\n # Disable coordinator\n DAGU_COORDINATOR_ENABLED=false dagu start-all\n # Distributed mode with coordinator on all interfaces\n dagu start-all --coordinator.host=0.0.0.0 --coordinator.port=50055\n # Production with both web and coordinator on all interfaces\n dagu start-all --host=0.0.0.0 --port=8080 --coordinator.host=0.0.0.0\n -A, --coordinator.advertise string Address to advertise in service registry (default: auto-detected hostname)\n -H, --coordinator.host string Coordinator gRPC server host (default: 127.0.0.1) (default \"127.0.0.1\")\n -P, --coordinator.port string Coordinator gRPC server port (default: 50055) (default \"50055\")\n", + "representation": "selected help lines" + }, + { + "command": "python3 -m sqlite3 --help", + "exit": 0, + "returned_output": "usage: python -m sqlite3 [-h] [-v] [filename] [sql]\n\nPython sqlite3 CLI\n\npositional arguments:\n filename SQLite database to open (defaults to ':memory:'). A new\n database is created if the file does not previously exist.\n sql An SQL query to execute. Any returned rows are printed to\n stdout.\n\noptions:\n -h, --help show this help message and exit\n -v, --version Print underlying SQLite library version\n", + "representation": "full returned output" + }, + { + "command": "ecal --help", + "exit": 255, + "returned_output": "usage: CALENDAR [[[DAY] MONTH] YEAR]\n", + "representation": "full returned output" + }, + { + "command": "ecal XNYS 11 2026", + "exit": 0, + "returned_output": " November 2026\n Su Mo Tu We Th Fr Sa\n\n[ 1] 2 3 4 5 6 [ 7]\n[ 8] 9 10 11 12 13 [14]\n[15] 16 17 18 19 20 [21]\n[22] 23 24 25 [26] 27 [28]\n[29] 30 \n\n", + "representation": "full returned output" + } + ], + "sqlite_library_executed": "3.53.1", + "sqlite_cli_lookup": { + "command": "lookup sqlite3 in PATH", + "present": false + }, + "sources": [ + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/runtime/runner.go:1267-1281,1502-1525,1196-1206,1633-1648", + "sha256": "95ec2ba0e869bc1fc3d2d25feb5b1af0371602d0af7073d4852381faaf5d7948" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmn/config/env.go:77-95", + "sha256": "5d852a9e96a17c37101602a96217b369bcdf79e0463e880e2cba85b9ffed3f81" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmn/config/loader.go:344-347,1311-1319", + "sha256": "eb7203d02fbf88a213bdee63261dccfefca376c358612630779b181069cfdec3" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmd/startall.go:32-45", + "sha256": "1b535400a80904fcceec79b16fa862b917d265170c5a31ae48cc8c856e32763e" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmd/history.go:568-602", + "sha256": "a972e448952c527fd602465c5c4fa43c008422f2c13d22b9b08ee94fff0a760e" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/040_backup.rst:197-206,679-703,787-806", + "sha256": "e7281c20f0ef27bdbbb5f59d12f64aef63fae33048c5bd6b086ff1973edd218e" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/045_working_with_repos.rst:482-521", + "sha256": "56bfbe763b9cae005700f9472f73f69acaa28eea07d6fe258471af729ffffcd6" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/050_restore.rst:55-73", + "sha256": "00d39803cfcb17158c2ee1d835de6351c902e66067d2493438831580e9b04251" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/060_forget.rst:225-233", + "sha256": "444be0adec93ad92810a4c6b363d8a5a7b77d84c88879e38408b1a7ea2eff614" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "src/shell.c.in:3737,3759,9074-9099", + "sha256": "bcceddfe1ab18e60c32cb9bbac7020ec22c03dbd68bd8dffe3599246560a46e3" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "src/backup.c:321-392", + "sha256": "ee4cbfc7d8afb311d12cc106d645764a3e18266694f003438b410bfa33764ef7" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "tool/sqlite3_rsync.c:13-40", + "sha256": "b9e54de43b69da85771de1acacd1ade4ad29d001be09b2f187dfb954eda2f0c8" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Modules/_sqlite/connection.c:2011-2016,2067-2102", + "sha256": "c33db9525cbc9887293c1cbdc03e0d8a3a3201e7dae98392452a40196e850070" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/sqlite3.rst:1107-1155", + "sha256": "f5be72345d2d37ffa53a82058b6054ed9052dda816a2e84639b96172fe3183b7" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/subprocess.rst:62-68", + "sha256": "2cb61f06f2fc41a60cf52c13a1bdaf52ff2a7962c55be96ec650bb12eb3c6845" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/hashlib.rst:267-302", + "sha256": "49df3195ff1acf0d114ed0288a15a43c924c9cd41564e0d956d40998ed6d6a82" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/fcntl.rst:139-149", + "sha256": "2d828976d383a49e1bd394921100078afd5412a50c498b5302682bbf2be58a8d" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "pyproject.toml:67-68", + "sha256": "0624504db59140714e6d12dd5039009e6517e20e88017c2e8989cf961645a266" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "exchange_calendars/ecal.py:100-149", + "sha256": "1744b47c0a502f5a3ae9baebdf5affac6027da68ce3ab954289691e7f03f6c8f" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "exchange_calendars/exchange_calendar.py:1012-1016,1257-1279", + "sha256": "7b8087892b698299fe448afbc4b3fb9300a34ba1605804999436525391c08de6" + }, + { + "pin": "systemd/systemd@v255", + "locator": "man/loginctl.xml:186-195", + "sha256": "b07e1b67f169a8d0b380f5738676390b2c5d6b4297bc6f1845aaf1e336d6a476" + }, + { + "pin": "systemd/systemd@v255", + "locator": "man/systemd.service.xml:818-836", + "sha256": "439322b120014971f37e0a96adc6cb827367846ff6c16f193e9ec4b15931033b" + } + ], + "release_checks": [ + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "tagged GitHub release body", + "entry_point_verified": "ecal is declared at pyproject.toml:67-68", + "release_body_sha256": "42187892bcb2fb2981cc93f0b0ca49247e2cfb1a299e01970e332c1b55034d06", + "summary": "Tagged 4.13.2 notes cover New Year holiday-name formatting and non-XNYS calendar updates; packaging/native help separately confirm the ecal entry point." + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "CHANGELOG.md:3", + "result": "Read tagged changelog; native backup help and doc/040_backup.rst:679-703 verify stdin-from-command.", + "sha256": "fbf2ba611d2821e6b965129ce455ae3d086dab29fc7552eb34610aff289bbf21" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "tagged GitHub release body", + "source_sha256": "333c6be589bdd718ad0b5e6a58ae5e8a47d76248169cdd69c6bdeff1095e5140", + "comparison_artifact": "blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json" + } + ], + "native_choices": [ + { + "glue": "session_day.py", + "upstream_outcome": "kept", + "checked": "Native ecal CLI, native is_session/session_close and Dagu preconditions/retries/handlers.", + "gap_citation": "gerrymanoim/exchange_calendars@4.13.2:pyproject.toml:67-68; exchange_calendars/ecal.py:100-149; exchange_calendars/exchange_calendar.py:1012-1016,1257-1279", + "why": "ecal renders a calendar; compose native session/close queries into the job eligibility/input-mtime token and leave skipping to Dagu." + }, + { + "glue": "journal_recovery.py", + "upstream_outcome": "kept", + "checked": "restic stdin-from-command, SQLite CLI .backup/.dump, SQLite sqlite3_rsync, CPython Online Backup and subprocess timeout.", + "gap_citation": "restic/restic@v0.19.1:doc/040_backup.rst:679-703; doc/045_working_with_repos.rst:482-521; doc/050_restore.rst:55-73; sqlite/sqlite@version-3.53.1:src/shell.c.in:9074-9099; tool/sqlite3_rsync.c:13-40", + "why": "Native operations do the copying/backups/checks/restores; consumer glue freezes the external oracle and required tables, rotates checks with a deadline and refuses controls." + }, + { + "glue": "journal_recovery.py rotation lock", + "upstream_outcome": "replaced", + "checked": "CPython native fcntl.flock", + "gap_citation": "python/cpython@v3.12.3:Doc/library/fcntl.rst:139-149", + "why": "Replace exclusive-create sentinels with nonblocking process-owned flock; stale file survives safely and no custom stale-lock recovery runs." + }, + { + "glue": "drill_process_restart.py", + "upstream_outcome": "kept", + "checked": "systemd Restart=on-failure, Dagu native history/status and native handlers/retries.", + "gap_citation": "systemd/systemd@v255:man/systemd.service.xml:818-836; dagucloud/dagu@v2.16.6:internal/cmd/history.go:568-602; internal/ir/status.go:133-149", + "why": "Upstreams own supervision and run records; identity/time/holiday guards and exact due-slot correlation are our acceptance procedure." + }, + { + "glue": "drill_local_recovery.py", + "upstream_outcome": "kept", + "checked": "Native Online Backup, restic backup/check/restore and .backup/.dump/sqlite3_rsync alternatives.", + "gap_citation": "python/cpython@v3.12.3:Modules/_sqlite/connection.c:2067-2102; restic/restic@v0.19.1:doc/050_restore.rst:55-73", + "why": "Synthetic concurrent WAL writers, failures preserving integrity/counts and path-free integration evidence are consumer acceptance rather than replacement backup tools." + } + ], + "stdin_assessment": { + "backup": "restic stores command stdout; sqlite3 .backup writes a destination DB and emits no DB stdout, so this command would archive an empty stream.", + "dump": "Logical SQL dump would require replay and a different completeness oracle; this contract freezes staged binary DB inventory before restic.", + "maintained_tool": "SQLite upstream sqlite3_rsync performs live replication; it does not replace the encrypted restic repository/consumer-oracle contract." + }, + "retained_decision": { + "locator": "coordinator decision-record-T-final.md:102", + "rule": "Keep restic and stdlib Online Backup; independent destination, key recovery and second-host consumer restore are decision 3.", + "litestream_context": "blueprints/us-equities/convergence-review/prompt.txt:F8 deferred Litestream in the prior hosting comparison. The final T row does not name Litestream; this repair does not reopen it." + }, + "corrections": [ + "Round 1 Jan-1 bounds falsely failed New Year holidays; real 4.13.2 tests reproduced three DateOutOfBounds errors and now pass bracketed queries.", + "SDK_ENV lock authority is adoption/sdk/requirements-linux-x86_64-py313.lock:241 at the repair base, not an unlanded runtime bundle.", + "exchange_calendars ships ecal; pyproject.toml:67-68 and installed CLI were checked, so no CLI-absence claim is made.", + "An initial source lookup under internal/cmn/env/env.go was wrong; rg located internal/cmn/config/env.go and lines 77-95 were read.", + "exchange_calendars CHANGELOG.md and etc/changes.rst returned HTTP 404 at 4.13.2; tagged GitHub release notes were read instead." + ], + "limitations": [ + "Source gap judgments are scoped inferences from documented native interfaces, not exhaustive ecosystem absence claims.", + "No SQLite CLI or sqlite3_rsync installation/trial; pinned SQLite source and installed CPython Online Backup are the evidence." + ] +} diff --git a/blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json b/blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json new file mode 100644 index 000000000..46e60991c --- /dev/null +++ b/blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json @@ -0,0 +1,56 @@ +{ + "schema_version": 1, + "round": "t-pr4-r3", + "observed_at_utc": "2026-10-05T07:58:37.912257+00:00", + "recorded_separately_at_utc": "2026-10-05T09:56:59.502538+00:00", + "scope": "Round-3 additions to the immutable round-2 native review. Earlier CLI/release observations remain in the parent review.", + "parent_review": { + "path": "blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json", + "sha256": "2f8c7896b7277264801fc4b701375c612574619a2b96190202396fb52b00e2e7", + "bytes": 15859 + }, + "sources": [ + { + "pin": "systemd/systemd@v255", + "locator": "man/systemctl.xml:541-549,2377-2388", + "sha256": "7f06242ce06d9d895bf7445e3d29680f8e6a7284004122b4e209097b8313b6d5" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/service/frontend/api/v1/dagruns.go:2066-2096", + "sha256": "f07b1b89d784251594bd68478a1120f88111a7f538081158826f7a99cb62f75f" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/service/frontend/api/v1/transformer.go:362,371", + "sha256": "923e6b2e76fa03473cde1c4ed82d9106acfc99f3e95e7fe961d21e0d11849d31" + } + ], + "native_choices": [ + { + "glue": "drill_process_restart.py", + "upstream_outcome": "kept", + "checked": "systemd Restart=on-failure and unit-addressed systemctl kill; Dagu history, REST run-details, original status records, handlers/retries.", + "gap_citation": "systemd/systemd@v255:man/systemctl.xml:541-549,2377-2388; dagucloud/dagu@v2.16.6:internal/service/frontend/api/v1/transformer.go:362,371", + "why": "Upstreams deliver the fault and supervise; acceptance glue guards identity/time/holiday and correlates the exact due slot. Original status bytes avoid HTTP/auth and require revalidation on pin moves." + }, + { + "glue": "drill_process_restart.py signal delivery", + "upstream_outcome": "replaced", + "checked": "Installed systemctl 255.4 help and systemd v255 unit-addressed kill.", + "gap_citation": "systemd/systemd@v255:man/systemctl.xml:541-549,2377-2388", + "why": "Use native systemctl --user kill --kill-whom=main --signal=SIGKILL rather than os.kill after a PID lookup." + } + ], + "corrections": [ + "Round 3 unavailable-input tokens exit 1 so native Dagu fails the run; holidays/before-close still skip.", + "Round 3 source journal_mode must be WAL unless explicit opt-in accepts rollback-journal writer blocking.", + "Round 3 control backups use a distinct host/tag; controls remain disposable-repository only." + ], + "native_review": { + "scope": "Unit-addressed fault delivery and supported run-details API; earlier native observations remain historical.", + "systemctl_version": "255.4", + "systemctl_help_exit": 0, + "systemctl_version_exit": 0 + } +} diff --git a/blueprints/us-equities/hosting/journal_recovery.py b/blueprints/us-equities/hosting/journal_recovery.py new file mode 100644 index 000000000..0076ed34b --- /dev/null +++ b/blueprints/us-equities/hosting/journal_recovery.py @@ -0,0 +1,441 @@ +#!/usr/bin/env python3 +"""Snapshot research SQLite journals, then run a local restic check/restore cycle. + +Supported primitives (integration policy and inventory comparison are ours): +python/cpython@v3.12.3:Doc/library/sqlite3.rst:1107-1155 (Online Backup API); +restic/restic@v0.19.1:doc/040_backup.rst:787-806 (all nonzero exits fail); +doc/045_working_with_repos.rst:482-521 (deterministic subset rotation); +doc/050_restore.rst:55-73 (explicit snapshot/subfolder restore). +Only explicit local paths are accepted; this script has no broker/client integration. +""" + +from __future__ import annotations + +import argparse +from contextlib import closing +from datetime import datetime, timedelta, timezone +import fcntl +import hashlib +import json +import os +from pathlib import Path +import re +import sqlite3 +import stat +import subprocess +import sys +import tempfile +import time + + +CHECK_PARTS = 7 +MAX_ROTATION_AGE = timedelta(days=8) +BACKUP_TIMEOUT = 120.0 +MAX_JOURNAL_BYTES = 256 * 1024 * 1024 +INVENTORY_FILE = "inventory.json" +SAFE_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z") + + +class RecoveryError(ValueError): + """A snapshot, native command, rotation or restore could not be accepted.""" + + +def json_bytes(value) -> bytes: + return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode() + + +def write_new(path: Path, data: bytes) -> None: + # CPython@v3.12.3:Doc/library/os.rst:996-1005,1077-1087: native fchmod/fsync. + with path.open("xb") as stream: + stream.write(data) + stream.flush() + os.fchmod(stream.fileno(), 0o400) + os.fsync(stream.fileno()) + parent = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(parent) + finally: + os.close(parent) + + +def file_identity(path: Path) -> dict: + if not stat.S_ISREG(path.lstat().st_mode): + raise RecoveryError("inventory requires regular files, without symlinks") + with path.open("rb") as stream: + digest = hashlib.file_digest(stream, "sha256").hexdigest() + return {"sha256": digest, "bytes": path.stat().st_size} + + +def journal_state(path: Path, tables: list[str]) -> dict: + if not tables or len(set(tables)) != len(tables) or any( + not SAFE_NAME.fullmatch(table) for table in tables + ): + raise RecoveryError("each journal requires distinct, explicit table names") + # mode=ro never creates a missing journal. Snapshot destinations use DELETE mode. + with closing(sqlite3.connect(path.resolve().as_uri() + "?mode=ro", uri=True)) as db: + integrity = [row[0] for row in db.execute("PRAGMA integrity_check")] + if integrity != ["ok"]: + raise RecoveryError("journal integrity_check failed") + counts = {table: db.execute(f'SELECT COUNT(*) FROM "{table}"').fetchone()[0] + for table in sorted(tables)} + return {"integrity_check": "ok", "required_tables": counts} + + +def copy_journal(source: Path, target: Path, max_bytes: int, *, allow_non_wal=False) -> dict: + """Use upstream Online Backup unchanged, with a single WAL read transaction. + + CPython@v3.12.3:Modules/_sqlite/connection.c:2013,2067-2102 passes pages=-1 + straight to sqlite3_backup_step and releases the GIL during that native call. + The caller bounds this worker with subprocess.run(timeout=...). + """ + with closing(sqlite3.connect(source.resolve().as_uri() + "?mode=ro", uri=True)) as src: + mode = src.execute("PRAGMA journal_mode").fetchone()[0].lower() + if mode != "wal" and not allow_non_wal: + raise RecoveryError("source must use WAL; --allow-non-wal explicitly accepts writer blocking") + src.execute("BEGIN") + size = src.execute("PRAGMA page_count").fetchone()[0] * src.execute("PRAGMA page_size").fetchone()[0] + if size > max_bytes: + raise RecoveryError("journal exceeds --max-journal-bytes") + with closing(sqlite3.connect(target)) as dst: + started = time.monotonic_ns() + src.backup(dst, pages=-1) + finished = time.monotonic_ns() + dst.execute("PRAGMA journal_mode=DELETE") + return {"backup_started_ns": started, "backup_finished_ns": finished, "pages": -1} + + +def snapshot_journals(journals: dict, stage: Path, oracle: Path, *, observation=None, + backup_timeout=BACKUP_TIMEOUT, max_journal_bytes=MAX_JOURNAL_BYTES, + allow_non_wal=False) -> dict: + """One Online Backup per journal; no cross-journal atomicity is promised. + + An isolated stdlib worker permits a hard wall timeout even within a native step. + Optional observation reports the native call interval to the offline fixture. + Failed stages remain for diagnosis; they never receive an accepted inventory. + """ + if backup_timeout <= 0 or max_journal_bytes <= 0: + raise RecoveryError("positive backup timeout and journal size limits are required") + if not journals or oracle.exists(): + raise RecoveryError("use nonempty journal selection and a new external inventory") + if stage.resolve() in oracle.resolve().parents: + raise RecoveryError("the trusted inventory must be outside staging") + for name, (source, tables) in journals.items(): + if not SAFE_NAME.fullmatch(name) or not tables: + raise RecoveryError("a journal name and required tables are mandatory") + if not source.is_file() or source.is_symlink(): + raise RecoveryError("journal source must be an existing regular file") + stage.mkdir(mode=0o700) + (stage / "journals").mkdir(mode=0o700) + records = [] + for name, (source, tables) in sorted(journals.items()): + target = stage / "journals" / f"{name}.sqlite3" + completed = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), "_copy", "--source", str(source.resolve()), + "--target", str(target.resolve()), "--max-journal-bytes", str(max_journal_bytes), + *(["--allow-non-wal"] if allow_non_wal else [])], + env={"PATH": "/usr/bin:/bin", "LC_ALL": "C", "TMPDIR": os.environ.get("TMPDIR", str(stage.parent))}, + capture_output=True, text=True, timeout=backup_timeout, check=False) + if completed.returncode: + raise RecoveryError("Online Backup worker refused: " + json.loads(completed.stdout)["reason"]) + if observation: + observation(name, json.loads(completed.stdout)) + state = journal_state(target, tables) + target.chmod(0o400) + records.append({"path": target.relative_to(stage).as_posix(), + **file_identity(target), **state}) + inventory = {"schema_version": 1, "files": records} + frozen = json_bytes(inventory) + write_new(stage / INVENTORY_FILE, frozen) + write_new(oracle, frozen) + verify(stage, oracle) + return inventory + + +def verify(root: Path, oracle: Path) -> dict: + """Trust the external frozen inventory, never the restore's own inventory.""" + frozen = oracle.read_bytes() + inventory = json.loads(frozen) + records = inventory.get("files", []) + if inventory.get("schema_version") != 1 or not records: + raise RecoveryError("empty or invalid frozen inventory") + expected = {INVENTORY_FILE} + for record in records: + relative = Path(record["path"]) + if relative.is_absolute() or ".." in relative.parts or record["path"] in expected: + raise RecoveryError("invalid or duplicate inventory path") + expected.add(record["path"]) + actual = set() + for path in root.rglob("*"): + mode = path.lstat().st_mode + if stat.S_ISDIR(mode): + continue + if not stat.S_ISREG(mode): + raise RecoveryError("restore contains a symlink or special file") + actual.add(path.relative_to(root).as_posix()) + if actual != expected: + raise RecoveryError("file inventory mismatch (missing or extra files)") + if (root / INVENTORY_FILE).read_bytes() != frozen: + raise RecoveryError("restored inventory differs from the external frozen oracle") + for record in records: + path = root / record["path"] + if file_identity(path) != {key: record[key] for key in ("sha256", "bytes")}: + raise RecoveryError("file sha256/bytes mismatch") + if journal_state(path, list(record["required_tables"])) != { + key: record[key] for key in ("integrity_check", "required_tables") + }: + raise RecoveryError("required journal state mismatch") + return {"matched_files": len(records), "matched_bytes": sum(r["bytes"] for r in records), + "inventory_sha256": hashlib.sha256(frozen).hexdigest(), "required_state": "matched"} + + +class Restic: + """Run the pinned executable with explicit local inputs and bounded commands.""" + + def __init__(self, binary: Path, repository: Path, password_file: Path, work: Path, + timeout: int = 600): + self.binary = binary.resolve() + self.repository = repository.resolve() + self.password_file = password_file.resolve() + self.work = work.resolve() + self.timeout = timeout + self.commands = [] + # Empty inherited environment: no remote repository, password or cloud settings. + self.env = {"HOME": str(work), "PATH": "/usr/bin:/bin", "LC_ALL": "C", + "TMPDIR": os.environ.get("TMPDIR", str(work)), "RESTIC_PROGRESS_FPS": "0"} + + def run(self, *args: str) -> str: + command = [str(self.binary), "--repo", str(self.repository), "--password-file", + str(self.password_file), "--cache-dir", str(self.work / "cache"), *args] + started = time.monotonic() + result = subprocess.run(command, cwd=self.work, env=self.env, text=True, + capture_output=True, timeout=self.timeout, check=False) + output = result.stdout + result.stderr + # Private raw returned output stays outside the checkout. Public proof scrubs paths. + (self.work / f"restic-{len(self.commands):02d}.log").write_text(output) + self.commands.append({"args": list(args), "exit": result.returncode, + "duration_s": round(time.monotonic() - started, 6), "output": output}) + (self.work / f"restic-{len(self.commands) - 1:02d}.result.json").write_bytes( + json_bytes(self.commands[-1])) + # restic@v0.19.1:doc/040_backup.rst:792-806: exit 3 is an INCOMPLETE snapshot. + if result.returncode != 0: + raise RecoveryError(f"restic {args[0]} refused (exit {result.returncode})") + return result.stdout + + +def check_rotation(restic, state_path: Path, *, parts=CHECK_PARTS, now=None, + full_check=False, clock=None) -> str: + """Daily n/t subsets with an eight-day maximum age; advance after native success. + + Run daily, including weekends/holidays; jitter and DST do not impose a per-step + 24h deadline. The eight-day bound gives daily runs DST/jitter slack; a rotation + exceeding it, including at rollover, requires a full read. + A separate state file belongs to this repository; concurrent checks fail closed. + """ + if not 1 <= parts <= CHECK_PARTS: + raise RecoveryError("rotation requires 1..7 parts (default 7)") + # Explicit 'now' is a deterministic test/drill clock. Real cycles also supply + # a completion clock, so a slow check cannot pass after the rotation deadline. + clock = clock or ((lambda: now) if now is not None else lambda: datetime.now(timezone.utc)) + now = now or clock() + if now.utcoffset() is None: + raise RecoveryError("rotation needs an aware UTC clock") + now = now.astimezone(timezone.utc) + lock = state_path.with_suffix(state_path.suffix + ".lock") + # Native process-owned flock replaces the crash-prone exclusive-create sentinel. + # CPython@v3.12.3:Doc/library/fcntl.rst:139-153. Keep the inode, even on exit. + with lock.open("a") as locked: + try: + fcntl.flock(locked, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError as error: + raise RecoveryError("rotation lock is held by another process") from error + try: + key = hashlib.sha256(str(restic.repository).encode()).hexdigest() + state = json.loads(state_path.read_text()) if state_path.exists() else { + "schema_version": 2, "parts": parts, "next": 1, "repository_key": key} + if state.get("parts") != parts or state.get("repository_key") != key: + raise RecoveryError("rotation state belongs to another repository or partition count") + previous = datetime.fromisoformat(state["last_success"]).astimezone(timezone.utc) if "last_success" in state else None + part = state["next"] + if type(part) is not int or not 1 <= part <= parts: + raise RecoveryError("invalid rotation cursor") + origin = datetime.fromisoformat(state["cycle_started_at"]).astimezone(timezone.utc) if "cycle_started_at" in state else now + overdue = (state.get("schema_version") != 2 or (part != 1 and "cycle_started_at" not in state) or + now < origin or (previous and (now < previous or now - previous > MAX_ROTATION_AGE)) or + now - origin > MAX_ROTATION_AGE) + if overdue: + if not full_check: + raise RecoveryError("rotation overdue or clock reversed; require --full-check") + if full_check: + restic.run("check", "--read-data") + part = 1 + if part == 1: + origin = now # The cycle-start instant, before this subset's native check. + subset = f"{part}/{parts}" + restic.run("check", f"--read-data-subset={subset}") + finished = clock().astimezone(timezone.utc) + if finished < now or finished - origin > MAX_ROTATION_AGE: + raise RecoveryError("rotation overdue or clock reversed after native check; require --full-check") + state.update(schema_version=2, next=part % parts + 1, + cycle_started_at=origin.isoformat(), last_success=finished.isoformat()) + # tempfile + replace is native atomic file publication, without stale .new refusal. + with tempfile.NamedTemporaryFile(dir=state_path.parent, delete=False) as stream: + temporary = Path(stream.name) + try: + stream.write(json_bytes(state)) + stream.flush() + os.fsync(stream.fileno()) + temporary.chmod(0o400) + os.replace(temporary, state_path) + finally: + temporary.unlink(missing_ok=True) + return subset + finally: + fcntl.flock(locked, fcntl.LOCK_UN) + + +def cycle(restic: Restic, journals: dict, state_path: Path, *, parts=CHECK_PARTS, + full_check=False, control=None, observation=None, + backup_timeout=BACKUP_TIMEOUT, max_journal_bytes=MAX_JOURNAL_BYTES, + allow_non_wal=False) -> dict: + cycle_started = datetime.now(timezone.utc) + stage, oracle, restored = (restic.work / name for name in ("stage", "frozen-inventory.json", "restored")) + if restored.exists() or not restic.repository.is_dir() or not restic.password_file.is_file(): + raise RecoveryError("use a fresh restore/work directory and an initialized local repository") + if restic.password_file.is_relative_to(stage) or restic.repository.is_relative_to(stage): + raise RecoveryError("repository and password file must be outside staging") + version_output = restic.run("version") + if not version_output.startswith("restic 0.19.1 "): + raise RecoveryError("restic must stay pinned to 0.19.1") + inventory = snapshot_journals(journals, stage, oracle, observation=observation, + backup_timeout=backup_timeout, max_journal_bytes=max_journal_bytes, + allow_non_wal=allow_non_wal) + verify(stage, oracle) # Refuse a staging change before attempting backup. + unreadable = stage / inventory["files"][0]["path"] + if control == "snapshot": + unreadable.chmod(0) # Keep another readable file, so restic can create an incomplete snapshot. + # restic/restic@v0.19.1:doc/040_backup.rst:197-206 groups parents by host/tags. + # Keep deliberately failing controls outside the production parent/forget group. + suffix = "-control" if control else "" + try: + output = restic.run("backup", str(stage), "--host", "equity-research-recovery" + suffix, + "--tag", "journal-recovery" + suffix, "--group-by", "host,tags", "--json") + finally: + unreadable.chmod(0o400) + if control == "snapshot": + raise RecoveryError("snapshot control did not provoke native refusal; control is not accepted") + verify(stage, oracle) # Detect changes during backup before accepting its snapshot ID. + summaries = [row for line in output.splitlines() if line.startswith("{") + for row in [json.loads(line)] if row.get("message_type") == "summary"] + if len(summaries) != 1 or not re.fullmatch(r"[a-f0-9]{64}", summaries[0].get("snapshot_id", "")): + raise RecoveryError("backup did not return exactly one complete snapshot ID") + snapshot_id = summaries[0]["snapshot_id"] + subset = check_rotation(restic, state_path, parts=parts, full_check=full_check, + now=cycle_started, clock=lambda: datetime.now(timezone.utc)) + # Restore the selected snapshot's staging subtree to a new directory; never use latest. + restic.run("restore", f"{snapshot_id}:{stage}", "--target", str(restored), + "--verify", "--overwrite", "never", "--json") + if control == "restore": + altered = restored / inventory["files"][0]["path"] + tables = list(inventory["files"][0]["required_tables"]) + before_state, before_identity = journal_state(altered, tables), file_identity(altered) + altered.chmod(0o600) + with closing(sqlite3.connect(altered)) as db: + current = db.execute("PRAGMA user_version").fetchone()[0] + db.execute(f"PRAGMA user_version={current ^ 1}") + unchanged = journal_state(altered, tables) == before_state + after_identity = file_identity(altered) + observation_control = {"mutation": "SQLite user_version header", "required_state_unchanged": unchanged, + "bytes_unchanged": after_identity["bytes"] == before_identity["bytes"], + "sha256_changed": after_identity["sha256"] != before_identity["sha256"]} + (restic.work / "restore-control.json").write_bytes(json_bytes(observation_control)) + if not all(observation_control[key] for key in ("required_state_unchanged", "bytes_unchanged", "sha256_changed")): + raise RecoveryError("restore control did not preserve counts/integrity/bytes while altering content") + compared = verify(restored, oracle) + if control: + raise RecoveryError("unknown or non-discriminating control") + return {"snapshot_id": snapshot_id, "checked_subset": subset, **compared} + + +def selections(journal_args, table_args) -> dict: + journals, tables = {}, {} + for argument in journal_args: + name, separator, path = argument.partition("=") + if not separator or not path or name in journals or not SAFE_NAME.fullmatch(name): + raise RecoveryError("use distinct --journal NAME=PATH arguments") + journals[name] = Path(path) + for argument in table_args: + name, separator, table = argument.partition("=") + if not separator or name not in journals or not SAFE_NAME.fullmatch(table): + raise RecoveryError("use --required-table NAME=TABLE for each selected journal") + tables.setdefault(name, []).append(table) + if not journals or set(tables) != set(journals): + raise RecoveryError("every selected journal needs explicit required tables") + return {name: (path, tables[name]) for name, path in journals.items()} + + +def main(argv=None) -> int: + os.umask(0o077) + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + for name in ("snapshot", "cycle"): + p = commands.add_parser(name) + p.add_argument("--journal", action="append", required=True) + p.add_argument("--required-table", action="append", required=True) + p.add_argument("--backup-timeout", type=float, default=BACKUP_TIMEOUT, + help="hard seconds per isolated Online Backup worker (default 120)") + p.add_argument("--max-journal-bytes", type=int, default=MAX_JOURNAL_BYTES, + help="maximum logical bytes per journal (default 256 MiB)") + p.add_argument("--allow-non-wal", action="store_true", + help="accept blocking rollback-journal writers for the entire bounded copy") + if name == "snapshot": + p.add_argument("--stage", type=Path, required=True) + p.add_argument("--inventory", type=Path, required=True) + else: + p.add_argument("--restic-bin", type=Path, required=True) + p.add_argument("--repository", type=Path, required=True) + p.add_argument("--password-file", type=Path, required=True) + p.add_argument("--work", type=Path, required=True) + p.add_argument("--rotation-state", type=Path, required=True) + p.add_argument("--check-parts", type=int, default=CHECK_PARTS) + p.add_argument("--full-check", action="store_true") + p.add_argument("--control", choices=("snapshot", "restore")) + p = commands.add_parser("verify") + p.add_argument("--restored", type=Path, required=True) + p.add_argument("--inventory", type=Path, required=True) + p = commands.add_parser("_copy", help="internal bounded native Online Backup worker") + p.add_argument("--source", type=Path, required=True) + p.add_argument("--target", type=Path, required=True) + p.add_argument("--max-journal-bytes", type=int, required=True) + p.add_argument("--allow-non-wal", action="store_true") + args = parser.parse_args(argv) + try: + if args.command == "_copy": + result = copy_journal(args.source, args.target, args.max_journal_bytes, + allow_non_wal=args.allow_non_wal) + elif args.command == "verify": + result = verify(args.restored, args.inventory) + else: + journals = selections(args.journal, args.required_table) + if args.command == "snapshot": + result = snapshot_journals(journals, args.stage, args.inventory, + backup_timeout=args.backup_timeout, max_journal_bytes=args.max_journal_bytes, + allow_non_wal=args.allow_non_wal) + else: + args.work.mkdir(mode=0o700) + runner = Restic(args.restic_bin, args.repository, args.password_file, args.work) + result = cycle(runner, journals, args.rotation_state, parts=args.check_parts, + full_check=args.full_check, control=args.control, + backup_timeout=args.backup_timeout, max_journal_bytes=args.max_journal_bytes, + allow_non_wal=args.allow_non_wal) + print(json.dumps({"result": "passed", **result}, sort_keys=True)) + return 0 + except (RecoveryError, OSError, sqlite3.Error, KeyError, TypeError, + subprocess.TimeoutExpired, json.JSONDecodeError) as error: + # No source path, repository path or password is emitted in the public result. + detail = str(error) if isinstance(error, RecoveryError) else type(error).__name__ + print(json.dumps({"result": "refused", "reason": detail}, sort_keys=True)) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/blueprints/us-equities/hosting/session_day.py b/blueprints/us-equities/hosting/session_day.py new file mode 100644 index 000000000..c5150c954 --- /dev/null +++ b/blueprints/us-equities/hosting/session_day.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Print the locked XNYS eligibility token for the Dagu step precondition. + +Source: gerrymanoim/exchange_calendars@4.13.2: +exchange_calendars/exchange_calendar.py:1012-1016,1263-1279; +exchange_calendars/exchange_calendar_xnys.py:157-165. +ecal.py:118-149 renders calendars rather than a close/freshness eligibility token. +Lock evidence: adoption/sdk/requirements-linux-x86_64-py313.lock:241. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime +from importlib.metadata import version +from pathlib import Path +from zoneinfo import ZoneInfo + + +CALENDAR_VERSION = "4.13.2" +MARKET_ZONE = ZoneInfo("America/New_York") + + +def eligibility(calendar, now: datetime, events: Path | None = None) -> str: + """Use today's New York date and actual session close, including early closes.""" + if now.utcoffset() is None: + raise ValueError("an explicit time zone is required") + day = now.astimezone(MARKET_ZONE).date().isoformat() + if not calendar.is_session(day): + return "non_session" + close = calendar.session_close(day) + if now < close: + return "before_close" + if events is not None: + # Producer contract: finish this session's local simulation and atomically + # publish LEAN_EVENTS after its close, by 16:25 New York. No data acquisition. + if not events.is_file(): + return "missing_input" + modified = events.stat().st_mtime + if modified < close.timestamp(): + return "stale_input" + if modified > now.timestamp(): + return "future_input" + # CPython@v3.12.3:Doc/library/datetime.rst:1249-1258 retains the New York tzinfo. + cutoff = now.astimezone(MARKET_ZONE).replace(hour=16, minute=25, second=0, microsecond=0) + if modified > cutoff.timestamp(): + return "late_input" + return "session" + + +def calendar_for(now: datetime): + """Bracket holidays before the first/after the last session of the year. + + exchange_calendars@4.13.2:exchange_calendar.py:1257-1279 uses session + boundaries for coverage. Genuine uncovered dates still raise DateOutOfBounds. + """ + import exchange_calendars + + if now.utcoffset() is None: + raise ValueError("an explicit time zone is required") + year = now.astimezone(MARKET_ZONE).year + return exchange_calendars.get_calendar( + "XNYS", start=f"{year - 1}-12-01", end=f"{year + 1}-01-31" + ) + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--at", help="optional RFC 3339 instant for an operator drill") + parser.add_argument("--events", type=Path, + help="require local input published after this session's close and by 16:25 New York") + args = parser.parse_args(argv) + # This normal DAG step fails on missing packages, wrong versions or calendar errors. + if version("exchange-calendars") != CALENDAR_VERSION: + raise RuntimeError("use the trading runtime's locked exchange-calendars 4.13.2") + now = datetime.fromisoformat(args.at) if args.at else datetime.now(MARKET_ZONE) + if now.utcoffset() is None: + raise ValueError("--at requires an explicit offset") + token = eligibility(calendar_for(now), now, args.events) + print(token) + # Unavailable session input must be visible to Dagu's failed-run history/handlers. + return 1 if token in {"missing_input", "stale_input", "future_input", "late_input"} else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/decisions/2026-10-05-trading-unattended-hosting-recovery.md b/docs/decisions/2026-10-05-trading-unattended-hosting-recovery.md new file mode 100644 index 000000000..9b328b5fa --- /dev/null +++ b/docs/decisions/2026-10-05-trading-unattended-hosting-recovery.md @@ -0,0 +1,417 @@ +# Unit T PR-4: unattended research hosting and journal recovery + +Date: 2026-10-05. Lane: trading. Builder base: +`4c897418fe35a030a1188ae447eaf31c893f8eff`. +Repair round 2 base: `b3a01ce273952260df0b552d5075997b737b71ad`, whose tree +matches the coordinator's round 1 commit. No commit or pin move by the builder. +Repair round 3 starts from `f417d2257869b8fdf90ad91b3246196a8b6a81f2`. +North-star action: run US-equities local research/evidence after each NYSE +session and preserve the research journals needed to reproduce its state. +This implements the coordinator's final `decision-record-T-final.md`, lines +102-103 (gates), 233-237 (rejections), 267-271 (overturn), 297 (PR-4), 351 +(residual) and 400-406 (Dagu/restic sources). The coordinator retains that record. + +## Decision and alternatives + +Retain Dagu **2.16.6** and restic **0.19.1**, without changing +`manifests/stack.json`. Run one `dagu start-all` process, disable the optional +coordinator in the config file, and deliver all four job variables in the +private DAG's `env:` block. The named schedule contract is +`nyse-post-close-evidence`, at 16:30 America/New_York on NYSE session days. +Early-close days keep the same execution time. The job processes existing local +simulation input and evidence; it does not connect to a broker. + +The final unit T record rejected these alternatives for this owned job: + +| Alternative | Reason; comparison that could reopen it | +| --- | --- | +| Prefect, Dagster, Temporal | Another orchestrator overlaps Dagu without a demonstrated research-job requirement (T rule D). A measured requirement Dagu cannot meet would justify a new comparison. | +| A second `dagu scheduler` unit | Two supervised units for one job; start-all already combines the needed services. Reopen if operating-host lifecycle acceptance misses due runs. | +| GitHub Actions market schedules | Dagu owns scheduling; that alternative's reliability was an unverified lead, not a tested advantage. A dated native reliability comparison could reopen it. | +| Coordinator setting in systemd `Environment=` | `/usr/bin/env -i` discards it. Keep the default override visible in the config file. | +| Coordinator setting as an `env -i` operand | Supported, but the config key keeps this choice visible in the checked-in example. | +| Copying active WAL journals; same-host-only repository | Copies can be inconsistent; the source host cannot provide off-host durability. Use Online Backup and keep the independent recovery gate open. | + +This bounded builder read search-first and modern-python guidance; the supplied +pins, stdlib/unittest contract and no-services/no-install envelope take precedence +over broader registry searches, research agents or new tool installation. No +general landscape sweep or host memory/context service query was performed. + +## Pinned sources and rules + +The [hosting recipe](../../blueprints/us-equities/hosting/README.md) gives the exact +operator commands and source-linked rule table. The verified original sources are: + +- `dagucloud/dagu@v2.16.6:internal/cmd/startall.go:32-45` (combined UI/scheduler, + optional coordinator), `internal/cmn/config/loader.go:1311-1319` (enabled + default). Installed `dagu start-all --help` and `dagu version` both returned + exit 0, with version 2.16.6. + [Start-all source](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/cmd/startall.go#L32). +- `coreutils/coreutils@v9.4:src/env.c:824-838` empties the inherited environment; + `dagucloud/dagu@v2.16.6:internal/spec/dag.go:1663-1673` builds explicit DAG + environment. This justifies placing `STACK_REPO`, `SDK_ENV`, `LEAN_EVENTS` + and `RESEARCH_OUTPUT` in the DAG rather than in discarded `Environment=` lines. + [env source](https://github.com/coreutils/coreutils/blob/v9.4/src/env.c#L824). +- `dagucloud/dagu@v2.16.6:internal/ir/schedule.go:281-305`, + `internal/cmn/schema/dag.schema.json:855-876` and `go.mod:65`; its locked + `robfig/cron@v3.0.1:parser.go:93-103` accepts `CRON_TZ`. The schedule name is + a documented contract and tag, since the pinned cron object has no name key. + [Timezone parser](https://github.com/robfig/cron/blob/v3.0.1/parser.go#L93). +- `gerrymanoim/exchange_calendars@4.13.2:exchange_calendars/exchange_calendar.py:1012-1016,1263-1279` + queries the session and actual close; + `exchange_calendars/exchange_calendar_xnys.py:157-165` defines 13:00 early + and 16:00 ordinary closes. At 16:30 both are complete. + [Calendar API](https://github.com/gerrymanoim/exchange_calendars/blob/4.13.2/exchange_calendars/exchange_calendar.py#L1263). +- `dagucloud/dagu@v2.16.6:internal/runtime/runner.go:1633-1648,1267-1281` + skips a step on a false precondition and propagates the skip to dependents. + A preceding normal calendar command exposes package/version/calendar failures; + a covered non-session token skips the research chain without failing it. + Bracket the queried year with prior December 1/following January 31, since + `exchange_calendar.py:1257-1279` measures coverage by its first/last sessions. + A January 1-to-December 31 window falsely raised DateOutOfBounds on New Year + holidays; real 4.13.2 regressions reproduced that before the correction. + [Precondition implementation](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/runtime/runner.go#L1633). +- `dagucloud/dagu@v2.16.6:internal/cmn/runenv/keys.go:12-13`, + `internal/runctx/context_env.go:28` supplies `DAG_RUN_ID`, used in output + filenames to preserve earlier evidence when the daily job repeats. + [Run identifier](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/cmn/runenv/keys.go#L12). +- `dagucloud/dagu@v2.16.6:internal/service/frontend/api/v1/dagruns.go:163-167` + guards the API with `permissions.run_dags`; its + `internal/service/scheduler/dag_executor.go:280-321` dispatches locally without + that API check. This source inference awaits operating-host observation of + the configured scheduler with `run_dags: false`. + [Local dispatch](https://github.com/dagucloud/dagu/blob/v2.16.6/internal/service/scheduler/dag_executor.go#L280). +- `python/cpython@v3.12.3:Doc/library/sqlite3.rst:1107-1155` documents concurrent + Online Backup and the exact `src.backup(dst, pages=..., progress=...)` example; + `Modules/_sqlite/connection.c:2067-2077` calls SQLite's Online Backup API. + [Backup example](https://github.com/python/cpython/blob/v3.12.3/Doc/library/sqlite3.rst#L1107). +- `restic/restic@v0.19.1:doc/040_backup.rst:787-806` defines complete success + (exit 0), fatal error (exit 1) and incomplete snapshots (exit 3). All nonzero + exits refuse acceptance, including 3. + [Exit codes](https://github.com/restic/restic/blob/v0.19.1/doc/040_backup.rst#L787). +- `restic/restic@v0.19.1:doc/045_working_with_repos.rst:482-521` defines + deterministic `n/t` partition checks. Our policy rotates all seven subsets + with an eight-day bound **per whole rotation**, persisting only successful checks; + an overdue rotation needs a successful full `--read-data` check before reset. + [Partitions](https://github.com/restic/restic/blob/v0.19.1/doc/045_working_with_repos.rst#L482). +- `restic/restic@v0.19.1:doc/050_restore.rst:55-73` restores an exact snapshot's + subtree. Our consumer verifies exact inventory, hashes and bytes against an + external oracle, then integrity and required-table counts as secondary checks. + [Restore syntax](https://github.com/restic/restic/blob/v0.19.1/doc/050_restore.rst#L55). +- `systemd/systemd@v255:man/systemd.service.xml:818-836` defines on-failure + restart. `man/systemctl.xml:541-549,2377-2388` supplies unit-addressed + `kill --kill-whom=main --signal=SIGKILL`. The later operator drill verifies + identity before native fault delivery and uses + Dagu's original status fields: + `dagucloud/dagu@v2.16.6:internal/cmd/history.go:568-602`, + `internal/persis/file/dagrun/dagrun.go:55-59`, + `internal/ir/run_status.go:156-179`, `internal/ir/status.go:10-18,133-149`. + History alone omits the slot/trigger, so it must be corroborated. + The supported REST run-details alternative exposes both fields at + `internal/service/frontend/api/v1/dagruns.go:2066-2096; transformer.go:362,371`. + Original `status.jsonl` is retained to hash/corroborate native bytes without + an HTTP/auth dependency; its internal format must be revalidated on pin moves. + [Restart policy](https://github.com/systemd/systemd/blob/v255/man/systemd.service.xml#L818). + +The SDK_ENV lock evidence at this base is +`adoption/sdk/requirements-linux-x86_64-py313.lock:241` (exchange-calendars 4.13.2). +Round 1 cited an unlanded runtime bundle; that is not main's SDK_ENV authority and +has been removed from the current recipe. Round 2 executes the existing locked +SDK for real-calendar tests; it installs no environment and adds no lock. + +Snapshot/restore glue and both drills belong in `hosting/`, which owns this +research job's recovery without adapter coupling. Each journal gets its own +Online Backup, an independently frozen file inventory and explicit required +tables. This does not provide atomic cross-journal transactions. No new timer +was enabled; the operator must run daily and keep every rotation within eight elapsed days. + +## Round 2 native reuse and repairs + +The 2026-10-05 standing rule applies on every new Linux/WSL operating host: use +clean maintained upstream installations, replacing only the consumer's integration +policy. The [native glue review](../../blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json) +records pinned installed help, release notes and source. Checked restic 0.19.1's +`--stdin-from-command`, SQLite 3.53.1's `.backup`/`.dump` and maintained +`sqlite3_rsync`, CPython's native backup/subprocess/flock, Dagu 2.16.6's +preconditions/retries/handlers and exchange_calendars 4.13.2's **ecal CLI**. +The final T off-host-recovery row **102** keeps restic with stdlib Online Backup; +the earlier hosting comparison deferred Litestream. This repair does not reopen +the selection. The final row itself does not name Litestream, so the review +records the earlier comparison separately rather than inventing a final-row quote. + +`restic/restic@v0.19.1:doc/040_backup.rst:679-703` captures a command's stdout. +SQLite `.backup` writes a file (`sqlite/sqlite@version-3.53.1:src/shell.c.in:9074-9099`), +so invoking it as a restic stdin producer would capture an empty stream. `.dump` +emits logical SQL (`src/shell.c.in:3759`), not the frozen binary file inventory. +SQLite's own `sqlite3_rsync` copies live databases +(`tool/sqlite3_rsync.c:13-40`), without replacing restic's encrypted repository +and the external consumer oracle. The existing CPython Online Backup API is the +native copying function; we do not rebuild it. + +### Why this glue exists + +| Kept glue | One-line pin-cited reason | +| --- | --- | +| `session_day.py` | exchange_calendars `@4.13.2:pyproject.toml:67-68; exchange_calendars/ecal.py:100-149` ships calendar rendering, leaving our post-close and input-freshness token to compose its native session/close API. | +| `journal_recovery.py` | restic `@v0.19.1:doc/040_backup.rst:679-703; doc/045_working_with_repos.rst:482-521; doc/050_restore.rst:55-73` supplies backup/check/restore, leaving the frozen external oracle, required tables, bounded rotation and controls as consumer policy. | +| `drill_process_restart.py` | systemd `@v255:man/systemctl.xml:541-549,2377-2388; man/systemd.service.xml:818-836` supplies unit-addressed kill and supervision; Dagu `@v2.16.6:internal/service/frontend/api/v1/transformer.go:362,371` supplies slot/trigger, leaving identity/time/holiday guards and exact-slot correlation as acceptance policy. | +| `drill_local_recovery.py` | CPython `@v3.12.3:Modules/_sqlite/connection.c:2067-2102` supplies copying and restic `@v0.19.1:doc/050_restore.rst:55-73` supplies restore, leaving synthetic concurrent writers, failure controls and sanitized evidence as our integration fixture. | + +**Replaced** the exclusive-create lock sentinel with native `fcntl.flock` +(`python/cpython@v3.12.3:Doc/library/fcntl.rst:139-149`); process exit releases +the lock even when its file remains. Dagu supplies dependency skipping, +preconditions (`internal/runtime/runner.go:1267-1281,1633-1648`), retries +(`:1502-1525`) and lifecycle handlers (`:1196-1206`) natively. No Python retry, +handler or scheduler implementation is added, and this job enables no retry +or alert handler policy. An independent alert path remains decision 2. + +The recovery worker calls native `Connection.backup(..., pages=-1)` in one read +transaction, permitting concurrent WAL writers while avoiding incremental-copy +restarts (`python/cpython@v3.12.3:Modules/_sqlite/connection.c:2013,2067-2102`). +It checks source `PRAGMA journal_mode` and refuses non-WAL journals unless +`--allow-non-wal` explicitly accepts rollback-journal writer blocking. The read +transaction holds a SHARED lock for the whole bounded copy; those writers can +receive `SQLITE_BUSY` at commit. Prefer quiescing writers before opting in +(`sqlite/sqlite@version-3.53.1:src/backup.c:346-354,383-389`). +That API has no hard wall-time parameter; `subprocess.run(timeout=...)` bounds +the isolated worker (`Doc/library/subprocess.rst:62-68`). The documented defaults are **120 seconds and 256 MiB per +journal**, with explicit `--backup-timeout`/`--max-journal-bytes` parameters; +failed/oversize/timed-out snapshots publish no inventory. Each per-journal +snapshot is consistent; simultaneous commits need not all appear in that snapshot. + +Rotation now timestamps the invocation before snapshot/backup, bounds completion +of the **whole seven-subset cycle** in UTC, and accepts 24 h + 60 s jitter and +the 2026-11-01 fall-back. It refuses incomplete overdue cycles, clock reversal, +over-eight-day idle gaps and old state until successful full checking. Backup +and any later forget policy group by **host,tags** +(`restic/restic@v0.19.1:doc/040_backup.rst:197-206; doc/060_forget.rst:225-233`), +so disposable staging paths share parent/retention groups. No forget/prune ran. +Control backups use a separate `equity-research-recovery-control` host and +`journal-recovery-control` tag, isolating incomplete snapshots from production +parents/retention. Controls are documented for disposable repositories only. + +The seven research-runtime passthrough entries are restored as well as the four +scheduled-job entries: Dagu's allowlist is at +`dagucloud/dagu@v2.16.6:internal/cmn/config/loader.go:344-347; internal/cmn/config/env.go:77-95`. +The research operator's simulation job must atomically publish `LEAN_EVENTS` +after the session's actual close **by 16:25 New York**. Its mtime must be between +close and check time. Missing/stale/future tokens exit 1 and fail the native run, +blocking its dependents and remaining visible to failed-history/handler queries. +Only covered non-session/before-close tokens retain a skipped research chain. +This availability check does not establish market-data +freshness or point-in-time correctness inside the file. + +The hosting README again carries the verified pinned-download/checksum recipe, +0600 private config/DAG, loopback exposure warning, auth transition note, +historical 2.17.2 artifact boundary and operator-only systemctl lifecycle steps. +It adds native `loginctl enable-linger` (`systemd/systemd@v255:man/loginctl.xml:186-195`) +and the caveat that linger cannot extend Windows/WSL VM lifetime. Portable +templates use plain `env TMPDIR=... nice -n 19 python3 ...`; the builder's actual +commands used RTK's fallback, recorded separately from the portable recipe. +Recovery uses Python **3.11+**, because `hashlib.file_digest` was added then +(`python/cpython@v3.12.3:Doc/library/hashlib.rst:267-302`); execution used 3.13.15. + +The drill resolves its scratch root at creation, scrubs lexical and canonical +forms and refuses any remaining absolute path before publishing `--output`. +The restore control now alters SQLite header content while integrity, required +counts and file bytes remain equal; external SHA-256 comparison still refuses it. +Tests cover symlinked roots, both controls, native lock contention/stale files, +slow/overdue rotation, real holiday/early-close calendars and native Dagu +skip/success chains, all on scratch directories without a scheduler. + +## Round 3 corrections and verification + +The cross-family read found seven new p2 defects and a missing manual run/cancel +recipe. Each behavioral regression was run against the old code first (exit 1), +then against its scoped repair (exit 0), including a native Dagu failure probe. +The hosting recipe restores native `start --run-id`, `stop --run-id` and +`history --status succeeded,failed,aborted` commands for operator inspection and +replay (`dagucloud/dagu@v2.16.6:internal/cmd/start.go:45-67; stop.go:20-38,53-91; +history.go:160-345; flags.go:463-481`; all three installed help commands exit 0). + +Native systemctl fault delivery replaces `os.kill`; identity guards remain. +The restart window reserves more than 150 seconds, rechecks immediately before +the fault and refuses a restart observed at/after due with a distinct reason. +Dagu does not replay missed slots when `catchup_window` is omitted +(`dagucloud/dagu@v2.16.6:internal/cmn/schema/dag.schema.json:151-153`). +No operating-host process was signalled during this repair. + +The historical round-2 receipt now names the published main base and explicitly +identifies its tested working tree by `integration_source_sha256`; its old source +hashes and results remain historical. The native test helper uses its private +scratch root even with caller `TMPDIR` unset. Round-3 verification uses the locked +SDK, so real-calendar tests execute rather than skip. Fresh native evidence is +recorded in the [native DAG artifact](../../blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json), +[recovery artifact](../../blueprints/us-equities/hosting/evidence/offline-proof-r3.json) and +[local-integration receipt](../../evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json); a private copy of +the deployed DAG uses synthetic post-close time/input, not a live market run. + +Round 4 restores the round-2 review's exact `f417d2257` bytes and its original +receipt hash/15859-byte pin. Round-3 additions have a separate review linked by +the round-3 receipt; its parent link retains the earlier native observations. +After a later-step failure, the operator replay window is the same New York date +after that session's close, using unchanged input with mtime after close and no +later than the check time or 16:25 New York. Missing, stale or late input failures +cannot be replayed for that session: correction after the run misses the cutoff. +Later runs skip or process another session. +Native `dagu status --run-id` must show every research node succeeded before +replay counts as recovery (`dagucloud/dagu@v2.16.6:internal/cmd/status.go:18-47`). +The restart drill refuses unit globs before fault delivery, because literals +refer to exactly one unit (`systemd/systemd@v255:man/systemctl.xml:1826-1833`). + +## Dagu 2.16.6 to 2.18.2 scheduler comparison + +Read both tagged source trees, the available intervening release notes at tags, +and the full `internal/service/scheduler/` diff. The +[comparison artifact](../../blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json) +records original source hashes, changes and exact locators. +[2.17.0](https://github.com/dagucloud/dagu/releases/tag/v2.17.0) describes +global queue pacing and queued retries (#2775); +[2.18.0](https://github.com/dagucloud/dagu/releases/tag/v2.18.0) describes +scheduler dotenv loading (#2937); +[2.18.2](https://github.com/dagucloud/dagu/releases/tag/v2.18.2), published +2026-10-04, describes busy-slot logging (#2980), optional signal propagation +and shutdown changes. Release metadata for v2.17.1 returned HTTP 404; no claim +about that missing release's contents is made. + +The production scheduler diff changes five files: environment preparation, +queued-condition state and failures, retry liveness, suspension migration and +shutdown, global queue admission and busy-slot logging. Relevant locators at +`dagucloud/dagu@v2.18.2` are +`internal/service/scheduler/dag_executor.go:468-475`, +`tick_planner.go:758-784,1560-1568,1658-1661`, +`scheduler.go:607-614,842-855`, `retry_scanner.go:165-181` and +`queue_dispatcher.go:929-950`. The cron model/parser, precondition mechanism, +catch-up and durable schedule state were separately compared as well. + +**No compared fix is required by this job's defined schedule.** It uses a +five-field cron, explicit DAG `env:`, local execution, no dotenv, no configured +global queue, no distributed workers, no retry policy, no SMTP and no catch-up +acceptance claim. Busy-slot additions improve observation without changing skip +guards. The dotenv fix would matter after switching to dotenv; optional graceful +signal propagation does not underwrite a SIGKILL process drill. This is source +comparison, not execution comparison or a claim that 2.16.6 is the newest pin. +Hold 2.16.6 provisionally and route any justified move through the foundation owner. + +The comparison also checked descriptor/interval additions in +`dagucloud/dagu@v2.18.2:internal/ir/schedule.go:17-19,281-345` and interrupted +precondition handling in `internal/runtime/condition.go:19-82`. Neither is needed +by the explicit five-field cron and single scalar eligibility comparison used +here; interrupted-precondition behavior remains outside the claimed acceptance. + +## Evidence, corrections and completeness critic + +The **round 2** [local integration receipt](../../evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json) +and [offline proof](../../blueprints/us-equities/hosting/evidence/offline-proof-r2.json) +retain a fresh native backup/check/restore of **16,531,456 journal bytes** and +independent committed-write observations within each single `pages=-1` native +call. Snapshot refusal is native exit **3**, procedure exit **1**; restore +mutation is native restore exit **0**, procedure exit **1**, with equal integrity, +required counts and bytes. The throwaway password was deleted. **42 tests** +passed, including six real-calendar cases, genuine DateOutOfBounds, freshness, +both controls, native lock contention/stale state, whole-cycle jitter/fall-back, +restart guards, symlinked scratch roots and Python 3.11-compatible syntax. + +The [native Dagu record](../../blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json) +retains non-session and stale-input skips plus an executed success chain, all +with native CLI exit **0**. Skips have node status **5**, successful steps **4**. +These are manual short-lived runs, not scheduler or host restart acceptance. +The verdict JSON has `nv_summary` without the referenced `nv[2]` recipe; the +probe therefore uses the pinned upstream scalar-precondition/dependency pattern. +Native `dagu validate` on the revised DAG also returned **0**. + +The pinned [cron Next probe record](../../blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json) +holds the tiny Go program and local v3.0.1 module prepared from tagged source. +Execution with module networking disabled returned **127**: `go` was not found +in PATH or the checked standard/ecosystem locations. No compiler was installed +under the source-only network constraint. The supplied expected results, +2026-11-02T21:30Z and 2027-03-15T20:30Z, are **not measured Next results**; this +uses the user's explicit option to record why the probe cannot run. + +Round 2 completeness critic: the native skip probes showed that Dagu can report +overall success while the research chain is skipped. The process-restart drill +now requires **every required research step** succeeded before counting the due +slot. Input mtime is only a producer availability contract, not proof of the +market dates/content. A strict per-step interval hid daily jitter and DST; a +whole-cycle UTC deadline plus completion check covers those cases. Sentinel +lock files could survive crashes; native process-owned locks address that gap. +During artifact publication the sanitizer first refused a relative `./` module +path; its absolute-path test now admits relative paths and has a regression. +The remaining modalities are actual operating-host lifecycle/independent alerts, +an executable cron Next probe, real consumer journal selection and state, +independent destination/key recovery, second-host restore and measured RTO. +They remain limits for the next hosting/recovery sweep. + +The historical **round 1** [local integration receipt](../../evidence/receipts/trading-unattended-hosting-recovery-20261005.json) +and [offline proof](../../blueprints/us-equities/hosting/evidence/offline-proof.json) +retain actual native restic output on scratch directories. Two synthetic WAL +journals were each written while their Online Backup was active, then restored +with exact bytes and required state. Subset 1/1 reads every pack in this small +proof. The unreadable-snapshot control observed native exit 3 and procedure exit +1; the missing-restored-file control observed native restore exit 0 and procedure +exit 1. The throwaway password was generated privately and deleted. Stdlib tests +exercise the default seven-part rotation, failed checks, stale-state refusal +and altered content preserving integrity and row counts. These are local +integration and synthetic tests, not an unchanged upstream suite. + +Corrections retained this turn: an initial lookup under `internal/scheduler/` +matched no files and was not an empty-diff result. `rg` located +`internal/service/scheduler/`, whose full tagged diff was then read. The calendar +tag is `4.13.2`; the `v4.13.2` URL returned HTTP 404 and was corrected against the +existing native calendar receipt and actual source. The pyproject location was +corrected as above. These source-location mistakes changed no pin or acceptance. + +Completeness critic: process liveness alone cannot attest a due slot, so the later +drill checks original scheduler trigger and slot. Same-count mutation bypasses +count/integrity-only checks; external hashes and a mutation test address that +gap. Missed partition executions lose coverage; persistent cursor, eight-day whole-cycle bound +and full reset address that gap. Remaining modalities are host reboot, +downtime/catch-up and duplicate-slot behavior, independent alerts, destination +and key loss, second-host consumer restore and measured recovery time. These +feed the next hosting/recovery lifecycle sweep and remain open. + +Round 6 on 2026-10-05 corrects six review threads without changing historical +proof or receipt bytes. The guard now fails visibly with `late_input` when +publication misses the documented 16:25 New York cutoff; replay uses that same +cutoff. Oracle publication applies mode 0400 before file fsync and syncs the +parent directory (`python/cpython@v3.12.3:Doc/library/os.rst:996-1005,1077-1087`). +Rotation checks the previous cycle origin even when its cursor has returned to +1; an overdue rollover requires a successful native full read before reset. +The previous seven-day bound rejected normal fall-back and daily drift at rollover. +Round 7 gives the daily seven-part cadence an eight-day UTC age bound, including +rollover: one day of slack covers the 25-hour fall-back day and 60 seconds of +daily drift. An origin or idle gap older than eight days still requires a full read. +The fall-back rollover and two-cycle jitter regressions failed against round 6 +before the bound changed, then passed; overdue cycles still refuse. +The restart drill parses its active file permission with upstream +PyYAML 6.0.3 (`yaml/pyyaml@6.0.3:lib/yaml/__init__.py:117-125`), including Dagu's +default and legacy override (`dagucloud/dagu@v2.16.6:internal/cmn/config/loader.go:586-611`). +Round 7 runs only this drill with the supported isolated `uv run --no-project +--python "$SDK_ENV/bin/python" --with PyYAML==6.0.3` overlay; the hash-locked +SDK_ENV stays unchanged and supplies the locked calendar. Source: +`astral-sh/uv@0.12.17:crates/uv/src/commands/project/run.rs:903-920,1040-1105`; +repository precedent: `tools/runtime-worker-evidence/checks.json:40-43`. +No YAML parser is rebuilt here. It parses and hashes the same captured native +status bytes, tolerating a replaced directory entry while polling. The round-3 +receipt is now discoverable in the receipt catalog with its original claim and +limitations. Focused regressions cover each correction; earlier receipts remain +dated evidence of the earlier behavior and do not attest these repairs. + +## Overturn and limits + +| T overturn row | Reopen when | +| --- | --- | +| Scheduler | A restart or reboot drill on the operating host misses a due run, or the 2.18.2 comparison finds a fix the drill depends on; bump through the foundation pin path first. | +| Coordinator setting | A Dagu pin move drops or renames `coordinator.enabled`; re-check `dagu start-all --help` on every move. | +| Recovery | A restore rehearsal fails the inventory/hash comparison or recovery-time objective, or the journal changes to a store with a different supported snapshot method. | +| Calendar | A dated comparison with official NYSE dates finds a material error in the grid, or exchange_calendars no longer covers the next session year. | + +The operating host and independent alert path remain **user decision 2**. +Neither operator drill was run against an operating service. A process-restart +drill is not reboot, missed-run or independent-alert acceptance. The same-host +drill cannot close the off-host-recovery gate: closure needs **user decision 3**, +an independent destination, key recovery and a restore by a consumer on a second +host under a selected and measured recovery-time objective. Scratch execution +and local template validation close neither gate. No broker credentials, live +mode, paid hosting, retention deletion, model calls, commit or push belong to +this unit. diff --git a/evidence/receipts/trading-unattended-hosting-recovery-20261005.json b/evidence/receipts/trading-unattended-hosting-recovery-20261005.json new file mode 100644 index 000000000..df3551699 --- /dev/null +++ b/evidence/receipts/trading-unattended-hosting-recovery-20261005.json @@ -0,0 +1,200 @@ +{ + "schema_version": 1, + "id": "trading-unattended-hosting-recovery-20261005", + "kind": "native_cli_e2e", + "component_ids": [ + "dagu", + "restic" + ], + "evidence_class": "local_integration", + "source_git_commit": "4c897418fe35a030a1188ae447eaf31c893f8eff", + "observed_at_utc": "2026-10-05T03:51:24.642607+00:00", + "claim": "Native restic 0.19.1 backed up two synthetic SQLite WAL journals copied by the stdlib Online Backup API during committed writes, read subset 1/1 (2/2 packs), restored 802816 identical journal bytes with matching external inventory and required state, and refused both snapshot and restore controls. Dagu 2.16.6 validated the scheduled research DAG offline; operating-host and off-host gates remain open.", + "limitations": [ + "Scratch local integration with synthetic research journals; not an unchanged upstream test suite or a real market/broker journal.", + "Per-journal Online Backup is independently consistent; no atomic cross-journal transaction or research consumer application restart was accepted.", + "Historical one-part check 1/1 read 2/2 packs. Round 1 enforced strict daily gaps in its synthetic rotation test; round 2 replaces that policy with a whole seven-subset rotation within seven days.", + "Operating host and independent alert path are user decision 2. No systemd unit was installed, enabled or started; no process-restart drill or scheduler was run.", + "A process-restart drill is not reboot, missed-run or independent-alert acceptance. permissions.run_dags scheduler behavior is source-reviewed and awaits the later host drill.", + "Off-host recovery stays open until user decision 3 chooses an independent destination and key recovery and a consumer restores on a second host against the independent inventory and required state.", + "Recovery-time objective, independent alerting and retention deletion remain unaccepted. No broker, credentials, live mode, model calls, commit or push was used.", + "Historical round 1 cited an unlanded calendar bundle. Round 2 uses the on-base SDK_ENV lock adoption/sdk/requirements-linux-x86_64-py313.lock:241 and executes the already installed exchange-calendars 4.13.2 for real-calendar regressions; no new installation is claimed." + ], + "tool_versions": { + "dagu": "2.16.6", + "restic": "0.19.1", + "python_executed": "3.13.15", + "cpython_reference": "v3.12.3", + "exchange_calendars_selected": "4.13.2" + }, + "upstream_sources": [ + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmd/startall.go:32-45; internal/cmn/config/loader.go:1311-1319; internal/spec/dag.go:1663-1673; internal/runtime/runner.go:1633-1648,1267-1281; internal/ir/schedule.go:281-305" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/040_backup.rst:787-806; doc/045_working_with_repos.rst:482-521; doc/050_restore.rst:55-73" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/sqlite3.rst:1107-1155; Modules/_sqlite/connection.c:2067-2077" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "exchange_calendars/exchange_calendar.py:1012-1016,1263-1279; exchange_calendars/exchange_calendar_xnys.py:157-165" + } + ], + "artifacts": [ + { + "path": "blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json", + "sha256": "c15ac08f09f324a6c1106f167c21cb013bc6c58fefbb04a3092ec68cc8cf43c7", + "bytes": 12307 + }, + { + "path": "blueprints/us-equities/hosting/evidence/dagu-validation.json", + "sha256": "ca499670b89b1dfc19745e1e8e5338196fc03e31a53f53c9dd7fa9bace7feccc", + "bytes": 444 + }, + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof.json", + "sha256": "1d2aea2c11578aab27fb20d6d549241cd9b0c1b7b709c9f5081025ff612a7ce4", + "bytes": 10383 + } + ], + "integration_source_sha256": { + "blueprints/us-equities/hosting/config.yaml.example": "46c546745cb2ac2a5fd72642241cb0e82d300ae628c740d7197d73ca80061848", + "blueprints/us-equities/hosting/dagu-equities.service.example": "327f5500f6d02d96508c35a1e512d7adcb2018427067a286b51603bebdd6bd41", + "blueprints/us-equities/hosting/equity-research-evidence.yaml": "cf9fcce2d08b6f0973fa7cf5d1070c5ac7d78befa39ae6a2ff76b03214a6e15d", + "blueprints/us-equities/hosting/session_day.py": "2e3e106c9273ab2e0b221d1510481e7d33e9fc33c25a1cabc10c0bf389113945", + "blueprints/us-equities/hosting/journal_recovery.py": "aac19598a40fd0c0df85b25174512b25121493bacce192010fed7e019fbfa0a4", + "blueprints/us-equities/hosting/drill_local_recovery.py": "fc8549c0178fc23fb04bdd9e247e3313e8613cb764b99cfa1f1563a8af02350c", + "blueprints/us-equities/hosting/drill_process_restart.py": "931ef44cc35da41ed8dd3b8c0a2919f9f2d2f98357fb9e5e33fa93dcc379521b", + "tests/test_us_equities_hosting.py": "935d03e8a00320904f414a2db9f958456415f68e4c11009fff78b53fe02ee566" + }, + "commands": [ + { + "cmd": "rtk env TMPDIR= nice -n 19 python3 blueprints/us-equities/hosting/drill_local_recovery.py --restic-bin --scratch-root --output ", + "exit": 0, + "returned_output": { + "representation": "parsed_json_lines", + "lines": [ + { + "checked_subset": "1/1", + "matched_journals": 2, + "restore_control_exit": 1, + "result": "passed", + "snapshot_control_exit": 1, + "throwaway_password_deleted": true + } + ] + }, + "started_at_utc": "2026-10-05T03:51:15.065593+00:00", + "completed_at_utc": "2026-10-05T03:51:24.642607+00:00", + "invocation_proxy": "Historical builder used rtk env through its fallback. This is an execution record, not a portable prerequisite; the current README recipes use plain env TMPDIR=... nice -n 19 python3 ... ." + }, + { + "cmd": "python3 -m unittest tests.test_us_equities_hosting", + "exit": 0, + "returned_output": "......................\n----------------------------------------------------------------------\nRan 22 tests in 1.842s\n\nOK\n", + "class": "stdlib local contract and synthetic fault tests" + }, + { + "cmd": "dagu validate --context local --dagu-home --config /config.yaml /blueprints/us-equities/hosting/equity-research-evidence.yaml", + "exit": 0, + "returned_output": "time=2026-10-04T23:56:06.918-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-04T23:56:07.022-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\n", + "class": "native offline structural validation; no DAG execution" + }, + { + "cmd": "dagu start-all --help", + "exit": 0, + "observed": "Installed help describes UI/scheduler/coordinator and the config-file coordinator override. No service started." + } + ], + "controls": { + "restore": { + "procedure_exit": 1, + "native_exits": [ + { + "command": "version", + "exit": 0 + }, + { + "command": "backup", + "exit": 0 + }, + { + "command": "check", + "exit": 0 + }, + { + "command": "restore", + "exit": 0 + } + ], + "returned_output": { + "reason": "file inventory mismatch (missing or extra files)", + "result": "refused" + } + }, + "snapshot": { + "procedure_exit": 1, + "native_exits": [ + { + "command": "version", + "exit": 0 + }, + { + "command": "backup", + "exit": 3 + } + ], + "returned_output": { + "reason": "restic backup refused (exit 3)", + "result": "refused" + } + } + }, + "independent_observation": [ + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 97 + }, + "sha256": "d3da8b7140827e3247d8a3bc647c9e9828d6b587fcf4bfcea7a170837bcc45f3" + }, + { + "bytes": 401408, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 97 + }, + "sha256": "7712822fae54114f0124caf9fb0d8604fcc9dddfda05e511aad364174df61159" + } + ], + "sanitization": "Native command args and output replace the private executable and scratch root with explicit placeholders. Password contents were never output; no host name, login name or host path is published. Native timestamps, restic snapshot IDs and fixture hashes remain. Raw returned output stays in private scratch custody. JSON command output is stored as parsed_json_lines records with all original JSON fields preserved; non-JSON lines remain text and raw formatting stays in private custody.", + "prior_attempts": [ + { + "command": "python3 -m unittest tests.test_us_equities_hosting", + "exit": 0, + "tests": 22, + "note": "Initial pass emitted a ResourceWarning for a test-only sqlite connection. The test now explicitly closes that connection; the retained final run passed without that warning." + } + ], + "operational_status": { + "unit_installed": false, + "scheduler_started": false, + "process_restart_drill_run": false, + "off_host_recovery_gate_closed": false, + "unattended_hosting_gate_closed": false, + "throwaway_password_deleted": true, + "pin_moved": false + }, + "review": { + "kind": "self", + "scope": "Original source comparison, independent restored bytes/state observation and completeness critic in the dated decision; coordinator review remains separate." + } +} diff --git a/evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json b/evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json new file mode 100644 index 000000000..448523271 --- /dev/null +++ b/evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json @@ -0,0 +1,537 @@ +{ + "schema_version": 1, + "id": "trading-unattended-hosting-recovery-r2-20261005", + "kind": "native_cli_e2e", + "component_ids": [ + "dagu", + "restic" + ], + "evidence_class": "local_integration", + "source_git_commit": "4c897418fe35a030a1188ae447eaf31c893f8eff", + "observed_at_utc": "2026-10-05T05:34:37.145949+00:00", + "claim": "Round 2 native restic 0.19.1 backed up two synthetic WAL journals using CPython Online Backup pages=-1 during independently observed commits, checked subset 1/1, restored 16531456 identical journal bytes against the external inventory and required state, and refused native-exit-3 snapshot and same-count/same-integrity content-mutation restore controls. Dagu 2.16.6 validated the revised DAG and executed native manual skip/success/stale-input probes. 42 stdlib tests passed including real exchange_calendars 4.13.2 cases. Cron Next was not executable: no Go compiler found. Operating-host and off-host gates remain open.", + "limitations": [ + "Scratch local integration and synthetic fixtures, not an unchanged upstream test suite, real market/broker state or operating-host acceptance.", + "Per-journal Online Backup is consistent independently; no transaction across journals. Commits during the native call need not appear in its read snapshot; comparison uses the frozen snapshot oracle.", + "Native check subset 1/1 reads every pack in this small proof. Whole seven-part/seven-day rotation, daily jitter, fall-back and deadlines are behavioural stdlib tests; operator execution is still required.", + "No scheduler or systemd unit installed, enabled or started; the operating-host restart/reboot/missed-run and independent-alert gates remain user decision 2.", + "Native Dagu probes use manual start, not scheduler dispatch. Scheduler run_dags:false behavior still needs the future host drill; a succeeded run with skipped research steps cannot count.", + "Off-host recovery requires user decision 3: independent destination, key recovery and second-host consumer restore under a chosen/measured recovery-time objective.", + "No retention deletion, broker, live credentials/mode, model calls, commit, push or manifest pin move.", + "Python 3.11 minimum is source-verified and syntax-checked. Execution used 3.13.15; no separate 3.11 interpreter was exercised.", + "robfig/cron v3.0.1 Next program/module were prepared from tagged source, but go run exited 127 because the compiler was not found; expected DST values are not measured results." + ], + "tool_versions": { + "dagu": "2.16.6", + "restic": "0.19.1", + "python_executed": "3.13.15", + "python_minimum": "3.11", + "exchange_calendars_executed": "4.13.2", + "sqlite_executed": "3.53.1", + "cpython_reference": "v3.12.3" + }, + "upstream_sources": [ + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/runtime/runner.go:1267-1281,1502-1525,1196-1206,1633-1648", + "sha256": "95ec2ba0e869bc1fc3d2d25feb5b1af0371602d0af7073d4852381faaf5d7948" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmn/config/env.go:77-95", + "sha256": "5d852a9e96a17c37101602a96217b369bcdf79e0463e880e2cba85b9ffed3f81" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmn/config/loader.go:344-347,1311-1319", + "sha256": "eb7203d02fbf88a213bdee63261dccfefca376c358612630779b181069cfdec3" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmd/startall.go:32-45", + "sha256": "1b535400a80904fcceec79b16fa862b917d265170c5a31ae48cc8c856e32763e" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/cmd/history.go:568-602", + "sha256": "a972e448952c527fd602465c5c4fa43c008422f2c13d22b9b08ee94fff0a760e" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/040_backup.rst:197-206,679-703,787-806", + "sha256": "e7281c20f0ef27bdbbb5f59d12f64aef63fae33048c5bd6b086ff1973edd218e" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/045_working_with_repos.rst:482-521", + "sha256": "56bfbe763b9cae005700f9472f73f69acaa28eea07d6fe258471af729ffffcd6" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/050_restore.rst:55-73", + "sha256": "00d39803cfcb17158c2ee1d835de6351c902e66067d2493438831580e9b04251" + }, + { + "pin": "restic/restic@v0.19.1", + "locator": "doc/060_forget.rst:225-233", + "sha256": "444be0adec93ad92810a4c6b363d8a5a7b77d84c88879e38408b1a7ea2eff614" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "src/shell.c.in:3737,3759,9074-9099", + "sha256": "bcceddfe1ab18e60c32cb9bbac7020ec22c03dbd68bd8dffe3599246560a46e3" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "src/backup.c:321-392", + "sha256": "ee4cbfc7d8afb311d12cc106d645764a3e18266694f003438b410bfa33764ef7" + }, + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "tool/sqlite3_rsync.c:13-40", + "sha256": "b9e54de43b69da85771de1acacd1ade4ad29d001be09b2f187dfb954eda2f0c8" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Modules/_sqlite/connection.c:2011-2016,2067-2102", + "sha256": "c33db9525cbc9887293c1cbdc03e0d8a3a3201e7dae98392452a40196e850070" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/sqlite3.rst:1107-1155", + "sha256": "f5be72345d2d37ffa53a82058b6054ed9052dda816a2e84639b96172fe3183b7" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/subprocess.rst:62-68", + "sha256": "2cb61f06f2fc41a60cf52c13a1bdaf52ff2a7962c55be96ec650bb12eb3c6845" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/hashlib.rst:267-302", + "sha256": "49df3195ff1acf0d114ed0288a15a43c924c9cd41564e0d956d40998ed6d6a82" + }, + { + "pin": "python/cpython@v3.12.3", + "locator": "Doc/library/fcntl.rst:139-149", + "sha256": "2d828976d383a49e1bd394921100078afd5412a50c498b5302682bbf2be58a8d" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "pyproject.toml:67-68", + "sha256": "0624504db59140714e6d12dd5039009e6517e20e88017c2e8989cf961645a266" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "exchange_calendars/ecal.py:100-149", + "sha256": "1744b47c0a502f5a3ae9baebdf5affac6027da68ce3ab954289691e7f03f6c8f" + }, + { + "pin": "gerrymanoim/exchange_calendars@4.13.2", + "locator": "exchange_calendars/exchange_calendar.py:1012-1016,1257-1279", + "sha256": "7b8087892b698299fe448afbc4b3fb9300a34ba1605804999436525391c08de6" + }, + { + "pin": "systemd/systemd@v255", + "locator": "man/loginctl.xml:186-195", + "sha256": "b07e1b67f169a8d0b380f5738676390b2c5d6b4297bc6f1845aaf1e336d6a476" + }, + { + "pin": "systemd/systemd@v255", + "locator": "man/systemd.service.xml:818-836", + "sha256": "439322b120014971f37e0a96adc6cb827367846ff6c16f193e9ec4b15931033b" + } + ], + "upstream_glue_choices": [ + { + "glue": "session_day.py", + "decision": "kept", + "checked": "Native ecal CLI, native is_session/session_close and Dagu preconditions/retries/handlers.", + "gap_citation": "gerrymanoim/exchange_calendars@4.13.2:pyproject.toml:67-68; exchange_calendars/ecal.py:100-149; exchange_calendars/exchange_calendar.py:1012-1016,1257-1279", + "why": "ecal renders a calendar; compose native session/close queries into the job eligibility/input-mtime token and leave skipping to Dagu." + }, + { + "glue": "journal_recovery.py", + "decision": "kept", + "checked": "restic stdin-from-command, SQLite CLI .backup/.dump, SQLite sqlite3_rsync, CPython Online Backup and subprocess timeout.", + "gap_citation": "restic/restic@v0.19.1:doc/040_backup.rst:679-703; doc/045_working_with_repos.rst:482-521; doc/050_restore.rst:55-73; sqlite/sqlite@version-3.53.1:src/shell.c.in:9074-9099; tool/sqlite3_rsync.c:13-40", + "why": "Native operations do the copying/backups/checks/restores; consumer glue freezes the external oracle and required tables, rotates checks with a deadline and refuses controls." + }, + { + "glue": "journal_recovery.py rotation lock", + "decision": "replaced", + "checked": "CPython native fcntl.flock", + "gap_citation": "python/cpython@v3.12.3:Doc/library/fcntl.rst:139-149", + "why": "Replace exclusive-create sentinels with nonblocking process-owned flock; stale file survives safely and no custom stale-lock recovery runs." + }, + { + "glue": "drill_process_restart.py", + "decision": "kept", + "checked": "systemd Restart=on-failure, Dagu native history/status and native handlers/retries.", + "gap_citation": "systemd/systemd@v255:man/systemd.service.xml:818-836; dagucloud/dagu@v2.16.6:internal/cmd/history.go:568-602; internal/ir/status.go:133-149", + "why": "Upstreams own supervision and run records; identity/time/holiday guards and exact due-slot correlation are our acceptance procedure." + }, + { + "glue": "drill_local_recovery.py", + "decision": "kept", + "checked": "Native Online Backup, restic backup/check/restore and .backup/.dump/sqlite3_rsync alternatives.", + "gap_citation": "python/cpython@v3.12.3:Modules/_sqlite/connection.c:2067-2102; restic/restic@v0.19.1:doc/050_restore.rst:55-73", + "why": "Synthetic concurrent WAL writers, failures preserving integrity/counts and path-free integration evidence are consumer acceptance rather than replacement backup tools." + } + ], + "artifacts": [ + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof-r2.json", + "sha256": "c80b455622e549c21e3a0df78ce300c0722182f529bcdae09ddf5ce5c25637b1", + "bytes": 10789 + }, + { + "path": "blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json", + "sha256": "86bf2cd9b2b6b698e4ea2de30d227b4e133a40d5de839e9fac92d821840b5f11", + "bytes": 21096 + }, + { + "path": "blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json", + "sha256": "2f8c7896b7277264801fc4b701375c612574619a2b96190202396fb52b00e2e7", + "bytes": 15859 + }, + { + "path": "blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json", + "sha256": "d397eb9d53692382de7e345783a8ac0585e3857f1fc6664b50fbf6d86ab5cc02", + "bytes": 2076 + }, + { + "path": "blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json", + "sha256": "c15ac08f09f324a6c1106f167c21cb013bc6c58fefbb04a3092ec68cc8cf43c7", + "bytes": 12307 + } + ], + "integration_source_sha256": { + "blueprints/us-equities/hosting/config.yaml.example": "a8bfd48d297f783c40338eeea473180144e9ddf5a8a6fe6011010dd9e732a24b", + "blueprints/us-equities/hosting/equity-research-evidence.yaml": "267fe130b5e8d10676cc998e690617bc6b119c89a6ec5fbbe8fe8771e93107b9", + "blueprints/us-equities/hosting/session_day.py": "eaab149b5825e37d31efdc35584ee112397e8a3b389bd9ee46a09e208a02762d", + "blueprints/us-equities/hosting/journal_recovery.py": "4a861e8db692017ba24b19894f276b6756aa25c9cfdbf56c882c8a3d58d608c7", + "blueprints/us-equities/hosting/drill_process_restart.py": "dc394c0b5a5cf61084a639bcab457f8cbb6c40a57bbfe12125b394da191e510b", + "blueprints/us-equities/hosting/drill_local_recovery.py": "405e1c8cc98876ba50b4ec47b46f9edf297a3bad8e7c429880190f20576ca6dd", + "tests/test_us_equities_hosting.py": "7e6367b3a0ab333278afa3ce796972d0aa2fcc1a9aca746662b610a1e9b01033" + }, + "commands": [ + { + "cmd": "env TMPDIR= nice -n 19 python3 blueprints/us-equities/hosting/drill_local_recovery.py --restic-bin --scratch-root --output ", + "exit": 0, + "invocation_proxy": "Builder prefixed the command with rtk env, which ran env through RTK fallback. The portable recipe does not require RTK.", + "returned_output": { + "representation": "parsed_json_lines", + "lines": [ + { + "checked_subset": "1/1", + "matched_journals": 2, + "restore_control_exit": 1, + "result": "passed", + "snapshot_control_exit": 1, + "throwaway_password_deleted": true + } + ] + }, + "started_at_utc": "2026-10-05T05:32:23.230897+00:00", + "completed_at_utc": "2026-10-05T05:32:33.774862+00:00" + }, + { + "cmd": "python3 -m unittest tests.test_us_equities_hosting -v", + "exit": 0, + "returned_output": "test_output_refuses_unsanitized_report_before_write (tests.test_us_equities_hosting.DrillGuardTests.test_output_refuses_unsanitized_report_before_write) ... ok\ntest_restart_pid_command_and_executable_identity_guards (tests.test_us_equities_hosting.DrillGuardTests.test_restart_pid_command_and_executable_identity_guards) ... ok\ntest_restart_refuses_holiday_and_calendar_failure_before_signalling (tests.test_us_equities_hosting.DrillGuardTests.test_restart_refuses_holiday_and_calendar_failure_before_signalling) ... ok\ntest_restart_time_window_refuses_wrong_slot_past_close_and_short_reserve (tests.test_us_equities_hosting.DrillGuardTests.test_restart_time_window_refuses_wrong_slot_past_close_and_short_reserve) ... ok\ntest_symlinked_scratch_root_scrubs_both_forms_and_resolves_at_creation (tests.test_us_equities_hosting.DrillGuardTests.test_symlinked_scratch_root_scrubs_both_forms_and_resolves_at_creation) ... ok\ntest_actual_close_is_checked_and_early_close_waits_until_daily_slot (tests.test_us_equities_hosting.HostingContractTests.test_actual_close_is_checked_and_early_close_waits_until_daily_slot) ... ok\ntest_coordinator_disabled_in_configuration (tests.test_us_equities_hosting.HostingContractTests.test_coordinator_disabled_in_configuration) ... ok\ntest_due_run_requires_slot_scheduler_and_success (tests.test_us_equities_hosting.HostingContractTests.test_due_run_requires_slot_scheduler_and_success) ... ok\ntest_input_mtime_must_follow_actual_session_close (tests.test_us_equities_hosting.HostingContractTests.test_input_mtime_must_follow_actual_session_close) ... ok\ntest_schedule_name_timezone_and_skipping_dependency_chain (tests.test_us_equities_hosting.HostingContractTests.test_schedule_name_timezone_and_skipping_dependency_chain) ... ok\ntest_scripts_use_python_311_compatible_syntax_and_native_digest (tests.test_us_equities_hosting.HostingContractTests.test_scripts_use_python_311_compatible_syntax_and_native_digest) ... ok\ntest_session_check_uses_new_york_date_and_skips_a_non_session (tests.test_us_equities_hosting.HostingContractTests.test_session_check_uses_new_york_date_and_skips_a_non_session) ... ok\ntest_unit_start_all_restart_and_job_environment_delivery (tests.test_us_equities_hosting.HostingContractTests.test_unit_start_all_restart_and_job_environment_delivery) ... ok\ntest_wrong_calendar_version_fails_in_the_normal_step (tests.test_us_equities_hosting.HostingContractTests.test_wrong_calendar_version_fails_in_the_normal_step) ... ok\ntest_both_controls_make_the_procedure_exit_nonzero (tests.test_us_equities_hosting.JournalRecoveryTests.test_both_controls_make_the_procedure_exit_nonzero) ... ok\ntest_cycle_clock_is_captured_before_snapshot_and_checks_native_completion (tests.test_us_equities_hosting.JournalRecoveryTests.test_cycle_clock_is_captured_before_snapshot_and_checks_native_completion) ... ok\ntest_empty_selection_and_unreviewed_tables_refused (tests.test_us_equities_hosting.JournalRecoveryTests.test_empty_selection_and_unreviewed_tables_refused) ... ok\ntest_every_nonzero_native_exit_including_three_is_refused (tests.test_us_equities_hosting.JournalRecoveryTests.test_every_nonzero_native_exit_including_three_is_refused) ... ok\ntest_failed_check_does_not_advance_rotation (tests.test_us_equities_hosting.JournalRecoveryTests.test_failed_check_does_not_advance_rotation) ... ok\ntest_hash_rejects_changed_contents_even_when_integrity_and_counts_match (tests.test_us_equities_hosting.JournalRecoveryTests.test_hash_rejects_changed_contents_even_when_integrity_and_counts_match) ... ok\ntest_missed_rotation_interval_requires_successful_full_read (tests.test_us_equities_hosting.JournalRecoveryTests.test_missed_rotation_interval_requires_successful_full_read) ... ok\ntest_missing_and_extra_files_refused (tests.test_us_equities_hosting.JournalRecoveryTests.test_missing_and_extra_files_refused) ... ok\ntest_missing_required_table_prevents_freezing (tests.test_us_equities_hosting.JournalRecoveryTests.test_missing_required_table_prevents_freezing) ... ok\ntest_native_lock_contention_refuses_and_stale_lock_after_exit_is_safe (tests.test_us_equities_hosting.JournalRecoveryTests.test_native_lock_contention_refuses_and_stale_lock_after_exit_is_safe) ... ok\ntest_online_backup_includes_committed_wal_state_and_freezes_hashes (tests.test_us_equities_hosting.JournalRecoveryTests.test_online_backup_includes_committed_wal_state_and_freezes_hashes) ... ok\ntest_online_backup_size_and_hard_time_limits_refuse_before_freezing (tests.test_us_equities_hosting.JournalRecoveryTests.test_online_backup_size_and_hard_time_limits_refuse_before_freezing) ... ok\ntest_restore_cannot_substitute_its_own_inventory (tests.test_us_equities_hosting.JournalRecoveryTests.test_restore_cannot_substitute_its_own_inventory) ... ok\ntest_rotation_across_fall_back_uses_utc_seven_day_cycle (tests.test_us_equities_hosting.JournalRecoveryTests.test_rotation_across_fall_back_uses_utc_seven_day_cycle) ... ok\ntest_rotation_allows_daily_jitter_over_multiple_whole_cycles (tests.test_us_equities_hosting.JournalRecoveryTests.test_rotation_allows_daily_jitter_over_multiple_whole_cycles) ... ok\ntest_rotation_reads_every_default_subset_within_seven_days (tests.test_us_equities_hosting.JournalRecoveryTests.test_rotation_reads_every_default_subset_within_seven_days) ... ok\ntest_rotation_refuses_other_repository_and_invalid_part_counts (tests.test_us_equities_hosting.JournalRecoveryTests.test_rotation_refuses_other_repository_and_invalid_part_counts) ... ok\ntest_row_count_is_a_secondary_required_state_check (tests.test_us_equities_hosting.JournalRecoveryTests.test_row_count_is_a_secondary_required_state_check) ... ok\ntest_slow_native_check_cannot_publish_state_after_cycle_deadline (tests.test_us_equities_hosting.JournalRecoveryTests.test_slow_native_check_cannot_publish_state_after_cycle_deadline) ... ok\ntest_snapshot_calls_backup_api_and_does_not_copy_source_files (tests.test_us_equities_hosting.JournalRecoveryTests.test_snapshot_calls_backup_api_and_does_not_copy_source_files) ... ok\ntest_stale_atomic_publication_file_does_not_reset_or_block_cursor (tests.test_us_equities_hosting.JournalRecoveryTests.test_stale_atomic_publication_file_does_not_reset_or_block_cursor) ... ok\ntest_symlink_is_refused (tests.test_us_equities_hosting.JournalRecoveryTests.test_symlink_is_refused) ... ok\ntest_whole_rotation_deadline_refuses_even_without_large_individual_gaps (tests.test_us_equities_hosting.JournalRecoveryTests.test_whole_rotation_deadline_refuses_even_without_large_individual_gaps) ... ok\ntest_native_non_session_skips_dependents (tests.test_us_equities_hosting.NativeDaguTests.test_native_non_session_skips_dependents) ... ok\ntest_native_session_executes_success_chain (tests.test_us_equities_hosting.NativeDaguTests.test_native_session_executes_success_chain) ... ok\ntest_native_stale_input_is_visible_and_skips_dependents (tests.test_us_equities_hosting.NativeDaguTests.test_native_stale_input_is_visible_and_skips_dependents) ... ok\ntest_holidays_and_early_closes_with_real_locked_calendar (tests.test_us_equities_hosting.RealCalendarTests.test_holidays_and_early_closes_with_real_locked_calendar) ... ok\ntest_truly_uncovered_date_is_a_hard_failure (tests.test_us_equities_hosting.RealCalendarTests.test_truly_uncovered_date_is_a_hard_failure) ... ok\n\n----------------------------------------------------------------------\nRan 42 tests in 7.782s\n\nOK\n", + "started_at_utc": "2026-10-05T05:34:28.663845+00:00", + "completed_at_utc": "2026-10-05T05:34:37.146883+00:00", + "class": "stdlib behavioural/contract tests, real pinned calendars and native manual Dagu fixture runs" + }, + { + "exit": 0, + "returned_output": "time=2026-10-05T01:30:30.475-04:00 level=INFO msg=\"Creating example DAGs for first-time users\" dir=/dags\ntime=2026-10-05T01:30:30.535-04:00 level=INFO msg=\"Example DAGs created successfully. Check the web UI to explore them!\"\n", + "dag_sha256": "267fe130b5e8d10676cc998e690617bc6b119c89a6ec5fbbe8fe8771e93107b9", + "cmd": "dagu validate --context local --dagu-home --config /config.yaml /blueprints/us-equities/hosting/equity-research-evidence.yaml" + }, + { + "cmd": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml (token=non_session)", + "exit": 0, + "native_node_statuses": { + "calendar_check": 4, + "nyse_session": 5, + "evidence": 5 + }, + "marker_created": false, + "class": "native manual Dagu fixture, no scheduler" + }, + { + "cmd": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml (token=session)", + "exit": 0, + "native_node_statuses": { + "calendar_check": 4, + "nyse_session": 4, + "evidence": 4 + }, + "marker_created": true, + "class": "native manual Dagu fixture, no scheduler" + }, + { + "cmd": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml (token=stale_input)", + "exit": 0, + "native_node_statuses": { + "calendar_check": 4, + "nyse_session": 5, + "evidence": 5 + }, + "marker_created": false, + "class": "native manual Dagu fixture, no scheduler" + }, + { + "cmd": "env GOPROXY=off GOSUMDB=off GOTOOLCHAIN=local go run main.go", + "exit": 127, + "returned_output": "nice: \u2018go\u2019: No such file or directory\n", + "class": "unavailable execution; permitted record-why alternative" + } + ], + "copy_observations": [ + { + "committed_writes_during_native_call": 6, + "journal": "evidence", + "native_copy_duration_s": 0.049027458, + "pages": -1 + }, + { + "committed_writes_during_native_call": 9, + "journal": "research", + "native_copy_duration_s": 0.052331414, + "pages": -1 + } + ], + "controls": { + "restore": { + "exit": 1, + "mutation_observation": { + "bytes_unchanged": true, + "mutation": "SQLite user_version header", + "required_state_unchanged": true, + "sha256_changed": true + }, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.012312, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-restore/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.83599, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 3, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 3, + "dirs_changed": 5, + "dirs_unmodified": 0, + "data_blobs": 1, + "tree_blobs": 9, + "data_added": 440024, + "data_added_packed": 4408, + "total_files_processed": 3, + "total_bytes_processed": 16532001, + "total_duration": 0.800409374, + "backup_start": "2026-10-05T01:32:31.299715116-04:00", + "backup_end": "2026-10-05T01:32:32.10012451-04:00", + "snapshot_id": "0f8689841dd2a17de838454f93f004cd439c69f8e9f685a1b76f0ced3945f2b0" + } + ] + } + }, + { + "args": [ + "check", + "--read-data-subset=1/1" + ], + "duration_s": 0.780648, + "exit": 0, + "output": "using temporary cache in /control-restore/cache/restic-check-cache-3829689771\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n[0:00] 100.00% 3 / 3 snapshots\nread group #1 of 6 data packs (out of total 6 packs in 1 groups\n[0:00] 100.00% 6 / 6 packs\nno errors were found\n" + }, + { + "args": [ + "restore", + "0f8689841dd2a17de838454f93f004cd439c69f8e9f685a1b76f0ced3945f2b0:/control-restore/stage", + "--target", + "/control-restore/restored", + "--verify", + "--overwrite", + "never", + "--json" + ], + "duration_s": 0.766378, + "exit": 0, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "total_files": 4, + "files_restored": 4, + "total_bytes": 16532001, + "bytes_restored": 16532001 + } + ] + } + } + ], + "returned": { + "reason": "file sha256/bytes mismatch", + "result": "refused" + } + }, + "snapshot": { + "exit": 1, + "native_commands": [ + { + "args": [ + "version" + ], + "duration_s": 0.015723, + "exit": 0, + "output": "restic 0.19.1 compiled with go1.26.4 on linux/amd64\n" + }, + { + "args": [ + "backup", + "/control-snapshot/stage", + "--host", + "equity-research-recovery", + "--tag", + "journal-recovery", + "--group-by", + "host,tags", + "--json" + ], + "duration_s": 0.786509, + "exit": 3, + "output": { + "representation": "parsed_json_lines", + "lines": [ + { + "message_type": "summary", + "files_new": 2, + "files_changed": 0, + "files_unmodified": 0, + "dirs_new": 3, + "dirs_changed": 5, + "dirs_unmodified": 0, + "data_blobs": 2, + "tree_blobs": 9, + "data_added": 439323, + "data_added_packed": 4764, + "total_files_processed": 2, + "total_bytes_processed": 8266273, + "total_duration": 0.759458587, + "backup_start": "2026-10-05T01:32:30.20588227-04:00", + "backup_end": "2026-10-05T01:32:30.965340857-04:00", + "snapshot_id": "32eac2d2e6d8e8a4b83723f540cf10a456cfc73011ef7ef7bf6882c65eb502fc" + }, + { + "message_type": "error", + "error": { + "message": "open /control-snapshot/stage/journals/evidence.sqlite3: permission denied" + }, + "during": "archival", + "item": "/control-snapshot/stage/journals/evidence.sqlite3" + }, + { + "message_type": "exit_error", + "code": 3, + "message": "Warning: at least one source file could not be read" + } + ] + } + } + ], + "returned": { + "reason": "restic backup refused (exit 3)", + "result": "refused" + } + } + }, + "independent_observation": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 103 + }, + "sha256": "2672f026b8cf69cbf716b07037e263be7d5787171e4ed8599e89f0759a935334" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 126 + }, + "sha256": "4809d2613be676b45b97bae9704f8e8c61b7a63b1450a009fcb6d623c3eb1f4f" + } + ], + "sanitization": "Canonical scratch root resolved at creation; lexical/canonical roots and executable paths scrubbed, then absolute-path assertions run before --output. New receipts contain no host path, login name or password; native fixture IDs/hashes and timestamps remain. Private raw output stays in scratch custody. JSON command output is stored as parsed_json_lines records with all original JSON fields preserved; non-JSON lines remain text and raw formatting stays in private custody.", + "prior_attempts": [ + { + "command": "real-calendar regression before fix", + "exit": 1, + "observation": "Three DateOutOfBounds failures: 2026-01-01, 2027-01-01, 2034-01-02. Same real-calendar cases passed after bracketing." + }, + { + "command": "python3 -m unittest tests.test_us_equities_hosting -v (intermediate contract update)", + "exit": 1, + "observation": "Legacy tests expected a missing-file restore control, removed sentinel and strict 24h cutoff; updated to new mutation control, native flock and whole-cycle bounds." + }, + { + "command": "python3 -m unittest tests.test_us_equities_hosting -v (before README repair)", + "exit": 1, + "observation": "One documentation contract failed because group-by host,tags had not yet been documented; final run passes." + }, + { + "command": "native probe artifact recording", + "exit": 1, + "observation": "Native Dagu probes and validate exited 0; publication guard refused the Go module relative ./ path. The guard now admits relative ./ paths and tests that case; cron record remains not_run." + }, + { + "command": "earlier round 2 local recovery drill", + "exit": 0, + "observation": "Passed before the relative-path sanitizer correction; final drill rerun with the final sanitizer and retained here." + } + ], + "operational_status": { + "unit_installed": false, + "scheduler_started": false, + "process_restart_drill_run": false, + "off_host_recovery_gate_closed": false, + "unattended_hosting_gate_closed": false, + "throwaway_password_deleted": true, + "pin_moved": false + }, + "review": { + "kind": "self", + "scope": "Original pinned source/native capability review, real calendar regression, native manual Dagu runs, native restic controls and independent bytes/state observations. Coordinator review remains separate." + }, + "source_revision": "working tree after round-2 repair; files pinned by integration_source_sha256" +} diff --git a/evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json b/evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json new file mode 100644 index 000000000..58db1397a --- /dev/null +++ b/evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json @@ -0,0 +1,407 @@ +{ + "schema_version": 1, + "id": "trading-unattended-hosting-recovery-r3-20261005", + "kind": "native_cli_e2e", + "round": "t-pr4-r3", + "component_ids": [ + "dagu", + "restic" + ], + "evidence_class": "local_integration", + "source_git_commit": "4c897418fe35a030a1188ae447eaf31c893f8eff", + "source_revision": "Working tree after round-3 repairs; tested files pinned by integration_source_sha256.", + "observed_at_utc": "2026-10-05T07:58:37.912257+00:00", + "claim": "All seven new p2 repairs have red/green regressions; native Dagu executes the six deployed steps and fails unavailable-input runs; native restic verifies WAL recovery and isolated failing controls.", + "tool_versions": { + "dagu": "2.16.6", + "restic": "0.19.1", + "python_executed": "3.13.15", + "python_minimum": "3.11", + "exchange_calendars_executed": "4.13.2", + "sqlite_executed": "3.53.1", + "systemctl_help_executed": "255.4" + }, + "integration_source_sha256": { + "blueprints/us-equities/hosting/config.yaml.example": "a8bfd48d297f783c40338eeea473180144e9ddf5a8a6fe6011010dd9e732a24b", + "blueprints/us-equities/hosting/dagu-equities.service.example": "327f5500f6d02d96508c35a1e512d7adcb2018427067a286b51603bebdd6bd41", + "blueprints/us-equities/hosting/equity-research-evidence.yaml": "6107eebb77ae8904ff6422b7e86f4c551dde36d35aa5d6e87b8f5d72b0c9397f", + "blueprints/us-equities/hosting/session_day.py": "73969582e6a675f894b681769eb8a99c1a142babb1dd81a5c2a4b377a9272e96", + "blueprints/us-equities/hosting/journal_recovery.py": "f90884e465634a7cbe741e53233294192fc64de5ded499b9337aeb187c1a112b", + "blueprints/us-equities/hosting/drill_process_restart.py": "2262abdbb9a05aefc77eceaeb0503a2053ded179b3825e3fb8e75dfb602c0701", + "blueprints/us-equities/hosting/drill_local_recovery.py": "405e1c8cc98876ba50b4ec47b46f9edf297a3bad8e7c429880190f20576ca6dd", + "blueprints/us-equities/data/summarize_backtest.py": "24690af7ab7066d041a33d952485ebe5a746fe787ee4fbd97c4369e6b26bed45", + "tests/test_us_equities_hosting.py": "06cd7a9ecae755b13001f2aae3bb7cfff0e0218ac4abb044480ab3f56a9c95b2" + }, + "artifacts": [ + { + "path": "blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json", + "sha256": "8119d2a1f7d281ed6309f2a46e552538cfc5250fd0af2d7cdd500a8711b8d43b", + "bytes": 3255 + }, + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof-r3.json", + "sha256": "1fb0eebddd33c15c880af008606301364bb62044a123492b86da96af00e83684", + "bytes": 11577 + }, + { + "path": "blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json", + "sha256": "79237a01bbe3d7c6997409e910894664aa0a41e8d36240689259e92a9673b5d2", + "bytes": 2790 + } + ], + "upstream_sources": [ + { + "pin": "sqlite/sqlite@version-3.53.1", + "locator": "src/backup.c:321-392", + "sha256": "ee4cbfc7d8afb311d12cc106d645764a3e18266694f003438b410bfa33764ef7" + }, + { + "pin": "systemd/systemd@v255", + "locator": "man/systemctl.xml:541-549,2377-2388", + "sha256": "7f06242ce06d9d895bf7445e3d29680f8e6a7284004122b4e209097b8313b6d5" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/service/frontend/api/v1/dagruns.go:2066-2096", + "sha256": "f07b1b89d784251594bd68478a1120f88111a7f538081158826f7a99cb62f75f" + }, + { + "pin": "dagucloud/dagu@v2.16.6", + "locator": "internal/service/frontend/api/v1/transformer.go:362,371", + "sha256": "923e6b2e76fa03473cde1c4ed82d9106acfc99f3e95e7fe961d21e0d11849d31" + } + ], + "red_green_regressions": [ + { + "test": "tests.test_us_equities_hosting.HostingContractTests.test_session_cli_fails_unavailable_inputs_but_keeps_calendar_skips", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "2764a350eb2781fdb1d6e26f46242a1af1e0bf042ab1b95c1536d1424258c6d0", + "green_output_sha256": "2567e3bce9c2d914b3ffb614f0a25e79d0b3c735c43f644116c16d79d759f282" + }, + { + "test": "tests.test_us_equities_hosting.DrillGuardTests.test_restart_fault_is_delivered_by_native_unit_addressed_kill", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "3b9c82416a5412cbff8ad674baaa0529445ea59035a1e1ea178e1bc2157bbe5c", + "green_output_sha256": "2d2a6a9a7cc7832312e68f2516eac3042699194e6abf0c43fa6a1c29cd72cccf" + }, + { + "test": "tests.test_us_equities_hosting.DrillGuardTests.test_restart_reserves_150_seconds_before_due_and_refuses_late_restart", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "61eaa291535a2d48497576856e87c4f57f5b4dbd621ed01feca09dadad82a337", + "green_output_sha256": "0e014d76c9f437c901211c27fe82875265a736ce2f8997f6a1cb6d2ce0e1bc9b" + }, + { + "test": "tests.test_us_equities_hosting.JournalRecoveryTests.test_snapshot_cli_refuses_rollback_sources_without_explicit_writer_blocking_opt_in", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "183045134a7f336b1a1ed2fd84bd91824fada4e7cf64744c277677b0f020b21c", + "green_output_sha256": "d1c27ce79bfa789454e30a9e3e3f37f7d5e681c86dc31d8c029a1a26d824c3cf" + }, + { + "test": "tests.test_us_equities_hosting.JournalRecoveryTests.test_control_backups_use_groups_distinct_from_production_retention", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "bb99bb2b2a35dbc0c0aba55d1c4a4f1ef1c433212186684fc263acbf2f6c8888", + "green_output_sha256": "4501a6447af718dd00f82e5ee56dfeeeb4b792ef0a9c5981d21bfd86835c2158" + }, + { + "test": "tests.test_us_equities_hosting.HostingContractTests.test_round2_receipt_names_published_base_and_hash_pinned_working_tree", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "021a8a053696b5e53f79eaa355720f24f56263b62ea13052e62df04b7ccbf329", + "green_output_sha256": "a3b07f0a76f733c360abf8b82f619260e05e1011fbd63cea308ed03f64fcdc90" + }, + { + "test": "tests.test_us_equities_hosting.NativeDaguTests.test_native_probe_uses_its_private_root_when_tmpdir_is_unset", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "540ae464d24b6b6421dcdb026ab4421c0b625d5c0003293f60debae25f704d61", + "green_output_sha256": "dc6028d9312392b0d83a0c0c8fa2271a7f9d7a80e5a95b8400935582dd82648a" + }, + { + "test": "tests.test_us_equities_hosting.NativeDaguTests.test_native_unavailable_input_fails_run_and_blocks_dependents", + "red_exit": 1, + "green_exit": 0, + "red_output_sha256": "77f578f30ed1ecb33126ce28d1b72e5e637e649059b296454ec7ff9e4e9b3af9", + "green_output_sha256": "ce3558a584121dee50105bf6ab73148f41923786cac666756327a0e5286baaaa" + } + ], + "commands": [ + { + "command": "python3 -m unittest tests.test_us_equities_hosting -v", + "exit": 0, + "duration_s": 11.575, + "raw_output_sha256": "2cb7120f6c2fc9d644fb8c492849a3990b7f3404ec701dc3f0d77f9457b33188", + "summary": [ + "Ran 50 tests in 11.070s", + "OK" + ], + "skipped": [] + }, + { + "command": "python3 -m unittest tests.test_us_equities_hosting.RealCalendarTests -v", + "exit": 0, + "duration_s": 3.996, + "raw_output_sha256": "4a5d414c419da8ed4dcbbe35b7d0a4fac4d60a441145ac9f186d32a823c3c0a7", + "summary": [ + "Ran 2 tests in 3.494s", + "OK" + ], + "skipped": [] + }, + { + "command": "python3 -m unittest tests.test_host_receipts -v", + "exit": 0, + "duration_s": 5.377, + "raw_output_sha256": "db057e871947f955b97cf1945df403c0ca371687f2109f25146e6d34c190f712", + "summary": [ + "Ran 184 tests in 5.269s", + "OK" + ], + "skipped": [] + }, + { + "command": "python3 -m unittest tests.test_evidence_manifest -v", + "exit": 0, + "duration_s": 0.058, + "raw_output_sha256": "32982b915f671d941f72e021bc50af1338e4e9498ea5424d3934ac7967c329ad", + "summary": [ + "Ran 5 tests in 0.009s", + "OK" + ], + "skipped": [] + }, + { + "command": "python3 -m unittest tests.test_validate -v", + "exit": 0, + "duration_s": 0.861, + "raw_output_sha256": "30b3852a069223d3bc34533823fcbdc62c5a6111126b7062c62902507d8b32cb", + "summary": [ + "Ran 68 tests in 0.784s", + "OK" + ], + "skipped": [] + } + ], + "native_dag_observation": { + "schema_version": 1, + "round": "t-pr4-r3", + "evidence_class": "local_integration", + "dagu_pin": "2.16.6", + "calendar_pin": "4.13.2", + "clock_fixture": "2026-11-27T21:30:00Z", + "events_mtime_fixture": "2026-11-27T19:00:00Z", + "scope": "Manual native run of all six deployed steps; synthetic historical LEAN input and post-close clock; no scheduler.", + "dag_template_sha256": "6107eebb77ae8904ff6422b7e86f4c551dde36d35aa5d6e87b8f5d72b0c9397f", + "private_copy_sha256": "e71ebf99aa29a86bf0eb18ab61a054ec24bdfcb67fed5538f2be22c4662884f9", + "commands": [ + { + "verb": "validate", + "exit": 0, + "duration_s": 0.210249, + "raw_output_sha256": "e542d658b185d65bc180fbfffc22127516c034643fd63cc45c4045a6f29bdf2c" + }, + { + "verb": "start", + "exit": 0, + "duration_s": 37.141351, + "raw_output_sha256": "63b019e05930b07cc64d36140722c96bca5fab3bf913f6506fd768523cc3908d" + } + ], + "native_run_status": 4, + "native_node_statuses": { + "calendar_check": 4, + "nyse_session": 4, + "prepare_output": 4, + "summarize": 4, + "baseline_evidence": 4, + "catalog_evidence": 4 + }, + "trigger_type": 2, + "timeout_sec": 180, + "history": { + "exit": 0, + "context": "local", + "from_utc": "2026-10-05T00:00:00Z", + "status_filter": "succeeded", + "limit": 1000, + "matched_run": true, + "row_count": 1 + }, + "parquet_count": 1, + "parquet_sha256": [ + "bcb285fabdedd0129b0d54dd7097688369695f5ed806fd3a89b26db9d33c65a8" + ], + "guard_output": "session", + "unavailable_input_probes": [ + { + "token": "missing_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "0b6641cc99565df6e909406e9fc0c1c7aa9000cdb7caed04e00c610efbbc4efb" + }, + { + "token": "stale_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "7fdf89042f9cfc5c0aea90cc9ac85b22f1c4c4d1909b72c2da3d5cb482ca8969" + }, + { + "token": "future_input", + "exit": 1, + "run_status": 2, + "node_statuses": { + "calendar_check": 2, + "nyse_session": 3, + "evidence": 3 + }, + "marker_created": false, + "output_token_observed": true, + "raw_sanitized_output_sha256": "452353a4995a5eabdd19bb7a6ec57cbab08ff3b0695a5f36a47d490647ba09a2" + } + ], + "completed_at_utc": "2026-10-05T07:50:43.143067+00:00", + "summarizer_returned_json": { + "scope": "LEAN bundled historical simulation, not broker fills", + "events": 1, + "distinct_orders": 1, + "statuses": { + "Filled": 1 + }, + "calendar": "XNYS", + "sample_session": "2013-10-07", + "session_open_utc": "2013-10-07T13:30:00+00:00", + "session_close_utc": "2013-10-07T20:00:00+00:00", + "source_sha256": "5d39b6390bf54295d3552923a5bc0ca7848271d300740702182a4c9aa53cad41", + "parquet_sha256": "bcb285fabdedd0129b0d54dd7097688369695f5ed806fd3a89b26db9d33c65a8" + } + }, + "recovery_observation": { + "drill_exit": 0, + "returned_json": { + "checked_subset": "1/1", + "matched_journals": 2, + "snapshot_control_exit": 1, + "restore_control_exit": 1, + "throwaway_password_deleted": true + }, + "independent_observation": [ + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/evidence.sqlite3", + "row_counts": { + "runs": 105 + }, + "sha256": "c49d676b4b63b431f9be9a7bd3aad252856c1ae1ca4b31dd294bd38671bd65a8" + }, + { + "bytes": 8265728, + "integrity_check": "ok", + "path": "journals/research.sqlite3", + "row_counts": { + "events": 127 + }, + "sha256": "01a99c624a13ea86cba2546191347b9e3b1dd121aee5cbd4e14766d044718f54" + } + ], + "control_groups": { + "host": "equity-research-recovery-control", + "tag": "journal-recovery-control", + "group_by": "host,tags" + }, + "native_snapshot_control_exit": 3, + "restore_control_mutation": { + "bytes_unchanged": true, + "mutation": "SQLite user_version header", + "required_state_unchanged": true, + "sha256_changed": true + } + }, + "operator_documentation": "Native manual start/cancel/history/replay recipes restored; no operator command or service operation executed.", + "source_corrections": [ + "Unavailable-input tokens fail the guard and run; only holidays and before-close skip.", + "Signal delivery uses native unit-addressed systemctl kill, with a 150-second reserve and late-slot refusal.", + "Source journal_mode is checked before copying; non-WAL needs an explicit writer-blocking opt-in.", + "Controls use distinct host/tag groups; historical round-2 source hashes are retained with corrected provenance." + ], + "limitations": [ + "Local integration and synthetic fixtures, not an unchanged upstream test suite or operating-host acceptance.", + "The deployed DAG copy uses a fixed post-close guard clock and a synthetic historical LEAN event; the native trigger remains manual.", + "Restart identity/window guards are tested with mocks. No systemctl manager operation, fault or service deployment ran.", + "Scheduler dispatch with run_dags:false, reboot, missed slots and independent alerts remain user decision 2.", + "Off-host recovery remains user decision 3: independent destination, key recovery and second-host consumer restore.", + "The independent final T row at line 102 was reread; it retains restic/Online Backup and does not name Litestream.", + "No Go compiler was found in PATH; the historical cron Next probe exit 127 remains unmeasured across DST.", + "No separate Python 3.11 interpreter was found in PATH; execution used 3.13.15.", + "Per-journal snapshots are consistent separately; there is no cross-journal transaction.", + "Native recovery reads subset 1/1; whole seven-part rotation remains behavioral fixture evidence.", + "No retention deletion, broker, credentials, live mode, model calls, commit, push or pin move." + ], + "invocation_proxy": "Builder shell commands used rtk env through its fallback, with TMPDIR outside the checkout and nice -n 19. Portable recipes use plain env.", + "native_command_exits": [ + { + "command": "dagu validate (private deployed DAG copy)", + "exit": 0 + }, + { + "command": "dagu start (private deployed DAG copy)", + "exit": 0 + }, + { + "command": "dagu history --context local --from --status succeeded --limit 1000 --format json equity-research-evidence", + "exit": 0 + }, + { + "command": "dagu start (real guard, missing_input)", + "exit": 1, + "expected_failure": true + }, + { + "command": "dagu start (real guard, stale_input)", + "exit": 1, + "expected_failure": true + }, + { + "command": "dagu start (real guard, future_input)", + "exit": 1, + "expected_failure": true + }, + { + "command": "drill_local_recovery.py --restic-bin --output ", + "exit": 0 + }, + { + "command": "restic backup (snapshot control)", + "exit": 3, + "expected_failure": true + }, + { + "command": "journal_recovery.py cycle --control snapshot", + "exit": 1, + "expected_failure": true + }, + { + "command": "journal_recovery.py cycle --control restore", + "exit": 1, + "expected_failure": true + } + ] +} diff --git a/manifests/evidence.json b/manifests/evidence.json index cbf76c00c..1a5ccdffe 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -4245,6 +4245,70 @@ "This is a sanitized historical native-model study on local fixtures, not an unchanged upstream test suite, a fresh provider run, host-install acceptance or independently attested billing telemetry." ], "path": "evidence/receipts/rtk-fold-harbor-20261004.json" + }, + { + "id": "trading-unattended-hosting-recovery-20261005", + "kind": "native_cli_e2e", + "component_ids": [ + "dagu", + "restic" + ], + "claim": "Native restic 0.19.1 backed up two synthetic SQLite WAL journals copied by the stdlib Online Backup API during committed writes, read subset 1/1 (2/2 packs), restored 802816 identical journal bytes with matching external inventory and required state, and refused both snapshot and restore controls. Dagu 2.16.6 validated the scheduled research DAG offline; operating-host and off-host gates remain open.", + "limitations": [ + "Scratch local integration with synthetic research journals; not an unchanged upstream test suite or a real market/broker journal.", + "Per-journal Online Backup is independently consistent; no atomic cross-journal transaction or research consumer application restart was accepted.", + "Historical one-part check 1/1 read 2/2 packs. Round 1 enforced strict daily gaps in its synthetic rotation test; round 2 replaces that policy with a whole seven-subset rotation within seven days.", + "Operating host and independent alert path are user decision 2. No systemd unit was installed, enabled or started; no process-restart drill or scheduler was run.", + "A process-restart drill is not reboot, missed-run or independent-alert acceptance. permissions.run_dags scheduler behavior is source-reviewed and awaits the later host drill.", + "Off-host recovery stays open until user decision 3 chooses an independent destination and key recovery and a consumer restores on a second host against the independent inventory and required state.", + "Recovery-time objective, independent alerting and retention deletion remain unaccepted. No broker, credentials, live mode, model calls, commit or push was used.", + "Historical round 1 cited an unlanded calendar bundle. Round 2 uses the on-base SDK_ENV lock adoption/sdk/requirements-linux-x86_64-py313.lock:241 and executes the already installed exchange-calendars 4.13.2 for real-calendar regressions; no new installation is claimed." + ], + "path": "evidence/receipts/trading-unattended-hosting-recovery-20261005.json" + }, + { + "id": "trading-unattended-hosting-recovery-r2-20261005", + "kind": "native_cli_e2e", + "component_ids": [ + "dagu", + "restic" + ], + "claim": "Round 2 native restic 0.19.1 backed up two synthetic WAL journals using CPython Online Backup pages=-1 during independently observed commits, checked subset 1/1, restored 16531456 identical journal bytes against the external inventory and required state, and refused native-exit-3 snapshot and same-count/same-integrity content-mutation restore controls. Dagu 2.16.6 validated the revised DAG and executed native manual skip/success/stale-input probes. 42 stdlib tests passed including real exchange_calendars 4.13.2 cases. Cron Next was not executable: no Go compiler found. Operating-host and off-host gates remain open.", + "limitations": [ + "Scratch local integration and synthetic fixtures, not an unchanged upstream test suite, real market/broker state or operating-host acceptance.", + "Per-journal Online Backup is consistent independently; no transaction across journals. Commits during the native call need not appear in its read snapshot; comparison uses the frozen snapshot oracle.", + "Native check subset 1/1 reads every pack in this small proof. Whole seven-part/seven-day rotation, daily jitter, fall-back and deadlines are behavioural stdlib tests; operator execution is still required.", + "No scheduler or systemd unit installed, enabled or started; the operating-host restart/reboot/missed-run and independent-alert gates remain user decision 2.", + "Native Dagu probes use manual start, not scheduler dispatch. Scheduler run_dags:false behavior still needs the future host drill; a succeeded run with skipped research steps cannot count.", + "Off-host recovery requires user decision 3: independent destination, key recovery and second-host consumer restore under a chosen/measured recovery-time objective.", + "No retention deletion, broker, live credentials/mode, model calls, commit, push or manifest pin move.", + "Python 3.11 minimum is source-verified and syntax-checked. Execution used 3.13.15; no separate 3.11 interpreter was exercised.", + "robfig/cron v3.0.1 Next program/module were prepared from tagged source, but go run exited 127 because the compiler was not found; expected DST values are not measured results." + ], + "path": "evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json" + }, + { + "id": "trading-unattended-hosting-recovery-r3-20261005", + "kind": "native_cli_e2e", + "component_ids": [ + "dagu", + "restic" + ], + "claim": "All seven new p2 repairs have red/green regressions; native Dagu executes the six deployed steps and fails unavailable-input runs; native restic verifies WAL recovery and isolated failing controls.", + "limitations": [ + "Local integration and synthetic fixtures, not an unchanged upstream test suite or operating-host acceptance.", + "The deployed DAG copy uses a fixed post-close guard clock and a synthetic historical LEAN event; the native trigger remains manual.", + "Restart identity/window guards are tested with mocks. No systemctl manager operation, fault or service deployment ran.", + "Scheduler dispatch with run_dags:false, reboot, missed slots and independent alerts remain user decision 2.", + "Off-host recovery remains user decision 3: independent destination, key recovery and second-host consumer restore.", + "The independent final T row at line 102 was reread; it retains restic/Online Backup and does not name Litestream.", + "No Go compiler was found in PATH; the historical cron Next probe exit 127 remains unmeasured across DST.", + "No separate Python 3.11 interpreter was found in PATH; execution used 3.13.15.", + "Per-journal snapshots are consistent separately; there is no cross-journal transaction.", + "Native recovery reads subset 1/1; whole seven-part rotation remains behavioral fixture evidence.", + "No retention deletion, broker, credentials, live mode, model calls, commit, push or pin move." + ], + "path": "evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json" } ], "files": [ @@ -14546,8 +14610,8 @@ }, { "path": "blueprints/us-equities/hosting/README.md", - "sha256": "77e0264bdcd7cf1dc700d22236f9f3ba3407b5e5f7a76df1d8883aac64992270", - "bytes": 7585 + "sha256": "b8957339e8e99f9711212bed7e41a509b2349703b595cfbeebe09ac8d5c02904", + "bytes": 32889 }, { "path": "blueprints/us-equities/hosting/backup/README.md", @@ -14576,24 +14640,94 @@ }, { "path": "blueprints/us-equities/hosting/config.yaml.example", - "sha256": "e411b4c53b531f4c1f44efd0d8c597b5229972bd163f3ba179a29daed4c14d9f", - "bytes": 369 + "sha256": "a8bfd48d297f783c40338eeea473180144e9ddf5a8a6fe6011010dd9e732a24b", + "bytes": 691 }, { "path": "blueprints/us-equities/hosting/dagu-equities.service.example", - "sha256": "b7e1814daa569c7b8484f600c13d98956ac3efb6d955e87a768591a38fa7d855", - "bytes": 441 + "sha256": "327f5500f6d02d96508c35a1e512d7adcb2018427067a286b51603bebdd6bd41", + "bytes": 863 + }, + { + "path": "blueprints/us-equities/hosting/drill_local_recovery.py", + "sha256": "2ea3cb7d176fbc561e23624d804df346e69df5a5c8ccd10b3485cd86b6a8e015", + "bytes": 12362 + }, + { + "path": "blueprints/us-equities/hosting/drill_process_restart.py", + "sha256": "5888fda35b878fc957340de5f2a8152e3f16eaa13e7b8513234e97c09465f001", + "bytes": 10995 }, { "path": "blueprints/us-equities/hosting/equity-research-evidence.yaml", - "sha256": "715f893dc329c40c6db6ffcbe23a308c5696e0ec791e49a4a38adfc627f0886d", - "bytes": 770 + "sha256": "6107eebb77ae8904ff6422b7e86f4c551dde36d35aa5d6e87b8f5d72b0c9397f", + "bytes": 2688 + }, + { + "path": "blueprints/us-equities/hosting/evidence/cron-next-probe-r2.json", + "sha256": "d397eb9d53692382de7e345783a8ac0585e3857f1fc6664b50fbf6d86ab5cc02", + "bytes": 2076 + }, + { + "path": "blueprints/us-equities/hosting/evidence/dagu-scheduler-comparison.json", + "sha256": "c15ac08f09f324a6c1106f167c21cb013bc6c58fefbb04a3092ec68cc8cf43c7", + "bytes": 12307 + }, + { + "path": "blueprints/us-equities/hosting/evidence/dagu-validation.json", + "sha256": "ca499670b89b1dfc19745e1e8e5338196fc03e31a53f53c9dd7fa9bace7feccc", + "bytes": 444 + }, + { + "path": "blueprints/us-equities/hosting/evidence/native-dagu-probes-r2.json", + "sha256": "86bf2cd9b2b6b698e4ea2de30d227b4e133a40d5de839e9fac92d821840b5f11", + "bytes": 21096 + }, + { + "path": "blueprints/us-equities/hosting/evidence/native-dagu-probes-r3.json", + "sha256": "8119d2a1f7d281ed6309f2a46e552538cfc5250fd0af2d7cdd500a8711b8d43b", + "bytes": 3255 + }, + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof-r2.json", + "sha256": "c80b455622e549c21e3a0df78ce300c0722182f529bcdae09ddf5ce5c25637b1", + "bytes": 10789 + }, + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof-r3.json", + "sha256": "1fb0eebddd33c15c880af008606301364bb62044a123492b86da96af00e83684", + "bytes": 11577 + }, + { + "path": "blueprints/us-equities/hosting/evidence/offline-proof.json", + "sha256": "1d2aea2c11578aab27fb20d6d549241cd9b0c1b7b709c9f5081025ff612a7ce4", + "bytes": 10383 + }, + { + "path": "blueprints/us-equities/hosting/evidence/upstream-glue-review-r2.json", + "sha256": "2f8c7896b7277264801fc4b701375c612574619a2b96190202396fb52b00e2e7", + "bytes": 15859 + }, + { + "path": "blueprints/us-equities/hosting/evidence/upstream-glue-review-r3.json", + "sha256": "79237a01bbe3d7c6997409e910894664aa0a41e8d36240689259e92a9673b5d2", + "bytes": 2790 + }, + { + "path": "blueprints/us-equities/hosting/journal_recovery.py", + "sha256": "ac9f179361b951eda229e8a9beb58c15ff03e4691aa0574263e5e9b5cabb2a7d", + "bytes": 24043 }, { "path": "blueprints/us-equities/hosting/receipt.json", "sha256": "a7311c35dfb25d6c8f5212435095c1bbc91cde407490a9d61bf895cd1fe9f6cc", "bytes": 4873 }, + { + "path": "blueprints/us-equities/hosting/session_day.py", + "sha256": "624de575e3e829c6086e696a0274dd7c2fc6d802b9343dac8ff8dda5943b4ffc", + "bytes": 3540 + }, { "path": "blueprints/us-equities/identity-readiness/README.md", "sha256": "2452c1c1b3639e490cae7c4b97177637f56d133116d2da1aedfc30ed9b0e7600", @@ -16704,6 +16838,11 @@ "sha256": "d7698470de9eb7ed30f0764f8b1b17acd8a926298b08630829e1b955ba177dfe", "bytes": 23344 }, + { + "path": "docs/decisions/2026-10-05-trading-unattended-hosting-recovery.md", + "sha256": "604998e8ff2290f91273ebe55c52165f7101cb956a00c56620c85e44fdf03a1d", + "bytes": 29820 + }, { "path": "docs/decisions/2026-10-05-typesafe-sdk-transport.md", "sha256": "d197ab297186fe55e6c83624c78a06ead264e00d66a7fe2d5e62cfc1b653dbd6", @@ -52054,6 +52193,21 @@ "sha256": "6c6edfde2da2c1292d36222372b62b2614b980337b72b399984a3a4b368cd37c", "bytes": 6386 }, + { + "path": "evidence/receipts/trading-unattended-hosting-recovery-20261005.json", + "sha256": "5162c8dfc2fa3ea8112589fe6f34ed9be7d403f0812a91c2334ee13b69f1a171", + "bytes": 9674 + }, + { + "path": "evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json", + "sha256": "46bcef63ef1ab2542f6c056da0bdd91c949f94e68e76881c690f0732196191c7", + "bytes": 30998 + }, + { + "path": "evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json", + "sha256": "4cb50ba3c449da96f241b5ed6b0c7aa51c71e98cce094ef73a319e15b45767c7", + "bytes": 16209 + }, { "path": "evidence/receipts/ultracode-token-routing-20260921.json", "sha256": "fcb3b15c9f4657f00b92a6c493b33b851136d416e0dbc0f37a8b67360bdedbcf", @@ -54324,6 +54478,11 @@ "sha256": "e60c417abd190f598a9ba1f248a1ea94ced9d1b2e13b265fd494266a9d427bca", "bytes": 121324 }, + { + "path": "tests/test_us_equities_hosting.py", + "sha256": "bd947c5a67c6265c3cae717dc0396ebe7fd1aa0919cdc5e89c8872b0c211f459", + "bytes": 55117 + }, { "path": "tests/test_validate.py", "sha256": "505b9e65352d8683e6f85848d6962836ea836e6d906151e9983587a1dbcb5021", diff --git a/tests/test_us_equities_hosting.py b/tests/test_us_equities_hosting.py new file mode 100644 index 000000000..eff5b65bb --- /dev/null +++ b/tests/test_us_equities_hosting.py @@ -0,0 +1,968 @@ +"""Stdlib contract and fault tests; native restic execution has a separate receipt. + +Calendar and restic doubles below test our integration, not upstream acceptance. +All mutable state belongs to a temporary directory; no service or broker is used. +""" + +from __future__ import annotations + +from contextlib import closing, contextmanager, nullcontext, redirect_stdout +from datetime import datetime, timedelta, timezone +from zoneinfo import ZoneInfo +import fcntl +import importlib.util +import io +import json +import os +from pathlib import Path +import re +import shlex +import shutil +import sqlite3 +import stat +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[1] +HERE = ROOT / "blueprints/us-equities/hosting" + + +def load(name): + spec = importlib.util.spec_from_file_location(name, HERE / f"{name}.py") + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +RECOVERY = load("journal_recovery") +SESSION = load("session_day") +RESTART = load("drill_process_restart") +DRILL = load("drill_local_recovery") + + +def calendar_python(): + """Use an already installed locked SDK; never resolve/install dependencies.""" + candidates = [sys.executable] + if os.environ.get("SDK_ENV"): + candidates.append(str(Path(os.environ["SDK_ENV"]) / "bin/python")) + for candidate in candidates: + try: + checked = subprocess.run( + [candidate, "-c", 'import exchange_calendars; from importlib.metadata import version; ' + 'assert version("exchange-calendars") == "4.13.2"'], + capture_output=True, timeout=30, check=False) + except OSError: + continue + if checked.returncode == 0: + return candidate + return None + + +CALENDAR_PYTHON = calendar_python() +DAGU = Path(os.environ.get("DAGU_BIN", Path.home() / ".local/share/codex-ecosystem/bin/dagu")) + + +@unittest.skipUnless(CALENDAR_PYTHON, "requires importable exchange-calendars 4.13.2 (or SDK_ENV)") +class RealCalendarTests(unittest.TestCase): + def test_holidays_and_early_closes_with_real_locked_calendar(self): + for day, expected in (("2026-01-01", "non_session"), ("2027-01-01", "non_session"), + ("2034-01-02", "non_session"), ("2026-11-26", "non_session"), + ("2026-11-27", "session"), ("2026-12-24", "session")): + with self.subTest(day=day): + result = subprocess.run( + [CALENDAR_PYTHON, str(HERE / "session_day.py"), "--at", day + "T16:30:00-05:00"], + capture_output=True, text=True, timeout=30, check=False) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout.strip(), expected) + + def test_truly_uncovered_date_is_a_hard_failure(self): + checked = subprocess.run( + [CALENDAR_PYTHON, "-c", 'import exchange_calendars as x; ' + 'from exchange_calendars.errors import DateOutOfBounds; ' + 'c=x.get_calendar("XNYS", start="2026-01-01", end="2026-12-31"); ' + 'c.is_session("2034-01-02")'], capture_output=True, text=True, timeout=30, check=False) + self.assertNotEqual(checked.returncode, 0) + self.assertIn("DateOutOfBounds", checked.stderr) + + +def native_dagu_probe(token, *, session_script=None): + """Native runner + original status, without UI/scheduler/services or network. + + Minimal reproduction of the deployed output/scalar-precondition/dependency + chain; Dagu@v2.16.6:internal/runtime/runner.go:1267-1281,1633-1648. + """ + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + root = Path(temporary).resolve() + marker = root / "ran" + config = root / "config.yaml" + config.write_text((HERE / "config.yaml.example").read_text()) + calendar_command = f"printf {token}" + if session_script is not None: + (root / "sdk").symlink_to(Path(CALENDAR_PYTHON).parent.parent, target_is_directory=True) + (root / "session_day.py").write_text(Path(session_script).read_text()) + events = root / "events.json" + if token != "missing_input": + events.write_text("[]") + instant = {"stale_input": "2026-11-26T22:00:00Z", + "future_input": "2026-11-27T21:31:00Z"}.get(token, "2026-11-27T19:00:00Z") + epoch = datetime.fromisoformat(instant).timestamp() + os.utime(events, (epoch, epoch)) + calendar_command = (f'"{root}/sdk/bin/{Path(CALENDAR_PYTHON).name}" "{root}/session_day.py" ' + f'--at 2026-11-27T21:30:00Z --events "{events}"') + dag = root / "native-chain.yaml" + dag.write_text(f'''type: graph +timeout_sec: 15 +artifacts: + enabled: false +steps: + - id: calendar_check + run: {json.dumps(calendar_command)} + output: SESSION_DAY + - id: nyse_session + preconditions: + - condition: '${{SESSION_DAY}}' + expected: session + run: 'true' + depends: [calendar_check] + - id: evidence + run: 'touch "{marker}"' + depends: [nyse_session] +''') + env = {"HOME": str(root), "PATH": "/usr/bin:/bin", "TMPDIR": str(root)} + version = subprocess.run([str(DAGU), "version"], env=env, cwd=root, + capture_output=True, text=True, timeout=15, check=False) + if version.returncode or "2.16.6" not in version.stdout: + raise AssertionError("native probe requires Dagu 2.16.6") + command = [str(DAGU), "start", "--context", "local", "--dagu-home", str(root), + "--config", str(config), str(dag)] + result = subprocess.run(command, env=env, cwd=root, capture_output=True, text=True, + timeout=30, check=False) + status_paths = list(root.rglob("status.jsonl")) + if len(status_paths) != 1: + raise AssertionError("one original native status record required") + status = RESTART.last_status(status_paths[0]) + proof = {"token": token, "version_exit": version.returncode, "version_output": version.stdout, + "command": "dagu start --context local --dagu-home --config /config.yaml /native-chain.yaml", + "exit": result.returncode, "output": result.stdout + result.stderr, + "native_status": status, "marker_created": marker.exists()} + return DRILL.sanitize(proof, root=root, lexical_root=root, binary=DAGU) + + +@unittest.skipUnless(DAGU.is_file(), "requires the installed Dagu 2.16.6 binary") +class NativeDaguTests(unittest.TestCase): + def check_probe(self, token, expected, marker): + proof = native_dagu_probe(token) + self.assertEqual(proof["exit"], 0, proof["output"]) + self.assertEqual(proof["native_status"]["status"], 4) + self.assertEqual({n["step"]["id"]: n["status"] for n in proof["native_status"]["nodes"]}, + {"calendar_check": 4, "nyse_session": expected, "evidence": expected}) + self.assertEqual(proof["marker_created"], marker) + return proof + + def test_native_non_session_skips_dependents(self): + self.check_probe("non_session", 5, False) + + def test_native_session_executes_success_chain(self): + self.check_probe("session", 4, True) + + def test_native_precondition_defense_in_depth_blocks_before_close(self): + # An exit-0 ineligible token must still block the research chain. + self.check_probe("before_close", 5, False) + + def test_native_probe_uses_its_private_root_when_tmpdir_is_unset(self): + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary, \ + mock.patch.dict(os.environ), mock.patch.object(tempfile, "tempdir", temporary): + os.environ.pop("TMPDIR", None) + proof = native_dagu_probe("session") + self.assertEqual(proof["exit"], 0, proof["output"]) + self.assertEqual(proof["native_status"]["status"], 4) + self.assertTrue(proof["marker_created"]) + + @unittest.skipUnless(CALENDAR_PYTHON, "requires the locked calendar SDK") + def test_native_probe_preserves_interpreter_basename_without_bin_python(self): + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + sdk = Path(temporary) / "fixture-sdk" + (sdk / "bin").mkdir(parents=True) + interpreter = sdk / "bin/python3" + interpreter.write_text('#!/bin/sh\nexec ' + shlex.quote(CALENDAR_PYTHON) + ' "$@"\n') + interpreter.chmod(0o700) + self.assertFalse((sdk / "bin/python").exists()) + # Keep the synthetic SDK inside the native probe's scrubbed root. + with mock.patch(__name__ + ".CALENDAR_PYTHON", str(interpreter)), \ + mock.patch.object(tempfile, "TemporaryDirectory", return_value=nullcontext(temporary)): + proof = native_dagu_probe("session", session_script=HERE / "session_day.py") + self.assertEqual(proof["exit"], 0, proof["output"]) + self.assertEqual(proof["native_status"]["status"], 4) + self.assertTrue(proof["marker_created"]) + + @unittest.skipUnless(CALENDAR_PYTHON, "requires the locked calendar SDK") + def test_native_unavailable_input_fails_run_and_blocks_dependents(self): + for token in ("missing_input", "stale_input", "future_input"): + with self.subTest(token=token): + proof = native_dagu_probe(token, session_script=HERE / "session_day.py") + self.assertNotEqual(proof["exit"], 0, proof["output"]) + self.assertEqual(proof["native_status"]["status"], 2) + self.assertEqual(proof["native_status"]["nodes"][0]["status"], 2) + self.assertIn(token, proof["output"]) + self.assertFalse(proof["marker_created"]) + + +class HostingContractTests(unittest.TestCase): + def test_coordinator_disabled_in_configuration(self): + config = (HERE / "config.yaml.example").read_text() + self.assertRegex(config, r"(?m)^coordinator:\n enabled: false$") + self.assertRegex(config, r"(?m)^ run_dags: false$") + for key in ("LEAN_RESULTS", "RESEARCH_WORKSPACE", "NATIVE_CODEX_HOME", "NATIVE_CODEX_BIN", + "NATIVE_CLAUDE_BIN", "NATIVE_RUNTIME_PATH", "SDK_OBSERVATION_DIR"): + self.assertRegex(config, rf"(?m)^ - {key}$") + + def test_unit_start_all_restart_and_job_environment_delivery(self): + unit = (HERE / "dagu-equities.service.example").read_text() + line = next(line for line in unit.splitlines() if line.startswith("ExecStart=")) + self.assertIn("=/usr/bin/env -i ", line) + self.assertIn("/path/to/dagu-2.16.6 start-all", line) + self.assertIn("--config /path/to/dagu-home/config.yaml", line) + self.assertRegex(unit, r"(?m)^Restart=on-failure$") + self.assertNotRegex(unit, r"(?m)^Environment=") + # env -i eliminates inherited Environment=; the private DAG explicitly owns all four. + dag = (HERE / "equity-research-evidence.yaml").read_text() + env_block = dag.split("\nenv:\n", 1)[1].split("\nworking_dir:", 1)[0] + for key in ("STACK_REPO", "SDK_ENV", "LEAN_EVENTS", "RESEARCH_OUTPUT"): + self.assertRegex(env_block, rf"(?m)^ {key}: /path/to/.+$") + + def test_schedule_name_timezone_and_skipping_dependency_chain(self): + dag = (HERE / "equity-research-evidence.yaml").read_text() + self.assertIn("tags: [nyse-post-close-evidence]", dag) + self.assertRegex(dag, r"(?m)^schedule: 'CRON_TZ=America/New_York 30 16 \* \* 1-5'$") + self.assertIn('"${SDK_ENV}/bin/python" "${STACK_REPO}/blueprints/us-equities/hosting/session_day.py"', dag) + self.assertIn("output: SESSION_DAY", dag) + self.assertIn("preconditions:\n - condition: '${SESSION_DAY}'\n expected: session", dag) + self.assertIn("depends: [calendar_check]", dag) + self.assertIn("depends: [nyse_session]", dag) + self.assertIn("depends: [prepare_output]", dag) + self.assertIn("order-events-${DAG_RUN_ID}.parquet", dag) + self.assertIn('--events "${LEAN_EVENTS}"', dag) + + def test_input_mtime_must_follow_actual_session_close(self): + calendar = mock.Mock() + calendar.is_session.return_value = True + calendar.session_close.return_value = datetime.fromisoformat("2026-11-27T18:00:00Z") + now = datetime.fromisoformat("2026-11-27T21:30:00Z") + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + events = Path(temporary) / "events.json" + self.assertEqual(SESSION.eligibility(calendar, now, events), "missing_input") + events.write_text("[]") + for instant, expected in (("2026-11-26T22:00:00Z", "stale_input"), + ("2026-11-27T18:01:00Z", "session"), + ("2026-11-27T21:31:00Z", "future_input")): + epoch = datetime.fromisoformat(instant).timestamp() + os.utime(events, (epoch, epoch)) + self.assertEqual(SESSION.eligibility(calendar, now, events), expected) + calendar.is_session.return_value = False + self.assertEqual(SESSION.eligibility(calendar, now, events), "non_session") + + def test_session_cli_fails_unavailable_inputs_but_keeps_calendar_skips(self): + calendar = mock.Mock() + calendar.is_session.return_value = True + calendar.session_close.return_value = datetime.fromisoformat("2026-11-27T18:00:00Z") + now = "2026-11-27T21:30:00Z" + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + events = Path(temporary) / "events.json" + for token, instant in (("missing_input", None), + ("stale_input", "2026-11-26T22:00:00Z"), + ("future_input", "2026-11-27T21:31:00Z")): + if instant: + events.write_text("[]") + epoch = datetime.fromisoformat(instant).timestamp() + os.utime(events, (epoch, epoch)) + output = io.StringIO() + with self.subTest(token=token), mock.patch.object(SESSION, "version", return_value="4.13.2"), \ + mock.patch.object(SESSION, "calendar_for", return_value=calendar), redirect_stdout(output): + self.assertEqual(SESSION.main(["--at", now, "--events", str(events)]), 1) + self.assertEqual(output.getvalue().strip(), token) + for token, at, is_session in (("non_session", now, False), + ("before_close", "2026-11-27T17:00:00Z", True)): + calendar.is_session.return_value = is_session + output = io.StringIO() + with self.subTest(token=token), mock.patch.object(SESSION, "version", return_value="4.13.2"), \ + mock.patch.object(SESSION, "calendar_for", return_value=calendar), redirect_stdout(output): + self.assertEqual(SESSION.main(["--at", at, "--events", str(events)]), 0) + self.assertEqual(output.getvalue().strip(), token) + + def test_wrong_calendar_version_fails_in_the_normal_step(self): + with mock.patch.object(SESSION, "version", return_value="0.0"): + with self.assertRaisesRegex(RuntimeError, "locked exchange-calendars"): + SESSION.main([]) + + def test_session_cli_enforces_the_1625_new_york_publication_cutoff(self): + calendar = mock.Mock() + calendar.is_session.return_value = True + calendar.session_close.return_value = datetime.fromisoformat("2026-10-05T20:00:00Z") + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + events = Path(temporary) / "events.json" + events.write_text("[]") + for instant, token, exit_code in (("2026-10-05T20:25:00Z", "session", 0), + ("2026-10-05T20:25:00.001Z", "late_input", 1), + ("2026-10-05T20:29:00Z", "late_input", 1)): + epoch = datetime.fromisoformat(instant).timestamp() + os.utime(events, (epoch, epoch)) + output = io.StringIO() + with self.subTest(instant=instant), mock.patch.object(SESSION, "version", return_value="4.13.2"), \ + mock.patch.object(SESSION, "calendar_for", return_value=calendar), redirect_stdout(output): + self.assertEqual(SESSION.main(["--at", "2026-10-05T20:30:00Z", "--events", str(events)]), exit_code) + self.assertEqual(output.getvalue().strip(), token) + + def test_session_check_uses_new_york_date_and_skips_a_non_session(self): + calendar = mock.Mock() + calendar.is_session.return_value = False + self.assertEqual(SESSION.eligibility(calendar, datetime.fromisoformat("2026-10-05T00:30:00Z")), + "non_session") + calendar.is_session.assert_called_once_with("2026-10-04") + calendar.session_close.assert_not_called() + + def test_actual_close_is_checked_and_early_close_waits_until_daily_slot(self): + calendar = mock.Mock() + calendar.is_session.return_value = True + calendar.session_close.return_value = datetime.fromisoformat("2026-11-27T18:00:00Z") + self.assertEqual(SESSION.eligibility(calendar, datetime.fromisoformat("2026-11-27T17:59:00Z")), + "before_close") + self.assertEqual(SESSION.eligibility(calendar, datetime.fromisoformat("2026-11-27T16:30:00-05:00")), + "session") + + def test_due_run_requires_slot_scheduler_and_success(self): + due = datetime.fromisoformat("2026-10-05T16:30:00-04:00") + native = {"name": "equity-research-evidence", "dagRunId": "fixture-run", "triggerType": 1, + "status": 4, "scheduleTime": "2026-10-05T20:30:00Z", + "nodes": [{"step": {"id": step}, "status": 4} for step in + ("calendar_check", "nyse_session", "prepare_output", "summarize", + "baseline_evidence", "catalog_evidence")]} + self.assertTrue(RESTART.due_run(native, due, {"fixture-run"})) + for change in ({"triggerType": 2}, {"status": 2}, {"scheduleTime": "2026-10-06T20:30:00Z"}, + {"name": "another-dag"}): + with self.subTest(change=change): + self.assertFalse(RESTART.due_run({**native, **change}, due, {"fixture-run"})) + skipped = json.loads(json.dumps(native)) + skipped["nodes"][1]["status"] = 5 + self.assertFalse(RESTART.due_run(skipped, due, {"fixture-run"})) + self.assertFalse(RESTART.due_run({**native, "nodes": []}, due, {"fixture-run"})) + + def test_scripts_use_python_311_compatible_syntax_and_native_digest(self): + import ast + self.assertTrue(hasattr(RECOVERY.hashlib, "file_digest")) + for name in ("journal_recovery", "session_day", "drill_process_restart", "drill_local_recovery"): + with self.subTest(script=name): + ast.parse((HERE / f"{name}.py").read_text(), feature_version=(3, 11)) + + def test_round2_receipt_names_published_base_and_hash_pinned_working_tree(self): + receipt = json.loads((ROOT / "evidence/receipts/trading-unattended-hosting-recovery-r2-20261005.json").read_text()) + self.assertTrue(receipt["source_git_commit"].startswith("4c897418f")) + self.assertEqual(receipt["source_revision"], + "working tree after round-2 repair; files pinned by integration_source_sha256") + self.assertTrue(receipt["integration_source_sha256"]) + + def test_historical_review_bytes_and_round3_artifact_custody(self): + import hashlib + review = HERE / "evidence/upstream-glue-review-r2.json" + expected = "2f8c7896b7277264801fc4b701375c612574619a2b96190202396fb52b00e2e7" + self.assertEqual(hashlib.sha256(review.read_bytes()).hexdigest(), expected) + self.assertEqual(review.stat().st_size, 15859) + for round_number in (2, 3): + receipt = json.loads((ROOT / f"evidence/receipts/trading-unattended-hosting-recovery-r{round_number}-20261005.json").read_text()) + relative = f"blueprints/us-equities/hosting/evidence/upstream-glue-review-r{round_number}.json" + artifact = next(a for a in receipt["artifacts"] if a["path"] == relative) + data = (ROOT / relative).read_bytes() + self.assertEqual(artifact["sha256"], hashlib.sha256(data).hexdigest()) + self.assertEqual(artifact["bytes"], len(data)) + + def test_round3_acceptance_receipt_is_discoverable_in_the_catalog(self): + relative = "evidence/receipts/trading-unattended-hosting-recovery-r3-20261005.json" + receipt = json.loads((ROOT / relative).read_text()) + catalog = json.loads((ROOT / "manifests/evidence.json").read_text()) + matches = [entry for entry in catalog["receipts"] if entry["id"] == receipt["id"]] + self.assertEqual(len(matches), 1) + entry = matches[0] + self.assertEqual(entry["path"], relative) + for field in ("kind", "component_ids", "claim", "limitations"): + self.assertEqual(entry[field], receipt[field]) + + +class DrillGuardTests(unittest.TestCase): + def restart_fixture(self, *, restart_at=None, kill_code=0, unit="dagu-equities.service", status_change=None, + config_text="permissions:\n run_dags: false\n"): + if not any(char in unit for char in "*?[]") and importlib.util.find_spec("yaml") is None: + self.skipTest("restart config check requires upstream PyYAML (recipe pins 6.0.3)") + due = datetime.fromisoformat("2026-10-05T16:30:00-04:00") + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + root = Path(temporary) + config = root / "config.yaml" + config.write_text(config_text) + history = root / "history" + history.mkdir() + binary = root / "dagu" + status = {"name": "equity-research-evidence", "dagRunId": "fixture-run", "triggerType": 1, + "status": 4, "scheduleTime": due.isoformat(), + "nodes": [{"step": {"id": step}, "status": 4} for step in + ("calendar_check", "nyse_session", "prepare_output", "summarize", + "baseline_evidence", "catalog_evidence")]} + status_path = history / "status.jsonl" + status_bytes = (json.dumps(status) + "\n").encode() + status_path.write_bytes(status_bytes) + self.status_fixture_sha256 = RECOVERY.hashlib.sha256(status_bytes).hexdigest() + native_open = Path.open + + @contextmanager + def changing_open(path, mode="r", *args, **kwargs): + with native_open(path, mode, *args, **kwargs) as stream: + yield stream + if path == status_path and mode in ("r", "rb"): + if status_change == "compact": + with native_open(path, "wb") as stream: + stream.write(b"{}\n") + elif status_change == "remove": + path.unlink() + before = {"ActiveState": "active", "Restart": "on-failure", "MainPID": "10", "NRestarts": "0", + "ExecStart": f"{binary} start-all --config {config}"} + after = {**before, "MainPID": "20", "NRestarts": "1"} + commands = [] + + def run(command, **kwargs): + commands.append(command) + return SimpleNamespace(returncode=kill_code if command[0] == "systemctl" else 0, + stdout="" if command[0] == "systemctl" else '[{"dagRunId":"fixture-run"}]') + + output = io.StringIO() + with mock.patch.object(RESTART, "datetime", wraps=datetime) as clock, \ + mock.patch.object(RESTART, "require_session"), \ + mock.patch.object(RESTART, "validate_process"), \ + mock.patch.object(RESTART, "unit_state", side_effect=[before, after]), \ + mock.patch.object(RESTART.subprocess, "run", side_effect=run), \ + mock.patch.object(RESTART.os, "kill") as kill, \ + mock.patch.object(RESTART.time, "monotonic", return_value=0), \ + (mock.patch.object(Path, "open", changing_open) if status_change else nullcontext()), \ + redirect_stdout(output): + clock.now.side_effect = [due - timedelta(minutes=5), due - timedelta(minutes=5), + restart_at or due - timedelta(minutes=4)] + result = RESTART.main(["--dagu-bin", str(binary), "--dagu-home", str(root), + "--config", str(config), "--dag-history", str(history), + "--due-at", due.isoformat(), "--unit", unit]) + return result, json.loads(output.getvalue()), commands, kill.called + + def test_restart_refuses_unit_patterns_before_fault_delivery(self): + for unit in ("dagu-*.service", "dagu-?.service", "dagu-[ab].service", "dagu-].service"): + with self.subTest(unit=unit): + result, observed, commands, pid_signal = self.restart_fixture(unit=unit) + self.assertEqual(result, 1, observed) + self.assertIn("glob", observed["reason"]) + self.assertEqual(commands, []) + self.assertFalse(pid_signal) + + def test_restart_permission_check_uses_the_active_yaml_value(self): + for config in ("# run_dags: false\npermissions:\n run_dags: true\n", + "inactive:\n run_dags: false\npermissions:\n run_dags: true\n", + "# run_dags: false\nhost: 127.0.0.1\n", + "description: |\n run_dags: false\npermissions:\n run_dags: true\n", + "permissions:\n run_dags: false\npermissionRunDags: true\n"): + with self.subTest(config=config): + result, observed, commands, pid_signal = self.restart_fixture(config_text=config) + self.assertEqual(result, 1, observed) + self.assertIn("read-only UI config", observed["reason"]) + self.assertEqual(commands, []) + self.assertFalse(pid_signal) + result, observed, _, _ = self.restart_fixture( + config_text="# run_dags: true\npermissions:\n run_dags: false\n") + self.assertEqual(result, 0, observed) + + def test_restart_fault_is_delivered_by_native_unit_addressed_kill(self): + result, observed, commands, pid_signal = self.restart_fixture() + self.assertEqual(result, 0, observed) + self.assertEqual(commands[0], ["systemctl", "--user", "kill", "--kill-whom=main", + "--signal=SIGKILL", "dagu-equities.service"]) + self.assertFalse(pid_signal) + + def test_restart_hashes_the_accepted_status_bytes_despite_compaction(self): + for change in ("compact", "remove"): + with self.subTest(change=change): + result, observed, commands, _ = self.restart_fixture(status_change=change) + self.assertEqual(result, 0, observed) + self.assertEqual(observed["native_status_sha256"], self.status_fixture_sha256) + + def test_restart_time_window_refuses_wrong_slot_past_close_and_short_reserve(self): + due = datetime.fromisoformat("2026-10-05T16:30:00-04:00") + RESTART.validate_window(due, due - timedelta(minutes=5), 900) + for slot, now, wait in ((due.replace(hour=15), due - timedelta(minutes=5), 900), + (due, due, 900), (due, due - timedelta(seconds=10), 900), + (due, due - timedelta(minutes=5), 400), + (due.replace(tzinfo=None), due - timedelta(minutes=5), 900), + (due, due - timedelta(minutes=5), 3601)): + with self.subTest(slot=slot, wait=wait), self.assertRaises(ValueError): + RESTART.validate_window(slot, now, wait) + + def test_restart_reserves_150_seconds_before_due_and_refuses_late_restart(self): + due = datetime.fromisoformat("2026-10-05T16:30:00-04:00") + for seconds in (20, 90, 149, 150): + with self.subTest(seconds=seconds), self.assertRaises(ValueError): + RESTART.validate_window(due, due - timedelta(seconds=seconds), 900) + RESTART.validate_window(due, due - timedelta(seconds=151), 900) + for restart_at in (due, due + timedelta(seconds=1)): + with self.subTest(restart_at=restart_at): + result, observed, commands, _ = self.restart_fixture(restart_at=restart_at) + self.assertEqual(result, 1, observed) + self.assertIn("at or after the due slot", observed["reason"]) + self.assertEqual(len(commands), 1) # No history polling after a missed slot. + + def test_restart_refuses_holiday_and_calendar_failure_before_signalling(self): + due = datetime.fromisoformat("2026-01-01T16:30:00-05:00") + for code, token in ((0, "non_session"), (0, "before_close"), (1, "")): + with mock.patch.object(RESTART.subprocess, "run", return_value=SimpleNamespace( + returncode=code, stdout=token)), mock.patch.object(RESTART.os, "kill") as kill: + with self.assertRaisesRegex(ValueError, "refused"): + RESTART.require_session(due) + kill.assert_not_called() + with mock.patch.object(RESTART, "validate_window"), mock.patch.object(RESTART.subprocess, "run", + return_value=SimpleNamespace(returncode=0, stdout="non_session")), \ + mock.patch.object(RESTART, "unit_state") as unit_state, mock.patch.object(RESTART.os, "kill") as kill, \ + redirect_stdout(io.StringIO()): + self.assertEqual(RESTART.main(["--dagu-bin", "unused", "--dagu-home", "unused", "--config", "unused", + "--dag-history", "unused", "--due-at", due.isoformat()]), 1) + unit_state.assert_not_called() + kill.assert_not_called() + + def test_restart_pid_command_and_executable_identity_guards(self): + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + root = Path(temporary) + binary = root / "dagu" + binary.touch() + process = root / "10" + process.mkdir() + (process / "cmdline").write_bytes(b"dagu\0start-all\0--config\0private-config\0") + (process / "exe").symlink_to(binary) + RESTART.validate_process(10, binary, root) + for pid, chosen in ((1, binary), (0, binary), (10, root / "different-binary")): + with self.subTest(pid=pid), self.assertRaises(ValueError): + RESTART.validate_process(pid, chosen, root) + (process / "cmdline").write_bytes(b"dagu\0server\0") + with self.assertRaises(ValueError): + RESTART.validate_process(10, binary, root) + + def test_symlinked_scratch_root_scrubs_both_forms_and_resolves_at_creation(self): + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + parent = Path(temporary) + actual = parent / "actual" + actual.mkdir() + link = parent / "linked" + link.symlink_to(actual, target_is_directory=True) + root, lexical = DRILL.scratch_directory(link) + self.assertEqual(root.parent, actual.resolve()) + self.assertEqual(lexical.parent, link) + binary = parent / "restic" + clean = DRILL.sanitize({"a": str(root / "restored"), "b": str(lexical / "stage"), + "binary": str(binary), "relative": "./journals/a.sqlite3"}, + root=root, lexical_root=lexical, binary=binary) + self.assertEqual(clean, {"a": "/restored", "b": "/stage", + "binary": "", "relative": "./journals/a.sqlite3"}) + with self.assertRaisesRegex(RECOVERY.RecoveryError, "absolute path"): + DRILL.sanitize({"leftover": str(parent / "not-in-root")}, root=root, + lexical_root=lexical, binary=binary) + for path in ("/unrecognized/path", "C:\\private\\file", "file:///private/db"): + with self.subTest(path=path), self.assertRaises(RECOVERY.RecoveryError): + DRILL.assert_sanitized({"leftover": path}) + + def test_output_refuses_unsanitized_report_before_write(self): + with tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) as temporary: + output = Path(temporary) / "published.json" + with mock.patch.object(DRILL, "drill", return_value={"leak": str(output)}), redirect_stdout(io.StringIO()): + self.assertEqual(DRILL.main(["--restic-bin", "unused", "--scratch-root", temporary, + "--output", str(output)]), 1) + self.assertFalse(output.exists()) + + +class FakeRestic: + """Synthetic native responses; only frozen staging files are copied for restore.""" + + def __init__(self, binary, repository, password, work, timeout=600): + self.repository = Path(repository).resolve() + self.work = Path(work).resolve() + self.password_file = Path(password).resolve() + self.commands = [] + self.fail_check = False + + def run(self, *args): + self.commands.append(args) + if args[0] == "version": + return "restic 0.19.1 compiled with fixture\n" + if args[0] == "backup": + self.stage = Path(args[1]) + if any(path.stat().st_mode & 0o777 == 0 for path in self.stage.rglob("*.sqlite3")): + raise RECOVERY.RecoveryError("restic backup refused (exit 3)") + return json.dumps({"message_type": "summary", "snapshot_id": "a" * 64}) + if args[0] == "restore": + shutil.copytree(self.stage, args[args.index("--target") + 1]) + if args[0] == "check" and self.fail_check: + raise RECOVERY.RecoveryError("restic check refused (exit 1)") + return "" + + +class JournalRecoveryTests(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory(dir=os.environ.get("TMPDIR")) + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.sources = {} + for name, table in (("research", "events"), ("evidence", "runs")): + path = self.root / f"{name}.sqlite3" + db = sqlite3.connect(path, isolation_level=None) + self.addCleanup(db.close) + db.execute("PRAGMA journal_mode=WAL") + db.execute(f"CREATE TABLE {table}(id INTEGER PRIMARY KEY, value TEXT)") + db.execute(f"INSERT INTO {table}(value) VALUES ('committed in WAL')") + self.sources[name] = (path, [table]) + self.stage = self.root / "stage" + self.oracle = self.root / "frozen.json" + (self.root / "repository").mkdir() + (self.root / "unused-password-file").touch() + + def snapshot(self): + return RECOVERY.snapshot_journals(self.sources, self.stage, self.oracle) + + def restore_fixture(self): + self.snapshot() + restored = self.root / "restored" + shutil.copytree(self.stage, restored) + return restored + + def runner(self): + return FakeRestic("unused", self.root / "repository", self.root / "unused-password-file", self.root) + + def test_online_backup_includes_committed_wal_state_and_freezes_hashes(self): + inventory = self.snapshot() + self.assertEqual(len(inventory["files"]), 2) + self.assertEqual((self.stage / "inventory.json").read_bytes(), self.oracle.read_bytes()) + self.assertEqual(RECOVERY.verify(self.stage, self.oracle)["matched_files"], 2) + for record in inventory["files"]: + self.assertEqual(list(record["required_tables"].values()), [1]) + self.assertRegex(record["sha256"], "^[a-f0-9]{64}$") + self.assertGreater(record["bytes"], 0) + self.assertEqual(record["integrity_check"], "ok") + self.assertFalse(list(self.stage.rglob("*-wal"))) + + def test_frozen_inventory_syncs_final_mode_and_parent_directory(self): + synced = [] + native_fsync = os.fsync + + def observe(fd): + status = os.fstat(fd) + synced.append(("directory" if stat.S_ISDIR(status.st_mode) else "file", + stat.S_IMODE(status.st_mode))) + native_fsync(fd) + + with mock.patch.object(RECOVERY.os, "fsync", side_effect=observe): + RECOVERY.write_new(self.oracle, b"frozen inventory\n") + self.assertEqual(self.oracle.read_bytes(), b"frozen inventory\n") + self.assertEqual([kind for kind, mode in synced], ["file", "directory"]) + self.assertEqual(synced[0][1], 0o400) + + def test_snapshot_calls_backup_api_and_does_not_copy_source_files(self): + import ast + tree = ast.parse((HERE / "journal_recovery.py").read_text()) + calls = [node.func.attr for node in ast.walk(tree) if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute)] + self.assertIn("backup", calls) + self.assertNotIn("copyfile", calls) + self.assertNotIn("copy2", calls) + backups = [node for node in ast.walk(tree) if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) and node.func.attr == "backup"] + self.assertEqual(len(backups), 1) + self.assertEqual(ast.literal_eval(next(k.value for k in backups[0].keywords if k.arg == "pages")), -1) + + def test_online_backup_size_and_hard_time_limits_refuse_before_freezing(self): + with self.assertRaisesRegex(RECOVERY.RecoveryError, "max-journal-bytes"): + RECOVERY.snapshot_journals(self.sources, self.stage, self.oracle, max_journal_bytes=1) + self.assertFalse(self.oracle.exists()) + with mock.patch.object(RECOVERY.subprocess, "run", side_effect=subprocess.TimeoutExpired("native backup", 1)): + with self.assertRaises(subprocess.TimeoutExpired): + RECOVERY.snapshot_journals(self.sources, self.root / "timeout-stage", self.oracle, + backup_timeout=1) + self.assertFalse(self.oracle.exists()) + + def test_snapshot_cli_refuses_rollback_sources_without_explicit_writer_blocking_opt_in(self): + source = self.root / "rollback.sqlite3" + with closing(sqlite3.connect(source)) as db: + self.assertEqual(db.execute("PRAGMA journal_mode").fetchone()[0], "delete") + db.execute("CREATE TABLE events(id INTEGER PRIMARY KEY)") + db.execute("INSERT INTO events VALUES (1)") + db.commit() + args = ["snapshot", "--journal", f"rollback={source}", "--required-table", "rollback=events"] + output = io.StringIO() + with redirect_stdout(output): + result = RECOVERY.main(args + ["--stage", str(self.stage), "--inventory", str(self.oracle)]) + self.assertEqual(result, 1, output.getvalue()) + self.assertIn("WAL", json.loads(output.getvalue())["reason"]) + self.assertFalse(self.oracle.exists()) + output = io.StringIO() + with redirect_stdout(output): + result = RECOVERY.main(args + ["--stage", str(self.root / "opt-in-stage"), + "--inventory", str(self.root / "opt-in-inventory.json"), + "--allow-non-wal"]) + self.assertEqual(result, 0, output.getvalue()) + self.assertEqual(json.loads(output.getvalue())["files"][0]["required_tables"], {"events": 1}) + + def test_missing_required_table_prevents_freezing(self): + path = self.sources["research"][0] + with self.assertRaises(sqlite3.Error): + RECOVERY.snapshot_journals({"research": (path, ["missing_table"])}, self.stage, self.oracle) + self.assertFalse(self.oracle.exists()) + + def test_empty_selection_and_unreviewed_tables_refused(self): + for selection in ({}, {"research": (self.sources["research"][0], [])}): + with self.subTest(selection=list(selection)): + with self.assertRaises(RECOVERY.RecoveryError): + RECOVERY.snapshot_journals(selection, self.stage, self.oracle) + + def test_hash_rejects_changed_contents_even_when_integrity_and_counts_match(self): + restored = self.restore_fixture() + journal = restored / "journals/research.sqlite3" + journal.chmod(0o600) + with closing(sqlite3.connect(journal)) as db: + with db: + db.execute("UPDATE events SET value='changed but complete schema'") + self.assertEqual(RECOVERY.journal_state(journal, ["events"]), + {"integrity_check": "ok", "required_tables": {"events": 1}}) + with self.assertRaisesRegex(RECOVERY.RecoveryError, "sha256/bytes"): + RECOVERY.verify(restored, self.oracle) + + def test_missing_and_extra_files_refused(self): + restored = self.restore_fixture() + extra = restored / "unselected.txt" + extra.touch() + with self.assertRaisesRegex(RECOVERY.RecoveryError, "inventory mismatch"): + RECOVERY.verify(restored, self.oracle) + extra.unlink() + (restored / "journals/evidence.sqlite3").unlink() + with self.assertRaisesRegex(RECOVERY.RecoveryError, "inventory mismatch"): + RECOVERY.verify(restored, self.oracle) + + def test_restore_cannot_substitute_its_own_inventory(self): + restored = self.restore_fixture() + (restored / "inventory.json").chmod(0o600) + (restored / "inventory.json").write_text('{"schema_version": 1, "files": []}\n') + with self.assertRaisesRegex(RECOVERY.RecoveryError, "external frozen oracle"): + RECOVERY.verify(restored, self.oracle) + + def test_symlink_is_refused(self): + restored = self.restore_fixture() + (restored / "linked").symlink_to(self.oracle) + with self.assertRaisesRegex(RECOVERY.RecoveryError, "symlink"): + RECOVERY.verify(restored, self.oracle) + + def test_row_count_is_a_secondary_required_state_check(self): + restored = self.restore_fixture() + inventory = json.loads(self.oracle.read_text()) + inventory["files"][0]["required_tables"]["runs"] = 2 + for path in (self.oracle, restored / "inventory.json"): + path.chmod(0o600) + path.write_bytes(RECOVERY.json_bytes(inventory)) + with self.assertRaisesRegex(RECOVERY.RecoveryError, "required journal state"): + RECOVERY.verify(restored, self.oracle) + + def test_every_nonzero_native_exit_including_three_is_refused(self): + runner = RECOVERY.Restic(Path("unused"), self.root / "repository", + self.root / "unused-password-file", self.root) + for code in (1, 2, 3, 10, 11, 12, 130): + with self.subTest(exit=code): + with mock.patch.object(RECOVERY.subprocess, "run", return_value=SimpleNamespace( + returncode=code, stdout="synthetic incomplete snapshot\n", stderr="")): + with self.assertRaisesRegex(RECOVERY.RecoveryError, f"exit {code}"): + runner.run("backup", "stage") + + def test_rotation_reads_every_default_subset_within_seven_days(self): + runner = self.runner() + state = self.root / "rotation.json" + first = datetime(2026, 10, 5, tzinfo=timezone.utc) + observed = [RECOVERY.check_rotation(runner, state, now=first + timedelta(days=i)) for i in range(7)] + self.assertEqual(observed, [f"{i}/7" for i in range(1, 8)]) + self.assertEqual(json.loads(state.read_text())["next"], 1) + self.assertEqual([args[1] for args in runner.commands], + [f"--read-data-subset={i}/7" for i in range(1, 8)]) + + def test_failed_check_does_not_advance_rotation(self): + runner = self.runner() + state = self.root / "rotation.json" + now = datetime.now(timezone.utc) + RECOVERY.check_rotation(runner, state, now=now) + before = state.read_bytes() + runner.fail_check = True + with self.assertRaises(RECOVERY.RecoveryError): + RECOVERY.check_rotation(runner, state, now=now + timedelta(hours=1)) + self.assertEqual(state.read_bytes(), before) + # Native flock is released even on failure; the inode deliberately remains. + self.assertEqual(RECOVERY.check_rotation(self.runner(), state, now=now + timedelta(hours=2)), "2/7") + + def test_missed_rotation_interval_requires_successful_full_read(self): + runner = self.runner() + state = self.root / "rotation.json" + now = datetime.now(timezone.utc) + RECOVERY.check_rotation(runner, state, now=now) + later = now + timedelta(days=8, seconds=1) + with self.assertRaisesRegex(RECOVERY.RecoveryError, "overdue"): + RECOVERY.check_rotation(runner, state, now=later) + self.assertEqual(RECOVERY.check_rotation(runner, state, now=later, full_check=True), "1/7") + self.assertIn(("check", "--read-data"), runner.commands) + + def test_completed_rotation_keeps_the_previous_deadline_at_rollover(self): + first = datetime(2026, 10, 5, tzinfo=timezone.utc) + for gap in (timedelta(days=8, seconds=1), timedelta(days=12)): + with self.subTest(gap=gap): + runner = self.runner() + state = self.root / f"rollover-{gap.total_seconds()}.json" + for i in range(7): + RECOVERY.check_rotation(runner, state, now=first + timedelta(days=i)) + before = state.read_bytes() + later = first + gap + with self.assertRaisesRegex(RECOVERY.RecoveryError, "overdue"): + RECOVERY.check_rotation(runner, state, now=later) + self.assertEqual(state.read_bytes(), before) + self.assertEqual(len(runner.commands), 7) + self.assertEqual(RECOVERY.check_rotation(runner, state, now=later, full_check=True), "1/7") + self.assertIn(("check", "--read-data"), runner.commands) + + def test_rotation_allows_daily_jitter_through_rollover(self): + runner = self.runner() + state = self.root / "jitter.json" + first = datetime(2026, 10, 26, 20, 30, tzinfo=timezone.utc) + times = [first + i * timedelta(hours=24, seconds=60) for i in range(14)] + observed = [RECOVERY.check_rotation(runner, state, now=instant) for instant in times] + self.assertEqual(observed, [f"{i}/7" for i in range(1, 8)] * 2) + self.assertNotIn(("check", "--read-data"), runner.commands) + self.assertEqual(json.loads(state.read_text())["cycle_started_at"], times[7].isoformat()) + + def test_daily_new_york_rotation_survives_fall_back_rollover(self): + runner = self.runner() + state = self.root / "dst.json" + times = [datetime(2026, 10, 29, 16, 30, tzinfo=ZoneInfo("America/New_York")) + timedelta(days=i) + for i in range(8)] + self.assertEqual(times[3].astimezone(timezone.utc) - times[2].astimezone(timezone.utc), timedelta(hours=25)) + self.assertEqual(times[7].astimezone(timezone.utc) - times[0].astimezone(timezone.utc), + timedelta(days=7, hours=1)) + observed = [RECOVERY.check_rotation(runner, state, now=instant) for instant in times] + self.assertEqual(observed, [f"{i}/7" for i in range(1, 8)] + ["1/7"]) + self.assertNotIn(("check", "--read-data"), runner.commands) + self.assertEqual(json.loads(state.read_text())["cycle_started_at"], times[7].astimezone(timezone.utc).isoformat()) + + def test_whole_rotation_deadline_refuses_even_without_large_individual_gaps(self): + runner = self.runner() + state = self.root / "slow.json" + first = datetime(2026, 10, 5, tzinfo=timezone.utc) + for i in range(6): + RECOVERY.check_rotation(runner, state, now=first + i * timedelta(hours=33)) + before = state.read_bytes() + with self.assertRaisesRegex(RECOVERY.RecoveryError, "overdue"): + RECOVERY.check_rotation(runner, state, now=first + 6 * timedelta(hours=33)) + self.assertEqual(state.read_bytes(), before) + + def test_cycle_clock_is_captured_before_snapshot_and_checks_native_completion(self): + runner = self.runner() + with mock.patch.object(RECOVERY, "check_rotation", return_value="1/7") as rotation: + RECOVERY.cycle(runner, self.sources, self.root / "time.json") + self.assertLessEqual(rotation.call_args.kwargs["now"], rotation.call_args.kwargs["clock"]()) + backup = next(args for args in runner.commands if args[0] == "backup") + self.assertEqual(backup[backup.index("--group-by") + 1], "host,tags") + self.assertIn("--group-by host,tags", (HERE / "README.md").read_text()) + + def test_slow_native_check_cannot_publish_state_after_cycle_deadline(self): + runner = self.runner() + first = datetime(2026, 10, 5, tzinfo=timezone.utc) + state = self.root / "slow-native.json" + RECOVERY.check_rotation(runner, state, now=first) + before = state.read_bytes() + with self.assertRaisesRegex(RECOVERY.RecoveryError, "after native check"): + RECOVERY.check_rotation(runner, state, now=first + timedelta(days=6), + clock=lambda: first + timedelta(days=8, seconds=1)) + self.assertEqual(state.read_bytes(), before) + + def test_native_lock_contention_refuses_and_stale_lock_after_exit_is_safe(self): + state = self.root / "locked.json" + lock = state.with_suffix(".json.lock") + command = [sys.executable, "-c", 'import fcntl,sys; f=open(sys.argv[1],"a"); ' + 'fcntl.flock(f,fcntl.LOCK_EX); print("locked",flush=True); sys.stdin.read()', str(lock)] + holder = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True) + try: + self.assertEqual(holder.stdout.readline().strip(), "locked") + with self.assertRaisesRegex(RECOVERY.RecoveryError, "held by another"): + RECOVERY.check_rotation(self.runner(), state) + self.assertFalse(state.exists()) + finally: + holder.communicate(input="", timeout=10) + self.assertEqual(holder.returncode, 0) + self.assertTrue(lock.exists()) + lock.write_text("stale owner information is not a live native lock") + self.assertEqual(RECOVERY.check_rotation(self.runner(), state), "1/7") + + def test_stale_atomic_publication_file_does_not_reset_or_block_cursor(self): + state = self.root / "atomic.json" + RECOVERY.check_rotation(self.runner(), state) + stale = state.with_suffix(".json.new") + stale.write_text("incomplete old write") + self.assertEqual(RECOVERY.check_rotation(self.runner(), state), "2/7") + self.assertEqual(stale.read_text(), "incomplete old write") + + def test_rotation_refuses_other_repository_and_invalid_part_counts(self): + runner = self.runner() + state = self.root / "rotation.json" + RECOVERY.check_rotation(runner, state) + runner.repository = self.root / "other-repository" + with self.assertRaisesRegex(RECOVERY.RecoveryError, "another repository"): + RECOVERY.check_rotation(runner, state) + for parts in (0, 8): + with self.assertRaises(RECOVERY.RecoveryError): + RECOVERY.check_rotation(runner, self.root / "new-rotation", parts=parts) + + def test_control_backups_use_groups_distinct_from_production_retention(self): + for control in (None, "snapshot", "restore"): + label = control or "production" + work = self.root / f"group-{label}" + work.mkdir() + runner = FakeRestic("unused", self.root / "repository", self.root / "unused-password-file", work) + with self.subTest(control=control): + if control: + with self.assertRaises(RECOVERY.RecoveryError): + RECOVERY.cycle(runner, self.sources, self.root / f"{label}-state.json", parts=1, control=control) + else: + RECOVERY.cycle(runner, self.sources, self.root / f"{label}-state.json", parts=1) + backup = next(args for args in runner.commands if args[0] == "backup") + suffix = "-control" if control else "" + self.assertEqual(backup[backup.index("--host") + 1], "equity-research-recovery" + suffix) + self.assertEqual(backup[backup.index("--tag") + 1], "journal-recovery" + suffix) + self.assertEqual(backup[backup.index("--group-by") + 1], "host,tags") + + def test_both_controls_make_the_procedure_exit_nonzero(self): + for control in ("snapshot", "restore"): + work = self.root / control + argv = ["cycle", "--restic-bin", "unused", "--repository", str(self.root / "repository"), + "--password-file", str(self.root / "unused-password-file"), "--work", str(work), + "--rotation-state", str(self.root / f"{control}-rotation.json"), "--control", control] + for name, (path, tables) in self.sources.items(): + argv.extend(["--journal", f"{name}={path}", "--required-table", f"{name}={tables[0]}"]) + with self.subTest(control=control), mock.patch.object(RECOVERY, "Restic", FakeRestic): + output = io.StringIO() + with redirect_stdout(output): + self.assertEqual(RECOVERY.main(argv), 1) + result = json.loads(output.getvalue()) + self.assertEqual(result["result"], "refused") + self.assertIn("exit 3" if control == "snapshot" else "sha256/bytes mismatch", result["reason"]) + if control == "snapshot": + self.assertFalse((work / "restored").exists()) + else: + mutation = json.loads((work / "restore-control.json").read_text()) + self.assertTrue(mutation["required_state_unchanged"]) + self.assertTrue(mutation["bytes_unchanged"]) + self.assertTrue(mutation["sha256_changed"]) + + +if __name__ == "__main__": + unittest.main()