Conversation
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
nv-yna
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
September 12, 2026 20:36
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #428 +/- ##
=======================================
Coverage ? 75.38%
=======================================
Files ? 110
Lines ? 16207
Branches ? 0
=======================================
Hits ? 12218
Misses ? 3989
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
|
Closing with the 2.0 merge (#407, part 18): tachometer, every worker, frontend, and service are named Slurm steps, and cleanup delivers SIGTERM with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tachometer shutdown currently sends SIGTERM to the local
srunclient. Slurm25.11.8 aborts the remote task with SIGKILL in response, so the scraper cannot flush and compact its capture. SRT then reports the launcher as having stopped gracefully without checking its exit code. This occurred in Hecate run579709: the scraper step ended0:9 and produced no final artifact.Record the scraper's exact Slurm job.step in a private file unique to its launch, then exec the scraper with its original arguments. A Tachometer-specific process class signals that recorded step, keeps
srunalive for the configured compaction grace, and checks the launcher exit code. Timeout escalation targets only the same step. Missing or invalid identity and signal-command failures are reported without falling back to launcher or allocation signals. Registry cleanup uses the same shutdown behavior.Validation:
make check: 2,178 passed, 2 skipped, 6 deselected. Final focused suite: 154 passed, including the added production-registration regression. Ruff and whitespace checks passed.TachometerStep.commandandTachometerProcess.terminateAPI. Step579710.37 completed0:0, and seven8-row shards reconciled to56 fully decoded rows in byte-identical local/uploaded final files.The cluster tests validate the signal mechanism and bounded scraper finalization; they do not certify production-scale compaction or resolve the separate Rust task-quiescence risk. Atomic shard publication is addressed independently by #427. Related open PR #407 by @ishandhanani, "[2.0] The full 2.0 stack: schema v2, services, graceful shutdown, observability, MCP, and the migrator (#386-#406 and after)", also introduces step signaling within its broader v2/services/cleanup migration, using name lookup and a launcher fallback. This patch keeps the main-line fix limited to Tachometer with an exact launch identity; it does not change #407.