P3-01: the Stage-2 supervised dataset + the pinned RiNALMo tokenizer adapter - #93
Merged
Merged
Conversation
…o tokenizer adapter (P3-01) Emits data/processed/stage2_dataset.parquet (DVC): 30,542 rows x 34 columns — 23,535 T-box positives + 7,007 unmasked §9.1 decoys — as RNA (T->U), each with its binary label, per-nucleotide boundary target, §8 aux targets, a dot-bracket pairing target for the P3-05 structure-consistency loss, and the ADR-0004 fold/parentage columns. Inputs (LOCAL, ~8 s, no GPU/network): master_clean_v0.parquet, labels_v0.parquet, the git-LFS split_assignments.parquet, decoys_v0.parquet. Three load-bearing choices, all stated in the module docstring: * flank_nt = 0 for BOTH classes and refused otherwise. The named inputs supply real genomic flank for positives only; a positives-only flank makes "sits in real genomic context" a separable shortcut for the binary head (PRD §5). * Folds inherited, never invented (fold_basis). A positive copies its own split row; a dinucleotide decoy inherits its parent's fold across every scheme (ADR-0004 D7, fail-closed on a missing parent); the 5,007 parentless decoys belong to no cluster or clade and are kept random-split-only with null clade columns and a null nested_train — the ADR-0004 D4 pattern extended to a class D4 does not name, flagged for sign-off at the P3-exit gate. * The pairing target is projected and measured. TBDB Structure is aligned to the gapped cmalign row, not the locus; requiring an exact UNIQUE anchor gives 18,269/23,535 (77.62 %) with zero no-hit and zero multi-hit rows. The rest render unaligned inserts lossily and carry a null target, never a substitute. Structure appears only as a target, never a Stage-2 input (PRD §6). Rejecting rows whose consensus structure pairs into a deleted column was measured wrong (it discarded 8,634 more records for 2.5 % of pairs); such pairs are now dropped and counted in n_pairs_dropped_gap. The tokenizer adapter pins RiNALMo's 28-token alphabet in pure Python so the build runs in envs/data.yml (multimolecule is ml-rna-only, ADR-0002 A4/A8) and the golden stays CI-runnable; 1022 is derived from max_position_embeddings 1024 minus <cls>/<eos>, and the pin is asserted equal to the live RnaTokenizer at the ADR-0002 A9 revision 2a71f6f9. Validation: PASS. 54 passed / 2 skipped in tbox-data; the 2 skips are the parity legs, 23 passed / 0 skipped in tbox-ml-rna. Round-trip exact on 100 real T-box loci + the alphabet fixture; every row has a fold, a parent and a closed-set fold_basis; max 552 tokens. ruff + black clean; snakemake --lint/-n clean. Seven sabotage bites, each caught by its named test, restores proved by cmp.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (4)
📝 WalkthroughWalkthroughStage 2 adds a pinned RiNALMo tokenizer, WUSS structure projection, supervised positive and decoy dataset assembly, invariant checks, digest generation, CLI outputs, Snakemake integration, and unit and golden regression tests. ChangesStage 2 dataset pipeline
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Inputs
participant build_dataset
participant tokenizer
participant DatasetFrame
participant ParquetOutputs
Inputs->>build_dataset: corpus, labels, splits, and decoys
build_dataset->>tokenizer: validate and encode RNA loci
tokenizer-->>build_dataset: token lengths and token IDs
build_dataset->>DatasetFrame: assemble positive and decoy rows
DatasetFrame-->>build_dataset: validated dataset and report
build_dataset->>ParquetOutputs: write Parquet, provenance, and audit JSON
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/tbox_finder/stage2/dataset.py`:
- Line 498: Update the parent_record_id assignment in the dataset row
construction to use the existing _text helper instead of str, ensuring missing
parent values remain null and _assert_dataset_invariants can reject them through
its parent_record_id gate.
- Around line 336-343: Update the anchor-matching logic around probe and locus
in the stage2 dataset function to reject an empty probe as REJECT_NO_ANCHOR, and
detect all overlapping probe occurrences rather than using str.count. Return
REJECT_NO_ANCHOR for no matches, REJECT_MULTI_ANCHOR for multiple matches
including overlaps, and retain the unique match offset; add regression tests
covering an empty probe and overlapping matches.
In `@workflow/rules/stage2.smk`:
- Around line 28-60: Update the stage2_dataset rule to declare corpus, labels,
split_table, and decoys as named input dependencies using their existing config
defaults, remove those source declarations from params, and pass them to the
command via the corresponding {input.<name>:q} references. Leave output,
provenance, directory, and other parameter handling unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a1038a92-0adc-4d68-b132-953e748fdeb9
⛔ Files ignored due to path filters (5)
analyses/phase3_log.qmdis excluded by!**/*.qmddata/processed/.gitignoreis excluded by!data/**data/processed/audits/stage2_dataset_report.jsonis excluded by!data/**data/processed/stage2_dataset.parquet.dvcis excluded by!data/**data/processed/stage2_dataset.provenance.jsonis excluded by!data/**
📒 Files selected for processing (8)
src/tbox_finder/stage2/__init__.pysrc/tbox_finder/stage2/dataset.pysrc/tbox_finder/stage2/tokenizer.pytests/fixtures/stage2_dataset/expected.sha256tests/golden/test_stage2_dataset_golden.pytests/unit/test_stage2_dataset.pytests/unit/test_stage2_tokenizer.pyworkflow/rules/stage2.smk
Comment on lines
+336
to
+343
| probe = "".join(ungapped_chars).replace("U", "T") | ||
| locus = str(locus_dna).upper() | ||
| hits = locus.count(probe) | ||
| if hits == 0: | ||
| return None, REJECT_NO_ANCHOR, 0 | ||
| if hits > 1: | ||
| return None, REJECT_MULTI_ANCHOR, 0 | ||
| offset = locus.find(probe) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
python - <<'PY'
locus, probe = "ATATA", "ATA"
print("count:", locus.count(probe))
i, n = locus.find(probe), 0
while i != -1:
n += 1
i = locus.find(probe, i + 1)
print("overlapping occurrences:", n)
PYRepository: bioedca/tbox-finder
Length of output: 193
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target function ---'
sed -n '260,365p' src/tbox_finder/stage2/dataset.py
printf '%s\n' '--- related constants and tests ---'
rg -n -C 3 'REJECT_(NO_ANCHOR|MULTI_ANCHOR)|project_structure_to_locus|all-gap|aligned_sequence|aligned_structure' src tests
printf '%s\n' '--- deterministic probe ---'
python3 - <<'PY'
cases = [
("ATATA", "ATA"),
("AAAA", "AA"),
("ATGC", ""),
]
for locus, probe in cases:
positions = []
pos = locus.find(probe)
step = 1 if probe else 1
while pos != -1:
positions.append(pos)
pos = locus.find(probe, pos + step)
print({
"locus": locus,
"probe": probe,
"str_count": locus.count(probe),
"overlap_aware_positions": positions,
"find": locus.find(probe),
})
PYRepository: bioedca/tbox-finder
Length of output: 12886
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target function ---'
sed -n '260,365p' src/tbox_finder/stage2/dataset.py
printf '%s\n' '--- related constants and tests ---'
rg -n -C 3 'REJECT_(NO_ANCHOR|MULTI_ANCHOR)|project_structure_to_locus|all-gap|aligned_sequence|aligned_structure' src tests
printf '%s\n' '--- deterministic probe ---'
python3 - <<'PY'
cases = [
("ATATA", "ATA"),
("AAAA", "AA"),
("ATGC", ""),
]
for locus, probe in cases:
positions = []
pos = locus.find(probe)
while pos != -1:
positions.append(pos)
pos = locus.find(probe, pos + 1)
print({
"locus": locus,
"probe": probe,
"str_count": locus.count(probe),
"overlap_aware_positions": positions,
"find": locus.find(probe),
})
PYRepository: bioedca/tbox-finder
Length of output: 12886
Use an overlap-aware anchor check.
str.count misses overlapping matches. For example, locus="ATATA" and probe="ATA" produce one count but two matches. The function then places the structure at the first offset. Reject overlapping matches as REJECT_MULTI_ANCHOR. Return REJECT_NO_ANCHOR when probe is empty. Add regression tests for both cases.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 342-342: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: locus.find(probe)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/tbox_finder/stage2/dataset.py` around lines 336 - 343, Update the
anchor-matching logic around probe and locus in the stage2 dataset function to
reject an empty probe as REJECT_NO_ANCHOR, and detect all overlapping probe
occurrences rather than using str.count. Return REJECT_NO_ANCHOR for no matches,
REJECT_MULTI_ANCHOR for multiple matches including overlaps, and retain the
unique match offset; add regression tests covering an empty probe and
overlapping matches.
Comment on lines
+28
to
+60
| rule stage2_dataset: | ||
| """Assemble `data/processed/stage2_dataset.parquet` (P3-01; PRD §6/§8/§10.2/§11).""" | ||
| output: | ||
| dataset=f"{_STAGE2_PROCESSED_DIR}/stage2_dataset.parquet", | ||
| provenance=f"{_STAGE2_PROCESSED_DIR}/stage2_dataset.provenance.json", | ||
| report=f"{_STAGE2_AUDIT_DIR}/stage2_dataset_report.json", | ||
| params: | ||
| # inputs + dirs derived from the outputs (not hardcoded prefixes) so | ||
| # `snakemake --lint` stays clean; the module writes both sidecars itself. | ||
| corpus=config.get("stage2_corpus", "data/processed/master_clean_v0.parquet"), | ||
| labels=config.get("stage2_labels", "data/processed/labels/labels_v0.parquet"), | ||
| split_table=config.get( | ||
| "stage2_split_table", "data/processed/splits/split_assignments.parquet" | ||
| ), | ||
| decoys=config.get("stage2_decoys", "data/processed/negatives/decoys_v0.parquet"), | ||
| out_dir=lambda wildcards, output: os.path.dirname(output.dataset), | ||
| audit_dir=lambda wildcards, output: os.path.dirname(output.report), | ||
| flank_nt=config.get("stage2_flank_nt", 0), | ||
| env_lock="envs/data.conda-lock.yml", | ||
| log: | ||
| "logs/stage2_dataset.log", | ||
| conda: | ||
| "../../envs/data.yml" | ||
| shell: | ||
| "python -m tbox_finder.stage2.dataset " | ||
| "--corpus {params.corpus:q} " | ||
| "--labels {params.labels:q} " | ||
| "--split-table {params.split_table:q} " | ||
| "--decoys {params.decoys:q} " | ||
| "--out-dir {params.out_dir:q} " | ||
| "--audit-dir {params.audit_dir:q} " | ||
| "--flank-nt {params.flank_nt} " | ||
| "--env-lock {params.env_lock:q} >{log} 2>&1" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '28,60p' workflow/rules/stage2.smk
rg -n -C 2 --glob '*.yml' --glob '*.yaml' \
'stage2_(corpus|labels|split_table|decoys)' .Repository: bioedca/tbox-finder
Length of output: 1781
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- tracked files ---'
git ls-files | sed -n '1,240p'
printf '%s\n' '--- source path references ---'
rg -n -C 3 \
'master_clean_v0\.parquet|labels_v0\.parquet|split_assignments\.parquet|decoys_v0\.parquet|stage2_dataset|stage2_(corpus|labels|split_table|decoys)' \
--glob '!*.lock' --glob '!*.jsonl' .
printf '%s\n' '--- workflow rule declarations and input sections ---'
rg -n -C 4 \
'^(rule|checkpoint|use rule)|^[[:space:]]+input:|^[[:space:]]+output:' \
workflow Snakefile* 2>/dev/null || true
printf '%s\n' '--- tracking metadata ---'
find . -maxdepth 4 -type f \( -name '*.dvc' -o -name '.gitattributes' -o -name 'dvc.yaml' -o -name 'dvc.lock' \) -print \
-exec sh -c 'for f do echo "--- $f"; sed -n "1,180p" "$f"; done' sh {} +Repository: bioedca/tbox-finder
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- workflow files ---'
git ls-files workflow | sed -n '1,160p'
printf '%s\n' '--- config and tracking files ---'
git ls-files | rg '(^|/)([^/]+/)*(config|conf|envs|data)(/|$)|\.dvc$|(^|/)\.gitattributes$|(^|/)dvc\.(yaml|lock)$' | sed -n '1,240p'
printf '%s\n' '--- relevant workflow references ---'
rg -n -C 5 \
'master_clean_v0\.parquet|labels_v0\.parquet|split_assignments\.parquet|decoys_v0\.parquet|stage2_dataset|stage2_(corpus|labels|split_table|decoys)' \
workflow conf config data Snakefile 2>/dev/null || true
printf '%s\n' '--- tracking metadata files ---'
git ls-files | rg '(^|/)([^/]+/)*(\.gitattributes|[^/]+\.dvc|dvc\.yaml|dvc\.lock)$' |
while IFS= read -r f; do
echo "--- $f"
sed -n '1,180p' "$f"
done
printf '%s\n' '--- stage2-related source files ---'
git ls-files | rg 'stage2|dataset' | sed -n '1,160p'Repository: bioedca/tbox-finder
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- stage2 rule and upstream output ranges ---'
sed -n '1,75p' workflow/rules/stage2.smk
sed -n '65,100p;325,360p;700,745p;530,565p' workflow/rules/data.smk
printf '%s\n' '--- workflow inclusion order ---'
sed -n '1,180p' workflow/Snakefile
printf '%s\n' '--- git attributes and DVC metadata ---'
sed -n '1,160p' .gitattributes
for f in \
data/processed/master_clean_v0.parquet.dvc \
data/processed/labels/labels_v0.parquet.dvc \
data/processed/negatives/decoys_v0.parquet.dvc \
data/interim/splits/split_assignments.parquet.dvc
do
echo "--- $f"
sed -n '1,40p' "$f"
done
printf '%s\n' '--- read-only structural verifier ---'
python3 - <<'PY'
from pathlib import Path
import re
stage2 = Path("workflow/rules/stage2.smk").read_text()
data = Path("workflow/rules/data.smk").read_text()
sources = {
"corpus": "data/processed/master_clean_v0.parquet",
"labels": "data/processed/labels/labels_v0.parquet",
"split_table": "data/processed/splits/split_assignments.parquet",
"decoys": "data/processed/negatives/decoys_v0.parquet",
}
print("stage2 input section:", bool(re.search(r"rule stage2_dataset:.*?\n input:", stage2, re.S)))
for name, path in sources.items():
print(f"{name}: param={bool(re.search(rf'^[ \t]+{name}=config\\.get', stage2, re.M))}, "
f"source_path_present={path in stage2}, "
f"input_ref={bool(re.search(rf'\\{{input\\.{name}:q\\}}', stage2))}")
for path in sources.values():
print(f"data.smk output for {path}:",
bool(re.search(rf'^[ \t]+(?:processed|labels|table|decoys)?=?[\"\\']?{re.escape(path)}[\"\\']?', data, re.M))
or path in data)
for dvc in (
"data/processed/master_clean_v0.parquet.dvc",
"data/processed/labels/labels_v0.parquet.dvc",
"data/processed/negatives/decoys_v0.parquet.dvc",
"data/interim/splits/split_assignments.parquet.dvc",
):
print(f"{dvc}: exists={Path(dvc).is_file()}")
PYRepository: bioedca/tbox-finder
Length of output: 14180
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
stage2_text = Path("workflow/rules/stage2.smk").read_text()
data_text = Path("workflow/rules/data.smk").read_text()
attributes = Path(".gitattributes").read_text()
start = stage2_text.index("rule stage2_dataset:")
rule_text = stage2_text[start:]
sources = {
"corpus": "data/processed/master_clean_v0.parquet",
"labels": "data/processed/labels/labels_v0.parquet",
"split_table": "data/processed/splits/split_assignments.parquet",
"decoys": "data/processed/negatives/decoys_v0.parquet",
}
print("stage2 has input section:",
bool(re.search(r"^[ \t]+input:", rule_text, re.M)))
for name, path in sources.items():
param_pattern = rf"^[ \t]+{re.escape(name)}=config\.get"
param_ref = "{params." + name + ":q}"
input_ref = "{input." + name + ":q}"
print(
name,
"param_declaration=", bool(re.search(param_pattern, rule_text, re.M)),
"params_shell_ref=", param_ref in rule_text,
"input_shell_ref=", input_ref in rule_text,
"source_path_in_rule=", path in rule_text,
)
upstream_markers = {
"corpus": 'processed="data/processed/master_clean_v0.parquet"',
"labels": 'labels=f"{_LABELS_DIR}/labels_v0.parquet"',
"split_table": 'table=f"{_PROCESSED_SPLITS_DIR}/split_assignments.parquet"',
"decoys": 'decoys=f"{_NEG_DIR}/decoys_v0.parquet"',
}
for name, marker in upstream_markers.items():
print("matching upstream output", name, "=", marker in data_text)
for path in (
"data/processed/master_clean_v0.parquet.dvc",
"data/processed/labels/labels_v0.parquet.dvc",
"data/processed/negatives/decoys_v0.parquet.dvc",
):
print("DVC sidecar", path, "=", Path(path).is_file())
print(
"split table Git-LFS attribute =",
"data/processed/splits/split_assignments.parquet filter=lfs" in attributes,
)
print(
"stage2 provenance records source hashes =",
all(path in Path("data/processed/stage2_dataset.provenance.json").read_text()
for path in sources.values()),
)
PYRepository: bioedca/tbox-finder
Length of output: 1042
Declare the source files as input dependencies.
stage2_dataset currently tracks the four DVC/Git-LFS sources only as params, so Snakemake cannot detect updated content or connect the matching upstream rules. Move corpus, labels, split_table, and decoys into named input entries and pass them through {input.<name>:q}. The provenance hashes do not affect Snakemake scheduling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@workflow/rules/stage2.smk` around lines 28 - 60, Update the stage2_dataset
rule to declare corpus, labels, split_table, and decoys as named input
dependencies using their existing config defaults, remove those source
declarations from params, and pass them to the command via the corresponding
{input.<name>:q} references. Leave output, provenance, directory, and other
parameter handling unchanged.
…stringified parent link
2 of 3 findings applied, 1 skipped with a tested reason.
* (major, applied) The pairing-target uniqueness check used `str.count`, which
counts NON-overlapping occurrences ("AAA".count("AA") == 1), so an ambiguous
probe could be reported unique and the structure silently placed at the first
offset — the exact mis-placement the rule exists to prevent. Replaced with an
overlap-aware scan. Re-measured over the full corpus: 0 of 23,535 records
differ between the two counts, anchored stays 18,269, and the artifact digest
is byte-identical — a latent hole closed, not a number moved.
* (minor, applied) parent_record_id was built with str(cell), which turns a null
into the present-looking "None" (pandas 2) / "nan" (pandas 3) and would sail
past the null gate as a fabricated parent link; now `_text`.
* (major, skipped with reason) Declaring the four source artifacts as Snakemake
`input:` was tested, not argued: it fails `snakemake --lint` ("Param ... is a
prefix of input or output file but hardcoded", the CI-blocking rule) and makes
`snakemake -n stage2_dataset` schedule a 6-job rebuild ending in
split_assignment_table — regenerating the committed ADR-0004 split table the
no-leakage gate is anchored on. DVC/git-LFS inputs stay out of the DAG, the
documented house convention (data.smk::derive_labels). Restored byte-identical.
Both fixes carry a new test and were sabotage-verified: reverting the overlap
scan fails only test_projection_refuses_an_OVERLAPPING_ambiguous_anchor;
reverting `_text` fails only test_a_null_parent_link_is_refused_not_stringified.
55 passed / 2 skipped in tbox-data (the 2 skips are the parity legs, 23 passed in
tbox-ml-rna); ruff + black clean; snakemake --lint clean and `-n stage2_dataset`
reports "Nothing to be done".
…he r1 comment Both minor findings applied. * `decode`/`id_to_token` cast with `int()`, and `int(6.9) == 6`, so a corrupted float id would silently decode as the nucleotide at 6 — the opposite of the docstring's "an id outside the vocabulary is an error, not a silent <unk>". Now `operator.index`, which still accepts the NumPy integers a parquet round-trip yields (asserted in the test) and refuses floats, strings and None. * Prose: the r1 comment said "scan every start position" while the code does two `find`s; corrected to describe the second search from `offset + 1`. No code change (the reviewer asked for the comment only). New test test_decode_refuses_a_non_integer_id, sabotage-verified: reverting `operator.index` to `int()` fails that test and nothing else. 57 passed / 2 skipped in tbox-data; 24 passed / 0 skipped in tbox-ml-rna; ruff + black clean.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Assembles
data/processed/stage2_dataset.parquet(DVC) — 30,542 rows × 34 columns: the 23,535 T-box positives plus the 7,007 unmasked §9.1 static decoys, as RNA (T→U transcribed), each carrying its binary label, per-nucleotide boundary target, §8 aux targets (regulatory mode / specifier codon / cognate AA / tRNA family), a dot-bracket pairing target for the P3-05 structure-consistency loss, and the ADR-0004 fold + parentage columns.PRD §6, §8, §10.2, §11 · ADR ADR-0004 (D4/D5/D7), ADR-0002 (A4/A8/A9) · Compute LOCAL, ~8 s, no GPU, no network.
The three load-bearing choices
1.
flank_nt = 0for both classes, refused otherwise. PRD §6 hands Stage 2 a "locus ± flank", but the named P3-01 inputs supply real genomic flank for positives only (P2-00'scontext_v0); the four decoy pools are generated, Rfam-sourced or shuffled and have none. A positives-only flank makes "is embedded in real genomic context" a perfectly separable shortcut for the binary head — the §5 circularity failure the project exists to avoid.flank_ntis a pinned parameter recorded per row and in provenance; a non-zero value raises until a matching flank exists for every pool. Structure appears only as a target column, never a Stage-2 input (PRD §6).2. Folds inherited, never invented. Every row records
fold_basis:corpus_record(23,535) — a positive copies every scheme column from its own split-table row;parent_record(2,000) — a dinucleotide-shuffled decoy inherits its parent's fold across every scheme (ADR-0004 D7's variant→parent→fold rule on the negative side), refused fail-closed if the parent is absent;decoy_pool_random(5,007) — decoys with no corpus parent (generated GC background, Rfam structured RNAs, tboxevo leader decoys) belong to no cluster and no clade, so no leave-clade-out holdout unit can contain them. They are handled the way ADR-0004 D4 handles taxonomy-incomplete positives: kept only in the random split, every clade-scheme column null,clade_holdout_eligibleFalse,fold_randomfrom a deterministic order-independent keyed hash.nested_trainis left null — whether such a decoy may enter the nested training fold is a P3-03 sampling policy, andTruehere would be a policy decision disguised as a data field.3. The pairing target is projected and measured, never assumed. TBDB's
Structure(WUSS) is aligned toSequence, the gapped cmalign row — not to the genomicFASTA_sequencelocus (median 238 vs 281 nt; co-extensive with each other on 99.996 % of records but with the locus on 0.5 %). The builder validates the alignment row, strips gaps, and requires the gap-stripped sequence to occur exactly once in the locus before mapping pairs through that offset. Measured over the full corpus: 18,269 / 23,535 (77.62 %) anchor, with zero no-hit and zero multi-hit rows — every clean-alphabet row anchors uniquely, an internal consistency check rather than a tuned threshold. The other 5,266 render unaligned inserts as~/[n]/*with the nucleotides elided (after sanitising the junk glyphs, 4,590 of 5,259 still have no locus match — measured), so they carrypairing_status = "unanchorable_alignment"and a null target. Nothing approximate or re-folded is substituted (CLAUDE.md §10.3).A first implementation rejected any record whose consensus structure paired into a column this member had deleted — discarding 8,634 further records. Measurement showed the rule was wrong:
Structureis a consensus line and only 20,854 of 850,029 pairs (2.5 %) have a deleted partner (per-row median 0.000, max 0.426). Such a pair is not formed in this sequence, so it is now dropped and counted (n_pairs_dropped_gap) — never re-based onto a neighbouring nucleotide, never grounds for discarding the row's other 97.5 % of real pairs.The tokenizer adapter
src/tbox_finder/stage2/tokenizer.pypins RiNALMo's 28-token alphabet in pure Python so the build runs inenvs/data.ymlalongside every other data-layer rule (multimoleculelives only inenvs/ml-rna.ymlper the ADR-0002 A4 env split and pullsdeepspeedat import), and so the golden test stays runnable in CI. The 1022-nucleotide bound is derived —RiNALMoConfig.max_position_embeddings1024 minus<cls>/<eos>— not copied from prose. Behaviour was read off the live tokenizer: upper-case,T→U, the 15 IUPAC codes plus- . * | ? X I Nin-vocabulary, everything else<unk>rather than dropped, so token length always equals sequence length and the per-nucleotide target stays index-aligned.Two guards fail in different environments, so neither is vacuous: the digest/id tests are stdlib-only (bare CI), and
test_pinned_vocab_matches_live_tokenizerasserts the pin equals the liveRnaTokenizerat the ADR-0002 A9 revision2a71f6f9…intbox-ml-rna.Validation — PASS
tbox-data; the 2 skips are themultimoleculeparity legs, which run 23 passed / 0 skipped intbox-ml-rna.decode(encode(s)) == sexact on all 100 real T-box loci of the committed ingest slice and on a hand-authored alphabet fixture.fold_random,parent_record_idand a closed-setfold_basis; longest row 550 nt / 552 tokens ≤ 1022.ruff check+black --checkclean;snakemake --lint"workflow is in a good condition";snakemake -nparses.tbox-dataandtbox-ml-rna; CI runs the real pytest.Seven sabotage bites,
__pycache__cleared each time and every restore proved bycmp(these files are new, sogit checkout --would be a silent no-op): wrong split row for a derived decoy →test_derived_decoy_inherits_every_scheme_value_from_its_parent(+ golden); gap-partner pair re-based instead of dropped →test_projection_drops_a_pair_whose_partner_is_a_deleted_column(+ golden); ambiguous anchor takes the first hit →test_projection_refuses_an_ambiguous_anchor; invented clade units on a parentless decoy →test_parentless_decoys_are_random_only_and_invent_no_clade(+ golden); a vocabulary id edited →test_vocab_digest_is_pinned(+3); the T→U transcription removed → 14 tests incl. the golden. The seventh edited a vocabulary id and recomputed the pinned digest so every CI-side guard passes — caught only by the live-parity test, proving that leg bites.Notes for review
require_full_join=False: 6 of the 100 ingest-fixture records predate the corpus de-duplication and have no split row. The refused counts (6 records, and the 4 dinucleotide decoys parented on them) are asserted, so the relaxation is pinned rather than silent — and it makes ADR-0004 D7's fail-closed rule fire on real data inside the golden.README.md/paper/manuscript.qmdare untouched: this step lands no analysis result, and the repo batches both at the phase-exit gate (the P2 precedent).dvc pushis deferred to the phase-exit gate (CLAUDE.md §5.2).https://claude.ai/code/session_01Tr67opmzHJAGBJPAb2aTdX
Summary by CodeRabbit
New Features
Tests