P3-08: the with/without-aux ablation on the calibrated binary head + the Stage-2 score producer

[bioedca]

imp.md P3-08. PRD §11 (with/without-aux check), §12, §2.3 GATE-2; ADR-0005 D11 (named posterior; 15 equal-mass debiased bins; ECE ≤ 0.05) + D16 (aux weighting); ADR-0004 A7 (test is GATE-2's split, calib is where T is fit).

The step had to build what it was specified to consume

imp.md scopes P3-08 as "LOCAL (analysis of already-trained arms)" — but the calibrated posteriors it analyses did not exist. Nothing in this repo read a Stage-2 checkpoint back: stage2_heads.pt and lora_adapter/ had write sites and zero readers, and train.py::evaluate thresholds tbox_logit at 0 (train.py:1161) and keeps only a scalar accuracy. So stage2/eval.py ships the score producer (load_stage2_checkpoint, score_rows) alongside the ablation entry. P3-07's block already assigned the producer here.

Result — gate.overall_pass = false, and that is the measurement

arm	calib wrong @ 0	T	test ECE	test AUPRC	median |logit|
aux1.0_lr1e-4 (production)	1 / 1,089	1.140627	0.005662 ✅	0.999824	8.26
aux0.0_lr1e-4 (no-aux)	0 / 1,089	none	undefined	0.999897	14.45

The no-aux arm cannot be calibrated at all. Its calib carve is perfectly separated, so the exact minimiser is the β→∞ (T→0) limit and P3-07's fit_temperature refuses it. The refusal is recorded as structured evidence and ece is None — never the T = 1 value, because a substituted temperature is indistinguishable from a measured one (CLAUDE.md §10.3). Two clauses are false in consequence; nothing was invented to make them true.

The production arm holds GATE-2 at 8.8× under the D11 gate. On the axis that survives, ΔAUPRC = −7.4e-5. Median |logit| 8.26 vs 14.45 is the mechanism: the aux terms shrink the head's margins and are what keep it calibratable.

⚠ The with-aux fit's entire signal for β is ONE misclassified calib row of 1,089. P3-10's GATE-2 grade currently hangs on that record. Carried to P3-09/P3-10.

Two §7 stops, neither resolved here

1. The gate's threshold was never pre-registered. imp.md grades "beyond a pre-registered tolerance"; no τ is pinned in PRD.md, any ADR, conf/ or src/. D16 admits an absolute reading (needs no new number — and it passes) and a delta reading (needs one). Both are computed, neither resolved: ablation.verdict = requires_signoff, and validate_report refuses a report that pins a tolerance itself.
2. No ADR pins a degenerate-limit rule, and there is no pre-registered fallback for an uncalibratable arm.

Design notes for review

• AUPRC is graded on logits, not the posterior. z ↦ z/T is strictly monotone so it cannot move average precision (asserted as a gate clause), but σ() saturates to exactly 1.0 at these margins and would tie the ranking away.
• The adapter load is measured, not assumed. PeftModel.from_pretrained only warns on missing keys and PEFT initialises lora_B to zero, so a silently-unloaded adapter is the untuned backbone producing a complete, plausible, meaningless report. Every adapter and head tensor is compared bit-exactly against the file, in both directions, and non-zero lora_B blocks are counted.
• Each arm is scored under the attention backend its own run recorded — re-resolving picks a different kernel on a different card, and on this backbone the eager path does not merely perturb numerics, it raises.
• NaN cluster_id becomes a singleton bootstrap block, never one giant block.

Validation

ruff 0.15.15 + black 25.11.0 clean. unit+golden+ml: 3,750 passed; the 14 failures are a strict subset of main's 31 in this local runner (comm -13 empty ⇒ zero regressions; CI runs the pinned stack). 23 sabotages, each RED against its named test, each restored byte-identically. Scoring: 2 arms × 7,201 rows, 12 min 33 s on the laptop RTX 4060, forward-only.

Summary by CodeRabbit

• New Features

  • Added a complete Stage 2 evaluation workflow for comparing models with and without auxiliary loss.
  • Added temperature calibration, ECE and AUPRC measurement, ranking checks, confidence intervals, and detailed evaluation reports.
  • Added strict checkpoint validation, deterministic scoring, cached-score reconciliation, and diagnostic exit statuses.
  • Added a recorded ablation report showing auxiliary weighting does not degrade the Gate-2 grade.

• Tests

  • Added coverage for checkpoint integrity, scoring invariance, calibration behavior, malformed inputs, cache validation, and report consistency.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 50 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6b01fa6d-92e0-48c4-a79e-e4c116981917

📥 Commits

Reviewing files that changed from the base of the PR and between a0abef7 and ce4210d.

⛔ Files ignored due to path filters (1)

• analyses/phase3_log.qmd is excluded by !**/*.qmd

📒 Files selected for processing (3)

• reports/stage2_aux_ablation.json
• src/tbox_finder/stage2/eval.py
• tests/unit/test_aux_ablation_check.py

📝 Walkthrough

Walkthrough

Added a Stage 2 evaluation pipeline for matched with-aux and no-aux arms. It performs strict checkpoint verification, deterministic scoring, temperature calibration, ECE/AUPRC grading, report validation, and committed ablation reporting.

Changes

Stage 2 auxiliary-ablation evaluation

Layer / File(s)	Summary
Metric and calibration grading src/tbox_finder/stage2/eval.py, tests/unit/test_aux_ablation_check.py	Adds score validation, rung and bootstrap handling, temperature fitting on calib, calibrated ECE, raw-logit AUPRC, confidence intervals, saturation diagnostics, and refusal handling.
Arm selection and report gates src/tbox_finder/stage2/eval.py, tests/unit/test_aux_ablation_check.py	Discovers matched checkpoint arms, compares absolute and delta readings, derives report clauses, and validates verdict and gate consistency.
Checkpoint loading and row scoring src/tbox_finder/stage2/eval.py, tests/ml/test_stage2_eval_smoke.py, tests/unit/test_aux_ablation_check.py	Verifies adapter, head, backbone, and LoRA integrity. Scores rows deterministically, reconciles cached logits, and provides CLI orchestration with validation diagnostics.
Committed ablation evidence reports/stage2_aux_ablation.json, tests/ml/test_stage2_eval_smoke.py	Adds arm metrics, calibration outcomes, checkpoint and dataset provenance, gate clauses, and validation of the committed report. No-aux calibration refusal makes the machinery gate false while the absolute ablation verdict passes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Checkpoints
  participant Dataset
  participant Evaluator
  participant Report
  CLI->>Checkpoints: discover and verify matched arms
  CLI->>Dataset: load rows and metadata
  CLI->>Evaluator: score rows and grade arms
  Evaluator->>Report: build and validate ablation report
  Report-->>CLI: gate result and diagnostics

Loading

Possibly related PRs

• bioedca/tbox-finder#94: Provides the calib split and calibration metadata consumed for temperature fitting and ECE grading.
• bioedca/tbox-finder#95: Provides the Stage2Model and checkpoint interfaces used for loading and scoring.
• bioedca/tbox-finder#99: Provides the Stage 2 training outputs and arm configuration artifacts evaluated here.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the auxiliary-loss ablation and the Stage-2 score producer, which are the PR's main changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch p3-08-aux-ablation

---

_{Comment @coderabbitai help to get the list of available commands.}

[bioedca]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (7)

tests/ml/test_stage2_eval_smoke.py (2)

318-334: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The test name promises a raw-logit check; the assertion only proves batching consistency.

Line 333 compares the batched score with a direct single-row forward pass. If a sigmoid were applied in score_rows and in the direct call, both sides would move together and this test would stay green. The docstring states the stronger property: the calibration stack must receive an unsquashed value.

An additional assertion that ties the scored value to the head's own output range would name the property directly. A blunt not 0.0 <= value <= 1.0 check risks flaking on the tiny randomly-initialised backbone, so prefer a structural check, for example asserting that score_rows returns exactly the tensor the head emits for a deliberately large-magnitude input.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ml/test_stage2_eval_smoke.py` around lines 318 - 334, Add a structural
raw-logit assertion to
test_the_scored_logit_is_the_raw_head_output_not_a_probability: use a
deliberately large-magnitude head output and verify score_rows preserves that
exact unsquashed value, rather than only matching a direct forward pass. Avoid a
generic [0, 1] range check, and retain the existing batching-consistency and
token-count assertions.

---

250-256: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The comment claims an empty base_model_name_or_path is accepted, but no assertion tests it.

Lines 250-251 state that a locally-constructed base records no name and must not be refused. The test then only sets LH.REPO_ID. The falsy branch of the guard in eval.py (Line 1373, if recorded_base and recorded_base != LH.REPO_ID) stays uncovered, so a change that refuses an empty value would keep this file green.

💚 Proposed addition

     config["base_model_name_or_path"] = LH.REPO_ID
     config_path.write_text(json.dumps(config), encoding="utf-8")
     assert _load(ckpt, torch)[1]["n_adapter_tensors_matched"] > 0
+
+    # ...and the empty case the comment above promises: no recorded name is not a
+    # mismatch, it is the absence of a claim.
+    config["base_model_name_or_path"] = None
+    config_path.write_text(json.dumps(config), encoding="utf-8")
+    assert _load(ckpt, torch)[1]["base_model_name_or_path"] is None

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ml/test_stage2_eval_smoke.py` around lines 250 - 256, Extend the
positive-control test around config["base_model_name_or_path"] to explicitly set
it to an empty value and assert _load still succeeds with matched adapter
tensors. Keep the existing LH.REPO_ID assertion, ensuring both the pinned
repository ID and empty recorded base paths are accepted.

src/tbox_finder/stage2/eval.py (3)

901-917: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Name the refusal when a run report omits aux_weight or lr.

Lines 916-917 index loss["aux_weight"] and config["lr"] directly. If a sweep report lacks either key, the caller sees a bare KeyError: 'aux_weight' with no arm name and no path. Every other refusal in this module names the file and the cause.

♻️ Proposed change

+        for holder, key, where in ((loss, "aux_weight", "config.loss"), (config, "lr", "config")):
+            if key not in holder:
+                raise ValueError(
+                    f"{report_path}: arm {arm_dir.name} records no {where}.{key}, so it "
+                    "cannot be placed in the ablation"
+                )
         arms[arm_dir.name] = {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/tbox_finder/stage2/eval.py` around lines 901 - 917, Update the arm record
construction around aux_weight and lr to validate that loss["aux_weight"] and
config["lr"] are present before indexing them, and raise the module’s
established error type with a message naming the arm and report path when either
is missing. Preserve the existing float conversion and successful-record
behavior.

---

1576-1592: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Build each padded batch on the CPU, then move it once.

Lines 1580-1584 allocate input_ids and attention_mask directly on target and then write one row at a time with torch.tensor(ids, device=target). On a CUDA target that is two small host-to-device copies per row, plus a device-side slice assignment per row. For 7201 rows per arm the launch overhead adds up and buys nothing.

Assembling the batch on the host and issuing one transfer per batch keeps the numerics identical.

⚡ Proposed change

-            input_ids = torch.full((len(chunk), width), TOK.PAD_ID, dtype=torch.long, device=target)
-            attention_mask = torch.zeros((len(chunk), width), dtype=torch.long, device=target)
-            for i, (n, _, ids) in enumerate(chunk):
-                input_ids[i, :n] = torch.tensor(ids, dtype=torch.long, device=target)
-                attention_mask[i, :n] = 1
+            input_ids = torch.full((len(chunk), width), TOK.PAD_ID, dtype=torch.long)
+            attention_mask = torch.zeros((len(chunk), width), dtype=torch.long)
+            for i, (n, _, ids) in enumerate(chunk):
+                input_ids[i, :n] = torch.tensor(ids, dtype=torch.long)
+                attention_mask[i, :n] = 1
+            input_ids = input_ids.to(target)
+            attention_mask = attention_mask.to(target)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/tbox_finder/stage2/eval.py` around lines 1576 - 1592, Update the batching
logic in the inference loop to allocate `input_ids` and `attention_mask` on the
CPU, populate all rows there, then transfer each completed tensor to `target`
once per batch before calling `model`. Preserve the existing padding values,
masks, tensor shapes, and logits/output behavior.

---

744-754: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Consider re-deriving reading_absolute.passes from the recorded ECE.

passes copies w["ece_gate_pass"] and validate_report then derives the verdict from passes. Nothing compares observed_ece against gate. A hand-edited artifact that sets ece: 0.2 and ece_gate_pass: true still validates, because no check closes that loop. The module states elsewhere that clauses must be re-derived rather than believed.

A cheap closure is to compute passes from the number the report already carries.

♻️ Proposed change in compare_arms

-    absolute_passes = bool(w["ece_gate_pass"]) if with_aux_ece_available else None
+    # Derived from the recorded ECE, not read back from the arm's own pass flag, so a
+    # report cannot carry a `passes` its own number contradicts.
+    absolute_passes = bool(M.gate2_ece_pass(w["ece"])) if with_aux_ece_available else None

A matching clause in validate_report (or a new clause in derive_clauses) that asserts reading_absolute.passes == (observed_ece <= gate) would make the guard complete.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/tbox_finder/stage2/eval.py` around lines 744 - 754, Update compare_arms
so reading_absolute.passes is re-derived from the recorded observed_ece and
ECE_GATE rather than copied from w["ece_gate_pass"], while preserving the
unavailable-ECE handling. Add or update validation in validate_report or
derive_clauses to assert that reading_absolute.passes matches the observed_ece
<= gate comparison.

tests/unit/test_aux_ablation_check.py (2)

750-827: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert that CLAUSE_SABOTAGE covers every derived clause.

The list currently names all 15 clauses. Nothing fails when someone adds a clause to derive_clauses and forgets to add its sabotage, so the new clause ships untested. One assertion closes that gap and keeps the sabotage suite honest as the clause set grows.

💚 Proposed test

+def test_every_derived_clause_has_a_sabotage() -> None:
+    """A clause with no sabotage is a clause nothing proves can go false."""
+    assert {name for name, _ in CLAUSE_SABOTAGE} == set(E.derive_clauses(_report()))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_aux_ablation_check.py` around lines 750 - 827, Update
test_every_clause_bites to assert that the clause names in CLAUSE_SABOTAGE
exactly match the keys returned by E.derive_clauses(_report()), including
detecting missing or extra sabotage entries. Keep the existing per-clause
sabotage assertions unchanged.

---

236-247: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The direct-fit reference encodes recalibrate's binary-to-two-class layout.

Lines 239-241 build the reference logits as [0, z] and compare the resulting temperature to grade_arm's at rel=1e-9. That exact equality holds only while R.temperature_scale uses the same two-column embedding for a scalar binary logit. If that module changes to a symmetric layout such as [-z/2, +z/2], this test fails for a reason unrelated to the rung-selection property it asserts.

A short comment naming the coupling, or asserting against R.temperature_scale on the test-rung slice instead of TEMP.fit_temperature, keeps the failure message pointing at the real cause.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_aux_ablation_check.py` around lines 236 - 247, The direct-fit
reference in the test around _arm and TEMP.fit_temperature is coupled to
recalibrate’s binary-to-two-class [0, z] embedding. Document that coupling with
a concise comment, or instead compute the expected temperature through
R.temperature_scale on the test-rung slice so the assertion remains valid if the
binary layout changes while still verifying rung selection.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/tbox_finder/stage2/eval.py`:
- Around line 1003-1012: Align the gate evaluation with the documented non-gated
--all-arms behavior: keep load/scoring clauses over all discovered arm_blocks,
but restrict calibration and grading clauses to the ablation pair by introducing
or reusing pair_blocks. Update the relevant calibs, stacks, graded_for_ece, and
graded clause calculations to read pair_blocks, while preserving the existing
all-arm reporting.

---

Nitpick comments:
In `@src/tbox_finder/stage2/eval.py`:
- Around line 901-917: Update the arm record construction around aux_weight and
lr to validate that loss["aux_weight"] and config["lr"] are present before
indexing them, and raise the module’s established error type with a message
naming the arm and report path when either is missing. Preserve the existing
float conversion and successful-record behavior.
- Around line 1576-1592: Update the batching logic in the inference loop to
allocate `input_ids` and `attention_mask` on the CPU, populate all rows there,
then transfer each completed tensor to `target` once per batch before calling
`model`. Preserve the existing padding values, masks, tensor shapes, and
logits/output behavior.
- Around line 744-754: Update compare_arms so reading_absolute.passes is
re-derived from the recorded observed_ece and ECE_GATE rather than copied from
w["ece_gate_pass"], while preserving the unavailable-ECE handling. Add or update
validation in validate_report or derive_clauses to assert that
reading_absolute.passes matches the observed_ece <= gate comparison.

In `@tests/ml/test_stage2_eval_smoke.py`:
- Around line 318-334: Add a structural raw-logit assertion to
test_the_scored_logit_is_the_raw_head_output_not_a_probability: use a
deliberately large-magnitude head output and verify score_rows preserves that
exact unsquashed value, rather than only matching a direct forward pass. Avoid a
generic [0, 1] range check, and retain the existing batching-consistency and
token-count assertions.
- Around line 250-256: Extend the positive-control test around
config["base_model_name_or_path"] to explicitly set it to an empty value and
assert _load still succeeds with matched adapter tensors. Keep the existing
LH.REPO_ID assertion, ensuring both the pinned repository ID and empty recorded
base paths are accepted.

In `@tests/unit/test_aux_ablation_check.py`:
- Around line 750-827: Update test_every_clause_bites to assert that the clause
names in CLAUSE_SABOTAGE exactly match the keys returned by
E.derive_clauses(_report()), including detecting missing or extra sabotage
entries. Keep the existing per-clause sabotage assertions unchanged.
- Around line 236-247: The direct-fit reference in the test around _arm and
TEMP.fit_temperature is coupled to recalibrate’s binary-to-two-class [0, z]
embedding. Document that coupling with a concise comment, or instead compute the
expected temperature through R.temperature_scale on the test-rung slice so the
assertion remains valid if the binary layout changes while still verifying rung
selection.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 996d4145-68ba-42cb-ac74-961543184ac8

📥 Commits

Reviewing files that changed from the base of the PR and between a4c075b and a0abef7.

⛔ Files ignored due to path filters (1)

• analyses/phase3_log.qmd is excluded by !**/*.qmd

📒 Files selected for processing (5)

• reports/p3/stage2_scores.json
• reports/stage2_aux_ablation.json
• src/tbox_finder/stage2/eval.py
• tests/ml/test_stage2_eval_smoke.py
• tests/unit/test_aux_ablation_check.py

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

--all-arms is documented as non-gated, but every discovered arm is graded here.

arm_blocks covers every entry of report["arms"]. With --all-arms, that is every trained arm, and each one must satisfy adapter_weights_verified_against_file, temperature_positive_and_converged, graded_object_is_pre_prior_shift and the rest. One unrelated arm that cannot be temperature-calibrated then turns gate.overall_pass false while the ablation pair is clean.

The CLI help at Line 1707 states the opposite: "score every trained arm, not just the ablation pair (reported, non-gated)".

Either restrict the clause set to the ablation pair, or correct the help text. The load/scoring clauses arguably should still cover every scored arm; the calibration clauses arguably should not.

🔧 One way to restrict the calibration and grade clauses to the ablation pair

     arm_blocks = [a for a in arms.values() if isinstance(a, Mapping)]
+    pair_names = [ablation.get("with_aux_arm"), ablation.get("no_aux_arm")]
+    pair_blocks = [a for n, a in arms.items() if n in pair_names and isinstance(a, Mapping)]

and then read pair_blocks for calibs, stacks, graded_for_ece and graded.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/tbox_finder/stage2/eval.py` around lines 1003 - 1012, Align the gate
evaluation with the documented non-gated --all-arms behavior: keep load/scoring
clauses over all discovered arm_blocks, but restrict calibration and grading
clauses to the ablation pair by introducing or reusing pair_blocks. Update the
relevant calibs, stacks, graded_for_ece, and graded clause calculations to read
pair_blocks, while preserving the existing all-arm reporting.