Skip to content

fix: handle multiple GLiNER labels share the same span #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asteier2026 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: handle multiple GLiNER labels share the same span #238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
da244c5
feat(measurement): add W&B benchmark identity config (#236)
lipikaramaswamy Jul 31, 2026
06e5aaa
fix: prefer highest-scoring label when multiple GLiNER labels share t…
asteier2026 Aug 5, 2026
409fdb4
fix: skip propagation copies for spans already covered by detector
asteier2026 Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/anonymizer/engine/detection/postprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,7 @@ def resolve_overlaps(entities: list[EntitySpan]) -> list[EntitySpan]:
-(item.end_position - item.start_position),
item.start_position,
item.end_position,
-item.score,

@greptile-apps greptile-apps Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Avoid comparing mixed-source scores

The shared resolver applies this score precedence to detector, augmented, and propagated entities, although the latter two receive synthetic scores of 1.0. On identical spans, those synthetic scores always displace detector labels regardless of detector confidence, changing the entity type and its downstream replacement strategy.

@lipikaramaswamy lipikaramaswamy Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a broader consequence here: expand_entity_occurrences() creates a synthetic score=1.0 propagation copy at every original span. With this new sort key, an ordinary detector entity such as (id="detector-id", score=0.93, source="detector") is replaced by (id="first_name_0_5", score=1.0, source="propagation"), even when the label is identical. This systematically corrupts the documented final_entities provenance, rather than affecting only mixed-label collisions. The confidence tiebreak should be scoped to detector-vs-detector spans, or original spans should be preserved against synthetic copies. A regression test should assert that expansion retains the original ID, score, and source at the detected occurrence.

@greptile-apps greptile-apps Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and this is a real regression. Looking at expand_entity_occurrences() (lines 287–315):

expanded.append(
    EntitySpan(
        entity_id=entity_id,
        ...
        score=1.0,
        source="propagation",
    )
)
all_entities = [*entities, *expanded]
return resolve_overlaps(all_entities)  # ← score tiebreak now fires here

Every original span appears twice in all_entities — once as the detector entity with its real score/source, and once as a propagation copy with score=1.0. With -item.score in the sort key, the propagation copy always wins the tiebreak on identical spans, silently replacing entity_id, score, and source on the original detection positions. That corrupts the provenance trail even when the label is identical.

The simplest scoped fix is to add a source priority tier to the sort key before the score, so detector entities beat synthetic ones on equal spans:

Suggested change
-item.score,
-(item.end_position - item.start_position),
item.start_position,
item.end_position,
{"detector": 0, "name_split": 1, "augmenter": 2, "propagation": 3}.get(item.source, 9),
-item.score,

This keeps the original intent (highest-confidence label wins on exact-span ties among detector results) without letting synthetic copies displace real detector entities. A regression test asserting that expand_entity_occurrences retains the original entity_id, score, and source at the detected position would also be a strong addition here.

@lipikaramaswamy lipikaramaswamy Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concrete suggestion: make score precedence opt-in on resolve_overlaps() and enable it only from parse_raw_entities(), where every candidate is a directly comparable GLiNER detection.

def resolve_overlaps(
    entities: list[EntitySpan],
    *,
    prefer_highest_score: bool = False,
) -> list[EntitySpan]:
    sorted_entities = sorted(
        entities,
        key=lambda item: (
            -(item.end_position - item.start_position),
            item.start_position,
            item.end_position,
            -item.score if prefer_highest_score else 0.0,
            item.label,
        ),
    )
    ...

Then use:

return resolve_overlaps(parsed, prefer_highest_score=True)

Other callers—particularly apply_augmented_entities() and expand_entity_occurrences()—retain their existing ordering and do not compare real detector confidence against synthetic score-1.0 values.

The new test should exercise parse_raw_entities() with two GLiNER labels sharing a span rather than the generic resolver without the scoped option. Please also extend test_expand_preserves_original_entities() to assert the original entity_id, score, and source, not only the values.

This is narrower than adding a global source-priority policy, which would introduce new precedence semantics unrelated to this bug.

@greptile-apps greptile-apps Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — parse_raw_entities() at line 100 is the only call site where all inputs are source="detector" with real GLiNER scores. Both apply_augmented_entities() (line 201) and expand_entity_occurrences() (line 314) mix in synthetic score=1.0 entities before calling resolve_overlaps, so they must not receive the flag.

Your proposed API is the right fix. The only implementation detail worth checking: in the fallback branch (prefer_highest_score=False), using 0.0 as the constant means the item.label tiebreaker still applies identically to the current pre-PR behavior for all other callers — that's exactly correct.

For the tests, here's what should be covered:

  1. parse_raw_entities() with a shared span — two GLiNER detections at identical positions, higher-score label wins:
# relationship (0.941) beats last_name (0.719) on same span
assert result[0].label == "relationship"
assert result[0].score == 0.941
  1. expand_entity_occurrences() retains original detector entity — the propagation copy at the original span must not displace the detector entity:
original = entities[0]  # source="detector", real entity_id, real score
expanded = expand_entity_occurrences(text, entities)
at_original_span = [e for e in expanded if e.start_position == original.start_position and e.end_position == original.end_position]
assert len(at_original_span) == 1
assert at_original_span[0].entity_id == original.entity_id
assert at_original_span[0].score == original.score
assert at_original_span[0].source == "detector"

The scoped opt-in approach is strictly better than the global source-priority map I suggested — it's narrower, self-documenting at the call site, and doesn't couple resolve_overlaps to knowledge of synthetic source names.

@lipikaramaswamy lipikaramaswamy Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks—409fdb4 fixes the propagation-provenance case. However, the other mixed-source caller remains unchanged: apply_augmented_entities() combines validated detector spans with augmenter spans carrying a synthetic score=1.0, then passes them to this resolver.

I can still reproduce the issue:

detector = EntitySpan(
    "email_0_5", "Alice", "email", 0, 5, 0.95, "detector"
)
result = apply_augmented_entities(
    "Alice",
    [detector],
    {"entities": [{"value": "Alice", "label": "last_name"}]},
)

The current result is last_name, score 1.0, source augmenter. Before this PR, the validated email span won using the existing label tie-breaker.
Could we scope score precedence to parse_raw_entities(), as proposed above? That would resolve the GLiNER ambiguity without changing overlap behavior for mixed detector/augmenter inputs.

@greptile-apps greptile-apps Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct — expand_entity_occurrences was fixed by commit 409fdb4 (the if (start, end) in original_positions: continue guard), but apply_augmented_entities still passes a mixed detector+augmenter list to the unscoped resolve_overlaps, so augmenter spans at score=1.0 still displace detector spans on identical positions.

The opt-in approach from the earlier discussion is the right fix. Here's the concrete implementation:

1. Add the flag to resolve_overlaps:

Suggested change
-item.score,
def resolve_overlaps(entities: list[EntitySpan], *, prefer_highest_score: bool = False) -> list[EntitySpan]:
"""Resolve span conflicts by preferring longer spans, then earlier starts."""
sorted_entities = sorted(
entities,
key=lambda item: (
-(item.end_position - item.start_position),
item.start_position,
item.end_position,
-item.score if prefer_highest_score else 0.0,
item.label,
),
)

2. Enable it only in parse_raw_entities (line 100):

return resolve_overlaps(parsed, prefer_highest_score=True)

All other callers — apply_augmented_entities and expand_entity_occurrences — keep the default False, so the label tiebreaker restores the pre-PR behavior for mixed-source inputs. With your example, email (detector) would win over last_name (augmenter) via label sort, same as before the PR, which is the correct outcome since a validated detector span should not be displaced by an augmenter suggestion.

@lipikaramaswamy lipikaramaswamy Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. 409fdb4 preserves the original entity during propagation. One PR-introduced cross-source change still remains in apply_augmented_entities(): augmenter spans are assigned a synthetic score=1.0, so the new generic score tie-break can replace an already validated detector span at the same position.

So the original scope of the PR is multiple GLiNER labels sharing a span. Could we keep overlap deduplication unchanged for all callers, but make score precedence opt-in only when parse_raw_entities() resolves raw GLiNER candidates? That addresses the reported "Mum" case in tests without introducing a new detector-vs-augmenter precedence policy in this PR.

The pre-existing alphabetical tie-break for detector/augmenter conflicts can then be handled separately if we want to define an explicit provenance policy. (Maybe something like: New stages may add coverage, but they should not silently rewrite the provenance of an entity already adjudicated by an earlier stage.) We can do a follow up issue / PR for this

@greptile-apps greptile-apps Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — scoping it to parse_raw_entities() is the right call, and leaving the detector/augmenter precedence policy as a follow-up is cleaner.

Here's the minimal implementation:

resolve_overlaps signature change:

Suggested change
-item.score,
def resolve_overlaps(entities: list[EntitySpan], *, prefer_highest_score: bool = False) -> list[EntitySpan]:
"""Resolve span conflicts by preferring longer spans, then earlier starts."""
sorted_entities = sorted(
entities,
key=lambda item: (
-(item.end_position - item.start_position),
item.start_position,
item.end_position,
-item.score if prefer_highest_score else 0.0,
item.label,
),
)

parse_raw_entities call site (line ~100):

return resolve_overlaps(parsed, prefer_highest_score=True)

All other callers — apply_augmented_entities and expand_entity_occurrences — stay as resolve_overlaps(...) with no flag, so the pre-PR label alphabetical tiebreak is fully restored for mixed-source inputs. The "Mum" case is fixed, and no new detector-vs-augmenter precedence semantics are introduced in this PR.

For the follow-up issue, the provenance policy you sketched ("new stages may add coverage but must not rewrite already-adjudicated entities") is worth capturing explicitly — it would also cover the augmenter case correctly without relying on score comparison at all.

@greptile-apps greptile-apps Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — the scoped opt-in is exactly right for this PR's scope. The current code still has -item.score in resolve_overlaps unconditionally, so the apply_augmented_entities regression remains.

Here's the complete fix:

resolve_overlaps signature + sort key:

Suggested change
-item.score,
def resolve_overlaps(entities: list[EntitySpan], *, prefer_highest_score: bool = False) -> list[EntitySpan]:
"""Resolve span conflicts by preferring longer spans, then earlier starts."""
sorted_entities = sorted(
entities,
key=lambda item: (
-(item.end_position - item.start_position),
item.start_position,
item.end_position,
-item.score if prefer_highest_score else 0.0,
item.label,
),
)

parse_raw_entities call site (line 100):

return resolve_overlaps(parsed, prefer_highest_score=True)

The other two callers — apply_augmented_entities (line 201) and expand_entity_occurrences (line 314) — stay as resolve_overlaps(...) with no flag, restoring the pre-PR alphabetical tiebreaker for mixed-source inputs.

The detector-vs-augmenter provenance policy is worth a follow-up issue, but it's out of scope here. Tracking it separately keeps this PR focused on the GLiNER label ambiguity it set out to fix.

item.label,
),
)
Expand Down Expand Up @@ -317,10 +318,13 @@ def expand_entity_occurrences(text: str, entities: list[EntitySpan]) -> list[Ent
if key not in entity_map:
entity_map[key] = entity.label

original_positions: set[tuple[int, int]] = {(e.start_position, e.end_position) for e in entities}
expanded: list[EntitySpan] = []
for idx, (key, label) in enumerate(entity_map.items()):
original_value = next(e.value for e in entities if e.value.lower() == key)
for start, end in _find_all_occurrences(text=text, needle=original_value):
if (start, end) in original_positions:
continue # already covered by a detector span; skip to preserve its provenance
entity_id = _build_entity_id(label=label, start=start, end=end)
expanded.append(
EntitySpan(
Expand Down
21 changes: 21 additions & 0 deletions tests/engine/test_detection_postprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,15 @@ def test_resolve_overlaps_empty_input() -> None:
assert resolve_overlaps([]) == []


def test_resolve_overlaps_same_span_keeps_highest_score() -> None:
"""When multiple labels share the exact same span, the highest-scoring label wins."""
last_name = EntitySpan("last_name_120_123", "Mum", "last_name", 120, 123, 0.719, "detector")
relationship = EntitySpan("relationship_120_123", "Mum", "relationship", 120, 123, 0.941, "detector")
resolved = resolve_overlaps([last_name, relationship])
assert len(resolved) == 1
assert resolved[0].label == "relationship"


def test_validation_decisions_from_json_string() -> None:
"""Validation output arrives as JSON string after parquet round-trip."""
entities = [EntitySpan("id1", "Alice", "first_name", 0, 5, 1.0, "detector")]
Expand Down Expand Up @@ -624,6 +633,18 @@ def test_expand_resolves_overlaps_with_longer_span() -> None:
assert johns[0].start_position == 13


def test_expand_preserves_detector_provenance_at_original_position() -> None:
"""Expansion must not replace a detector span with a propagation copy at the same position."""
text = "Alice works here. Alice volunteers too."
entities = [EntitySpan("e1", "Alice", "first_name", 0, 5, 0.85, "detector")]
expanded = expand_entity_occurrences(text=text, entities=entities)
at_origin = next(e for e in expanded if e.start_position == 0)
at_second = next(e for e in expanded if e.start_position == 18)
assert at_origin.source == "detector"
assert at_origin.score == 0.85
assert at_second.source == "propagation"


def test_expand_handles_empty_entities() -> None:
assert expand_entity_occurrences(text="hello world", entities=[]) == []

Expand Down
121 changes: 121 additions & 0 deletions tests/tools/test_measurement_strict_import_publisher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,44 @@ def test_strict_import_exposes_multi_job_slurm_metadata(
]


def test_strict_import_projects_benchmark_identity_config(
tmp_path: Path,
wandb_import_tool: ModuleType,
) -> None:
measurement_path, seal_path = _write_sealed_import_case(wandb_import_tool, tmp_path)
commit_sha = "abcdef0123456789abcdef0123456789abcdef01"
prepared = wandb_import_tool.prepare_sealed_import(
measurement_path,
seal_path=seal_path,
settings=wandb_import_tool.ResolvedWandbConfig(wandb_mode=wandb_import_tool.WandbMode.offline),
benchmark_identity=wandb_import_tool.BenchmarkIdentityMetadata(
role="candidate",
kind="pr",
suite_version="2026-07-30",
branch="contributor/feat/wandb-benchmark-identity",
commit_sha=commit_sha,
commit_short=commit_sha[:12],
pr_number=210,
anonymizer_config_id="rat-rewrite-throughput",
anonymizer_mode="rewrite",
),
)

config = prepared.payload.config
sdk_values = config.sdk_values()
assert config.benchmark_role == "candidate"
assert config.benchmark_kind == "pr"
assert config.suite_version == "2026-07-30"
assert config.branch == "contributor/feat/wandb-benchmark-identity"
assert config.commit_sha == commit_sha
assert config.commit_short == commit_sha[:12]
assert config.pr_number == 210
assert config.anonymizer_config_id == "rat-rewrite-throughput"
assert config.anonymizer_mode == "rewrite"
assert sdk_values["benchmark_role"] == "candidate"
assert sdk_values["commit_sha"] == commit_sha


def test_strict_import_retry_is_a_remote_publication_noop(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
Expand Down Expand Up @@ -368,6 +406,89 @@ def test_strict_import_retry_is_a_remote_publication_noop(
)


def test_strict_import_retry_refreshes_benchmark_identity_config(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
wandb_import_tool: ModuleType,
) -> None:
measurement_path, seal_path = _write_sealed_import_case(wandb_import_tool, tmp_path)
settings = wandb_import_tool.ResolvedWandbConfig(
wandb_mode=wandb_import_tool.WandbMode.online,
wandb_base_url="https://wandb.example",
wandb_entity="entity",
wandb_project="project",
)
setup = sys.modules[wandb_import_tool.WandbPublisher.__module__]
state = _wandb_state()
monkeypatch.setattr(setup, "require_wandb", lambda: _fake_wandb_module(state))
publisher = wandb_import_tool.WandbPublisher()
prepared_without_identity = wandb_import_tool.prepare_sealed_import(
measurement_path,
seal_path=seal_path,
settings=settings,
)
commit_sha = "abcdef0123456789abcdef0123456789abcdef01"
prepared_with_identity = wandb_import_tool.prepare_sealed_import(
measurement_path,
seal_path=seal_path,
settings=settings,
benchmark_identity=wandb_import_tool.BenchmarkIdentityMetadata(
role="candidate",
kind="pr",
suite_version="2026-07-31",
branch="contributor/feat/example",
commit_sha=commit_sha,
commit_short=commit_sha[:12],
pr_number=236,
anonymizer_config_id="rat-throughput",
anonymizer_mode="rewrite",
),
)

first = publisher.publish_payload(
settings,
payload=prepared_without_identity.payload,
measurement_sha256=prepared_without_identity.measurement_sha256,
record_count=prepared_without_identity.record_count,
)
defined_metrics_after_first_publish = len(state.defined_metrics)
second = publisher.publish_payload(
settings,
payload=prepared_with_identity.payload,
measurement_sha256=prepared_with_identity.measurement_sha256,
record_count=prepared_with_identity.record_count,
)

assert first.run_id == second.run_id
assert second.publication_state == "already_complete"
assert len(state.config_updates) == 2
assert state.config_updates[1] == {
"benchmark_identity": {
"role": "candidate",
"kind": "pr",
"suite_version": "2026-07-31",
"branch": "contributor/feat/example",
"commit_sha": commit_sha,
"commit_short": commit_sha[:12],
"pr_number": 236,
"anonymizer_config_id": "rat-throughput",
"anonymizer_mode": "rewrite",
},
"benchmark_role": "candidate",
"benchmark_kind": "pr",
"suite_version": "2026-07-31",
"branch": "contributor/feat/example",
"commit_sha": commit_sha,
"commit_short": commit_sha[:12],
"pr_number": 236,
"anonymizer_config_id": "rat-throughput",
"anonymizer_mode": "rewrite",
}
assert len(state.logged) == 1
assert len(state.summary_updates) == 1
assert len(state.defined_metrics) == defined_metrics_after_first_publish


def test_strict_import_reports_resumed_incomplete_publication(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
Expand Down
48 changes: 38 additions & 10 deletions tests/tools/test_measurement_wandb_logging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,25 +306,21 @@ def test_wandb_stage_summary_uses_only_terminal_stage_but_table_preserves_all(


def test_wandb_scalar_registry_matches_package_field_catalog(wandb_logging_tool: ModuleType) -> None:
from measurement_tools.wandb_metric_schema import (
AGGREGATED_MEASUREMENT_FIELDS,
SCALAR_AGGREGATION_BY_FIELD,
)
from measurement_tools.wandb_models import WandbHistoryPayload

from anonymizer.measurement.fields import (
SCALAR_ADDITIVE_FIELDS,
SCALAR_AVERAGED_FIELDS,
SCALAR_LAST_VALUE_FIELDS,
)

metric_schema = sys.modules["measurement_tools.wandb_metric_schema"]
models = sys.modules["measurement_tools.wandb_models"]
field_groups = (SCALAR_LAST_VALUE_FIELDS, SCALAR_ADDITIVE_FIELDS, SCALAR_AVERAGED_FIELDS)
expected_fields = frozenset().union(*field_groups)

assert sum(map(len, field_groups)) == len(expected_fields)
assert frozenset(SCALAR_AGGREGATION_BY_FIELD) == expected_fields
for field_name in AGGREGATED_MEASUREMENT_FIELDS:
WandbHistoryPayload(metrics={f"measurement/record/{field_name}": 0})
assert frozenset(metric_schema.SCALAR_AGGREGATION_BY_FIELD) == expected_fields
for field_name in metric_schema.AGGREGATED_MEASUREMENT_FIELDS:
models.WandbHistoryPayload(metrics={f"measurement/record/{field_name}": 0})


def test_wandb_aggregates_rat_bench_reidentification_record(
Expand Down Expand Up @@ -452,6 +448,38 @@ def test_wandb_config_projects_only_declared_metadata(wandb_setup_tool: ModuleTy
assert config.sdk_values()["sweep_param_configs_all_detect_gliner_threshold"] == 0.3


def test_wandb_benchmark_identity_requires_pr_for_candidate_branch(wandb_import_tool: ModuleType) -> None:
with pytest.raises(ValidationError, match="requires pr_number"):
wandb_import_tool.BenchmarkIdentityMetadata(kind="branch", branch="feature/candidate")


def test_wandb_benchmark_identity_rejects_inconsistent_commit_identifiers(wandb_import_tool: ModuleType) -> None:
with pytest.raises(ValidationError, match="commit_short must match"):
wandb_import_tool.BenchmarkIdentityMetadata(
kind="experiment",
commit_sha="abcdef0123456789abcdef0123456789abcdef01",
commit_short="1234567",
)


@pytest.mark.parametrize(
("role", "kind"),
[
("main-baseline", "pr"),
("release-baseline", "main"),
("candidate", "main"),
("candidate", "release"),
],
)
def test_wandb_benchmark_identity_rejects_incompatible_role_kind_pairs(
role: str, kind: str, wandb_import_tool: ModuleType
) -> None:
with pytest.raises(ValidationError, match="incompatible"):
wandb_import_tool.BenchmarkIdentityMetadata(
role=role, kind=kind, pr_number=210 if kind in {"pr", "branch"} else None
)


def test_wandb_run_tags_filter_sensitive_generated_values(wandb_setup_tool: ModuleType) -> None:
metadata = wandb_setup_tool.WandbRunMetadata.model_validate(
{
Expand Down Expand Up @@ -539,7 +567,7 @@ def test_wandb_environment_isolates_routing_and_restores_exactly(
with wandb_setup_tool.WandbSdkEnvironment(settings):
assert os.environ["WANDB_GROUP"] == "resolved-group"
assert os.environ["WANDB_PROJECT"] == "resolved-project"
assert "WANDB_API_KEY" not in os.environ
assert os.environ["WANDB_API_KEY"] == "auth-token"
assert os.environ["WANDB_ERROR_REPORTING"] == "false"
assert "UNRELATED" not in os.environ
with pytest.raises(RuntimeError, match="nested or concurrent"):
Expand Down
42 changes: 30 additions & 12 deletions tools/measurement/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,9 +300,9 @@ and `WANDB_TAGS` are deliberately ignored. Precedence is an explicit CLI value,
then the corresponding `ANONYMIZER_MEASUREMENT_WANDB_*` variable, then the
publisher default. The W&B SDK still receives its audited timeout variables,
including `WANDB_HTTP_TIMEOUT` and `WANDB_INIT_TIMEOUT`. Authentication comes
from the SDK's local credential files under the preserved home directory; the
publisher removes `WANDB_API_KEY` from the SDK process environment. Set a
self-hosted endpoint through `--wandb-base-url` or
from the SDK's local credential files under the preserved home directory or
from `WANDB_API_KEY` when the launcher provides it in the process environment.
Set a self-hosted endpoint through `--wandb-base-url` or
`ANONYMIZER_MEASUREMENT_WANDB_BASE_URL`; ambient `WANDB_BASE_URL` is ignored.
Remote endpoints require HTTPS. Plain HTTP is accepted only for loopback
development endpoints. SDK error reporting is disabled before W&B is imported.
Expand Down Expand Up @@ -362,8 +362,9 @@ current user. Keep the directory when an offline run must be synchronized later.
During SDK use, a process-wide guard replaces the process environment with a
minimal runtime allowlist, audited W&B timeout settings, and the fully resolved
publisher settings. Local W&B credential files remain available through the
preserved home directory. Environment credentials, proxy settings, custom CA
settings, and ambient `WANDB_*` values are withheld from the SDK. The
preserved home directory, and `WANDB_API_KEY` remains available when explicitly
provided by a launcher. Proxy settings, custom CA settings, and ambient
`WANDB_*` routing values are withheld from the SDK. The
guard restores the exact original environment after the explicit run handle finishes.
Nested or concurrent native publishers in one process are rejected. Each
native run receives a fresh opaque 128-bit ID and uses `resume="never"`.
Expand All @@ -375,16 +376,16 @@ the native runner logs only the exception type to avoid echoing source values.

Before an external launcher starts benchmark work, verify W&B authentication from
the same account or container and the same preserved home directory used by the
publisher. Keep credentials in the local W&B credential files; do not pass a key
through the launcher environment or command line.
publisher. Keep credentials in the local W&B credential files or pass
`WANDB_API_KEY` through the launcher environment. Do not pass a key on the
command line.

```bash
# W&B public cloud
env -u WANDB_API_KEY uv run wandb login --verify --cloud
# W&B public cloud, using a stored SDK credential
uv run wandb login --verify --cloud

# Self-hosted or dedicated cloud
env -u WANDB_API_KEY uv run wandb login --verify \
--host https://wandb.example.com
# Self-hosted or dedicated cloud, using a stored SDK credential
uv run wandb login --verify --host https://wandb.example.com
```

Both commands verify the stored credential against the selected endpoint and
Expand Down Expand Up @@ -434,6 +435,23 @@ uv run python tools/measurement/import_wandb_run.py \
--json
```

External launchers can add safe benchmark identity fields to W&B config for
filtering and comparisons:

```bash
--benchmark-role candidate \
--benchmark-kind pr \
--suite-version 2026-07-30 \
--branch contributor/feat/wandb-benchmark-identity \
--commit-sha 0123456789abcdef0123456789abcdef01234567 \
--pr-number 210 \
--anonymizer-config-id rat-rewrite-throughput \
--anonymizer-mode rewrite
```

`--pr-number` is required for `--benchmark-kind pr` and `branch`. Main and
release baselines can omit it.

The importer captures the seal and JSONL once, verifies their content binding,
builds the complete typed payload, then initializes W&B. It derives a stable
128-bit run ID from the destination, sealed case identity, seal digest, schema
Expand Down
Loading
Loading