fix(langfuse): reject right-prefix template keys, not just wrong prefixes

[Solitud1nem]

What does this PR do?

The placeholder guard from #26320 checks only that credentials start with pk-lf- / sk-lf-. But the plugin README and the website env-var reference both print the keys as literally pk-lf-... / sk-lf-... — and that exact copy-paste passes the prefix check on today's main: the SDK constructs a client without complaint and every trace dies at flush time with zero log output, which is precisely the silent failure the guard was built to catch. The new regression test proves it (test_docs_template_keys_warn_and_skip fails on main).

_validate_langfuse_key() now looks past the prefix. A remainder that is a template (..., ***, empty — i.e. a bare prefix) or contains an unambiguous template fragment (placeholder, your-, test-key, changeme, example, unset, dummy, xxx) gets the same one-shot warning and _INIT_FAILED short-circuit as a wrong-prefix value. False positives are structurally impossible for issued keys: Langfuse remainders are UUID-shaped (hex digits and dashes), and every fragment in the list contains at least one non-hex letter — a property the tests pin. A side effect worth noting: the docs' own pk-lf-... template is now caught, so the copy-paste footgun closes without touching the docs or their translations.

Context on the report chain: #51399 has six duplicates filed after #26320 merged. Some of that stream is likely version lag (triage noted "the fix ships once a release tag carries #26320"), so I'm not claiming every duplicate hit this exact hole — but the hole itself is real and test-proven above, and it's the most likely paste value in existence.

Deliberately out of scope, two things: (1) surfacing runtime ingestion failures — #62882 is already reworking that path, and this PR stays clear of it; (2) the adjacent gap Bartok9 described on #52389: a revoked or invalid key with a real UUID shape can't be caught statically at all — that needs a live auth probe (e.g. the SDK's auth_check()) with its own network/offline tradeoffs, and deserves its own PR.

Related Issue

Fixes #51399

Duplicate chain (all marked duplicates of #51399 by triage; canonical original #22763 is closed): #52377, #52389, #57949, #60308, #60961, #63787.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• plugins/observability/langfuse/__init__.py — _validate_langfuse_key() gains a remainder check after the existing prefix check: whole-value templates by exact match, conservative fragments by substring, case-insensitive; new _PLACEHOLDER_REMAINDERS / _PLACEHOLDER_FRAGMENTS constants with the false-positive reasoning documented inline
• tests/plugins/test_langfuse_plugin.py — 13 new tests in the existing TestPlaceholderKeyDetection class: 10 parametrized right-prefix templates rejected (including the docs' literal pk-lf-...), 2 UUID-shaped real keys still accepted, and an end-to-end _get_langfuse() case proving the docs copy-paste now warns and never constructs a client

How to Test

1. HERMES_LANGFUSE_PUBLIC_KEY=pk-lf-... + HERMES_LANGFUSE_SECRET_KEY=sk-lf-... (the literal values from the plugin README), run any traced operation.
2. Before: plugin initializes silently, Langfuse dashboard stays empty. After: one-shot warning naming both env vars ("has the 'pk-lf-' prefix but looks like a template value"), plugin short-circuits.
3. Real keys (pk-lf-<uuid>): behavior unchanged.
4. pytest tests/plugins/test_langfuse_plugin.py -q — 61 passed; the 11 new rejection tests fail on main.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate (checked all 37 closed + 21 open langfuse PRs: the only prior attempt at this validator, fix(observability/langfuse): surface placeholder & runtime ingestion failures (#60961) #60974, was author-closed unreviewed after 3 minutes; open fix(langfuse): repair silent trace-loss chain in observability plugin #62882 targets a different root cause — cross-thread OTEL tokens — in a different region of the file)
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run the affected suite: tests/plugins/test_langfuse_plugin.py — 61 passed
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Windows 11

Documentation & Housekeeping

• I've updated relevant documentation — N/A by design: the docs' pk-lf-... template is now caught by the guard, closing the copy-paste footgun without churning docs and their translations
• I've updated cli-config.yaml.example — N/A (no config keys)
• I've updated CONTRIBUTING.md or AGENTS.md — N/A
• I've considered cross-platform impact — platform-neutral string validation
• I've updated tool descriptions/schemas — N/A

Screenshots / Logs

Fail-before / pass-after on today's main:

# without the fix
FAILED ...test_validate_langfuse_key_rejects_right_prefix_templates[...pk-lf-...] (x10)
FAILED ...test_docs_template_keys_warn_and_skip
11 failed, 50 passed

# with the fix
61 passed

[teknium1]

Thanks for isolating the residual right-prefix case. Current main's _validate_langfuse_key() accepts every value beginning with pk-lf- / sk-lf- at plugins/observability/langfuse/__init__.py:141-142, while the bundled README and environment-variable reference both show literal pk-lf-... / sk-lf-... examples (plugins/observability/langfuse/README.md:24-25; website/docs/reference/environment-variables.md:185-186). The patch extends the existing validator and preserves the existing one-shot _INIT_FAILED short-circuit in _get_langfuse().

The PR base has no current-main changes in either touched file, so this is mechanically salvageable without target-region conflict.

Automated hermes-sweeper review.

[PRATHAMESH75]

Reviewed in the context of #66960 (another placeholder-key silent-failure report), which #66960 was triaged as a duplicate of #51399.

This PR does fix the outstanding class of the bug. The root cause is correctly identified: the guard from #26320 checks only value.startswith('pk-lf-' / 'sk-lf-'), so the docs' own copy-paste templates (pk-lf-..., sk-lf-...) sail through and then die silently at flush time. _validate_langfuse_key() now also inspects the remainder after the prefix and rejects bare/template values (_PLACEHOLDER_REMAINDERS) and template fragments (_PLACEHOLDER_FRAGMENTS), returning the same warning + _INIT_FAILED short-circuit path.

Coverage vs the reports is solid:

• The reproduction case that kept the duplicates coming — pk-lf-... / sk-lf-... straight from the README/website — is now caught (test_docs_template_keys_warn_and_skip, and plugins/observability/langfuse/__init__.py remainder check).
• False-positive safety for real keys is pinned explicitly (test_validate_langfuse_key_accepts_real_shaped_keys): Langfuse issues UUID-shaped remainders (hex + dashes), and every fragment in the list contains at least one non-hex letter, so an issued key can't be flagged. Nice invariant, and it's tested rather than asserted in prose only.
• Re Langfuse SDK plugin: placeholder API key silent failure #66960's literal examples (sk-..., your-key-here, sk-placeholder): those are all wrong-prefix values, already handled by the pre-existing not value.startswith(expected) branch on main. So main + this PR together cover the full placeholder class Langfuse SDK plugin: placeholder API key silent failure #66960 describes — this PR closes the remaining right-prefix gap.

Minor things to weigh (none blocking):

1. xxx is the shortest, loosest fragment (3 chars, substring match). It's safe against hex/dash remainders by the stated invariant, but if Langfuse ever issued a non-UUID remainder it's the one most likely to collide. Given the documented key shape it's fine today; just the fragment I'd keep an eye on if key formats ever change.
2. This intentionally implements the warn + short-circuit design rather than the active startup ping some reports (incl. Langfuse SDK plugin: placeholder API key silent failure #66960) floated ("test ping to the Langfuse server"). That's the right call IMO — a network probe at init adds latency/failure modes and the template-detection approach catches the actual reported misconfigurations without it — worth stating explicitly in the issue so the "ideal: ping" suggestion is consciously declined, not overlooked.

Overall: correct root cause, matches the reproduction, well-tested (both the reject and accept directions), and minimal-footprint. LGTM for closing the #51399 / #66960 placeholder-key line once merged.

[Solitud1nem]

Reviewed in the context of #66960 (another placeholder-key silent-failure report), which #66960 was triaged as a duplicate of #51399.

This PR does fix the outstanding class of the bug. The root cause is correctly identified: the guard from #26320 checks only value.startswith('pk-lf-' / 'sk-lf-'), so the docs' own copy-paste templates (pk-lf-..., sk-lf-...) sail through and then die silently at flush time. _validate_langfuse_key() now also inspects the remainder after the prefix and rejects bare/template values (_PLACEHOLDER_REMAINDERS) and template fragments (_PLACEHOLDER_FRAGMENTS), returning the same warning + _INIT_FAILED short-circuit path.

Coverage vs the reports is solid:

• The reproduction case that kept the duplicates coming — pk-lf-... / sk-lf-... straight from the README/website — is now caught (test_docs_template_keys_warn_and_skip, and plugins/observability/langfuse/__init__.py remainder check).
• False-positive safety for real keys is pinned explicitly (test_validate_langfuse_key_accepts_real_shaped_keys): Langfuse issues UUID-shaped remainders (hex + dashes), and every fragment in the list contains at least one non-hex letter, so an issued key can't be flagged. Nice invariant, and it's tested rather than asserted in prose only.
• Re Langfuse SDK plugin: placeholder API key silent failure #66960's literal examples (sk-..., your-key-here, sk-placeholder): those are all wrong-prefix values, already handled by the pre-existing not value.startswith(expected) branch on main. So main + this PR together cover the full placeholder class Langfuse SDK plugin: placeholder API key silent failure #66960 describes — this PR closes the remaining right-prefix gap.

Minor things to weigh (none blocking):

1. xxx is the shortest, loosest fragment (3 chars, substring match). It's safe against hex/dash remainders by the stated invariant, but if Langfuse ever issued a non-UUID remainder it's the one most likely to collide. Given the documented key shape it's fine today; just the fragment I'd keep an eye on if key formats ever change.
2. This intentionally implements the warn + short-circuit design rather than the active startup ping some reports (incl. Langfuse SDK plugin: placeholder API key silent failure #66960) floated ("test ping to the Langfuse server"). That's the right call IMO — a network probe at init adds latency/failure modes and the template-detection approach catches the actual reported misconfigurations without it — worth stating explicitly in the issue so the "ideal: ping" suggestion is consciously declined, not overlooked.

Overall: correct root cause, matches the reproduction, well-tested (both the reject and accept directions), and minimal-footprint. LGTM for closing the #51399 / #66960 placeholder-key line once merged.

Thanks — that's a genuinely careful read, and you caught the two things worth calling out.

On xxx: agreed it's the loosest fragment, and it's the one I'd revisit first. The invariant it leans on is that Langfuse issues UUID-shaped remainders — hex digits and dashes — and x isn't a hex digit, so no issued key can contain it. That's why test_validate_langfuse_key_accepts_real_shaped_keys pins the accept direction rather than leaving it to prose. Worth being explicit about the failure mode though: if the issued-key shape ever stops being UUID-like, xxx isn't the only fragment at risk — the whole remainder heuristic would need rechecking, and that test is the thing that would fail first and tell us.

On the startup ping — declining it deliberately, and putting that on the record here so it's a decision rather than an oversight. A network call at plugin init buys a narrow win (catching a well-formed but revoked key) for a real cost: latency on every startup, a new failure mode when Langfuse is unreachable or slow, and ambiguity about what to do when the probe times out — none of which the reported misconfigurations need. Every reproduction in #51399, #66960 and the rest of the cluster is a template value copy-pasted from our own docs, and that's detectable offline, which is what this PR does. If someone later wants revoked-key detection, that's a separate change with its own trade-off discussion — #52389 is the thread where real-shaped-but-revoked keys came up, and it would need a live auth_check(), not this guard.

And yes — your read on #66960 matches mine: its literal examples are wrong-prefix values already caught on main, so main plus this PR cover the full class it describes.

[alt-glitch]

This was generated by AI during triage.

Related: #51399 tracks the placeholder-key reports, #29340 probes well-shaped invalid credentials, and closed #60974 also covered runtime-ingestion failure surfacing. This focused patch addresses the remaining right-prefix template case.

[andrexibiza]

Independent verification (swarm dedup campaign, worker t_82825c05) — premise re-verified against current main after the Jul 29 suite-wide test prune (6b81590, 46,820 -> 28,106 test functions):

1. Gap still live on main. Reproduced with a direct probe on dd600d1: _validate_langfuse_key('HERMES_LANGFUSE_PUBLIC_KEY', 'pk-lf-...') returns None — the docs-literal template passes the prefix-only check and the SDK silently drops traces at flush. Wrong-prefix values (placeholder, test-key) correctly warn.
2. RED: all 11 new tests in this PR fail against main's plugin code (11 failed, 50 deselected) — matches the PR's own claim.
3. Merge check: PR head (2b8b595) merges cleanly into current main via 3-way merge — no conflicts despite the prune deleting test_validate_langfuse_key_unknown_name_passes, the anchor the new tests insert after.
4. GREEN: merged state runs 37 passed on tests/plugins/test_langfuse_plugin.py, including all 11 new template-rejection tests.

Post-prune green-light confirmed: this PR is the correct fix for the #51399 cluster's residual right-prefix template gap.

[andrexibiza]

Independent verification against current main (2026-08-02, commit dd600d1) — cluster worker for #51399 dedup campaign.

Premise confirmed on current main

• _validate_langfuse_key() at plugins/observability/langfuse/init.py:141-142 still returns None for every value with the right prefix. The docs' literal copy-paste templates pk-lf-... / sk-lf-... pass the guard, the SDK client is constructed, and traces die silently at flush time — exactly the #51399 failure mode. The merged #26320 guard only covered wrong-prefix values.

Empirical test evidence (Windows 11, Python 3.12.12, pytest 9.1.1)

• Clean main (dd600d1): 24 passed, no right-prefix coverage.
• 3-way merge of this PR onto current main (git merge-tree --write-tree origin/main pr64309-head): clean tree, GitHub reports MERGEABLE; the new tests land inside TestPlaceholderKeyDetection.
• Merged tree, full plugin suite: 37 passed (24 baseline + 13 new: 10 parametrized right-prefix rejects, 2 UUID-shaped accepts, 1 docs-template end-to-end warn-and-skip).
• The 13 new tests run against the clean-main plugin (PR tests only, no plugin change): 11 failed — the 10 right-prefix template rejections plus test_docs_template_keys_warn_and_skip — proving the regression tests cover the actual reported hole, not a strawman. The 2 accepts passed as expected.

Notes for the author / reviewer

• The PR's test-file base drifted after suite-wide test prune 6b81590 removed test_validate_langfuse_key_unknown_name_passes (the anchor the test hunks sit on): git apply of the PR diff fails on the test file, but the 3-way merge is clean and all tests pass in the merged result. A rebase/merge of main is advisable for hygiene but is not merge-blocking.
• Matches the keep_open review (teknium1, Jul 16): extension of the existing validator, preserves the one-shot _INIT_FAILED short-circuit, no target-region conflict.

Verdict: green-light. This is the agreed best fix for the #51399 cluster (right-prefix template keys); merge when a maintainer approves.

[Solitud1nem]

Superseded by #77373 — please merge that one instead.

Same change: every added and removed line is identical, and the commit there is mine with authorship intact. Only the context differs, because that branch sits on a post-prune base.

Re-verified today on main 82c6aca: the gap is live on a clean tree ('pk-lf-...' still accepted, 24 passed), and in the tree merged with 77373's head the templates are rejected while UUID-shaped keys still pass (37 passed).

I will close this once #77373 lands. If a maintainer would rather clear it from the queue now, go ahead — the fix is not lost.