fix(observability): NeMo Relay plugin-config teardown failure-safety + post-finalize guard (follow-up to #41551)

[kshitijk4poor]

What does this PR do?

Two follow-up hardening fixes on top of the bundled NeMo Relay observability plugin lifecycle work in #41551, found while reviewing that PR. Stacks on #41551 (branched from its head).

1. clear()-failure no longer strands the runtime

_clear_plugins_toml flipped _plugin_config_initialized / _plugin_config_needs_reinit only after _resolve_awaitable(clear()) returned. close_session catches and logs a raising clear(), so on failure the flags stayed initialized=True / needs_reinit=False — and no later session would ever reinitialize plugins.toml or activate the direct fallbacks. The exporter was silently dead for the rest of the process.

Fix: flip the flags in a finally, so a partial/failed clear still re-arms reinit (or the direct fallback) on the next session start.

2. ensure_session no longer resurrects a finalized session

Any hook that funnels through ensure_session after the terminal close_session for an id re-initialized plugins.toml and pushed a scope that was never popped — silently re-arming an exporter for an ended session (and leaking the scope). on_session_end is the named example.

Fix: track a bounded set of recently-finalized session ids. A stray hook for a finalized id returns a detached, unstored _SessionState (no reinit, no scope push). An explicit on_session_start (create=True) clears the marker, so a genuine restart of the same id still works.

Note on severity: (2) has no production trigger today — the plugin on_session_end hook isn't invoked anywhere in cli.py/gateway/, and per-turn hooks fire during run_conversation, before finalize. It's defense against a future hook addition. (1) is a real latent bug that triggers whenever the relay's clear() raises.

Verification

• _resolve_awaitable repro: drove both failure modes — clear-failure recovers (reinit fires on next start), stray post-finalize hook produces no resurrection and balanced scope.push/scope.pop.
• Added 3 regression tests; the first two fail against the pre-fix code (verified by reverting the prod change and re-running).
• tests/plugins/ full suite green (1089 passed), ruff clean. Behavior-preserving for all existing paths (24 prior nemo_relay tests still pass).

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• Tests (adding or improving test coverage)

Related

Follow-up to #41551 (and its parent #38232). Branched from #41551's head — merge that first, or treat these two commits as fixups to fold in.

[kshitijk4poor]

Closing — these two are mnajafian-nv's to address on #41551 directly (already flagged inline there). Opened this prematurely; the warnings stand as review comments on #41551, not a separate PR.