feat(studio): instrument the audio FX rack, including work an agent did

[vanceingalls]

Nothing in the audio FX feature reported anything, so every question about it — which presets get used, whether the named jobs beat a bare peaking, whether anybody touches the one-knob profiles — was unanswerable.

Dashboard: Audio FX rack — usage (1986431, project 356858). 12 HogQL tiles. Every query verified to execute; the reach tile already returns its real denominator (~25–30k studio installs/week). The audio_fx tiles stay empty until a build carrying these events ships.

The events

studio:audio_fx_* through the existing trackStudioEvent transport, defined in one module (audioFxTelemetry.ts) rather than scattered across six components, because the shapes have to agree for a dashboard to join them.

Presets applied / removed / amount / automated / auditioned; nodes added (with via: job|effect|eq), removed, bypassed, moved; parameters committed (with surface: knob|details); profile knobs; carve changes; the leveller; the Tone EQ. track_kind rides on all of them, so every answer can be read per material.

Two rules the module states and enforces:

• Nothing user-authored leaves the browser. No element ids, no filenames, no chain JSON, no parameter values in the provenance payload — only catalogue identifiers, counts and booleans. Tested.
• Commit, not preview. Every rack control has a continuous preview path and a commit path; wiring the preview would emit tens of events per drag and describe mouse travel rather than decisions. The shelf's hover-audition is deduped to once per preset per shelf-open for the same reason.

Agent attribution

The harder half, in two tiers.

1. The CLI already fingerprints twelve agent runtimes from its own environment, and Studio cannot see any of it. So buildCliIdentityScript now publishes the resolved category as window.__HF_CLI_AGENT_RUNTIME, and both studio session-property builders attach it to every event — not just audio ones, since "person or agent" is worth asking of any feature. Gated on telemetry being on, unlike the canary decisions: those exist so Studio does not enrol, this only labels an event, and with telemetry off there is no event to label.

2. An agent asked to fix a mix does not drive the panel — it edits the composition HTML or runs scripts/carve.mjs, and none of the events above will ever fire. So the panel watches the chain's own shape and reports audio_fx_chain_observed when it changes, carrying how many edits this session had made. A chain that moved while that counter stood still moved because something outside the studio moved it. The write path is wrapped once rather than counted at ten callsites — the count is only meaningful if it is exhaustive.

agent_runtime is encoded as "none" rather than omitted, on both transports: most sessions are people, that is a finding, and this project's telemetry has already produced one wrong conclusion from comparing a populated sentinel against an absence.

Verified

• The real detector returns claude_code in an agent-driven shell.
• Injection across 5 cases, including the script-tag escape and the telemetry-off case.
• The studio accessor across 5 cases, including the empty-string-is-not-a-value case.
• Event shapes across 11 cases, including an assertion that no chain contents reach the payload.
• Every dashboard query executes.

3,739 studio tests, 311 CLI telemetry/server tests, lint and tsc clean.

Not verified, and why

No live PostHog event from this checkout. browserTelemetryAllowed() is false under Vite dev, and the CLI's isDevMode() is true whenever it runs from .ts source. Those guards exist to stop developers polluting production telemetry, and I did not defeat them to produce a demo event. First real data arrives when a released build runs this code.

Left out deliberately

• scripts/carve.mjs is not instrumented. Its output is already identifiable by the fromCarve tag on the nodes it writes, so tier 2 covers it without giving a script context its own policy, id and transport to get wrong.
• Render-side fx properties on render_complete would be a good cross-check — separate change.

🤖 Generated with Claude Code

[terencecho]

Verdict: LGTM_WITH_NITS

Empirical claim verification (@ head 9f028bf)

• One-module telemetry (audioFxTelemetry.ts): verified — packages/studio/src/components/editor/audioFxTelemetry.ts:1-234 centralises the audio_fx_* name-space and every callsite (FxSection, FxNodeRow, FxPresetMenu, AudioFxGroup) imports from it.
• "Nothing user-authored leaves the browser": verified for the provenance path — chainShape at audioFxTelemetry.ts:71-92 emits only counts / booleans, and audioFxTelemetry.test.ts:73-76 asserts the serialised payload contains neither "n1" nor "3000". Note the rule as stated in the PR body is scoped to provenance; trackParamCommitted (audioFxTelemetry.ts:145-152) does ship numeric knob values — this matches the "a key without a value would just tell you somebody touched frequency" argument and is consistent with the module's own docstring.
• Agent-runtime plumbing: verified end-to-end. CLI publishes at packages/cli/src/server/telemetryIdentity.ts:160-179 (gated on telemetryShouldTrack()); Studio reads via packages/studio/src/telemetry/agentRuntime.ts:29-42 (memoised, empty-string coerced to null); attaches on every event via studioTelemetry.ts:46 and on the system meta via system.ts:54.
• Shelf audition dedup: propertyPanelFxPresetMenu.tsx:75-82 — useRef(new Set) resets on unmount, one report per preset per shelf-open.

OLD-assumption counterfactuals

• resolveAgentRuntime empty string → null fails under OLD "return raw verbatim" — would emit "" and "none" as two rows.
• memoization test fails under OLD non-cached lookup — a mid-session window.__HF_CLI_AGENT_RUNTIME overwrite would split a session.
• escapes </script> fails under OLD naïve string concat in buildCliIdentityScript.
• chainShape counts distinct presets fails under OLD "count preset nodes" — three telephone/voice-clean nodes across two presets would read as 3.
• authored_outside=false when panelEdits>0 fails under OLD "flag every observation" — every panel edit would double-log as agent work.
• stays silent when telemetry off, even under an agent fails under OLD unconditional publish — leaves a marker in the page of an opted-out user.

Cross-PR coupling

• HfAudioFxNode not extended: no changes to packages/core/src/audioFx.ts. Attribution reads existing fromPreset / fromCarve / fromEq / fromLeveller flags — the parse/serialize allow-list concern does not apply.
• Attempt/result pairing: N/A — every event is a committed action, not a two-phase funnel. preset_auditioned → preset_applied is a sequential hover-then-decide analytic, not attempt/result.
• Retryable-persist black-out: N/A — every emit is BEFORE the persist / mutate call (propertyPanelFxSection.tsx:322-329, 389-393, 471-473, 549, 588, 804-805; AudioFxGroup leveller at :685, :795). No throw-swallowed retries.
• Write-key: phc_zjjbX0PnWxERXrMHhkEJWj9A9BhGVLRReICgsfTMmpx in studioTelemetry.ts:6 — pre-existing, unchanged.

Attribution reliability

Tier-1 spoofable (a user can export CLAUDECODE=1) but low-consequence — mislabels a single self-driven session. Tier-2 (authored_outside = panelEdits === 0) also fires on external edits from any source (another tab, collab, git-pull HMR) — treat "authored_outside" as "not-this-panel", not literally "an agent".

Nits

• AudioFxGroup chain-fingerprint at propertyPanelAudioFxGroup.tsx:224-228 omits fromEq / fromLeveller / enabled from the shape hash — an external bypass-toggle or leveller swap could go unreported. Match nodeOrigin's discriminants for symmetry.
• Same block: useEffect has no dep array — runs on every render. Guarded, but wasteful.

CI

Preflight (lint + format) in preview-regression: FAILURE. Given wa-20* stack tip and recurring oxfmt --check drift (parent memory), highly likely pre-existing stack drift, not this PR. Regression / player-perf / WIP all green. Would call LGTM_PENDING_CI if the Preflight log confirms non-3229 files; on the assumption it's the standard stack condition, LGTM stands.

— Review by tai (pr-review)

[miga-heygen]

Review: feat(studio): instrument the audio FX rack, including work an agent did — #3229

Verdict: LGTM

Comprehensive telemetry tip — 16 event types under audio_fx_* covering the full rack lifecycle: discover (audition), adopt (apply/add), configure (param/profile/amount), organize (move/bypass/automate), remove, and attribute (chain_observed + agent_runtime).

Privacy is clean. Module header states "Nothing user-authored leaves the browser." Implementation follows through: FxEventProperties is typed as scalar-only (no objects/arrays), chainShape() reduces to origin counts and preset counts (never node contents), $ip: null on every batch. The privacy test explicitly verifies param values and node IDs don't appear in serialized shapes.

Prior-slice integration is correct. presetAmount tracked on wet/dry commit, profiles on one-knob commit with surface: "knob" | "details" distinction (directly answers whether the abstraction gets used), copy layer used for display only (no copy text leaks into events), family classification from tints, node origin flags check all four provenance markers.

chain_observed attribution is clever. authored_outside: panelEdits === 0 discriminates agent-written chains (HTML edits bypass the panel) from panel-written chains. The panelEdits counter resets after each observation, so interleaved agent + human edits are correctly discriminated.

Two minor findings (non-blocking):

1. trackParamAutomated is the only event function missing FxTrackContext — it won't emit track_kind, making it a gap in any dashboard that groups rack events by material type. Every other event function follows the pattern.

2. nodeOrigin with fromEq is implemented but not tested (four other branches are). One-line test case would close it.

agentRuntime memoization, trackPresetAuditioned dedup via useRef(new Set()), chainShape comparison short-circuit — all correct for performance. The XSS test for encodeInlineScriptValue with </script> payload is a nice touch.

---

Review by Miga

🤖 Generated with Claude Code