feat(external-context): Add opt-in auto recall for administrator-owned Mem0 dialects

docs/design/external-context-mem0-auto-recall.md

@@ -0,0 +1,197 @@
+# Mem0 External Context Auto Recall
+
+**Status:** Implemented
+
+**Date:** 2026-09-04
+
+## Decision
+
+Add an administrator-installed `UserPromptSubmit` command Hook to the public
+Mem0 External Context package. The Hook reuses the administrator-owned
+`DialectV1`, bounded request engine, and untrusted result envelope without
+changing Qwen Core or the default Extension manifest.
+
+The retrieval profiles are mutually exclusive:
+
+- **On-demand:** `InstanceConfigV2`, the default Extension manifest, and the
+  `context_search({ query })` MCP tool.
+- **Auto recall:** `InstanceConfigV3`, an administrator-installed Hook, and no
+  Mem0 External Context MCP server.
+
+The default manifest does not register the Hook. Installing or upgrading the
+Extension therefore cannot cause automatic prompt forwarding or add a process
+spawn to existing user turns. Auto recall requires both a v3 configuration and
+an explicit Hook registration in an administrator-controlled `QWEN_HOME`.
+
+## Scope
+
+### Goals
+
+- Perform at most one provider search for an eligible user submission.
+- Preserve the v2 MCP configuration and `context_search` contract.
+- Keep the endpoint, credential, scope, dialect, and repository binding outside
+  model control.
+- Use only `submitted_prompt` provenance captured before model-bound expansion.
+- Reduce accidental credential forwarding before a query leaves the host.
+- Inject only bounded, structured, untrusted user-layer context.
+- Fail open with bounded latency and no integration-generated request logs.
+
+### Non-goals
+
+- Memory creation, update, deletion, ingestion, or automatic memory extraction.
+- Built-in provider presets or provider-specific dialects.
+- Qwen Core changes or conditional Extension-manifest features.
+- DLP, user authentication, tenant authorization, or compliance audit.
+- Input paths that do not provide `submitted_prompt`.
+- Retry, redirect, cache, protocol probing, or a persistent Hook process.
+- Preventing indirect prompt injection from retrieved content.
+
+## Configuration
+
+`InstanceConfigV2` remains the exact on-demand schema. Auto recall uses a
+separate canonical schema and `schemaVersion: 3`:
+
+```json
+{
+  "schemaVersion": 3,
+  "autoRecall": {
+    "repositoryRoot": "/absolute/path/to/repository"
+  },
+  "dialectPath": "/etc/qwen/external-context/memory.dialect.json",
+  "endpoint": {
+    "origin": "https://memory.example.com",
+    "basePath": "",
+    "allowInsecureHttp": false
+  },
+  "credentialEnv": "MEMORY_API_KEY",
+  "scope": {
+    "userId": "repository-memory"
+  },
+  "timeoutMs": 1500
+}
+```
+
+The Hook and MCP entry points share
+`QWEN_EXTERNAL_CONTEXT_MEM0_CONFIG`, but accept different configuration
+versions. The MCP loader accepts only v2 and the Hook loader accepts only v3.
+This rejects accidental cross-mode use while preserving existing deployments.
+The auto-recall timeout must be from 100 through 5000 milliseconds.
+
+`repositoryRoot` must be an existing absolute directory and cannot be a
+filesystem root. The loader resolves it through `realpath`. Each event `cwd` is
+also resolved through `realpath`; retrieval runs only for the configured root
+or a descendant. The repository check prevents accidental corpus reuse after a
+directory change. Provider-side credentials and authorization remain the
+actual security boundary.
+
+`DialectV1` is unchanged. It remains a closed administrator-owned grammar and
+cannot define arbitrary headers, templates, transformations, or write methods.
+
+## Runtime flow
+
+Each eligible invocation starts a new Node process:
+
+1. Read at most 1 MiB of Hook JSON from stdin.
+2. Require `hook_event_name: "UserPromptSubmit"`, a non-empty
+   `submitted_prompt`, and a string `cwd`.
+3. Load and validate `InstanceConfigV3`, `DialectV1`, the canonical repository
+   root, and the named credential.
+4. Resolve `cwd` and skip retrieval when it is outside the configured root.
+5. Remove fenced code, the exact configured credential, and common secret
+   shapes; collapse whitespace and keep at most 512 Unicode code points.
+6. Call the existing request engine once with the configured timeout.
+7. Return at most five results through `UserPromptSubmit.additionalContext` as
+   the existing `untrusted_external_context` envelope.
+8. Emit `{}` for missing provenance, mismatched paths, empty results, invalid
+   configuration, timeouts, transport failures, or invalid provider responses.
+
+The Hook never reads or falls back to the legacy `prompt` field. Eligibility
+depends on `submitted_prompt`, not the input transport: supported TUI
+submissions and headless CLI user turns supply it, including `qwen -p` and
+stream-json user messages from SDK clients. Events without this field, such as
+tool-result continuations, skip retrieval. Do not infer a TUI-only origin or
+exclude a transport merely from its name.
+
+Configuration and dialect paths must resolve to regular files and are bounded
+to 64 KiB. Nonblocking open and descriptor validation reject FIFOs before a
+filesystem worker can block waiting for a writer. The long-running MCP
+process reads them once at startup. The command Hook reads them once per
+eligible invocation, so administrator file changes apply to the next eligible
+submission; changing its environment or Hook registration requires restarting
+Qwen.
+
+## Bounds and failure semantics
+
+- Sanitizer input: 4096 Unicode code points.
+- Provider query: 512 Unicode code points.
+- Provider timeout: 100-5000 ms.
+- Internal Hook wall-clock budget: 6500 ms.
+- Qwen command-Hook timeout: 8000 ms.
+- Provider response: 1 MiB before JSON parsing.
+- Output: five items, 1000 Unicode code points per content field, and 4000
+  JavaScript code units for the serialized envelope.
+
+There is no retry, redirect, or cache. The Hook writes exactly one JSON object
+to stdout and emits no integration-generated stderr. Once the pinned Node
+entry point starts, handled failures return `{}` with exit code zero. The
+executable flushes stdout and explicitly exits so abandoned connections do not
+keep the event loop alive after the result is ready. Secret-assignment
+matching starts at identifier boundaries and checks the keyword separately from
+the assignment suffix, avoiding overlapping scans on repeated-keyword inputs.
+A launcher failure before Node starts or an outer Qwen timeout follows the
+command-Hook runner's own error policy; ordinary runner timeouts are nonfatal
+but delay the turn. Administrators must validate the fixed binary and bundle
+paths before rollout.
+
+Sanitization is a best-effort reduction, not DLP. The external provider may log
+the sanitized query. Retrieved content is sent to the model provider and may be
+persisted in the session transcript. The untrusted envelope and Qwen's reserved
+Hook-context wrapper preserve provenance but do not prevent the model from
+following malicious retrieved instructions.
+
+## Deployment
+
+The package ships `dist/auto-recall.js`, the v3 schema, and unbranded POSIX and
+Windows Hook examples. It ships no provider preset or provider dialect.
+
+The administrator installs a pinned package version at a stable absolute path,
+creates the v3 instance and dialect files outside ordinary workspaces, injects
+the configuration path and credential through the managed process environment,
+and copies the applicable Hook definition into an administrator-controlled
+`QWEN_HOME/settings.json`.
+
+This registration opts in every eligible input handled by that launcher,
+including headless and stream-json user turns. Automation can disable all Hooks
+with `--bare`, `--safe-mode`, or `disableAllHooks: true` in managed settings
+before startup. The two flags also change which customizations are loaded.
+When automation needs other Hooks, use a separate controlled `QWEN_HOME`
+without this Hook and omit its configuration and credential from the
+automation environment. A protocol-level TUI-only filter would require a
+separate Core/CLI provenance change and is outside this package-only design.
+
+The auto-recall process must not enable the package's default Extension
+manifest or configure another on-demand Mem0 MCP server. Otherwise one turn
+could produce both a deterministic Hook request and a model-selected MCP
+request. A separate pinned installation path is preferred for the Hook-only
+profile.
+
+Rollback removes the Hook registration and credential from the managed
+launcher and restarts Qwen. It does not delete provider records or access logs.
+
+## Verification
+
+Unit tests cover strict v2/v3 parsing, canonical roots, containment, missing
+provenance, legacy-prompt isolation, input limits, credential patterns, Unicode
+bounds, one-request behavior, fail-open output, timeouts, and final context
+bounds. Package test commands build the shipped bundles before running tests.
+Local subprocess tests execute the Hook bundle with a fake provider and cover
+configuration, `DialectV1`, exact outbound requests, flushed Hook stdout, and
+successful process exit during a stalled TLS handshake and rejection of instance
+or dialect FIFOs. Repeated secret-keyword
+near misses are tested in bundle-importing subprocesses with a real deadline.
+
+Package verification builds both entry points and inspects `npm pack --dry-run`
+to confirm that the tarball contains the runtime, schemas, manifest,
+documentation, and unbranded Hook examples, with no provider-specific data.
+The default manifest test continues to require exactly `context_search` and no
+Hook registration.

docs/design/external-context-mem0-extension.md

@@ -1,6 +1,6 @@
 # Administrator-configured Mem0 External Context Extension
 
-**Status:** PR2 implementation
+**Status:** Implemented base integration; optional Auto Recall follow-up implemented
 
 **Date:** 2026-08-28
 
@@ -23,12 +23,20 @@ absolute dialect path. The dialect file describes request and response
 differences within the existing closed `DialectV1` grammar. Its `id` is an
 administrator-managed audit label and has no registry or file-name semantics.
 
-External Context MCP Profile v1 remains the only public Qwen interoperability
-boundary. Qwen Core does not gain a provider registry, public provider SDK,
+External Context MCP Profile v1 remains the on-demand interoperability
+boundary. The optional automatic mode uses Qwen's existing
+`UserPromptSubmit` Hook contract; it does not add a provider-specific Qwen
+interface. Qwen Core does not gain a provider registry, public provider SDK,
 dynamic module loading, or new third-party cases in its private
 `ProviderConfig` union. The existing direct integration remains available for
 compatibility and is not modified by this design.
 
+The default Extension manifest remains MCP-only. Administrators may separately
+install the package's opt-in `UserPromptSubmit` command Hook to retrieve context
+before a prompt. That deployment mode is specified in
+[Opt-in Auto Recall for Administrator-configured Mem0](./external-context-mem0-auto-recall.md)
+and is intentionally not enabled by installing the Extension.
+
 ## Goals
 
 - Let administrators connect retrieval-only compatible services without a
@@ -37,6 +45,8 @@ compatibility and is not modified by this design.
   endpoint, credential, scope, timeout, and dialect remain administrator-owned.
 - Preserve the bounded request engine, response normalization, and failure
   behavior already implemented by the Extension.
+- Preserve zero automatic retrieval for the default Extension installation
+  while allowing an administrator-owned Hook profile to opt in explicitly.
 - Give protocols outside the closed grammar a clear path to a separate local
   or remote MCP Extension.
 
@@ -45,10 +55,11 @@ compatibility and is not modified by this design.
 - Publish provider presets, provider-specific contract fixtures, or live
   service credentials.
 - Add ordinary Qwen settings or Extension settings for the configuration path.
-- Define arbitrary request templates, JSONPath, scripting, custom headers, or
-  executable hooks.
+- Define arbitrary request templates, JSONPath, scripting, or custom headers.
 - Probe or fall back between upstream protocol versions.
-- Add memory creation, update, deletion, Auto Recall, redirects, or retries.
+- Enable Auto Recall by default or combine the on-demand MCP and automatic Hook
+  surfaces in one installation profile.
+- Add memory creation, update, deletion, redirects, or retries.
 - Migrate or remove the existing direct External Context integration.
 
 ## Architecture
@@ -61,21 +72,25 @@ flowchart LR
     D["Administrator-owned DialectV1"] --> E
     E --> R["Bounded request engine"]
     R --> S["Compatible HTTP service"]
+    H["Optional administrator-owned UserPromptSubmit Hook"] --> R
+    A["Administrator-owned InstanceConfigV3"] --> H
 ```
 
-The local Extension owns configuration loading and HTTP translation. Qwen sees
-only the MCP profile. A service that cannot fit the bounded dialect grammar
-owns a separate MCP implementation instead of expanding the grammar or Qwen
-Core.
+The local package owns configuration loading and HTTP translation. On-demand
+retrieval reaches it through the MCP profile; the optional automatic mode
+reaches it through the existing command-Hook contract. A service that cannot
+fit the bounded dialect grammar owns a separate MCP implementation instead of
+expanding the grammar or Qwen Core.
 
 ## Version model
 
 Four independent version axes remain explicit:
 
 1. **MCP Profile version** defines the Qwen-to-Extension tool contract. This
    Extension implements External Context MCP Profile v1.
-2. **Instance schema version** defines administrator binding. This design uses
-   `schemaVersion: 2`.
+2. **Instance schema version** defines administrator binding. The MCP runtime
+   uses `schemaVersion: 2`; the separate Auto Recall Hook profile uses
+   `schemaVersion: 3` so each entry point rejects the other's configuration.
 3. **Dialect version** defines interpretation of the closed request and
    response grammar. This design keeps `dialectVersion: 1` unchanged.
 4. **Upstream API version** belongs to the service and appears only in the
@@ -175,9 +190,9 @@ The grammar stays deliberately closed:
 - `threshold` and `rerank` are typed fields, not request fragments.
 
 The dialect cannot define arbitrary headers, body interpolation, JSONPath,
-code, environment-variable expansion, redirects, or response transformations.
-Its `id` does not have to match its file name or any value in the instance
-file. Administrators own its naming and versioning policy.
+code, environment-variable expansion, redirects, response transformations, or
+trigger behavior. Its `id` does not have to match its file name or any value in
+the instance file. Administrators own its naming and versioning policy.
 
 ## Startup and failure behavior
 
@@ -217,16 +232,20 @@ query, credential, or upstream response.
 
 ## Retrieval-only boundary
 
-The manifest exposes exactly `context_search`. A dialect cannot enable memory
-creation, update, deletion, or Auto Recall. Write protocols require a separate
-future profile or Extension because their idempotency, duplication, timeout,
-and authorization semantics do not fit the retrieval grammar.
+The default manifest exposes exactly `context_search` and contains no Hooks. A
+dialect cannot enable memory creation, update, deletion, or Auto Recall. The
+optional administrator-installed Hook changes only when the same bounded
+retrieval runs; it does not add write operations or expand the dialect grammar.
+Write protocols require a separate future profile or Extension because their
+idempotency, duplication, timeout, and authorization semantics do not fit the
+retrieval grammar.
 
 ## Packaging and service ownership
 
-The npm package publishes only the bundled runtime, canonical schemas,
-Extension manifest, and README. It contains no administrator dialect, provider
-preset, provider identifier, or provider-specific contract fixture.
+The npm package publishes only the bundled MCP and Auto Recall entry points,
+canonical schemas, Extension manifest, unbranded Hook configuration examples,
+and README. It contains no administrator dialect, provider preset, provider
+identifier, or provider-specific contract fixture.
 
 The public package name is `@qwen-code/external-context-mem0`. Its package and
 Extension manifest versions follow the Qwen Code release version and are
@@ -265,16 +284,20 @@ built-in provider rollout for either case.
    and profile boundary.
 4. **Distribution follow-up:** Publish the self-contained Extension through the
    normal Qwen Code npm release without adding provider data or Core wiring.
-5. Design any portable write capability separately.
+5. **Auto Recall follow-up:** Add a separately installed, administrator-owned
+   Hook profile while leaving the default manifest MCP-only.
+6. Design any portable write capability separately.
 
 There is no Qwen-maintained provider-preset PR3. Administrators own compatible
 dialect data; incompatible protocols use their own MCP Extension.
 
 ## Verification
 
-Verification covers both canonical schemas; 64 KiB file limits; unavailable,
+Verification covers all canonical schemas; 64 KiB file limits; unavailable,
 malformed, unsupported, relative, and semantically invalid configurations;
 credential ordering; synthetic GET and POST request contracts; response
-normalization; the MCP tool surface; real stdio MCP startup against a local
-synthetic HTTP service; restart-only reload behavior; package contents; build,
-typecheck, lint, and tests. No verification contacts a live provider service.
+normalization; the MCP tool surface; Hook provenance, repository containment,
+sanitization, fail-open behavior, and wall-clock bounds; real stdio MCP and Hook
+startup against local synthetic HTTP services; reload behavior; package
+contents; build, typecheck, lint, and tests. No verification contacts a live
+provider service.