feat(cli): apply shared daemon write-routing policy

[fatkobra]

What does this PR do?

Dependency

What does this PR do?

Builds on the shared routing-policy foundation merged in #2027 and the hook-routing implementation merged in #2030.

Contributes to #1963.

This is the third PR in the staged hook and CLI gateway rollout.

It applies the shared direct / prefer / require policy to routine CLI write operations:

• mempalace mine;
• mempalace sweep;
• mempalace sync;
• the optional post-setup mine run by mempalace init.

It also makes sweep a first-class daemon job and preserves the one-time project scan performed by init when that mine is submitted to the daemon.

Policy behavior

direct

Use the existing direct in-process execution path.

prefer

Submit through the local daemon.

Interactive CLI commands are allowed to start the daemon if it is not already running.

require

Submit through the local daemon.

Interactive CLI commands are allowed to start the daemon if it is not already running.

Because CLI commands can start the daemon, both prefer and require normally select it. The distinction remains important for hook callers, which cannot safely cold-start the daemon.

Explicit overrides

Force daemon routing:

mempalace mine ./project --daemon

Force direct execution:

mempalace mine ./project --direct

The options are mutually exclusive and override environment/config policy.

The same options are available for:

• init;
• mine;
• sweep;
• sync.

Background jobs

--background now follows the selected route rather than requiring the literal --daemon flag.

For example, this works when CLI routing is configured as prefer or require:

mempalace mine ./project --background

A direct route with --background exits with an explicit configuration error.

No ambiguous fallback

Once daemon submission begins, an error never causes the command to rerun directly.

The daemon may have durably accepted the job before the client observed a timeout or transport failure. Retrying the operation directly could duplicate memories or race the accepted job.

Post-init mine

mempalace init already scans the project once to show the user a file-count and size estimate.

When the accepted mine is daemon-routed, this PR forwards that exact scanned file list in the daemon payload. The daemon therefore does not need to walk the project a second time.

Sweep support

Before this PR, sweep was direct-only.

This PR adds daemon service support for:

• single transcript files;
• recursively swept directories;
• normal result summaries;
• partial directory failures;
• lock-contention errors;
• backend and palace isolation.

Backward compatibility and rollout

The default CLI policy remains direct.

Existing users therefore receive no silent execution-topology change.

A supervised Tier 3 deployment can opt in with:

MEMPALACE_CLI_WRITE_ROUTING=prefer

or prohibit direct routing with:

MEMPALACE_CLI_WRITE_ROUTING=require

Configuration file:

{
  "write_routing": {
    "cli": "require"
  }
}

The production default can be reconsidered after the stacked rollout is reviewed and merged.

Maintenance exclusions

The following remain outside ordinary write routing:

• repair;
• migration;
• wing migration;
• index rebuild;
• closet compression;
• embedder-identity changes.

These operations may replace indexes, rewrite broad metadata sets, or require cached handles to close. They need an exclusive-maintenance policy rather than an ordinary daemon queue job.

Safety properties

• require never falls back to a direct writer.
• Explicit --direct remains available as a deliberate emergency/debug escape hatch.
• Explicit --daemon remains backward compatible.
• Background mode requires a daemon-selected route.
• Submission errors never cause direct retry.
• The init pre-scan is preserved across daemon routing.
• Sweep jobs share the existing daemon writer serialization.
• No changes are made to mine_palace_lock().
• Hook routing from feat(hooks): apply shared daemon write-routing policy #2030 is not modified.

Tests

Coverage includes:

• CLI direct/prefer/require resolution;
• explicit --daemon and --direct precedence;
• parser-level mutual exclusion;
• direct-background rejection;
• background use under daemon policies;
• mine daemon and direct paths;
• no fallback after daemon-submission failure;
• sync daemon routing;
• sweep daemon routing;
• post-init daemon routing with pre-scanned files;
• post-init direct behavior;
• daemon file and directory sweep execution;
• partial sweep failure reporting;
• daemon job dispatch;
• existing CLI, daemon, policy, config, hook, shell, and writer-lock regressions.

Run:

python3 -m ruff format --check .
python3 -m ruff check .
python3 -m pytest tests/test_cli_write_routing.py tests/test_cli.py tests/test_daemon.py tests/test_write_routing.py tests/test_config.py -q
python3 -m pytest tests/test_hook_write_routing.py tests/test_hooks_cli.py tests/test_hooks_shell.py tests/test_hooks_bash_compat.py -q
python -m pytest tests/ -v

Rollout sequence

1. Shared policy foundation — feat(routing): add shared daemon write-routing policy #2027, merged.
2. Hook routing — feat(hooks): apply shared daemon write-routing policy #2030, merged.
3. Routine CLI routing — this PR.
4. Exclusive maintenance behavior — separate follow-up.
5. Production-default decision — maintainer decision after the stack is validated.

References

Contributes to the Tier 3 rollout in #1963.

Builds on #2027 and #2030, both merged.

Uses the daemon gateway direction from #1976 and #1270.

Related to #1888, #2002, #2026, and #2028.

How to test

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check .)

[fatkobra]

Restacked this PR onto current develop after #2030 merged.

The rebase completed without conflicts. The branch now contains exactly one PR-specific commit: the routine CLI-routing implementation. The shared routing-policy foundation from #2027 and the hook-routing implementation from #2030 are inherited from develop.

Post-restack validation:

• Ruff format check passed;
• Ruff lint passed;
• focused CLI, daemon, routing-policy, and config tests passed;
• merged hook-routing regression tests passed;
• writer-lock safety regressions passed;
• the PR-specific diff contains only the CLI-routing implementation, tests, and documentation;
• no hook implementation or palace-lock files are changed.

CI reran against the restacked branch and is green.

[igorls]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

[fatkobra]

I will fix it and make it ready @igorls

[fatkobra]

@igorls rebased this branch onto current develop and resolved the CLI and
service conflicts.

The rebased branch remains one PR-specific commit and preserves current
upstream behavior, including:

• source-adapter mining;
• normalized no-flag projects mode;
• HTTP-hub mine forwarding;
• no direct retry after an ambiguous hub submission;
• current daemon lock-deferral reporting;
• per-job palace and backend environment restoration;
• the already merged hook-routing policy.

The shared routing policy now applies to the intended routine CLI
operations:

• init post-setup mining;
• mine;
• sweep;
• sync.

Post-init daemon mining continues to use the already scanned file list
rather than scanning the project twice.

The daemon service now supports sweep jobs for both transcript files and
directories, including partial-failure and lock-contention reporting.

The pre-scanned-file daemon payload is validated before mining:

• it is accepted only for projects-mode mining;
• it cannot be combined with a source adapter;
• entries must be non-empty strings;
• paths must remain inside the project root.

Validation completed:

• repository-wide Ruff formatting and lint checks;
• focused CLI, daemon, routing-policy, configuration, source-adapter, hook,
  shell, service, and sweeper tests;
• the complete CI-shaped test suite with coverage;
• GitHub CI is green.

The PR-specific diff remains limited to the CLI-routing implementation,
tests, and documentation. No hook implementation or low-level palace-lock
files are changed.

The rebased history was pushed with --force-with-lease.