perf(sessions): page archived sidebar rows

[santastabber]

Summary

• page archived sidebar sessions behind an archived_limit query parameter so the initial archive toggle no longer fetches/renders the entire archived history
• keep archive counts accurate while returning only the first archived page plus visible rows
• avoid redundant sidebar count rendering work when server-provided bucket counts are present
• add regressions for archive paging, cache-key separation, frontend query params, and filtered "load more" behavior

Verification

• python3 -m py_compile api/routes.py api/route_session_list_cache.py
• node --check static/sessions.js
• git diff --check
• ./scripts/test.sh tests/test_session_list_long_history_perf.py tests/test_sidebar_session_partition.py tests/test_issue4766_sidebar_source_pushdown.py tests/test_session_sidebar_cache.py -q → 47 passed
• gitleaks git --no-banner --redact --timeout 90 --report-format json --report-path /tmp/gitleaks-webui-archive.json --log-opts <base>..<head> . → 0 findings
• trufflehog git file://<temp-clone> --since-commit <base> --branch HEAD --max-depth=1 --include-paths <changed-paths> --json --no-update --force-skip-archives --force-skip-binaries --detector-timeout 10s → 0 findings
• coderabbit review --agent --type committed --base origin/master → 0 findings after addressing one minor local finding

[greptile-apps]

Greptile Summary

This PR introduces server-side pagination for the archived sessions sidebar: the initial archive toggle now requests only the first page of archived sessions (default 100, up to 2000) via a new archived_limit query parameter, and a "Load more" button lets users page through history incrementally. When a search query or project filter is active, the limit is dropped so all archived sessions are available for client-side filtering.

• Backend (routes.py, route_session_list_cache.py): adds _query_positive_int, threads archived_limit/archived_offset through the cache key and payload builder, keeps archived_count/webui_session_count/cli_session_count accurate over the full (unpaged) set, and serialises the limit back into the response so the client knows what was applied.
• Frontend (sessions.js): builds the archived_limit query parameter in _sessionListQueryString, resets it on archive-toggle open, shows a "Load more" affordance when more archives exist, and triggers an uncapped refetch when search becomes active (then a capped refetch when it clears), optimising the tab-count render by skipping the cross-bucket pass when the server already provides authoritative counts.

Confidence Score: 5/5

Safe to merge — the change is additive and well-isolated, with accurate counts preserved over the full unpaged dataset and comprehensive regression coverage for the new paging, cache-key, and refetch behaviours.

All counts (archived_count, webui_session_count, cli_session_count) are computed before the archived slice is applied, so paging never produces stale totals. Cache-key separation is verified by test. The search/project-filter refetch cycle is exercised end-to-end. No correctness or data-loss issues were found.

No files require special attention.

Important Files Changed

Filename	Overview
api/routes.py	Adds _query_positive_int helper and threads archived_limit/archived_offset through _build_session_list_cache_payload; counts (webui_session_count, archived_count) are correctly computed on the full unpaged set before slicing.
api/route_session_list_cache.py	Extends _session_list_cache_key with normalized archived_limit and archived_offset so different page sizes get separate cache slots; normalization logic mirrors the payload builder.
static/sessions.js	Adds _sessionArchivePagingFilterActive, archive limit computation in _sessionListQueryString, _syncArchivedSearchPagingRefresh for refetch-on-search-change, and the Load more DOM element; server-count optimisation replaces the expensive cross-bucket render with a null sentinel.
tests/test_session_list_long_history_perf.py	Adds regression for visible-rows preservation under limit, cache-key separation across limit values, and JS-level query string assertions; covers the new constants and load-more behaviour.
tests/test_issue4766_sidebar_source_pushdown.py	Extended to cover archived_limit in query strings, search-active suppression, project-filter suppression, cap behaviour, and refetch cycle on search enter/clear.
tests/test_sidebar_session_partition.py	Updated source-count assertions to match the null-sentinel conditional, and adds archive load-more DOM assertions.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant U as User
    participant JS as sessions.js
    participant API as /api/sessions
    participant Cache as Cache Builder

    U->>JS: Toggle Show archived
    JS->>JS: "_archivedRowsLoadedLimit = PAGE_SIZE (100)"
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=100"
    API->>Cache: "_build_session_list_cache_payload(archived_limit=100)"
    Cache-->>API: visible rows + first 100 archived + accurate counts
    API-->>JS: "sessions, archived_count, webui_session_count, archived_limit=100"
    JS->>JS: _applySessionListPayload stores _archivedWebuiCount and _serverWebuiSessionCount
    JS->>JS: renderSessionListFromCache shows Load more button

    U->>JS: Click Load more
    JS->>JS: "_archivedRowsLoadedLimit += PAGE_SIZE -> 200"
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=200"
    API-->>JS: visible rows + first 200 archived + counts
    JS->>JS: renderSessionListFromCache updates sidebar

    U->>JS: Type search query
    JS->>JS: _syncArchivedSearchPagingRefresh detects queryActive changed
    JS->>API: "GET /api/sessions?include_archived=1 (no archived_limit)"
    API-->>JS: ALL sessions + counts
    JS->>JS: client-side title filter applied, no Load more button shown

    U->>JS: Clear search
    JS->>JS: _syncArchivedSearchPagingRefresh detects queryActive cleared
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=200"
    API-->>JS: visible rows + first 200 archived + counts
    JS->>JS: renderSessionListFromCache restores paged view

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant U as User
    participant JS as sessions.js
    participant API as /api/sessions
    participant Cache as Cache Builder

    U->>JS: Toggle Show archived
    JS->>JS: "_archivedRowsLoadedLimit = PAGE_SIZE (100)"
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=100"
    API->>Cache: "_build_session_list_cache_payload(archived_limit=100)"
    Cache-->>API: visible rows + first 100 archived + accurate counts
    API-->>JS: "sessions, archived_count, webui_session_count, archived_limit=100"
    JS->>JS: _applySessionListPayload stores _archivedWebuiCount and _serverWebuiSessionCount
    JS->>JS: renderSessionListFromCache shows Load more button

    U->>JS: Click Load more
    JS->>JS: "_archivedRowsLoadedLimit += PAGE_SIZE -> 200"
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=200"
    API-->>JS: visible rows + first 200 archived + counts
    JS->>JS: renderSessionListFromCache updates sidebar

    U->>JS: Type search query
    JS->>JS: _syncArchivedSearchPagingRefresh detects queryActive changed
    JS->>API: "GET /api/sessions?include_archived=1 (no archived_limit)"
    API-->>JS: ALL sessions + counts
    JS->>JS: client-side title filter applied, no Load more button shown

    U->>JS: Clear search
    JS->>JS: _syncArchivedSearchPagingRefresh detects queryActive cleared
    JS->>API: "GET /api/sessions?include_archived=1&archived_limit=200"
    API-->>JS: visible rows + first 200 archived + counts
    JS->>JS: renderSessionListFromCache restores paged view

Loading

_{Reviews (3): Last reviewed commit: "perf(sessions): page archived sidebar ro..." | Re-trigger Greptile}

[nesquena-hermes]

🔬 Gate certification — RED ⛔ (correct paging, but a SILENT search-archived gap — reproduced)

Certified head: sha:f6d174fb9c71 (rebased 15cc7470) · PR: #5200 · santastabber — "perf(sessions): page archived sidebar rows" · crit=2 perf
Verdict: The pagination is correctly built — visible rows always render, cache key includes the page params, counts stay accurate, bounds enforced. But it introduces a SILENT functional regression: with "Show archived" on and a search or project filter active, archived rows are capped server-side at the first page while the (filter-aware-hidden) load-more can't fetch more — so a search silently misses archived matches past page 1. Codex reproduced it; Opus's COMMENT missed it; I confirmed it in the source. One targeted fix away.

What I ran (isolated worktree /tmp/wt-gate-5200, rebased onto current master)

Gate	Result
Codex (reproduce)	SHIP ONLY WITH FIXES — SILENT: filtered archived rows past page 1 unreachable
Opus (full review)	COMMENT — no blockers; different minor nits (didn't catch the search gap)
Full pytest suite (-p no:xdist, rebased)	11105 passed, 0 failed (green — but no test covers search+archived+page-2, so it masks the gap)
PR's own + touched tests	30 passed
node --check sessions.js	OK

⛔ Blocking (SILENT, reproduced) — search/filter over archived silently drops matches past page 1

_sessionListQueryString() (sessions.js:1849-1851) sets archived_limit whenever _showArchived is on — but not gated on filter state. Meanwhile filtering is client-side, and the archived "Load more" control is shown only if(_showArchived && !archivePagingFilterActive) where archivePagingFilterActive = Boolean(searchQueryRaw || _activeProject) (sessions.js:~6476). So when "Show archived" + a search or project filter are both active:

• the server caps archived rows at the first page (archived_limit),
• the client filters only that first page,
• and the load-more that would fetch the rest is hidden → there is no way to reach archived sessions matching the search/filter beyond page 1.

Before this PR all archived rows were loaded, so search found them all. This is a silent search-correctness regression ("my archived session isn't in search results" with no indication) — exactly the existing-flow-reliability class Nathan prioritizes. Moderate likelihood (power users with large archives who search/filter with archived shown).

Fix-spec (Codex, correct): do not send archived_limit when a search/project filter is active (($('sessionSearch').value||'').trim() || _activeProject) — fetch the full archived set for client-side filtering — OR make the load-more filter-aware so filtered archived pagination can complete. Add a test for "search with archived shown finds a match on archived page 2."

What's correct (the paging core is good — keep it)

• ✅ Visible/active rows always render regardless of archived paging (scoped = visible_rows_for_page + archived[offset:offset+limit]) — test-covered.
• ✅ Cache key includes normalized archived_limit/archived_offset (route_session_list_cache.py) — no stale cross-page serving; dedicated test asserts the key varies. No WAL-sidecar/state.db staleness regression.
• ✅ Bounds enforced (archived_limit ≤ 2000, archived_offset ≤ 200000; negatives clamped). No perf: /api/sessions ships 465 rows (342KB) when the sidebar displays 48 #4766 partition regression (30 tests pass).
• ✅ Pagination slice is off-by-one-clean; client re-sort makes server reordering safe (Opus verified).

Opus's non-blocking nits (fold in with the fix)

1. 2000 archived "Load more" soft dead-end (button stays visible but server clamps to 2000 → same rows; extremely rare) — hide the button once the cap is hit.

2. Error-fallback mismatch: bad-type archived_limit → cache-key falls back to None, payload to 0 (unreachable via HTTP; align them).
3. Redundant ... or 0 on archived_offset. 4. No CHANGELOG (maintainer adds at merge).

Recommendation to the next agent

Bounce for the search-under-filter archived fix (Codex's #1), then re-gate. The paging perf work is correct and well-tested; the gap is that archived paging + client-side filtering interact badly (filtered matches past page 1 become unreachable). Fix is targeted (gate archived_limit on no-active-filter, or filter-aware pagination) + a regression test. Fold in Opus's >2000 dead-end + fallback-mismatch nits while there. I applied gate-fail; the search gap is a reproduced SILENT regression, not a nit. Cert valid only at sha:f6d174fb9c71.

---

Gate-certifier layer (warm-up → gate → release). I do not merge/tag/deploy/close. Cert valid only at sha:f6d174fb9c71; a new push invalidates it → re-gate.

[santastabber]

Updated this PR to address the review/gate feedback.

Changes in the latest head:

• Archived paging is disabled while search or project filters are active, so archived matches beyond the first page are not silently hidden.
• The direct-call bad archived_limit fallback now matches the cache-key behavior.
• Removed the redundant archived_offset fallback.
• Replaced the rendered-count 0 sentinel with a documented null sentinel and added/updated focused regressions.

Verification on the updated head:

• python3 -m py_compile api/routes.py api/route_session_list_cache.py
• node --check static/sessions.js
• git diff --check
• Focused sidebar/session tests: 47 passed
• CodeRabbit local review: 0 findings
• Gitleaks PR-range scan: 0 findings
• TruffleHog changed-path scan on a temp clone: 0 findings

[nesquena-hermes]

Re-gated the convergence — the gaps from the first gate read are addressed (paging disabled under search/project filters at the renderSessionList level, cache-key/direct-call fallback aligned, null sentinel + regressions). Thanks for the rework. But the re-gate (Codex, reproduced) found one remaining path the filter-guard misses, so one more turn:

The remaining issue (SILENT — archived rows unreachable): the archived-paging suppression is applied when renderSessionList() refetches /api/sessions (sessions.js:1838), but the normal search-input path doesn't go through there. filterSessions() (sessions.js:5088) only calls renderSessionListFromCache() + /api/sessions/search — it never refetches /api/sessions, so archived_limit is never lifted while searching. And the search endpoint can return match_type:"title" rows, but sessions.js:5106-5108 discards everything except content matches — so an archived title / session-id match beyond the first archived page is unreachable (the load-more button is hidden at sessions.js:6441-6442 during search). Net: with "Show archived" on, typing a query that matches an archived session's title past page 1 shows nothing.

Fix-spec (Codex, sound):

1. static/sessions.js: when _showArchived and search becomes active, trigger/debounce renderSessionList({deferWhileInteracting:false}) so /api/sessions refetches without archived_limit (lift the cap while searching archived). Refetch again when search clears to restore the paged archive cache.
2. Add a regression: with "Show archived" on, an archived title match beyond the first archived page must appear after typing the search.

The renderSessionList-path guard is correct now; this is just the filterSessions() search-input entry point that bypasses it. Re-gate on re-push. Thanks @santastabber.

[santastabber]

Updated again for the latest gate comment.

Additional fixes in this head:

• filterSessions() now triggers an uncapped /api/sessions refetch when archived search becomes active, so archived title/session-id matches beyond the first archived page are reachable.
• Clearing the search refetches again to restore normal paged archive payloads.
• Added a focused Node regression for the archived search start/clear transition.
• Added a 2000-row client-side cap for archive paging and hides the load-more affordance once the cap is reached.

Verification on the updated head:

• python3 -m py_compile api/routes.py api/route_session_list_cache.py
• node --check static/sessions.js
• git diff --check
• Focused sidebar/session tests: 48 passed
• CodeRabbit local review: 0 findings
• Gitleaks PR-range scan: 0 findings
• TruffleHog changed-path scan on a temp clone: 0 findings

[nesquena-hermes]

Shipped in v0.51.757 (via #5241). Thanks @santastabber — archived sidebar rows are now paged (no more loading thousands at once), and the search-input gap I flagged is fixed: with Show-archived on, starting a search transparently refetches without the archive cap so title/id matches beyond the first archived page are reachable, restoring normal paging when search clears. Converged from the gate-fail; re-gated Codex SAFE (no refetch thrash — render-generation guard; default paging + 2000-row cap intact), full suite 11075 passed. Closes #5200.