Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .qwen/e2e-tests/2026-07-28-serve-large-text-range-read.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Serve large-text range reads

## Automated verification

Run the focused Core range and service tests:

```bash
cd packages/core
npx vitest run src/utils/read-text-range.test.ts src/services/fileSystemService.test.ts
```

Run the WorkspaceFileSystem, ACP adapter, and HTTP route tests:

```bash
cd packages/cli
npx vitest run src/serve/fs/workspace-file-system.test.ts
npx vitest run src/serve/bridge-file-system-adapter.test.ts
npx vitest run src/serve/routes/workspace-file-read.test.ts
```

Then run repository verification:

```bash
npm run lint
npm run typecheck
npm run build
```

## Manual scenario

1. Put a valid UTF-8 CSV larger than 256 KiB in the bound workspace. The
original reproduction used a 406,892-byte, 5,000-row CSV.
2. Start `qwen serve` and connect an ACP session to that workspace.
3. Ask the agent to call `read_file` with `limit: 20` and read the first 20
lines.
4. Confirm the first `read_file` call succeeds with that finite limit and
returns the requested CSV lines.
5. Confirm the agent does not need a shell command such as `head`, `sed`, or
`awk` as a fallback.
6. Request a later finite line window and confirm it also succeeds without
returning more than 256 KiB.

## Regression checks

- No-limit, line-only, maxBytes-only, and line-plus-maxBytes requests against
the same large file remain `file_too_large`.
- A finite line window over a large binary file remains `binary_file`.
- A large non-UTF-8 text window remains `file_too_large` with a UTF-8
conversion hint.
- A supported non-UTF-8 file within the full-snapshot cap still obeys
`maxBytes` after its content is decoded to UTF-8.
- A partial large-file response reports the complete `sizeBytes`, sets
`truncated: true`, omits the full-file hash, and exposes
`originalLineCount: null` until EOF is known.
- Replacing the pathname, appending, truncating, or overwriting the opened file
during the read is rejected instead of returning a mixed or stale result.
- A deep offset beyond 10 MiB still succeeds when the request has a finite
line limit.

## Baseline status

Before the fix, Core could read the requested range from the 406,892-byte CSV,
but Serve rejected the file at its 256 KiB full-snapshot gate before slicing.
The focused automated tests cover the corrected Core, WorkspaceFileSystem, ACP,
and HTTP paths; the manual ACP/model scenario remains the release smoke test.
94 changes: 94 additions & 0 deletions docs/design/serve-large-text-range-consistency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Serve large-text range consistency

## Context

Serve now streams finite-limit text windows from files larger than
`MAX_READ_BYTES`. A line-only or maxBytes-only request does not unlock this
path. The workspace boundary opens the file once, reads through that handle,
and returns partial metadata without a full-file hash.

An open file descriptor fixes the inode, but it does not freeze the inode's
bytes. Node also does not synchronize file-system operations that modify the
same file concurrently. A reader can therefore observe bytes written after
`open`, including an in-place rewrite with the same inode and size.

Issue #7946 requires files changed or replaced during a read to remain
rejected. Append-tolerant reads are not part of that contract.

## Decision

Large streamed windows use the file's open-time stat as their snapshot
baseline:

1. The initial `lstat` and open-time `fstat` must identify the same regular
file with the same device, inode, size, modification time, and change time.
2. After streaming, both `fstat` and pathname `lstat` must retain that
identity and version, and the pathname must not be a symlink.
3. The caller closes the handle in `finally`, after all read and stability
checks.
4. A mismatch detected by these stability checks returns `hash_mismatch`.

Content validation errors are captured until the post-read checks finish, so a
concurrent mutation that also causes decoding to fail still returns
`hash_mismatch`. Without a mutation, binary content remains `binary_file` and
large non-UTF-8 text remains `file_too_large` with a conversion hint.

This matches the existing full-snapshot stability policy. It intentionally
rejects concurrent appends: size growth proves that the open-time snapshot was
not stable, while inode identity alone cannot prove that the original prefix
was unchanged.

Reliable append tolerance would require a separate snapshot mechanism or a
second bounded read that verifies every byte used to locate and produce the
requested line window. That additional I/O and protocol policy are outside
this bug fix.

## Range-reader cleanup

A caller-owned file handle always selects the streaming path. The separate
`forceStreaming` switch and handle-buffering fast path are therefore removed.
The handle chunk reader limits positional reads to the caller's captured file
size, so a read cannot cross the open-time EOF, and reuses one 512 KiB buffer
because each chunk is decoded synchronously before the generator advances.

There is no fixed scan-byte budget: line offsets require scanning from byte
zero, so deep windows remain O(file size). The finite line limit and
`MAX_READ_BYTES` cap bound returned content and memory, while cancellation is
checked between reads. A future scan-cost policy needs a cursor or equivalent
continuation contract instead of silently making valid deep offsets
unreachable.

Serve derives `lineEnding` from the returned content, matching its existing
full-snapshot behavior. Core can continue reporting file-level metadata for its
other consumers.

Every large-file window keeps `truncated: true`, even when the scan happens to
reach EOF. This boundary uses the flag to distinguish a window without a
full-file hash from a complete snapshot that is safe to treat as whole-file
content; it does not mean only that decoded characters were omitted.

## Consumers

All Serve callers resolve through the selected workspace runtime before
reaching this boundary:

- `GET /file`
- ACP HTTP `_qwen/file/read`
- the injected ACP `readTextFile` adapter

Windowless reads used by workspace setup retain the existing 256 KiB
full-snapshot refusal.

## Verification

- A mixed-EOL large file reports the ending style present in the returned
slice.
- Concurrent append, truncation, same-size rewrite, pathname replacement, and
symlink replacement are rejected.
- Handle-bound range reads never use the full-buffer fast path and reuse their
streaming buffer.
- A deep offset beyond 10 MiB succeeds with a finite line limit.
- No-limit, line-only, and maxBytes-only requests remain behind the 256 KiB
full-snapshot gate.
- Existing output, encoding, binary, hash, and line-count limits remain
unchanged.
Loading
Loading