feat: add token-budget based file reading with intelligent preview

[daniel-lxs]

Problem

Files can fill the entire context window, and tiktoken crashes with RuntimeError: unreachable on files >5MB.

Solution

Token-budget based file reading with multi-layer protection against context overflow and tokenizer crashes.

How It Works

File Read Request
       ↓
Check file size
       ↓
   < 100KB? → Read normally (fast path)
       ↓
   > 5MB? → Preview first 100KB (crash prevention)
       ↓
Count tokens → Within budget? → Full content
                     ↓
                Exceeds budget? → Truncate to fit

Implementation

Layer 1: Fast Path (< 100KB)

• Skip validation for small files - zero overhead ⚡

Layer 2: Token Validation (100KB - 5MB)

• Dynamic budget: (contextWindow - currentTokens) * 0.6
• Real token counting, smart truncation if needed 🎯

Layer 3: Preview Mode (> 5MB)

• Return 100KB preview to prevent crashes
• Suggests using line_range for targeted reading 👁️

Layer 4: Error Recovery

• Catch tokenizer unreachable errors gracefully
• Fallback to 100KB preview instead of crashing 🛡️

Benefits

✅ Dynamic budget based on actual context (no magic numbers)
✅ Real token counting using existing tokenizer
✅ 100KB previews for large files
✅ Graceful error handling prevents conversation crashes
✅ Simple (~160 lines) vs complex heuristics
✅ 17 comprehensive tests covering all scenarios

Testing

All 17 tests passing: fast path, budget validation, preview mode, error recovery, edge cases, unicode support.

Related

Closes #6667

---

Important

Introduces token-budget based file reading in readFileTool.ts to handle large files efficiently, with new functions for budget validation and content truncation, and comprehensive tests.

• Behavior:
  • Implements token-budget based file reading in readFileTool.ts to prevent context overflow and tokenizer crashes.
  • Introduces validateFileTokenBudget and truncateFileContent in fileTokenBudget.ts for handling large files.
  • Handles files <100KB normally, previews first 100KB for files >5MB, and truncates content if it exceeds token budget.
• Testing:
  • Adds fileTokenBudget.spec.ts with 17 tests covering scenarios like small files, large files, budget validation, and error handling.
  • Updates line-counter.spec.ts to test line and token counting with error handling.
• Misc:
  • Updates line-counter.ts to include token estimation alongside line counting.
  • Modifies readFileTool.spec.ts to test new file reading behavior.

^{This description was created by for 169fb35. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

Starting my review of this PR—comments incoming soon! 🚀

Follow Along on Roo Code Cloud

---

Issues to Address

All previously flagged issues have been resolved:

• The lines attribute shows totalLines from the original file even after content is truncated (see inline comment on readFileTool.ts:619)
• Empty lines are being filtered out when counting displayed lines, causing incorrect line count in the lines attribute (see inline comment on readFileTool.ts:620-621)
• Line counting logic has off-by-one error for files ending with newline characters (see inline comment on readFileTool.ts:620)

Latest Review: No new issues found. All previous concerns have been properly addressed.