feat(website): Stream pack progress via NDJSON#1421
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds real-time progress reporting to the pack operation by implementing NDJSON streaming. The backend now emits progress events during cloning, extraction, and processing. Frontend components, composables, and API layers are updated to accept and display progress callbacks throughout the operation lifecycle. Changes
Sequence DiagramsequenceDiagram
participant Client as Browser Client
participant Composable as usePackRequest
participant APIClient as packRepository API
participant Server as Backend Stream
participant Processor as Pack Processor
Client->>Composable: submitRequest()
Composable->>APIClient: packRepository(request, callbacks)
APIClient->>Server: POST /api/pack
Server->>Processor: Clone/Extract/Process
Processor->>Server: onProgress('cloning', message)
Server->>APIClient: Stream {type:'progress', stage:'cloning'}
APIClient->>Composable: callbacks.onProgress('cloning', message)
Composable->>Client: Update progressStage, progressMessage
Client->>Client: Re-render with progress
Processor->>Server: onProgress('extracting', message)
Server->>APIClient: Stream {type:'progress', stage:'extracting'}
APIClient->>Composable: callbacks.onProgress('extracting', message)
Composable->>Client: Update progressStage, progressMessage
Client->>Client: Re-render with progress
Processor->>Server: onProgress('processing', message)
Server->>APIClient: Stream {type:'progress', stage:'processing'}
APIClient->>Composable: callbacks.onProgress('processing', message)
Composable->>Client: Update progressStage, progressMessage
Client->>Client: Re-render with progress
Processor->>Server: Return PackResult
Server->>APIClient: Stream {type:'result', data:PackResult}
APIClient->>Composable: Resolve with result
Composable->>Client: Update result state
Client->>Client: Display final output
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
⚡ Performance Benchmark
Details
History0239f83 fix(website): Default detail message to '...' to prevent layout shift
e5bc33a fix(server): Harden git clone with GIT_TERMINAL_PROMPT and -- separator
0982c6e style(website): Keep main message fixed, show progress as subtitle
e236196 fix(server): Remove tokenCountTree option
3bca5d6 fix(server): Fix allFiles mapping to match FileInfo type
0e6ba32 perf(server): Throttle progress events to 200ms intervals
bba4cba feat(server): Enable detailed pack progress messages
902f87d feat(website): Stream pack progress via NDJSON
3cff58c [autofix.ci] apply automated fixes
0f4d5b7 fix(website): Address PR review feedback
7480cad feat(website): Stream pack progress via NDJSON
|
Deploying repomix with
|
| Latest commit: |
992e979
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://97b405c0.repomix.pages.dev |
| Branch Preview URL: | https://feat-website-pack-progress-s-viwi.repomix.pages.dev |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Code Review — PR #1421Nice feature! NDJSON streaming is a solid choice for real-time progress — simpler than SSE, no WebSocket overhead, and proxy-friendly. The two-phase architecture (validation → JSON errors, processing → NDJSON stream) is clean. A few items to consider: Issues1. Unused The 2. This is called on every request and affects the global logger singleton. Under concurrent requests, this creates a race condition — one request's log level setting affects all others. Since 3. Dynamic
4. Error messages may leak internal details — Raw SuggestionIn throw new AppError(
'Failed to clone repository. The repository may not be public or the URL may be invalid.',
500,
);
// Log the raw error separately for debugging5. No error handling around If the server sends malformed NDJSON (e.g., truncated response on connection drop), Suggestionlet event: StreamEvent;
try {
event = JSON.parse(line) as StreamEvent;
} catch {
throw new ApiError('Failed to parse server response');
}Minor / Nice-to-have6. Compression disabled for entire response —
7. No cancellation propagation — If the client aborts mid-stream, the git clone and Looks Good
Overall this is well-structured. The main actionable items are #2 (global state mutation) and #3 (dynamic import consistency). The rest are hardening suggestions. 🤖 Generated with Claude Code |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
yamadashy
force-pushed
the
feat/website-pack-progress-streaming-v2
branch
from
0f4d5b7 to
3c9d098
Compare
Add real-time progress streaming to the pack endpoint using Hono's stream() helper with NDJSON format. Users now see stage-specific messages during processing instead of a static "Processing repository..." Server changes: - packAction uses Hono stream() with NDJSON (one JSON per line) - processRemoteRepo split into git clone + runDefaultAction for separate cloning/processing stages - processZipFile reports extracting/processing stages - Content-Encoding: identity skips compress for real-time delivery Client changes: - packRepository parses NDJSON stream with onProgress callback - TryItLoading displays stage messages (Checking cache, Cloning repository, Processing files, etc.) - message field prepared for future detailed progress from pack() Progress stages: - URL: cache-check → cloning → processing → result - ZIP: extracting → processing → result Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
yamadashy
force-pushed
the
feat/website-pack-progress-streaming-v2
branch
from
3cff58c to
902f87d
Compare
Update to latest main which includes progressCallback support in runDefaultAction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pass progressCallback to runDefaultAction so users see fine-grained progress during processing (e.g., "Searching for files...", "Collecting files...", "Generating output..."). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Detailed pack progress messages (from pack()) can fire at high frequency. Throttle to one event per 200ms within the same stage, while always sending immediately when the stage changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use fileCharCounts instead of fileTokenCounts for allFiles, matching the FileInfo interface which requires charCount, not tokenCount. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No longer needed since file selection uses charCount instead of tokenCount (aligned with #1422). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Display stage message (e.g., "Processing files...") as the main text, with the detailed pack() message (e.g., "Collecting files...") shown below in smaller gray text. Long messages are truncated to 60 characters with ellipsis. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Main text stays as "Processing repository..." (same as main branch). Stage and detailed messages (e.g., "Cloning repository...", "Searching for files...") are shown below in smaller gray text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Set GIT_TERMINAL_PROMPT=0 to prevent git from hanging on credential prompts for private repos (fail fast instead of waiting for timeout) - Add '--' separator before URL to prevent argument injection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Always show the detail line with '...' as fallback when no message is available, avoiding a layout jump when messages start arriving. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Code Review —
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Add return statement so the Promise from onProgress propagates to defaultAction's error handler (Promise.resolve().catch()), preventing unhandled promise rejections when stream writes fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Missing await on cleanupTempDirectory causes fire-and-forget cleanup with potential unhandled rejection
At website/server/src/domains/pack/processZipFile.ts:130, cleanupTempDirectory(tempDirPath) is called without await, unlike the properly awaited cleanup in the new remoteRepo.ts:154. The cleanupTempDirectory function (website/server/src/domains/pack/utils/fileUtils.ts:147-159) is async and can throw an AppError if the path validation fails. Without await, this becomes an unhandled promise rejection. Additionally, the subsequent await fs.unlink(outputFilePath) could race with the directory removal since the cleanup hasn't completed yet.
(Refers to line 130)
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| // Clone the repository | ||
| await onProgress?.('cloning'); | ||
| const parsed = parseRemoteValue(repoUrl); |
There was a problem hiding this comment.
🔴 parseRemoteValue called without refs loses branch resolution for multi-segment branch URLs
In the new remoteRepo.ts, parseRemoteValue(repoUrl) is called without fetching remote refs first (website/server/src/domains/pack/remoteRepo.ts:51). The old code used runCli which dispatched to runRemoteAction (src/cli/actions/remoteAction.ts:178-188), which fetched refs via getRemoteRefs() and passed them to parseRemoteValue(repoUrl, refs). The refs parameter in git-url-parse is critical for resolving multi-segment branch names from URLs like https://github.com/user/repo/tree/feature/my-branch. Without refs, git-url-parse may parse only feature as the ref instead of feature/my-branch, causing the clone to either fail (if feature branch doesn't exist) or clone the wrong branch. Users commonly copy such URLs directly from their GitHub browser tab.
Prompt for agents
In website/server/src/domains/pack/remoteRepo.ts, the function processRemoteRepo calls parseRemoteValue(repoUrl) at line 51 without providing remote refs. The old code path (via runCli -> runRemoteAction in src/cli/actions/remoteAction.ts:178-188) fetched remote refs using getRemoteRefs(parsedUrl) before calling parseRemoteValue(repoUrl, refs). Without refs, git-url-parse cannot correctly resolve multi-segment branch names from URLs like https://github.com/user/repo/tree/feature/my-branch.
To fix this, before calling parseRemoteValue, fetch refs from the remote repository similar to how remoteAction.ts does it:
1. First do a preliminary parse (or extract the base repo URL) to get the clone URL
2. Call getRemoteRefs() (from src/core/git/gitRemoteHandle.ts) to fetch available refs
3. Pass the refs to parseRemoteValue(repoUrl, refs)
Alternatively, if adding getRemoteRefs as a dependency is undesirable for the server, you could add a progress stage like 'repository-fetch' for this step, which the TryItLoading.vue component already has a message for.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Add real-time progress streaming to the pack endpoint using Hono's
stream()helper with NDJSON format. Users now see stage-specific messages during processing instead of a static "Processing repository..."Server changes
packActionuses Honostream()with NDJSON (one JSON object per line)processRemoteReporefactored: manualgit clone+runDefaultActionfor separate cloning/processing stages (removesrunClidependency)processZipFilereports extracting/processing stagesContent-Encoding: identityskips compress middleware for real-time streaming deliveryClient changes
packRepositoryparses NDJSON stream withonProgresscallbackTryItLoadingdisplays stage-specific messagesmessagefield prepared for future detailed progress frompack()(depends on feat(core): Add progressCallback parameter to runDefaultAction #1420)Progress stages
Future enhancement
After #1420 is merged and repomix dependency is updated,
runDefaultAction'sprogressCallbackwill enable detailed messages like "Searching for files...", "Collecting files...", "Generating output..." during the processing stage.Checklist
npm run testnpm run lint