feat(cloudflare): add optional R2 batch uploads via rclone for cache population

[ghost]

Batch upload R2 cache using rclone

This PR proposes adding support for optional batch uploading R2 cache using rclone.
Based on the solution discussed in Issue #866

Details

Automatic selection between batch vs standard upload

The CLI now detects when all of R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, and CLOUDFLARE_ACCOUNT_ID are set in the environment and switches to batch upload mode. Otherwise, it falls back to the standard (Wrangler-based) upload method.

Graceful fallback on errors

If batch upload fails (e.g. due to rclone errors), the flow will log a warning and transparently revert to the default upload approach.

Staging and parallel transfers

Assets are copied into a temporary staging directory, then uploaded using rclone copy with concurrency settings (--transfers=32, --checkers=16) for performance.

Related issues

• [BUG] Slow Wrangler file by file cache populating #866
• [BUG] R2/KV Incrememental cache doesn't work when prerendering a large number of routes #637

[changeset-bot]

🦋 Changeset detected

Latest commit: 3ec1364

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@opennextjs/cloudflare	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vicb]

@krzysztof-palka-monogo thank you so much for this PR, it will please many of our users!

I only took a quick look and I'll do a full review on Monday.

I have some preliminary comments/thoughts:

• I think we should rename --rcloneBatch. First we do not want to expose the implementation (rclone) as it might change - we can keep mention to rclone and in the doc. Ideally we can re-use cacheChunkSize
• rclone.js was last released 3y ago - Ideally we could find a more recent and maintained implementation (but we should not block the PR on that, it could be a follow-up or we can wait until we actually have an issue)
• We should info log when users are not using batch upload and have i.e. > 10 assets. Another option could be to batch upload by default and fallback to slow/safe on error (again happy if this addressed as a follow up)

[ghost]

Good morning @vicb

Here’s the justification for keeping an explicit --rcloneBatch (or similarly named) flag and retaining the existing upload mechanism as the default:

1. Requires external configuration
   Rclone is not a drop-in replacement — it depends on a separate config file (the rclone config, credentials, remotes, etc). For users who have never used rclone, they must explicitly create and supply that config. That means it is not “just work” out of the box.

2. Avoids a breaking change
   If we made rclone-based behavior the default, we risk breaking setups for users who do not have configured rclone. Keeping the current behavior as default ensures backward compatibility.

3. Explicit intent
   Having a named flag (e.g. --rcloneBatch) clearly signals the use of rclone and keeps any implementation coupling explicit (rather than hidden). It also gives future flexibility: if one day we swap in a different batch uploader (or abstraction), the explicit flag boundary helps isolate that change.

[vicb]

Here’s the justification for keeping an explicit --rcloneBatch (or similarly named) flag and retaining the existing upload mechanism as the default:

We definitely don't want "rclone" in the flag. We could use another mechanism, i.e. remote bindings.

1. Requires external configuration
   Rclone is not a drop-in replacement — it depends on a separate config file (the rclone config, credentials, remotes, etc). For users who have never used rclone, they must explicitly create and supply that config. That means it is not “just work” out of the box.

IMO we should drop the need for an external config in favor of using env vars - the CLI could dynamically create a temp config file for rclone.js as it doesn't seem to support env vars.

2. Avoids a breaking change
   If we made rclone-based behavior the default, we risk breaking setups for users who do not have configured rclone. Keeping the current behavior as default ensures backward compatibility.

We can use rclone when configured and fallback to the current mechanism - but log that there is faster way and link some docs.

3. Explicit intent
   Having a named flag (e.g. --rcloneBatch) clearly signals the use of rclone and keeps any implementation coupling explicit (rather than hidden). It also gives future flexibility: if one day we swap in a different batch uploader (or abstraction), the explicit flag boundary helps isolate that change.

I disagree here.
Users want something that works (fast), they should not have to care about how it is implemented.

[ghost]

Hi @vicb

I've updated the PR to align with your suggestions:

• Removed --rcloneBatch flag and all rclone references from user-facing API
• Implemented environment variable configuration (R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_ACCOUNT_ID)
• Batch upload now activates automatically when credentials are detected
• Fallback to wrangler uploads when credentials aren't provided
• Temp config files are created dynamically with secure permissions and
  auto-cleanup

Documentation now focuses on "batch upload" as an optional performance feature, not implementation details.

Please let me know if thats will meet the requirements and if so could we run "Publish prereleases" action to properly test the changes in some real applications 😄

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@opennextjs/cloudflare@925

commit: 3ec1364

[ghost]

Hi @vicb

I will be glad for code review.

[vicb]

Note sure I'll have time for a full review today, sorry.

A couple comments:

• R2_ACCOUNT_ID should rather be CLOUDFLARE_ACCOUNT_ID as it is not specific to R2

• Thanks for implementing the env vars based solution for CI. I'm wondering if we can improve the story for local dev. It would be nice to get the vars from .env / .dev.vars files but it would mean adding vars that are only used in local dev and that might confuse users. Maybe we could look for a local <pjt>/rclone.conf and fallback to env vars when it's not here. What do you think?

[ghost]

Hello @vicb

I've addressed both points:

1. Renamed R2_ACCOUNT_ID to CLOUDFLARE_ACCOUNT_ID
2. The CLI now loads environment variables from a .env file in the project root.

This keeps the implementation simple while making local dev more convenient. WDYT?

[vicb]

This keeps the implementation simple while making local dev more convenient. WDYT?

I don't think there is an ideal solution for .env vs a local config file.

That being said, Next.js uses .env files with this loading order so we should make sure we are consistent.

I'll take a deeper look today and comment on how we could do that

[ghost]

I’ll be away for a week starting tomorrow, so I need to wrap up this functionality today 😅
Since our ideas have been evolving, could we prioritize only essential changes now and handle the rest in a follow-up iteration?

[vicb]

Added a few nits.

The main point is that I think we should use the npm package if it supports copy.

Thanks for all your work on this!

[ghost]

Hello @vicb I addressed all the comments.

[vicb]

Hello @vicb I addressed all the comments.

I think there is one opened comment about rclone.promises

[ghost]

@vicb could you please clarify what changes exactly you are requesting?
I already replied to your earlier comment about the rclone npm package, which is abandoned.

[ghost]

I’m honestly starting to feel a bit sabotaged 😞

[vicb]

I already replied to your earlier comment about the rclone npm package, which is abandoned.

Either it's abandonned and we should not use it or we use it.

The alternative is to use any of the other more recent package on npm.

I’m honestly starting to feel a bit sabotaged 😞

Sorry you feel that way.
That's a package we plan to maintain long-term so we take it the time it needs to review and merge the PRs.

[vicb]

Many thanks @krzysztof-palka-monogo for the PR and enduring my reviews :)

[ghost]

Huge relief :) Thanks a lot for the approval and merge!