feat(cliproxyapi): persist usage-export across restarts via R2 - #1577
Conversation
|
You do not have enough credits to review this pull request. Please purchase more credits to continue. |
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughSummary by CodeRabbit
WalkthroughTwo shell script files are modified to add S3 persistence capabilities to the cliproxyapi service. Three new helper functions are added to construct S3 URIs and perform conditional uploads/downloads of usage export files. The startup script is updated to hydrate the usage export from S3 on initialization and persist it back to S3 after successful exports. Changes
Sequence DiagramsequenceDiagram
participant Startup as start.sh
participant FS as File System
participant S3 as Object Store (S3)
participant Export as Export Process
Startup->>Startup: Check for S3 credentials
alt Credentials Present
Startup->>S3: Request USAGE_EXPORT_FILE
alt File Exists in S3
S3-->>FS: Download to local path
end
end
Startup->>Export: Execute export request
Export->>FS: Generate usage-export.json
Export-->>Startup: Export complete
alt Credentials Present
Startup->>FS: Read USAGE_EXPORT_FILE
Startup->>S3: Upload to object store
S3-->>Startup: Upload acknowledged
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Mesa DescriptionTL;DRPersist What changed?
Description generated by Mesa. Update settings |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements S3-based persistence for usage export data by adding upload and download utilities to the common script and integrating them into the startup and export processes. The review feedback recommends using shell parameter expansion for mandatory environment variables to ensure better error reporting and consistency with existing patterns.
| AWS_ACCESS_KEY_ID="$OBJECTSTORE_ACCESS_KEY" \ | ||
| AWS_SECRET_ACCESS_KEY="$OBJECTSTORE_SECRET_KEY" \ | ||
| @aws@ s3 cp \ | ||
| --endpoint-url="$OBJECTSTORE_ENDPOINT" \ |
There was a problem hiding this comment.
For consistency with cliproxy_s3_sync and to provide better error reporting if variables are unexpectedly empty, consider using the ${VAR:?} expansion syntax for credentials and the endpoint.
| AWS_ACCESS_KEY_ID="$OBJECTSTORE_ACCESS_KEY" \ | |
| AWS_SECRET_ACCESS_KEY="$OBJECTSTORE_SECRET_KEY" \ | |
| @aws@ s3 cp \ | |
| --endpoint-url="$OBJECTSTORE_ENDPOINT" \ | |
| AWS_ACCESS_KEY_ID="${OBJECTSTORE_ACCESS_KEY:?OBJECTSTORE_ACCESS_KEY is required}" \ | |
| AWS_SECRET_ACCESS_KEY="${OBJECTSTORE_SECRET_KEY:?OBJECTSTORE_SECRET_KEY is required}" \ | |
| @aws@ s3 cp \ | |
| --endpoint-url="${OBJECTSTORE_ENDPOINT:?OBJECTSTORE_ENDPOINT is required}" \ |
References
- Maintain consistency with established patterns for writing scripts that are extracted from Nix expressions.
| AWS_ACCESS_KEY_ID="$OBJECTSTORE_ACCESS_KEY" \ | ||
| AWS_SECRET_ACCESS_KEY="$OBJECTSTORE_SECRET_KEY" \ | ||
| @aws@ s3 cp \ | ||
| --endpoint-url="$OBJECTSTORE_ENDPOINT" \ |
There was a problem hiding this comment.
For consistency with cliproxy_s3_sync and to provide better error reporting if variables are unexpectedly empty, consider using the ${VAR:?} expansion syntax for credentials and the endpoint.
| AWS_ACCESS_KEY_ID="$OBJECTSTORE_ACCESS_KEY" \ | |
| AWS_SECRET_ACCESS_KEY="$OBJECTSTORE_SECRET_KEY" \ | |
| @aws@ s3 cp \ | |
| --endpoint-url="$OBJECTSTORE_ENDPOINT" \ | |
| AWS_ACCESS_KEY_ID="${OBJECTSTORE_ACCESS_KEY:?OBJECTSTORE_ACCESS_KEY is required}" \ | |
| AWS_SECRET_ACCESS_KEY="${OBJECTSTORE_SECRET_KEY:?OBJECTSTORE_SECRET_KEY is required}" \ | |
| @aws@ s3 cp \ | |
| --endpoint-url="${OBJECTSTORE_ENDPOINT:?OBJECTSTORE_ENDPOINT is required}" \ |
References
- Maintain consistency with established patterns for writing scripts that are extracted from Nix expressions.
1 participant
Summary
cliproxy_upload_usage_to_s3andcliproxy_download_usage_from_s3helpers incommon.sh, plus a fixed-keycliproxy_usage_s3_uri(s3://$OBJECTSTORE_BUCKET/usage-export.json).start.sh's existingusage_exporttrap now uploads~/.cli-proxy-api/usage-export.jsonto R2 right after the management API export.usage-export.jsonfrom R2 only if it's missing locally, so the post-bootusage_importcan replay stats on a fresh machine.Test plan
make switchapplies cleanly on macOSlaunchctl kickstart -k gui/$(id -u)/org.nixos.cliproxyapi) and confirms3://cliproxyapi/usage-export.jsonupdates~/.cli-proxy-api/usage-export.json, restart, confirm it gets pulled from R2 and/v0/management/usage/exportreturns prior countersSummary by cubic
Persist usage metrics across restarts by storing usage-export.json in Cloudflare R2 (S3-compatible). On shutdown we upload the export; on startup we hydrate it if missing so usage replay works on fresh machines.
Written for commit 2f44fc7. Summary will update on new commits.