-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support local cache server/daemon #6690
Comments
This seems like a mix of a few different asks:
(If I'm understanding correctly, and these are all separate things, it would be best to file separate issues, but let's start by getting on the same page first!) |
does each developer have an api key somehow? how is access managed?
communicate with a storage provider, as mentioned in https://ducktors.github.io/turborepo-remote-cache/supported-storage-providers.html
the demo PR is a good example.
a daemon, yeah. because ultimately the existence of a local cache server which communicates with remote storage should be transparent to developers.
See the demo PR -- it is what uploads and downloads cache from the remote storage.
yes, exactly, end-users shouldn't be aware of it.
only usability and security issues (part of what I'm trying to address by having each developer manage their keys to the "remote storage", having their own s3 keys). I didn't mention it specifically, but we're not using Vercel hosted cache. That could be where a disconnect could come from in this convo. Some big companies prefer to manage their own infra because its way cheaper. (S3 for storing turbo's cache assets, for example -- but with S3, we need a strategy to make it so only CI has write access, and developers only have read access -- which is the crux of the whole problem -- it's totally possible there is an easier way to achieve that goal)
not quite <3 |
From reading your PR, I'm a little confused by "manage the daemon myself (gross, lots of code)". The PR doesn't seem to be a huge amount of code? Of course it's a proof of concept but I'm a little confused as to where the code will be needed. That could just be my naive estimation. As for alternatives, I do think you could accomplish this by hosting your own remote cache and creating authorization rules that prevent some users from writing to the cache. Granted, that would be "gross, lots of code" and perhaps some hosting burden. We have discussed the possibility of adding authorization to the cache. We can definitely discuss it more and return with some decisions on our end. To answer the actual question:
You can run For adding hooks into the daemon lifecycle, we can discuss that at our next team meeting. Hope that was helpful! TL;DR: We'll discuss your requests and get back to you. |
Hi @NullVoxPopuli, we discussed your proposal. Here's our thoughts:
|
ye! If a cache server could work with any number of tokens / with per-token access (read-only by default, opt one token in to write), that would be STELLAR ✨ |
Perhaps I'm mistaken, but I think the tokens are really up to your cache server's implementation. If you want, you could store some users as having read-only permissions and others as having read and write. From there, the cache server can read the token and return either enabled or disabled for the caching status. Now there is some limitations in that we currently don't support the cache server returning a caching status that indicates it's read-only (we can look into that!), but if you use the Basically, here's the flow:
|
yeah, that's about what @fox1t was saying here: ducktors/turborepo-remote-cache#199 (comment) I'll close this since it seems everyone is in agreement on design for how this feature should work 🎉 thanks!! |
Which project is this feature idea for?
Turborepo
Describe the feature you'd like to request
Turborepo already has a daemon, so I think it kinda makes sense to support a local cache server hooking in to that daemon.
I'm currently using https://github.com/ducktors/turborepo-remote-cache
Which works great, but at the moment, for uploading to S3, we can't have devs manage their own S3 keys.
Describe the solution you'd like
Ideally, we want to
this is easily managed with AWS IAM -- but we need to be able to run a cache server locally as a daemon.
Additionally, we need a way to force the daemon to be booted before
turbo
runs anything.there are situations right now where turbo will run without the daemon, and as a monorepo grows, we really need that to run.
Describe alternatives you've considered
afaict there are two alternatives
The text was updated successfully, but these errors were encountered: