-
-
Notifications
You must be signed in to change notification settings - Fork 265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support S3-compatible provider as storage backend #317
Conversation
This is awesome! |
How about
I'd define
I'd stick with official AWS SDK |
Do you mean something like this? storage_type = "local" # default value if omitted
[server]
hostname = "https://example-host.com"
port = 8080
data_dir = "/data/podsync/"
# ... storage_type = "s3"
[server]
hostname = "https://example-bucket-name.s3-website-us-west-2.amazonaws.com"
[s3]
endpoint = "https://s3.us-west-2.amazonaws.com"
region = "us-west-2"
bucket = "example-bucket-name"
# ... |
May be something like [storage]
type="local|s3"
[storage.s3]
...
[storage.local]
... WDYT? |
What would be under |
I think [storage]
# type="local" by default
[storage.local]
data_dir = "/data/podsync/" and remove |
Sounds good to me.
…Sent from my iPhone
On Jun 11, 2022, at 4:38 PM, Contextualist ***@***.***> wrote:
I think storage.local.data_dir makes more sense in the current model anyway. Maybe we could deprecate server.data_dir: emit a warning to the user and encourage the following settings
[storage]
# type="local" by default
[storage.local]
data_dir = "/data/podsync/"
and remove server.data_dir in the future. These changes will be transparent to the Docker users.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
This also deprecate `server.data_dir` in favor of `storage.local.data_dir`
Everything is ready except for documentation updates (README.md, config.toml.example, cloud_formation.yml). Since the |
Next release will include these changes, so it’s fine to include documentation updates too. |
Ok, done! |
Thanks! |
Fix two regressions introduced by PR #317
Next release will include these changes, so it’s fine to include documentation updates too.
…Sent from my iPhone
On Jun 12, 2022, at 11:37 AM, Contextualist ***@***.***> wrote:
Everything is ready except for documentation updates (README.md, config.toml.example, cloud_formation.yml). Since the storage section is a major change to the config, should we postpone the documentation change until the next release?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
Currently, this is a working implementation, but still work and discussion are needed. This resolves #159 and achieve part of the goals in #180.
To try this out, set a config as the following:
And set the access key and secret in environment variables
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
(credential should also works per aws-sdk-go's documentation, but I haven't tried it out).Optional: use CLI arg
--headless
to quit after first round of updates.To be discussed:
ShouldRenamed to--update-once
be a config entry instead? Should we rename it?--headless
Where should theIn the news3
config entry be located?storage
config sections3
config entries?fs.Storage
interface includehttp.FileSystem
?Dependency aws/aws-sdk-go is quite heavy (compiled binary size is increased from 4MB to 22MB); should we use a lighter implementation for the S3 API (e.g. rhnvrm/simples3)?No changeTo do:
fs.S3
Any other discussions are welcome. I hope this PR can contribute some ideas on the S3 backend. If you already have something better, feel free to close this in favor of another PR!