Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Add global_prefix and prefix_folders options #28

Merged
merged 2 commits into from
Feb 22, 2023

Conversation

nvaatstra
Copy link
Contributor

@nvaatstra nvaatstra commented Feb 19, 2023

Adds 2 configuration options to simpleblob, to offer extra control over handling prefixes within a bucket:

  • global_prefix is a prefix applied to all operations, allowing work within a prefix seamlessly (Default: "" - matches current v0.2.1 behaviour)
  • prefix_folders can be enabled to prevent List operations from including all items in subfolders (Default: false - differs from v0.2.1 behaviour, but this is not expected to affect anyone)

Tests have been updated to cover both options.

Examples - Recursive

(Some items below refer to Recursive, which is the same as !PrefixFolders. We renamed it)

If a bucket contains the following:

- bar1
- bar2
- foo/bar3
- foo/bar4

Then a List() using Recursive unset or false will yield (this is the current v0.2.1 behaviour):

- bar1
- bar2
- foo/

If Recursive is set to true, then a List() will yield:

- bar1
- bar2
- foo/bar3
- foo/bar4

Examples - GlobalPrefix
If a bucket contains the following:

- bar1
- bar2
- foo/bar3
- foo/bar4

Then a List() using GlobalPrefix unset or "" will yield (this is the current v0.2.1 behaviour):

- bar1
- bar2
- foo/

If GlobalPrefix is set to "foo/", then a List() will yield:

- bar3
- bar4

And a Store() action for name bar5, followed by List() will yield:

- bar3
- bar4
- bar5

The actual contents of the bucket will now be:

- bar1
- bar2
- foo/bar3
- foo/bar4
- foo/bar5

Examples - GlobalPrefix & Recursive
global_prefix = "foo/" and recursive = true will result in a List() returning:

- foo/bar3
- foo/bar4
- foo/bar5

global_prefix = "" and recursive = true will result in a List() returning:

- bar1
- bar2
- foo/bar3
- foo/bar4
- foo/bar5

GlobalPrefix string `yaml:"global_prefix"`

// Recursive can be enabled to make List operations recurse into nested prefixes
Recursive bool `yaml:"recursive"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to make listing all filenames with any slashes the default, instead of requiring a per-backend opt-in.

So instead of Recursive we would provide an option to opt-out of the recursion,

Copy link
Contributor Author

@nvaatstra nvaatstra Feb 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below commit removes Recursive and replaces it with PrefixFolders.

PrefixFolders can be enabled to make List operations show nested prefixes as folders instead of recursively listing all contents of nested prefixes.

Tests updated accordingly.

Examples - PrefixFolders
If a bucket contains the following:

- bar1
- bar2
- foo/bar3
- foo/bar4

Then a List() using PrefixFolders unset or false will yield:

- bar1
- bar2
- foo/bar3
- foo/bar4

If PrefixFolders is set to true, then a List() will yield:

- bar1
- bar2
- foo/

@wojas wojas merged commit 75237ca into PowerDNS:main Feb 22, 2023
@wojas wojas added this to the v0.2.2 milestone Feb 23, 2023
@wojas wojas changed the title S3: GlobalPrefix and Recursive S3: Add global_prefix and prefix_folders options Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants