Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Implement Library.get_key_path for NFS and Azure storages #2172

Merged
merged 5 commits into from
Feb 13, 2025

Conversation

maxim-morozov
Copy link
Collaborator

Reference Issues/PRs

Expose Library.get_key_path for NFS and Azure storages.

What does this implement or fix?

get_key_path will be used to determine where each key is located. This can help to set hard quotas for the platforms that support path based quotas, like VAST.

Any other comments?

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

res.emplace_back(found_key);
}, "");

for(auto vk=res.begin(); vk!=res.end(); ++vk) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for (auto vk : res)

@poodlewars
Copy link
Collaborator

poodlewars commented Feb 11, 2025

Are some changes to create Python bindings missing? I don't understand how you're planning to use this. I also don't follow how these functions will help with quotas?

@maxim-morozov
Copy link
Collaborator Author

Are some changes to create Python bindings missing? I don't understand how you're planning to use this. I also don't follow how these functions will help with quotas?

The usage is like this:

lib_tool = LibraryTool(lib._library)
data_keys = lib_tool.find_keys(KeyType.TABLE_DATA)
print(lib_tool.get_key_path(data_keys[0]))

For quotas we will obtain the path to the data tdata and will impose the quota on this path.

@poodlewars
Copy link
Collaborator

Are some changes to create Python bindings missing? I don't understand how you're planning to use this. I also don't follow how these functions will help with quotas?

The usage is like this:

lib_tool = LibraryTool(lib._library)
data_keys = lib_tool.find_keys(KeyType.TABLE_DATA)
print(lib_tool.get_key_path(data_keys[0]))

For quotas we will obtain the path to the data tdata and will impose the quota on this path.

Can you include a test that does that please? I think you're missing Python bindings for your new library tool API.

std::vector<VariantKey> res;

store.iterate_type(KeyType::TABLE_DATA, [&](VariantKey &&found_key) {
res.emplace_back(found_key);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
res.emplace_back(found_key);
res.emplace_back(std::move(found_key));

res.emplace_back(found_key);
}, "");

for(auto vk: res) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for(auto vk: res) {
for(const auto& vk: res) {


auto store = get_storage();
store->iterate_type(KeyType::TABLE_DATA, [&](VariantKey &&found_key) {
res.emplace_back(found_key);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
res.emplace_back(found_key);
res.emplace_back(std::move(found_key));

res.emplace_back(found_key);
}, "");

for(auto vk : res) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for(auto vk : res) {
for(const auto& vk : res) {

@maxim-morozov maxim-morozov merged commit f89fd12 into master Feb 13, 2025
149 checks passed
@maxim-morozov maxim-morozov deleted the expose_key_path branch February 13, 2025 20:33
jamesmunro pushed a commit that referenced this pull request Feb 25, 2025
#### Reference Issues/PRs
Expose `Library.get_key_path` for NFS and Azure storages.

#### What does this implement or fix?
get_key_path will be used to determine where each key is located. This
can help to set hard quotas for the platforms that support path based
quotas, like VAST.

#### Any other comments?

#### Checklist

<details>
  <summary>
   Checklist for code changes...
  </summary>
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [x] Are API changes highlighted in the PR description?
- [x] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Maxim Morozov <[email protected]>
@maxim-morozov maxim-morozov changed the title Enhancement: Implement for NFS and Azure storages Enhancement: Implement Library.get_key_path for NFS and Azure storages Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants