test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 #801
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
changed the title
jjerphan
changed the title
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
commented
Aug 29, 2023
jjerphan
changed the title
jjerphan
changed the title
willdealtry
approved these changes
Aug 29, 2023
jjerphan
changed the title
jjerphan
changed the title
Closed
willdealtry
approved these changes
Aug 30, 2023
jjerphan
added
enhancement
poodlewars
pushed a commit
that referenced
this pull request
Aug 30, 2023
…ation for pandas 2.0 (#801)
G-D-Petrov
pushed a commit
that referenced
this pull request
Sep 17, 2023
…ation for pandas 2.0 (#801)
G-D-Petrov
pushed a commit
that referenced
this pull request
Sep 20, 2023
Fix formatting with Black Remotve black formatting for now@ Add necesary imports Modify some integration tests to comply with persistent storages Modify some integration tests to comply with persistent storages Fix some errors Fix errors in stress delete Patch an issue in test_symbol_list_read_only_compaction_needed when using lmdb Do a sanity build after the refactor Do a sanity build after the refactor Fix soem typos Fix some typos Fix some typos Fix soem typos Fix some typos in column stats Fix a broken test Resolve some build problems Fix the real_storage factory when we are passing in lmdb_config option Fix broken tests Clean up Fix error in path_prefix Fix names around interop tests Revert changes to python unit tests Revert support for real_s3_uri in object_storage_uri_incl_bucket for now Fix black formatting Test the refactor of the persistant storage path_prefix Fix pattern for format testing Fix formatting for black Fix for regualar tests taht use real storages Test a more granulated approach Test with cahnging the local_test name TEst with a single shared path_prefix Attempt to retry some of the failing tests Add import for retry Try to separate into even more granulated folders Try to separate into even more granulated folders Make sure that we are not adding a backslash at the end of the path_prefix Add tests for the oldest versions; enable all logs Fix logic for the oldest version compat testing@ Fix logic for the oldest version compat testing@ Fix AWS log print Try to not split the pytests Fix test_stress_delete for persistent storage Fix problem with path_prefixes issues #798 and #806 Revert prefix fix and move it to the python layer Cleanup the PR Cleanup PR Address PR Comments Fix broken test test_list_symbols_regex Fix path for real_s3 arctic_client fixture Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Bugfix: NativeVersionStore.batch_write_metadata should throw an exception if there are errors, not return a DataError object Make black happy Fix black formatting Add handling of unspecified ca path in azure uri Add test and fix hint Skip azure test in conda Add dep on packaging Required for: ``` from arcticdb.util._versions import IS_PANDAS_TWO ``` but seems to have been missed. Enhancement 737: Support empty-type columns in QueryBuilder operations test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 (#801) Fix get_num_rows for NativeVersionStore (#800) Fixes #750 Highlight notable design decisions. Added the necessary read_options to the read_descriptor call. Also added a new test to test the API Co-authored-by: Georgi Petrov <[email protected]> test: Remove test for fallback to pickle (#805) S3 and Azure: Do not save sensitive or ephemeral config in the config library PR Comments conda-build: Adapt C++ test suite for Linux (#713) Docs - update release number Planned release number for storage config datafix now 3.0.0 conda-build: Pin cmake Signed-off-by: Julien Jerphanion <[email protected]> Update releasing.md (#817) Add warning describing that promoting a release candidate will deploy outdated documentation to the site. <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Highlight notable design decisions. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings and documentation? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Fix/229 LMDB configurable map size (#811) <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Fixes #229 Backwards compatible. You can now create an LMDB connection with a map size specified in the form `[0-9]+(KB|MB|GB|TB)`. See the Arctic `__init__.py` docstring for details. ``` ac = Arctic("lmdb:///home/user/my_lmdb?map_size=2GB") ``` Highlight notable design decisions. We allow users to set an LMDB mapsize in their Arctic URI. We put this option in the URI since it must be set before the LMDB env is open. Since in our model the Arctic instance is what holds the LMDB env open, this fits the lifetimes well. <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [x] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> --------- Co-authored-by: aseaton <[email protected]> get description batch to catch all the exceptions read metadata batch to catch all the exceptions write batch to catch all the exceptions Bugfix: NativeVersionStore batch_read_metadata and batch_write should never return dataerror objects maint: Add Security Policy (#823) Highlight notable design decisions. This adds a minimal Security Policy. This is highly inspired from other projects', such as [scikit-learn's](https://github.com/ory/examples/blob/master/SECURITY.md), [ory's](https://github.com/ory/examples/blob/master/SECURITY.md), and [Microsoft's](https://github.com/microsoft/.github/blob/main/SECURITY.md). For now, only the last version is supported, with version being explicitly enumerated in the table. We could also just mentioned that the latest minor version is the only one supported (with its patched versions). Moreover, we could have another security policy based on [the Common Vulnerability Scoring Systems](https://www.first.org/cvss/) such as [ory's](https://github.com/ory/examples/blob/master/SECURITY.md). If we want to have a complete Security Policy, then it might be worth adopting one which is similar to [CSAN's](https://gitlab.in2p3.fr/csan/csan/-/blob/master/SECURITY.md). <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Signed-off-by: Julien Jerphanion <[email protected]> fix/774 Rename ArcticNativeNotYetImplemented in backwards compatible way Fix docs after exception rename uri support for mongodb Co-authored-by: Alex Seaton <[email protected]> Spinning extra mongo server instance for testing Update python/arcticdb/util/test.py Co-authored-by: Alex Seaton <[email protected]> Minor revert Analysis workflow (#783) (#784) Implements a workflow for code coverage that will be extended to support other analysis tasks Addresses issue #638 --------- Co-authored-by: Georgi Petrov <[email protected]> Fix publish step (#825) Highlight notable design decisions. Add the same fix with conditionals to the publish step Documentation for LMDB map size Fix formatting Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Fix merge artefact Update test_arctic.py Fix build tests issues Fix tests that rely on arctic_client Address PR Comments Fix path to storage_test in conftest Hardcode the latest supported version to 3.0.0 Fix bug in test for reading many symbols Address PR Comments Create new fixture for persistent storages; cleanup some storage tests
G-D-Petrov
pushed a commit
that referenced
this pull request
Sep 21, 2023
Fix formatting with Black Remotve black formatting for now@ Add necesary imports Modify some integration tests to comply with persistent storages Modify some integration tests to comply with persistent storages Fix some errors Fix errors in stress delete Patch an issue in test_symbol_list_read_only_compaction_needed when using lmdb Do a sanity build after the refactor Do a sanity build after the refactor Fix soem typos Fix some typos Fix some typos Fix soem typos Fix some typos in column stats Fix a broken test Resolve some build problems Fix the real_storage factory when we are passing in lmdb_config option Fix broken tests Clean up Fix error in path_prefix Fix names around interop tests Revert changes to python unit tests Revert support for real_s3_uri in object_storage_uri_incl_bucket for now Fix black formatting Test the refactor of the persistant storage path_prefix Fix pattern for format testing Fix formatting for black Fix for regualar tests taht use real storages Test a more granulated approach Test with cahnging the local_test name TEst with a single shared path_prefix Attempt to retry some of the failing tests Add import for retry Try to separate into even more granulated folders Try to separate into even more granulated folders Make sure that we are not adding a backslash at the end of the path_prefix Add tests for the oldest versions; enable all logs Fix logic for the oldest version compat testing@ Fix logic for the oldest version compat testing@ Fix AWS log print Try to not split the pytests Fix test_stress_delete for persistent storage Fix problem with path_prefixes issues #798 and #806 Revert prefix fix and move it to the python layer Cleanup the PR Cleanup PR Address PR Comments Fix broken test test_list_symbols_regex Fix path for real_s3 arctic_client fixture Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Bugfix: NativeVersionStore.batch_write_metadata should throw an exception if there are errors, not return a DataError object Make black happy Fix black formatting Add handling of unspecified ca path in azure uri Add test and fix hint Skip azure test in conda Add dep on packaging Required for: ``` from arcticdb.util._versions import IS_PANDAS_TWO ``` but seems to have been missed. Enhancement 737: Support empty-type columns in QueryBuilder operations test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 (#801) Fix get_num_rows for NativeVersionStore (#800) Fixes #750 Highlight notable design decisions. Added the necessary read_options to the read_descriptor call. Also added a new test to test the API Co-authored-by: Georgi Petrov <[email protected]> test: Remove test for fallback to pickle (#805) S3 and Azure: Do not save sensitive or ephemeral config in the config library PR Comments conda-build: Adapt C++ test suite for Linux (#713) Docs - update release number Planned release number for storage config datafix now 3.0.0 conda-build: Pin cmake Signed-off-by: Julien Jerphanion <[email protected]> Update releasing.md (#817) Add warning describing that promoting a release candidate will deploy outdated documentation to the site. <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Highlight notable design decisions. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings and documentation? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Fix/229 LMDB configurable map size (#811) <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Fixes #229 Backwards compatible. You can now create an LMDB connection with a map size specified in the form `[0-9]+(KB|MB|GB|TB)`. See the Arctic `__init__.py` docstring for details. ``` ac = Arctic("lmdb:///home/user/my_lmdb?map_size=2GB") ``` Highlight notable design decisions. We allow users to set an LMDB mapsize in their Arctic URI. We put this option in the URI since it must be set before the LMDB env is open. Since in our model the Arctic instance is what holds the LMDB env open, this fits the lifetimes well. <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [x] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> --------- Co-authored-by: aseaton <[email protected]> get description batch to catch all the exceptions read metadata batch to catch all the exceptions write batch to catch all the exceptions Bugfix: NativeVersionStore batch_read_metadata and batch_write should never return dataerror objects maint: Add Security Policy (#823) Highlight notable design decisions. This adds a minimal Security Policy. This is highly inspired from other projects', such as [scikit-learn's](https://github.com/ory/examples/blob/master/SECURITY.md), [ory's](https://github.com/ory/examples/blob/master/SECURITY.md), and [Microsoft's](https://github.com/microsoft/.github/blob/main/SECURITY.md). For now, only the last version is supported, with version being explicitly enumerated in the table. We could also just mentioned that the latest minor version is the only one supported (with its patched versions). Moreover, we could have another security policy based on [the Common Vulnerability Scoring Systems](https://www.first.org/cvss/) such as [ory's](https://github.com/ory/examples/blob/master/SECURITY.md). If we want to have a complete Security Policy, then it might be worth adopting one which is similar to [CSAN's](https://gitlab.in2p3.fr/csan/csan/-/blob/master/SECURITY.md). <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Signed-off-by: Julien Jerphanion <[email protected]> fix/774 Rename ArcticNativeNotYetImplemented in backwards compatible way Fix docs after exception rename uri support for mongodb Co-authored-by: Alex Seaton <[email protected]> Spinning extra mongo server instance for testing Update python/arcticdb/util/test.py Co-authored-by: Alex Seaton <[email protected]> Minor revert Analysis workflow (#783) (#784) Implements a workflow for code coverage that will be extended to support other analysis tasks Addresses issue #638 --------- Co-authored-by: Georgi Petrov <[email protected]> Fix publish step (#825) Highlight notable design decisions. Add the same fix with conditionals to the publish step Documentation for LMDB map size Fix formatting Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Fix merge artefact Update test_arctic.py Fix build tests issues Fix tests that rely on arctic_client Address PR Comments Fix path to storage_test in conftest Hardcode the latest supported version to 3.0.0 Fix bug in test for reading many symbols Address PR Comments Create new fixture for persistent storages; cleanup some storage tests Fix bad merges after rebase Remove check for azure support Remove unnecessary condition in arctic_client fixture@ Fix broken tests
G-D-Petrov
pushed a commit
that referenced
this pull request
Sep 25, 2023
Fix formatting with Black Remotve black formatting for now@ Add necesary imports Modify some integration tests to comply with persistent storages Modify some integration tests to comply with persistent storages Fix some errors Fix errors in stress delete Patch an issue in test_symbol_list_read_only_compaction_needed when using lmdb Do a sanity build after the refactor Do a sanity build after the refactor Fix soem typos Fix some typos Fix some typos Fix soem typos Fix some typos in column stats Fix a broken test Resolve some build problems Fix the real_storage factory when we are passing in lmdb_config option Fix broken tests Clean up Fix error in path_prefix Fix names around interop tests Revert changes to python unit tests Revert support for real_s3_uri in object_storage_uri_incl_bucket for now Fix black formatting Test the refactor of the persistant storage path_prefix Fix pattern for format testing Fix formatting for black Fix for regualar tests taht use real storages Test a more granulated approach Test with cahnging the local_test name TEst with a single shared path_prefix Attempt to retry some of the failing tests Add import for retry Try to separate into even more granulated folders Try to separate into even more granulated folders Make sure that we are not adding a backslash at the end of the path_prefix Add tests for the oldest versions; enable all logs Fix logic for the oldest version compat testing@ Fix logic for the oldest version compat testing@ Fix AWS log print Try to not split the pytests Fix test_stress_delete for persistent storage Fix problem with path_prefixes issues #798 and #806 Revert prefix fix and move it to the python layer Cleanup the PR Cleanup PR Address PR Comments Fix broken test test_list_symbols_regex Fix path for real_s3 arctic_client fixture Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Bugfix: NativeVersionStore.batch_write_metadata should throw an exception if there are errors, not return a DataError object Make black happy Fix black formatting Add handling of unspecified ca path in azure uri Add test and fix hint Skip azure test in conda Add dep on packaging Required for: ``` from arcticdb.util._versions import IS_PANDAS_TWO ``` but seems to have been missed. Enhancement 737: Support empty-type columns in QueryBuilder operations test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 (#801) Fix get_num_rows for NativeVersionStore (#800) Fixes #750 Highlight notable design decisions. Added the necessary read_options to the read_descriptor call. Also added a new test to test the API Co-authored-by: Georgi Petrov <[email protected]> test: Remove test for fallback to pickle (#805) S3 and Azure: Do not save sensitive or ephemeral config in the config library PR Comments conda-build: Adapt C++ test suite for Linux (#713) Docs - update release number Planned release number for storage config datafix now 3.0.0 conda-build: Pin cmake Signed-off-by: Julien Jerphanion <[email protected]> Update releasing.md (#817) Add warning describing that promoting a release candidate will deploy outdated documentation to the site. <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Highlight notable design decisions. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings and documentation? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Fix/229 LMDB configurable map size (#811) <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Fixes #229 Backwards compatible. You can now create an LMDB connection with a map size specified in the form `[0-9]+(KB|MB|GB|TB)`. See the Arctic `__init__.py` docstring for details. ``` ac = Arctic("lmdb:///home/user/my_lmdb?map_size=2GB") ``` Highlight notable design decisions. We allow users to set an LMDB mapsize in their Arctic URI. We put this option in the URI since it must be set before the LMDB env is open. Since in our model the Arctic instance is what holds the LMDB env open, this fits the lifetimes well. <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [x] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> --------- Co-authored-by: aseaton <[email protected]> get description batch to catch all the exceptions read metadata batch to catch all the exceptions write batch to catch all the exceptions Bugfix: NativeVersionStore batch_read_metadata and batch_write should never return dataerror objects maint: Add Security Policy (#823) Highlight notable design decisions. This adds a minimal Security Policy. This is highly inspired from other projects', such as [scikit-learn's](https://github.com/ory/examples/blob/master/SECURITY.md), [ory's](https://github.com/ory/examples/blob/master/SECURITY.md), and [Microsoft's](https://github.com/microsoft/.github/blob/main/SECURITY.md). For now, only the last version is supported, with version being explicitly enumerated in the table. We could also just mentioned that the latest minor version is the only one supported (with its patched versions). Moreover, we could have another security policy based on [the Common Vulnerability Scoring Systems](https://www.first.org/cvss/) such as [ory's](https://github.com/ory/examples/blob/master/SECURITY.md). If we want to have a complete Security Policy, then it might be worth adopting one which is similar to [CSAN's](https://gitlab.in2p3.fr/csan/csan/-/blob/master/SECURITY.md). <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Signed-off-by: Julien Jerphanion <[email protected]> fix/774 Rename ArcticNativeNotYetImplemented in backwards compatible way Fix docs after exception rename uri support for mongodb Co-authored-by: Alex Seaton <[email protected]> Spinning extra mongo server instance for testing Update python/arcticdb/util/test.py Co-authored-by: Alex Seaton <[email protected]> Minor revert Analysis workflow (#783) (#784) Implements a workflow for code coverage that will be extended to support other analysis tasks Addresses issue #638 --------- Co-authored-by: Georgi Petrov <[email protected]> Fix publish step (#825) Highlight notable design decisions. Add the same fix with conditionals to the publish step Documentation for LMDB map size Fix formatting Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Fix merge artefact Update test_arctic.py Fix build tests issues Fix tests that rely on arctic_client Address PR Comments Fix path to storage_test in conftest Hardcode the latest supported version to 3.0.0 Fix bug in test for reading many symbols Address PR Comments Create new fixture for persistent storages; cleanup some storage tests Fix bad merges after rebase Remove check for azure support Remove unnecessary condition in arctic_client fixture@ Fix broken tests
G-D-Petrov
pushed a commit
that referenced
this pull request
Sep 25, 2023
Fix formatting with Black Remotve black formatting for now@ Add necesary imports Modify some integration tests to comply with persistent storages Modify some integration tests to comply with persistent storages Fix some errors Fix errors in stress delete Patch an issue in test_symbol_list_read_only_compaction_needed when using lmdb Do a sanity build after the refactor Do a sanity build after the refactor Fix soem typos Fix some typos Fix some typos Fix soem typos Fix some typos in column stats Fix a broken test Resolve some build problems Fix the real_storage factory when we are passing in lmdb_config option Fix broken tests Clean up Fix error in path_prefix Fix names around interop tests Revert changes to python unit tests Revert support for real_s3_uri in object_storage_uri_incl_bucket for now Fix black formatting Test the refactor of the persistant storage path_prefix Fix pattern for format testing Fix formatting for black Fix for regualar tests taht use real storages Test a more granulated approach Test with cahnging the local_test name TEst with a single shared path_prefix Attempt to retry some of the failing tests Add import for retry Try to separate into even more granulated folders Try to separate into even more granulated folders Make sure that we are not adding a backslash at the end of the path_prefix Add tests for the oldest versions; enable all logs Fix logic for the oldest version compat testing@ Fix logic for the oldest version compat testing@ Fix AWS log print Try to not split the pytests Fix test_stress_delete for persistent storage Fix problem with path_prefixes issues #798 and #806 Revert prefix fix and move it to the python layer Cleanup the PR Cleanup PR Address PR Comments Fix broken test test_list_symbols_regex Fix path for real_s3 arctic_client fixture Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Bugfix: NativeVersionStore.batch_write_metadata should throw an exception if there are errors, not return a DataError object Make black happy Fix black formatting Add handling of unspecified ca path in azure uri Add test and fix hint Skip azure test in conda Add dep on packaging Required for: ``` from arcticdb.util._versions import IS_PANDAS_TWO ``` but seems to have been missed. Enhancement 737: Support empty-type columns in QueryBuilder operations test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 (#801) Fix get_num_rows for NativeVersionStore (#800) Fixes #750 Highlight notable design decisions. Added the necessary read_options to the read_descriptor call. Also added a new test to test the API Co-authored-by: Georgi Petrov <[email protected]> test: Remove test for fallback to pickle (#805) S3 and Azure: Do not save sensitive or ephemeral config in the config library PR Comments conda-build: Adapt C++ test suite for Linux (#713) Docs - update release number Planned release number for storage config datafix now 3.0.0 conda-build: Pin cmake Signed-off-by: Julien Jerphanion <[email protected]> Update releasing.md (#817) Add warning describing that promoting a release candidate will deploy outdated documentation to the site. <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Highlight notable design decisions. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings and documentation? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Fix/229 LMDB configurable map size (#811) <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Fixes #229 Backwards compatible. You can now create an LMDB connection with a map size specified in the form `[0-9]+(KB|MB|GB|TB)`. See the Arctic `__init__.py` docstring for details. ``` ac = Arctic("lmdb:///home/user/my_lmdb?map_size=2GB") ``` Highlight notable design decisions. We allow users to set an LMDB mapsize in their Arctic URI. We put this option in the URI since it must be set before the LMDB env is open. Since in our model the Arctic instance is what holds the LMDB env open, this fits the lifetimes well. <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [x] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> --------- Co-authored-by: aseaton <[email protected]> get description batch to catch all the exceptions read metadata batch to catch all the exceptions write batch to catch all the exceptions Bugfix: NativeVersionStore batch_read_metadata and batch_write should never return dataerror objects maint: Add Security Policy (#823) Highlight notable design decisions. This adds a minimal Security Policy. This is highly inspired from other projects', such as [scikit-learn's](https://github.com/ory/examples/blob/master/SECURITY.md), [ory's](https://github.com/ory/examples/blob/master/SECURITY.md), and [Microsoft's](https://github.com/microsoft/.github/blob/main/SECURITY.md). For now, only the last version is supported, with version being explicitly enumerated in the table. We could also just mentioned that the latest minor version is the only one supported (with its patched versions). Moreover, we could have another security policy based on [the Common Vulnerability Scoring Systems](https://www.first.org/cvss/) such as [ory's](https://github.com/ory/examples/blob/master/SECURITY.md). If we want to have a complete Security Policy, then it might be worth adopting one which is similar to [CSAN's](https://gitlab.in2p3.fr/csan/csan/-/blob/master/SECURITY.md). <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Signed-off-by: Julien Jerphanion <[email protected]> fix/774 Rename ArcticNativeNotYetImplemented in backwards compatible way Fix docs after exception rename uri support for mongodb Co-authored-by: Alex Seaton <[email protected]> Spinning extra mongo server instance for testing Update python/arcticdb/util/test.py Co-authored-by: Alex Seaton <[email protected]> Minor revert Analysis workflow (#783) (#784) Implements a workflow for code coverage that will be extended to support other analysis tasks Addresses issue #638 --------- Co-authored-by: Georgi Petrov <[email protected]> Fix publish step (#825) Highlight notable design decisions. Add the same fix with conditionals to the publish step Documentation for LMDB map size Fix formatting Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Fix merge artefact Update test_arctic.py Fix build tests issues Fix tests that rely on arctic_client Address PR Comments Fix path to storage_test in conftest Hardcode the latest supported version to 3.0.0 Fix bug in test for reading many symbols Address PR Comments Create new fixture for persistent storages; cleanup some storage tests Fix bad merges after rebase Remove check for azure support Remove unnecessary condition in arctic_client fixture@ Fix broken tests Extend persistent storages to more tests Fix formatting with Black Remotve black formatting for now@ Add necesary imports Modify some integration tests to comply with persistent storages Modify some integration tests to comply with persistent storages Fix some errors Fix errors in stress delete Patch an issue in test_symbol_list_read_only_compaction_needed when using lmdb Do a sanity build after the refactor Do a sanity build after the refactor Fix soem typos Fix some typos Fix some typos Fix soem typos Fix some typos in column stats Fix a broken test Resolve some build problems Fix the real_storage factory when we are passing in lmdb_config option Fix broken tests Clean up Fix error in path_prefix Fix names around interop tests Revert changes to python unit tests Revert support for real_s3_uri in object_storage_uri_incl_bucket for now Fix black formatting Test the refactor of the persistant storage path_prefix Fix pattern for format testing Fix formatting for black Fix for regualar tests taht use real storages Test a more granulated approach Test with cahnging the local_test name TEst with a single shared path_prefix Attempt to retry some of the failing tests Add import for retry Try to separate into even more granulated folders Try to separate into even more granulated folders Make sure that we are not adding a backslash at the end of the path_prefix Add tests for the oldest versions; enable all logs Fix logic for the oldest version compat testing@ Fix logic for the oldest version compat testing@ Fix AWS log print Try to not split the pytests Fix test_stress_delete for persistent storage Fix problem with path_prefixes issues #798 and #806 Revert prefix fix and move it to the python layer Cleanup the PR Cleanup PR Address PR Comments Fix broken test test_list_symbols_regex Fix path for real_s3 arctic_client fixture Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Bugfix: NativeVersionStore.batch_write_metadata should throw an exception if there are errors, not return a DataError object Make black happy Fix black formatting Add handling of unspecified ca path in azure uri Add test and fix hint Skip azure test in conda Add dep on packaging Required for: ``` from arcticdb.util._versions import IS_PANDAS_TWO ``` but seems to have been missed. Enhancement 737: Support empty-type columns in QueryBuilder operations test: Test the absence of implace modification on datetime64 normalization for pandas 2.0 (#801) Fix get_num_rows for NativeVersionStore (#800) Fixes #750 Highlight notable design decisions. Added the necessary read_options to the read_descriptor call. Also added a new test to test the API Co-authored-by: Georgi Petrov <[email protected]> test: Remove test for fallback to pickle (#805) S3 and Azure: Do not save sensitive or ephemeral config in the config library PR Comments conda-build: Adapt C++ test suite for Linux (#713) Docs - update release number Planned release number for storage config datafix now 3.0.0 conda-build: Pin cmake Signed-off-by: Julien Jerphanion <[email protected]> Update releasing.md (#817) Add warning describing that promoting a release candidate will deploy outdated documentation to the site. <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Highlight notable design decisions. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings and documentation? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Fix/229 LMDB configurable map size (#811) <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Fixes #229 Backwards compatible. You can now create an LMDB connection with a map size specified in the form `[0-9]+(KB|MB|GB|TB)`. See the Arctic `__init__.py` docstring for details. ``` ac = Arctic("lmdb:///home/user/my_lmdb?map_size=2GB") ``` Highlight notable design decisions. We allow users to set an LMDB mapsize in their Arctic URI. We put this option in the URI since it must be set before the LMDB env is open. Since in our model the Arctic instance is what holds the LMDB env open, this fits the lifetimes well. <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [x] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> --------- Co-authored-by: aseaton <[email protected]> get description batch to catch all the exceptions read metadata batch to catch all the exceptions write batch to catch all the exceptions Bugfix: NativeVersionStore batch_read_metadata and batch_write should never return dataerror objects maint: Add Security Policy (#823) Highlight notable design decisions. This adds a minimal Security Policy. This is highly inspired from other projects', such as [scikit-learn's](https://github.com/ory/examples/blob/master/SECURITY.md), [ory's](https://github.com/ory/examples/blob/master/SECURITY.md), and [Microsoft's](https://github.com/microsoft/.github/blob/main/SECURITY.md). For now, only the last version is supported, with version being explicitly enumerated in the table. We could also just mentioned that the latest minor version is the only one supported (with its patched versions). Moreover, we could have another security policy based on [the Common Vulnerability Scoring Systems](https://www.first.org/cvss/) such as [ory's](https://github.com/ory/examples/blob/master/SECURITY.md). If we want to have a complete Security Policy, then it might be worth adopting one which is similar to [CSAN's](https://gitlab.in2p3.fr/csan/csan/-/blob/master/SECURITY.md). <details> <summary> Checklist for code changes... </summary> - [x] Have you updated the relevant docstrings and documentation? - [x] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [x] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [x] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> Signed-off-by: Julien Jerphanion <[email protected]> fix/774 Rename ArcticNativeNotYetImplemented in backwards compatible way Fix docs after exception rename uri support for mongodb Co-authored-by: Alex Seaton <[email protected]> Spinning extra mongo server instance for testing Update python/arcticdb/util/test.py Co-authored-by: Alex Seaton <[email protected]> Minor revert Analysis workflow (#783) (#784) Implements a workflow for code coverage that will be extended to support other analysis tasks Addresses issue #638 --------- Co-authored-by: Georgi Petrov <[email protected]> Fix publish step (#825) Highlight notable design decisions. Add the same fix with conditionals to the publish step Documentation for LMDB map size Fix formatting Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Update build.yml Fix the test_separation_between_libraries_with_prefixes test when we have a path_prefix present Add new tests Fix / in script call Fix path_prefix for mongo client Fix path_prefix for mongo client Skip path_prefix test when testing with mongo Add some tests with more lifelike data Fix black formatting Fix formatting Fix merge artefact Update test_arctic.py Fix build tests issues Fix tests that rely on arctic_client Address PR Comments Fix path to storage_test in conftest Hardcode the latest supported version to 3.0.0 Fix bug in test for reading many symbols Address PR Comments Create new fixture for persistent storages; cleanup some storage tests Fix bad merges after rebase Remove check for azure support Remove unnecessary condition in arctic_client fixture@ Fix broken tests
vasil-pashov
pushed a commit
that referenced
this pull request
Oct 18, 2023
…ation for pandas 2.0 (#801)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Follow-up of #540.
What does this implement/fix? How does it work (high level)? Highlight notable design decisions.
See this thread https://github.com/man-group/ArcticDB/pull/540/files#r1288498933.
Any other comments?
Checklist
Checklist for code changes...