Remove minhash conditional for 25.02 #558
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Praateek <[email protected]>
Signed-off-by: Praateek <[email protected]>
praateekmahajan
commented
Feb 18, 2025
|
|
||
| # TODO: remove when dask min version gets bumped | ||
| DASK_SHUFFLE_METHOD_ARG = _dask_version > parse_version("2024.1.0") | ||
| DASK_P2P_ERROR = _dask_version < parse_version("2023.10.0") | ||
| DASK_SHUFFLE_CAST_DTYPE = _dask_version > parse_version("2023.12.0") | ||
| DASK_CUDF_PARQUET_READ_INCONSISTENT_SCHEMA = _dask_version > parse_version("2024.12") | ||
| DASK_CUDF_PARQUET_READ_INCONSISTENT_SCHEMA = _dask_cudf_version > parse_version( |
There was a problem hiding this comment.
this was a bug we had.. added condition for 25.02 in the hope rapidsai/cudf#17554 gets merged for 25.04
Signed-off-by: Praateek <[email protected]>
sarahyurick
reviewed
Feb 18, 2025
nemo_curator/_compat.py
Outdated
|
|
||
| # TODO: remove when dask min version gets bumped | ||
| DASK_SHUFFLE_METHOD_ARG = _dask_version > parse_version("2024.1.0") | ||
| DASK_P2P_ERROR = _dask_version < parse_version("2023.10.0") | ||
| DASK_SHUFFLE_CAST_DTYPE = _dask_version > parse_version("2023.12.0") | ||
| DASK_CUDF_PARQUET_READ_INCONSISTENT_SCHEMA = _dask_version > parse_version("2024.12") | ||
| DASK_CUDF_PARQUET_READ_INCONSISTENT_SCHEMA = _dask_cudf_version > parse_version( | ||
| "2025.02" |
There was a problem hiding this comment.
Suggested change
| "2025.02" | |
| "25.2.0" |
|
We should still hold off on merging this since one mode of installation is the NeMo-FW container that is currently tied to Rapids 24.10 (for 0.7.0). I believe the next release should include Rapids 25.02 or newer, but until then, this logic might fail in those containers. |
Signed-off-by: Praateek <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Since now stable points to 25.02 we can remove the old conditionla logic we had to handle various minhash api's.
Also fixes an issue with dask_cudf.read_parquet inconssitent schema, we hope rapidsai/cudf#17554 get's merged in 25.04
Closes #557.
Usage
# Add snippet demonstrating usageChecklist