feat: Introduce NePS-Spaces #248
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced tests for various aspects of the NEPS space, including: - `test_search_space__nos_like.py`: Tests for the resolution of a complex pipeline with unary, binary, and ternary operations. - `test_search_space__recursion.py`: Validates recursive operations within the pipeline, ensuring correct factor propagation. - `test_search_space__resampled.py`: Tests for resampling behavior in both float and integer pipelines, ensuring shared and unique values are handled correctly. - `test_search_space__reuse_arch_elements.py`: Verifies the reuse of architectural elements in pipelines, ensuring correct operation and configuration string generation. - `utils.py`: Added utility functions to generate possible configuration strings for pipelines, aiding in testing and validation.
…odule - Changed all instances of `space` to `neps_space` in test files to reflect the new import structure. - Updated class definitions to inherit from `neps_space.Pipeline` instead of `space.Pipeline`. - Modified the creation of operations and categorical choices to use `neps_space.Operation` and `neps_space.Categorical`. - Adjusted the resolution and conversion functions to utilize the new `neps_space` methods. - Ensured consistency across all test cases for the new structure and functionality.
- Added a new module `bracket_optimizer.py` that implements a bracket-based optimization strategy for multi-fidelity configurations in NePS spaces. - Introduced the `_BracketOptimizer` class to handle the optimization process, including sampling configurations from a prior band and managing different fidelity levels. - Implemented various bracket types such as successive halving, hyperband, ASHA, and async hyperband. - Updated `priorband.py` to utilize the new sampling strategies and refactored the prior sampling logic. - Created a new `sampling.py` module to define various samplers, including `OnlyPredefinedValuesSampler`, `RandomSampler`, `PriorOrFallbackSampler`, and mutation-based samplers. - Modified existing tests to accommodate the new optimizer and sampling strategies, ensuring compatibility with the updated structure. - Refactored import paths in test files to align with the new module organization.
- Updated import statements to use `neps.space.neps_spaces.parameters` for all relevant classes and functions in test files. - Refactored test classes and methods to replace direct usage of `neps_space` with the new parameters module, ensuring consistency across the test suite. - Adjusted the instantiation of various operations and categorical choices to align with the new structure. - Ensured that all references to operations, floats, integers, and categorical choices are correctly sourced from the updated parameters module.
- Updated imports in test files to directly import necessary classes from neps.space.neps_spaces.parameters. - Refactored class definitions to inherit directly from Pipeline and other relevant classes. - Replaced instances of parameter class references with direct class references for improved readability. - Ensured consistency across test files by applying similar refactoring patterns.
…pipeline adjustments
- Updated imports for optimizers in test_neps_integration.py to use the new neps.optimizers module. - Replaced references to old RandomSearch and ComplexRandomSearch algorithms with their new counterparts from neps_algorithms. - Adjusted the optimizer imports in test_neps_integration_priorband__max_cost.py and test_neps_integration_priorband__max_evals.py to align with the new structure. - Ensured all instances of priorband optimizers are updated to use the new neps_algorithms module.
…eration classes to improve clarity and consistency
…arity and consistency - Updated docstrings in SamplingResolutionContext to use Args and Returns sections for better readability. - Changed parameter names from `prior_index` to `prior` in Categorical class to enhance clarity. - Introduced a new function `convert_confidence_level` to handle string to ConfidenceLevel conversion. - Updated tests to reflect changes in parameter names from `prior_index` to `prior`. - Enhanced docstrings across various classes and methods in the sampling module for consistency.
… and convert_confidence_level function
…es and enhancing links to related topics
…ing clarity, and improving formatting in code examples
… HPO classes - Updated the image segmentation example to utilize HPO_Float and HPO_Integer for hyperparameter definitions. - Refactored test cases to replace old parameter classes with HPO variants, ensuring compatibility with the new hyperparameter optimization framework. - Added log sampling for NePS-Integer - Adjusted imports in test files to reflect changes in the neps library structure. - Enhanced the configuration encoder tests to validate the new HPO parameter types. - Ensured all relevant tests are passing with the updated hyperparameter definitions.
…md file - Corrected links in getting_started.md, evaluate_pipeline.md, and neps_run.md to point to neps_spaces.md. - Updated navigation in mkdocs.yml to reflect the removal of pipeline_space.md. - Enhanced clarity in algorithms.py and grid.py by updating parameter references. - Added validation in parameters.py to ensure prior_confidence is set when prior is defined.
…earch, and custom optimizer in NePS
…ct optimizer reference
…ecture search; rename examples for consistency
- Introduced a new `string_formatter` module to handle the pretty formatting of Operation objects, replacing the previous `operation_formatter`. - Updated `IOSampler` to use a new `format_choice` function for better representation of categorical choices. - Modified the `Summary` class to utilize the new `format_value` function for pipeline configurations. - Enhanced example scripts to improve readability and maintainability, including better logging and formatting. - Added comprehensive tests for the new string formatting functionality, ensuring correct representation of various operation types and structures. - Removed legacy code and references to the old formatting approach, streamlining the codebase.
- Introduced a helper function `_format_prior_confidence` for consistent formatting of prior confidence values in `string_formatter.py`. - Updated various instances in the codebase to utilize the new `_format_prior_confidence` function, improving code readability and maintainability. - Replaced `neps.Resampled(...)` calls with the new `.resample()` method across multiple example scripts and test files, ensuring consistency in how resampling is handled. - Cleaned up formatting in test cases to enhance clarity and maintain a consistent style. - Adjusted the formatting of operation strings in `test_string_formatter.py` to ensure proper indentation and alignment.
- Replaced instances of `Fidelity` with `IntegerFidelity` in various files to standardize fidelity parameter usage. - Updated the string formatting function calls from `operation_to_string` to `format_value` across multiple test files for consistency. - Cleaned up code formatting and improved readability by adding or adjusting whitespace in several locations. - Enhanced the `ask_and_tell_example.py` and `freeze_thaw.py` scripts to reflect the new fidelity parameter structure. - Adjusted test cases to ensure compatibility with the updated parameter types and formatting functions.
… add test for serialization
…ility for new algorithms
…etter readability
…neps_local_and_incumbent functions
…mprove error handling
…across optimizers and related functions feat: Enhance budget logging in DefaultWorker to provide detailed resource usage status
…ing rung for improved sampling
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request updates the documentation to standardize and modernize how search spaces are defined and referenced in NePS. The main change is moving from dictionary-based search space definitions to using classes that inherit from
neps.PipelineSpace.Migration to class-based search space definitions:
docs/getting_started.md,docs/index.md, anddocs/reference/neps_run.mdto use a class inheriting fromneps.PipelineSpaceinstead of a dictionary for defining the search space. This includes updating parameter definitions to useneps.Float,neps.Integer, andneps.Categorical, etc. [1] [2] [3]Reference and terminology updates:
neps.run(evaluate_pipeline, ExampleSpace())instead of passing a dictionary. [1] [2] [3]Minor formatting and clarity improvements: