feat: aviary environments - #1466
Merged
Merged
Conversation
marta-sd
self-requested a review
June 17, 2026 07:13
marta-sd
reviewed
Jun 17, 2026
| datasets: | ||
| - name: train | ||
| type: train | ||
| jsonl_fpath: environments/aviary_bbh/data/gsm8k_train.jsonl |
Contributor
There was a problem hiding this comment.
Suggested change
| jsonl_fpath: environments/aviary_bbh/data/gsm8k_train.jsonl | |
| jsonl_fpath: environments/aviary_gsm8k/data/gsm8k_train.jsonl |
| license: Apache 2.0 | ||
| - name: validation | ||
| type: validation | ||
| jsonl_fpath: environments/aviary_bbh/data/gsm8k_validation.jsonl |
Contributor
There was a problem hiding this comment.
Suggested change
| jsonl_fpath: environments/aviary_bbh/data/gsm8k_validation.jsonl | |
| jsonl_fpath: environments/aviary_gsm8k/data/gsm8k_validation.jsonl |
| datasets: | ||
| - name: train | ||
| type: train | ||
| jsonl_fpath: environments/aviary_bbh/data/hotpotqa_train.jsonl |
Contributor
There was a problem hiding this comment.
Suggested change
| jsonl_fpath: environments/aviary_bbh/data/hotpotqa_train.jsonl | |
| jsonl_fpath: environments/aviary_hotpotqa/data/hotpotqa_train.jsonl |
| license: Apache 2.0 | ||
| - name: validation | ||
| type: validation | ||
| jsonl_fpath: environments/aviary_bbh/data/hotpotqa_validation.jsonl |
Contributor
There was a problem hiding this comment.
Suggested change
| jsonl_fpath: environments/aviary_bbh/data/hotpotqa_validation.jsonl | |
| jsonl_fpath: environments/aviary_hotpotqa/data/hotpotqa_validation.jsonl |
| ## Remote Approach: | ||
| To run the remote approach, first launch the dataset server implemented in the [hypotest](https://github.com/EdisonScientific/hypotest) repository. Documentation on how to run the dataset server can be found [here](https://github.com/EdisonScientific/hypotest/blob/main/README.md). | ||
|
|
||
| Then, prepare your Gym data with the task_idx values of the problems you would like to train/evaluate on. An example dataset is provided for reference in [data/bbh_train_example.jsonl](data/bbh_train_example.jsonl). |
Contributor
There was a problem hiding this comment.
Suggested change
| Then, prepare your Gym data with the task_idx values of the problems you would like to train/evaluate on. An example dataset is provided for reference in [data/bbh_train_example.jsonl](data/bbh_train_example.jsonl). | |
| Then, prepare your Gym data with the task_idx values of the problems you would like to train/evaluate on. An example dataset is provided for reference in [data/example.jsonl](data/example.jsonl). |
| BixBench-Hypothesis is a dataset proposed by Edison Scientific to measure LLM capabilities for testing hypotheses in bioinformatics contexts. Edison Scientific and NVIDIA have also collaborated to release BBH-Train, an RL training dataset meant to improve model capabilities on bioinformatics-related data analysis. | ||
|
|
||
| There are two methods to running BBH with NeMo-Gym: the remote approach and the bundled approach. The remote approach hosts the environment sandboxes as an external service that NeMo-Gym can communicate with, serving as a modular method for running train/inference jobs. The bundled approach colocates the environment sandboxes on the Gym/RL nodes, serving as an efficient and fully packaged method for running train/inference jobs. | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| The instructions below assume that you have a running vLLM server and `policy_base_url`, `policy_model_name`, and `policy_api_key` specified in your `env.yaml`. See [documentation](https://docs.nvidia.com/nemo/gym/reference/configuration#local-configuration-envyaml) for details. |
| # BixBench (Aviary) | ||
|
|
||
| Adapts the BixBench [Aviary environment](https://github.com/Future-House/aviary) into NeMo Gym. Implements the [BixBench dataset](https://arxiv.org/abs/2503.00096) an environment with execution of a Jupyter notebook. Also serves as an example for how to implement notebook-backed environments for other scientific computational tasks. | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| The instructions below assume that you have a running vLLM server and `policy_base_url`, `policy_model_name`, and `policy_api_key` specified in your `env.yaml`. See [documentation](https://docs.nvidia.com/nemo/gym/reference/configuration#local-configuration-envyaml) for details. |
| # GSM8k (Aviary) | ||
|
|
||
| Adapts the GSM8K [Aviary environment](https://github.com/Future-House/aviary) into NeMo Gym. Implements [GSM8k](https://arxiv.org/abs/2110.14168) as an environment equipped with a calculator tool. | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| The instructions below assume that you have a running vLLM server and `policy_base_url`, `policy_model_name`, and `policy_api_key` specified in your `env.yaml`. See [documentation](https://docs.nvidia.com/nemo/gym/reference/configuration#local-configuration-envyaml) for details. |
| # HotPotQA (Aviary) | ||
|
|
||
| Adapts the HotPotQA [Aviary environment](https://github.com/Future-House/aviary) into NeMo Gym. The HotPotQA environment asks agents to perform multi-hop question answering on the [HotPotQA dataset](https://aclanthology.org/D18-1259/) with a wikipedia search tool. | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| The instructions below assume that you have a running vLLM server and `policy_base_url`, `policy_model_name`, and `policy_api_key` specified in your `env.yaml`. See [documentation](https://docs.nvidia.com/nemo/gym/reference/configuration#local-configuration-envyaml) for details. |
Contributor
Author
|
/ok |
Contributor
Author
|
/claude review |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-cmunley1-aviary-env.docs.buildwithfern.com/nemo/gym |
cmunley1
force-pushed
the
cmunley1/aviary-env
branch
from
August 25, 2026 17:46
d53fe28 to
0bd8ee2
Compare
Contributor
Author
|
/claude review |
Contributor
Author
|
/ok |
Contributor
Author
|
/ok to test 0bd8ee2 |
Contributor
|
SHIP — no reliability concerns. Reviewed as a restructuring (Aviary configs/data moved from Code changes checked:
Config/data: GSM8K |
Contributor
Author
|
/ok to test caa201b |
Contributor
Author
|
/ok to test 54effb8 |
Contributor
Author
|
/ok to test 373970c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
moves gsm8k w calculator, hotpotqa w search, bixbench and bbh to environments