-
Notifications
You must be signed in to change notification settings - Fork 401
Remove prepare_harbor_tasks.py; use harbor-private adapters #982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -50,7 +50,6 @@ Docker Network (swe-net) | |||||
| | `swe_agent_function.py` | Custom agent function — dispatches to Harbor server, returns env metadata | | ||||||
| | `generate.py` | Reward function, agent metrics aggregation, `RolloutFn` | | ||||||
| | `download_and_process_data.py` | Download from HuggingFace or local JSONL, convert to Miles format | | ||||||
| | `prepare_harbor_tasks.py` | Convert Miles JSONL to Harbor task directories (generic fallback) | | ||||||
|
|
||||||
| ## Step-by-Step Setup | ||||||
|
|
||||||
|
|
@@ -117,6 +116,20 @@ pip install harbor | |||||
|
|
||||||
| ### Step 4: Prepare data and Harbor task directories | ||||||
|
|
||||||
| Harbor task directories are prepared on the agent server side using **harbor adapters**. Each adapter converts a specific dataset into Harbor's 4-file task format. For example, to prepare SWE-bench tasks: | ||||||
|
|
||||||
| ```bash | ||||||
| # On the agent server (CPU machine), inside the harbor repo: | ||||||
| cd $CWD/harbor/adapters/swebench && uv sync | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The path
Suggested change
|
||||||
|
|
||||||
| # Generate Harbor task directories for all SWE-bench Verified instances | ||||||
| uv run run_adapter.py --task-dir $HARBOR_TASKS_DIR --all | ||||||
| ``` | ||||||
|
|
||||||
| This uses the `swebench` Python package to produce correct Docker image names and Dockerfiles for each instance. Other adapters (e.g. `adapters/swe-gym`) follow the same pattern. | ||||||
|
|
||||||
| To prepare training data on the Miles side: | ||||||
|
|
||||||
| ```bash | ||||||
| # Inside miles container: | ||||||
|
|
||||||
|
|
@@ -127,10 +140,6 @@ python download_and_process_data.py --input /data/tb.jsonl --output tb.jsonl \ | |||||
|
|
||||||
| # Merge into one mixed JSONL | ||||||
| cat swe.jsonl tb.jsonl > mixed.jsonl | ||||||
|
|
||||||
| # Create Harbor task dirs (for custom data without a Harbor adapter) | ||||||
| python prepare_harbor_tasks.py --input my.jsonl --output /root/harbor_tasks/ \ | ||||||
| --docker-network swe-net | ||||||
| ``` | ||||||
|
|
||||||
| Each Harbor task directory contains 4 files: | ||||||
|
|
@@ -294,7 +303,7 @@ Agent containers need to resolve the Miles container's hostname. Ensure: | |||||
|
|
||||||
| ### `TaskNotFound` error | ||||||
|
|
||||||
| The task directory for the given `instance_id` doesn't exist under `HARBOR_TASKS_DIR`. Run the appropriate Harbor adapter or `prepare_harbor_tasks.py` first. | ||||||
| The task directory for the given `instance_id` doesn't exist under `HARBOR_TASKS_DIR`. Run the appropriate harbor adapter first (e.g. `adapters/swebench/run_adapter.py` for SWE-bench tasks). | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||
|
|
||||||
| ### SGLang engines OOM (`Not enough memory`) | ||||||
|
|
||||||
|
|
||||||
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR description mentions confirming that no other files reference
prepare_harbor_tasks.py. However, there is still a stale reference to this script in the docstring ofdownload_and_process_data.py(line 8). Please ensure this is also updated to maintain consistency across the repository.