diff --git a/openapi/README.md b/openapi/README.md index acd1619433..0657895783 100644 --- a/openapi/README.md +++ b/openapi/README.md @@ -2,7 +2,7 @@ ## Generation -To generate the `openapi.yaml` file that corresponds exactly to what is implemented in each microservice, we start each microservice and fetch the `/openapi.json` file generated by the FastAPI server. +To generate the `openapi.yaml` files, the generator imports each FastAPI app in an isolated subprocess and calls its `.openapi()` method directly (no running server), then applies a series of schema fixes and validation passes. To generate the updated OpenAPI schema, run: @@ -10,27 +10,34 @@ To generate the updated OpenAPI schema, run: uv run --frozen python -m script.generate_openapi_spec ``` -## OpenAPI Specs Overview +## Output layout -The following table lists all the OpenAPI specifications that are merged into the final `openapi.yaml` file, along with their sources: +The generator no longer emits one spec per microservice and merges them. It now produces: -| Service | Source | Output File | -|---------|---------|-------------| -| Entity Store | Generated from `entity_store.server:app` | `entity-store.openapi.yaml` | -| Evaluator | Generated from `evaluator.server:app` | `evaluator.openapi.yaml` | -| Guardrails | Generated from `guardrails.app:app` | `guardrails.openapi.yaml` | -| Customization | Generated from `nemo-customizer-plugin` contributor routes | `customization.openapi.yaml` | -| Deployment Management | Direct copy from `deployment.openapi.yaml` | `deployment-management.openapi.yaml` | -| Jobs | Generated from `jobs.api.server:app` | `jobs.openapi.yaml` | -| Data Designer | Generated from `data_designer.api.server:app` | `data-designer.openapi.yaml` | -| Auditor | Generated from `auditor.server:app` | `auditor.openapi.yaml` | -| Safe Synthesizer API | Generated from `safe_synthesizer_api.server:app` | `safe-synthesizer.openapi.yaml` | -| Intake | Generated from `src.main:app` | `intake.openapi.yaml` | -| Models | Generated from `models.api.server:app` | `models.openapi.yaml` | -| Inference Gateway | Generated from `inference_gateway.api.server:create_app` | `inference-gateway.openapi.yaml` | -| Platform Common | Generated from `nemoplatform.server:app` | `nmp-common.openapi.yaml` | +**One aggregate platform spec**, built from the platform runner (`nmp.platform_runner.server:create_platform_openapi_app`) with plugin services deliberately excluded (`NEMO_PLUGIN_SERVICES_ALLOWLIST=""` — see `SERVICES` in `script/generate_openapi_spec.py`). This aggregate covers the core platform services (entities, jobs, models, inference gateway, secrets, files, platform-common, etc.) and lands in: -All specs are merged with the `--keep-versions` flag to preserve version information in the final `openapi.yaml`. +| File | Contents | +|------|----------| +| `openapi/openapi.yaml` | Final merged GA + EA platform spec | +| `openapi/ga/openapi.yaml` | GA-only platform spec | +| `openapi/ea/openapi.yaml` | EA-only platform spec | +| `openapi/ga/individual/platform.openapi.yaml` | The platform spec before GA/EA merge | + +**One spec per opted-in plugin**, written next to each plugin — never merged into the platform spec. A plugin opts in by declaring a `[tool.nemo.openapi]` table in its own `pyproject.toml`; `discover_plugins()` (`script/openapi_helper/plugin_config.py`) enumerates those, builds each plugin's FastAPI app via the convention loader (or a `factory_override`), and emits: + +| Plugin | Output File | +|--------|-------------| +| Agents | `plugins/nemo-agents/openapi/openapi.yaml` | +| Auditor | `plugins/nemo-auditor/openapi/openapi.yaml` | +| Customization | `plugins/nemo-customizer/openapi/openapi.yaml` | +| Data Designer | `plugins/nemo-data-designer/openapi/openapi.yaml` | +| Deployments | `plugins/nemo-deployments/openapi/openapi.yaml` | +| Evaluator | `plugins/nemo-evaluator/openapi/openapi.yaml` | +| Safe Synthesizer | `plugins/nemo-safe-synthesizer/openapi/openapi.yaml` | + +The Customization spec is assembled at generation time from whichever customization contributors (`nemo.customization.contributors` entry points — e.g. `automodel`, `rl`, `unsloth`) are installed in the workspace, so its route surface depends on the synced environment. To add a new plugin to this list, add an (empty is fine) `[tool.nemo.openapi]` table to its `pyproject.toml`; if the plugin has more than one `nemo.services` entry point, set `service_name` in that table to disambiguate. + +The platform GA and EA specs are merged with the `--keep-versions` flag to preserve version information in the final `openapi.yaml`. ### Conflicts @@ -54,4 +61,4 @@ Currently, the examples are manually generated from notebooks. A robust system t ## NOTE -Ideally, the `nmp-common.openapi.yaml` file would be the platform OpenAPI spec directly, but for various reasons, it is not. We should get to that point soon. Until then, we take this merging approach to have a spec that corresponds 100% to what is implemented. +The aggregate platform spec is now generated directly from the platform runner (`nmp.platform_runner.server`), so it corresponds to what is actually mounted at runtime. Only the GA and EA variants of that single platform spec are merged; individual core services are no longer emitted and merged separately. diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/routes.py b/packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/routes.py index cf606122c4..e90a711468 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/routes.py +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/routes.py @@ -209,9 +209,18 @@ def _derive_job_type(job_cls: type["NemoJob"]) -> str: """PascalCase form of ``job_cls.name`` for OpenAPI schema names. ``"generate"`` → ``"Generate"``; ``"metric-eval"`` → ``"MetricEval"``; - ``"raw_job"`` → ``"RawJob"``. + ``"raw_job"`` → ``"RawJob"``; ``"automodel.jobs"`` → ``"AutomodelJobs"``. + + Splits on ``.`` as well as ``-``/``_`` so a dot never reaches the generated + schema class name (``{job_type}JobRequest``). Were a dot to survive, Pydantic + would encode it as a ``__`` separator in the OpenAPI ref name, and schema-name + normalization (which strips everything before the last ``__`` to drop module + namespaces) would then collapse per-backend names like ``automodel.jobs`` and + ``rl.jobs`` to a single ``jobsJobRequest``, making every backend's request + body alias the first one. Folding the dot into the PascalCase name keeps the + discriminator inside the final segment so each backend gets a distinct schema. """ - parts = job_cls.name.replace("_", "-").split("-") + parts = job_cls.name.replace("_", "-").replace(".", "-").split("-") return "".join(part[:1].upper() + part[1:] for part in parts if part) diff --git a/packages/nemo_platform_plugin/tests/test_jobs_routes.py b/packages/nemo_platform_plugin/tests/test_jobs_routes.py index e114a4a32d..7dcb53bf81 100644 --- a/packages/nemo_platform_plugin/tests/test_jobs_routes.py +++ b/packages/nemo_platform_plugin/tests/test_jobs_routes.py @@ -201,6 +201,73 @@ def run(self, config: dict) -> dict: assert _derive_job_type(J) == "RawJob" + def test_dotted_name_folds_into_pascalcase(self) -> None: + # A dot must not survive into the schema class name: Pydantic renders it + # as a ``__`` separator in the OpenAPI ref, which schema-name + # normalization then strips, collapsing per-backend names (e.g. + # "automodel.jobs" and "rl.jobs") to a single "jobsJobRequest". + class J(NemoJob): + name = "automodel.jobs" + spec_schema = _WidgetSpec + + def run(self, config: dict) -> dict: + return config + + job_type = _derive_job_type(J) + assert job_type == "AutomodelJobs" + assert "." not in job_type + + +class TestPerJobTypeSchemaNaming: + """Guard the end-to-end path where dotted job names used to collapse. + + Two backends whose ``name`` differs only before the dot (``alpha.jobs`` / + ``beta.jobs``) must produce distinct request schemas that survive + ``tweak_spec``'s schema-name normalization, each referencing its own spec + schema. Before the ``_derive_job_type`` dot fix, both collapsed to a single + ``jobsJobRequest`` and every backend's POST body aliased the first one. + """ + + def test_dotted_backends_keep_distinct_request_schemas(self) -> None: + from fastapi import FastAPI + from nmp.common.api.utils import tweak_spec + + class AlphaSpec(BaseModel): + alpha_field: str + + class BetaSpec(BaseModel): + beta_field: int + + def _make_job(job_name: str, spec_cls: type[BaseModel]) -> type[NemoJob]: + class _J(NemoJob): + name = job_name + spec_schema = spec_cls + + def run(self, config: dict) -> dict: + return config + + @classmethod + async def compile(cls, **kwargs): # pragma: no cover - not invoked by openapi() + raise NotImplementedError + + return _J + + app = FastAPI() + app.include_router(add_job_routes(_make_job("alpha.jobs", AlphaSpec))) + app.include_router(add_job_routes(_make_job("beta.jobs", BetaSpec))) + + spec = tweak_spec(app.openapi()) + schemas = spec["components"]["schemas"] + + request_keys = {k for k in schemas if k.endswith("JobRequest")} + assert request_keys == {"AlphaJobsJobRequest", "BetaJobsJobRequest"} + + def _spec_ref(request_key: str) -> str: + return schemas[request_key]["properties"]["spec"]["$ref"].split("/")[-1] + + assert _spec_ref("AlphaJobsJobRequest") == "AlphaSpec" + assert _spec_ref("BetaJobsJobRequest") == "BetaSpec" + # --------------------------------------------------------------------------- # _adapt_to_spec diff --git a/packages/nmp_customization_common/src/nmp/customization_common/contributor/base.py b/packages/nmp_customization_common/src/nmp/customization_common/contributor/base.py index 65775e4ebe..fa81e1a6fd 100644 --- a/packages/nmp_customization_common/src/nmp/customization_common/contributor/base.py +++ b/packages/nmp_customization_common/src/nmp/customization_common/contributor/base.py @@ -15,8 +15,7 @@ from typing import Any, Callable, ClassVar import typer -from fastapi import APIRouter -from nemo_platform_plugin.authz import AuthzScope, CallerKind, path_rule +from nemo_platform_plugin.authz import AuthzScope from nemo_platform_plugin.customization_contributor import CustomizationContributorSDKResources from nemo_platform_plugin.jobs.api_factory import JobRouteOption from nemo_platform_plugin.jobs.routes import add_job_routes @@ -55,21 +54,18 @@ def _get_config(self) -> Any: raise NotImplementedError def get_routers(self) -> list[RouterSpec]: - """Health endpoint + ``add_job_routes`` for the backend job collection. + """``add_job_routes`` for the backend job collection. - HTTP authz is derived from the ``@path_rule``-decorated routes: the health - endpoint is authenticated-but-permissionless, and the job collection's - permissions (``customization..jobs.*``) are stamped onto the factory - routes via the ``customization`` :class:`AuthzScope` (scope ``customization``, - permission namespace deepened to ``customization..jobs``). + The job collection's permissions (``customization..jobs.*``) are + stamped onto the factory routes via the ``customization`` + :class:`AuthzScope` (scope ``customization``, permission namespace + deepened to ``customization..jobs``). + + Backend health is not exposed per contributor — the customization router + reports a single ``/apis/customization/v2/healthz`` that enumerates the + registered contributors. """ config = self._get_config() - router = APIRouter() - - @router.get("/healthz") - @path_rule(callers=[CallerKind.PRINCIPAL], permissions=[]) - async def healthz() -> dict[str, str]: - return {"backend": self.name, "status": "ok"} jobs_router = add_job_routes( self.job_cls, @@ -81,12 +77,6 @@ async def healthz() -> dict[str, str]: ) return [ - RouterSpec( - router=router, - prefix=f"/v2/workspaces/{{workspace}}/{self.name}", - tag=self._title, - description=f"{self._title} contributor health.", - ), RouterSpec( router=jobs_router, prefix="/v2/workspaces/{workspace}", diff --git a/packages/nmp_customization_common/src/nmp/customization_common/sdk/client.py b/packages/nmp_customization_common/src/nmp/customization_common/sdk/client.py index ec02f3054a..d7c94783c8 100644 --- a/packages/nmp_customization_common/src/nmp/customization_common/sdk/client.py +++ b/packages/nmp_customization_common/src/nmp/customization_common/sdk/client.py @@ -168,13 +168,6 @@ def __init__(self, platform: PlatformClient) -> None: self._platform = platform self._http_client = platform._client - def _healthz_url(self) -> str: - return url( - self._platform, - f"v2/workspaces/{{workspace}}/{self.backend}/healthz", - self._platform.workspace, - ) - def _new_record(self, payload: Any) -> JobRecord: return self.record_schema.model_validate(payload) @@ -182,18 +175,6 @@ def _new_record(self, payload: Any) -> JobRecord: class JobsResource(_JobsResourceBase): """Sync SDK namespace at ``client.customization..jobs``.""" - def plugin_status(self) -> dict[str, object]: - """Return contributor health from the customization service.""" - response = self._http_client.get( - self._healthz_url(), - headers=platform_default_headers(self._platform), - ) - response.raise_for_status() - payload = response.json() - if not isinstance(payload, dict): - raise TypeError(f"{self.backend} health response must be a JSON object.") - return {str(key): value for key, value in payload.items()} - def create( self, spec: BaseModel, @@ -241,18 +222,6 @@ def get_job_resource(self, job_name: str, workspace: str | None = None) -> JobRe class AsyncJobsResource(_JobsResourceBase): """Async SDK namespace at ``client.customization..jobs``.""" - async def plugin_status(self) -> dict[str, object]: - """Return contributor health from the customization service.""" - response = await self._http_client.get( - self._healthz_url(), - headers=platform_default_headers(self._platform), - ) - response.raise_for_status() - payload = response.json() - if not isinstance(payload, dict): - raise TypeError(f"{self.backend} health response must be a JSON object.") - return {str(key): value for key, value in payload.items()} - async def create( self, spec: BaseModel, diff --git a/plugins/nemo-automodel/tests/test_api.py b/plugins/nemo-automodel/tests/test_api.py index 7a34ad188d..560b83387e 100644 --- a/plugins/nemo-automodel/tests/test_api.py +++ b/plugins/nemo-automodel/tests/test_api.py @@ -31,19 +31,19 @@ def _make_automodel_app() -> FastAPI: return app -def test_automodel_healthz_under_workspace() -> None: - client = TestClient(_make_automodel_app()) - response = client.get("/v2/workspaces/test-ws/automodel/healthz") - assert response.status_code == 200 - assert response.json() == {"backend": "automodel", "status": "ok"} - - def test_automodel_jobs_collection_path() -> None: paths = _route_paths(_make_automodel_app()) assert "/v2/workspaces/{workspace}/automodel/jobs" in paths -def test_customization_router_merges_automodel(monkeypatch: pytest.MonkeyPatch) -> None: +def test_contributor_exposes_no_per_backend_healthz() -> None: + """Backend health is not exposed per contributor; the customization router + reports a single ``/v2/healthz`` that enumerates contributors instead.""" + paths = _route_paths(_make_automodel_app()) + assert not any(p.endswith("/healthz") for p in paths) + + +def test_customization_router_healthz_lists_contributors(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr( "nemo_customizer.router.discover_customization_contributors", lambda: {"automodel": AutomodelContributor()}, @@ -55,8 +55,7 @@ def test_customization_router_merges_automodel(monkeypatch: pytest.MonkeyPatch) app.include_router(spec.router, prefix=prefix) client = TestClient(app) - assert client.get("/healthz").json()["contributors"] == ["automodel"] - assert client.get("/v2/workspaces/ws-a/automodel/healthz").status_code == 200 + assert client.get("/v2/healthz").json()["contributors"] == ["automodel"] def test_workspace_isolation_list_uses_path_segment() -> None: @@ -64,4 +63,3 @@ def test_workspace_isolation_list_uses_path_segment() -> None: app = _make_automodel_app() paths = _route_paths(app) assert "/v2/workspaces/{workspace}/automodel/jobs" in paths - assert "/v2/workspaces/{workspace}/automodel/healthz" in paths diff --git a/plugins/nemo-automodel/tests/test_contributor.py b/plugins/nemo-automodel/tests/test_contributor.py index bbe078dda0..567307a58c 100644 --- a/plugins/nemo-automodel/tests/test_contributor.py +++ b/plugins/nemo-automodel/tests/test_contributor.py @@ -28,7 +28,6 @@ def test_contributor_mounts_job_collection() -> None: app.include_router(spec.router, prefix=spec.prefix) paths = _route_paths(app) - assert "/v2/workspaces/{workspace}/automodel/healthz" in paths assert "/v2/workspaces/{workspace}/automodel/jobs" in paths diff --git a/plugins/nemo-customizer/openapi/openapi.yaml b/plugins/nemo-customizer/openapi/openapi.yaml new file mode 100644 index 0000000000..4afce2002a --- /dev/null +++ b/plugins/nemo-customizer/openapi/openapi.yaml @@ -0,0 +1,2960 @@ +openapi: 3.1.0 +info: + title: customization (plugin) + version: 0.0.0 +paths: + /apis/customization/v2/healthz: + get: + tags: + - Customization + summary: Healthz + operationId: healthz_apis_customization_v2_healthz_get + responses: + '200': + description: Successful Response + content: + application/json: + schema: + additionalProperties: true + type: object + title: Response Healthz Apis Customization V2 Healthz Get + /apis/customization/v2/workspaces/{workspace}/automodel/jobs: + post: + tags: + - Automodel Jobs + summary: Create Job + operationId: create_job_apis_customization_v2_workspaces__workspace__automodel_jobs_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/AutomodelJobsJobRequest' + responses: + '201': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AutomodelJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + get: + tags: + - Automodel Jobs + summary: List Jobs + operationId: list_jobs_apis_customization_v2_workspaces__workspace__automodel_jobs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: page + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page number. + default: 1 + title: Page + description: Page number. + - name: page_size + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page size. + default: 10 + title: Page Size + description: Page size. + - name: sort + in: query + required: false + schema: + allOf: + - $ref: '#/components/schemas/AutomodelJobsJobsSortField' + description: The field to sort by. To sort in decreasing order, use `-` + in front of the field name. + default: -created_at + description: The field to sort by. To sort in decreasing order, use `-` in + front of the field name. + - in: query + name: filter + style: deepObject + required: false + explode: true + schema: + $ref: '#/components/schemas/AutomodelJobsJobsListFilter' + description: Filter jobs on various criteria. + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AutomodelJobsJobsPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{job}/results/{name}: + get: + tags: + - Automodel Jobs + summary: Get Job Result + operationId: get_job_result_apis_customization_v2_workspaces__workspace__automodel_jobs__job__results__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{job}/results/{name}/download: + get: + tags: + - Automodel Jobs + summary: Download Job Result + operationId: download_job_result_apis_customization_v2_workspaces__workspace__automodel_jobs__job__results__name__download_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/octet-stream: + schema: + type: string + format: binary + '404': + description: Not Found + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{name}: + get: + tags: + - Automodel Jobs + summary: Get Job + operationId: get_job_apis_customization_v2_workspaces__workspace__automodel_jobs__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AutomodelJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: + tags: + - Automodel Jobs + summary: Delete Job + operationId: delete_job_apis_customization_v2_workspaces__workspace__automodel_jobs__name__delete + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '204': + description: Successful Response + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{name}/cancel: + post: + tags: + - Automodel Jobs + summary: Cancel Job + operationId: cancel_job_apis_customization_v2_workspaces__workspace__automodel_jobs__name__cancel_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AutomodelJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{name}/logs: + get: + tags: + - Automodel Jobs + summary: Get Job Logs + operationId: get_job_logs_apis_customization_v2_workspaces__workspace__automodel_jobs__name__logs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + - name: limit + in: query + required: false + schema: + title: Limit + type: integer + - name: page_cursor + in: query + required: false + schema: + title: Page Cursor + type: string + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobLogPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{name}/results: + get: + tags: + - Automodel Jobs + summary: List Job Results + operationId: list_job_results_apis_customization_v2_workspaces__workspace__automodel_jobs__name__results_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobListResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/automodel/jobs/{name}/status: + get: + tags: + - Automodel Jobs + summary: Get Job Status + operationId: get_job_status_apis_customization_v2_workspaces__workspace__automodel_jobs__name__status_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobStatusResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs: + post: + tags: + - Rl Jobs + summary: Create Job + operationId: create_job_apis_customization_v2_workspaces__workspace__rl_jobs_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/RlJobsJobRequest' + responses: + '201': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/RlJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + get: + tags: + - Rl Jobs + summary: List Jobs + operationId: list_jobs_apis_customization_v2_workspaces__workspace__rl_jobs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: page + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page number. + default: 1 + title: Page + description: Page number. + - name: page_size + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page size. + default: 10 + title: Page Size + description: Page size. + - name: sort + in: query + required: false + schema: + allOf: + - $ref: '#/components/schemas/RlJobsJobsSortField' + description: The field to sort by. To sort in decreasing order, use `-` + in front of the field name. + default: -created_at + description: The field to sort by. To sort in decreasing order, use `-` in + front of the field name. + - in: query + name: filter + style: deepObject + required: false + explode: true + schema: + $ref: '#/components/schemas/RlJobsJobsListFilter' + description: Filter jobs on various criteria. + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/RlJobsJobsPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{job}/results/{name}: + get: + tags: + - Rl Jobs + summary: Get Job Result + operationId: get_job_result_apis_customization_v2_workspaces__workspace__rl_jobs__job__results__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{job}/results/{name}/download: + get: + tags: + - Rl Jobs + summary: Download Job Result + operationId: download_job_result_apis_customization_v2_workspaces__workspace__rl_jobs__job__results__name__download_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/octet-stream: + schema: + type: string + format: binary + '404': + description: Not Found + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}: + get: + tags: + - Rl Jobs + summary: Get Job + operationId: get_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/RlJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: + tags: + - Rl Jobs + summary: Delete Job + operationId: delete_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__delete + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '204': + description: Successful Response + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/cancel: + post: + tags: + - Rl Jobs + summary: Cancel Job + operationId: cancel_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__cancel_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/RlJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/logs: + get: + tags: + - Rl Jobs + summary: Get Job Logs + operationId: get_job_logs_apis_customization_v2_workspaces__workspace__rl_jobs__name__logs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + - name: limit + in: query + required: false + schema: + title: Limit + type: integer + - name: page_cursor + in: query + required: false + schema: + title: Page Cursor + type: string + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobLogPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/results: + get: + tags: + - Rl Jobs + summary: List Job Results + operationId: list_job_results_apis_customization_v2_workspaces__workspace__rl_jobs__name__results_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobListResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/status: + get: + tags: + - Rl Jobs + summary: Get Job Status + operationId: get_job_status_apis_customization_v2_workspaces__workspace__rl_jobs__name__status_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobStatusResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs: + post: + tags: + - Unsloth Jobs + summary: Create Job + operationId: create_job_apis_customization_v2_workspaces__workspace__unsloth_jobs_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJobRequest' + responses: + '201': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + get: + tags: + - Unsloth Jobs + summary: List Jobs + operationId: list_jobs_apis_customization_v2_workspaces__workspace__unsloth_jobs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: page + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page number. + default: 1 + title: Page + description: Page number. + - name: page_size + in: query + required: false + schema: + type: integer + exclusiveMinimum: 0 + description: Page size. + default: 10 + title: Page Size + description: Page size. + - name: sort + in: query + required: false + schema: + allOf: + - $ref: '#/components/schemas/UnslothJobsJobsSortField' + description: The field to sort by. To sort in decreasing order, use `-` + in front of the field name. + default: -created_at + description: The field to sort by. To sort in decreasing order, use `-` in + front of the field name. + - in: query + name: filter + style: deepObject + required: false + explode: true + schema: + $ref: '#/components/schemas/UnslothJobsJobsListFilter' + description: Filter jobs on various criteria. + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJobsPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}: + get: + tags: + - Unsloth Jobs + summary: Get Job Result + operationId: get_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}/download: + get: + tags: + - Unsloth Jobs + summary: Download Job Result + operationId: download_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__download_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: job + in: path + required: true + schema: + type: string + title: Job + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/octet-stream: + schema: + type: string + format: binary + '404': + description: Not Found + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}: + get: + tags: + - Unsloth Jobs + summary: Get Job + operationId: get_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: + tags: + - Unsloth Jobs + summary: Delete Job + operationId: delete_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__delete + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '204': + description: Successful Response + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/cancel: + post: + tags: + - Unsloth Jobs + summary: Cancel Job + operationId: cancel_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__cancel_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/logs: + get: + tags: + - Unsloth Jobs + summary: Get Job Logs + operationId: get_job_logs_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__logs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + - name: limit + in: query + required: false + schema: + title: Limit + type: integer + - name: page_cursor + in: query + required: false + schema: + title: Page Cursor + type: string + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobLogPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/results: + get: + tags: + - Unsloth Jobs + summary: List Job Results + operationId: list_job_results_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__results_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobListResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/status: + get: + tags: + - Unsloth Jobs + summary: Get Job Status + operationId: get_job_status_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__status_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobStatusResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' +components: + schemas: + AutomodelJobInput: + properties: + name: + title: Name + type: string + model: + type: string + title: Model + dataset: + $ref: '#/components/schemas/DatasetSpec' + training: + $ref: '#/components/schemas/TrainingSpec' + schedule: + $ref: '#/components/schemas/ScheduleSpec' + batch: + $ref: '#/components/schemas/BatchSpec' + optimizer: + $ref: '#/components/schemas/OptimizerSpec' + parallelism: + $ref: '#/components/schemas/ParallelismSpec' + output: + $ref: '#/components/schemas/OutputRequest' + integrations: + $ref: '#/components/schemas/IntegrationsSpecInput' + additionalProperties: false + type: object + required: + - model + - dataset + - training + title: AutomodelJobInput + description: POST body / CLI JSON. + AutomodelJobOutput: + properties: + name: + title: Name + type: string + model: + type: string + title: Model + dataset: + $ref: '#/components/schemas/DatasetSpec' + training: + $ref: '#/components/schemas/TrainingSpec' + schedule: + $ref: '#/components/schemas/ScheduleSpec' + batch: + $ref: '#/components/schemas/BatchSpec' + optimizer: + $ref: '#/components/schemas/OptimizerSpec' + parallelism: + $ref: '#/components/schemas/ParallelismSpec' + output: + $ref: '#/components/schemas/OutputResponse' + integrations: + $ref: '#/components/schemas/IntegrationsSpecOutput' + additionalProperties: false + type: object + required: + - model + - dataset + - training + - schedule + - batch + - optimizer + - parallelism + - output + title: AutomodelJobOutput + description: Stored canonical spec after ``to_spec()``. + AutomodelJobsJob: + properties: + id: + title: Id + type: string + name: + type: string + title: Name + description: + title: Description + type: string + project: + title: Project + type: string + workspace: + title: Workspace + type: string + created_at: + title: Created At + type: string + format: date-time + updated_at: + title: Updated At + type: string + format: date-time + spec: + $ref: '#/components/schemas/AutomodelJobOutput' + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + title: Status Details + additionalProperties: true + type: object + error_details: + title: Error Details + additionalProperties: true + type: object + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - name + - spec + title: AutomodelJobsJob + AutomodelJobsJobRequest: + properties: + name: + title: Name + type: string + description: + title: Description + type: string + project: + title: Project + type: string + spec: + $ref: '#/components/schemas/AutomodelJobInput' + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - spec + title: AutomodelJobsJobRequest + AutomodelJobsJobsListFilter: + additionalProperties: false + properties: + created_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs created at 'gte' datetime or 'lte' datetime. + name: + anyOf: + - $ref: '#/components/schemas/StringFilter' + - type: string + description: Name of the job. + title: Name + workspace: + description: Workspace of the job. + title: Workspace + type: string + project: + description: Project containing the job. + title: Project + type: string + status: + allOf: + - $ref: '#/components/schemas/PlatformJobStatus' + description: The current status. + updated_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs updated at 'gte' datetime or 'lte' datetime. + title: AutomodelJobsJobsListFilter + type: object + AutomodelJobsJobsPage: + properties: + data: + items: + $ref: '#/components/schemas/AutomodelJobsJob' + type: array + title: Data + pagination: + allOf: + - $ref: '#/components/schemas/PaginationData' + description: Pagination information. + sort: + title: Sort + description: The field on which the results are sorted. + type: string + filter: + title: Filter + description: Filtering information. + additionalProperties: true + type: object + type: object + required: + - data + title: AutomodelJobsJobsPage + AutomodelJobsJobsSortField: + type: string + enum: + - created_at + - -created_at + - updated_at + - -updated_at + title: AutomodelJobsJobsSortField + BatchSpec: + properties: + global_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Global Batch Size + default: 8 + micro_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Micro Batch Size + default: 1 + sequence_packing: + type: boolean + title: Sequence Packing + default: false + sequence_packing_max_samples: + type: integer + exclusiveMinimum: 0.0 + title: Sequence Packing Max Samples + description: Samples analyzed to estimate the optimal pack size when packing + is enabled. + default: 1000 + additionalProperties: false + type: object + title: BatchSpec + DPOTraining: + properties: + optimizer_type: + allOf: + - $ref: '#/components/schemas/OptimizerType' + description: "Optimizer + LR-scheduler combination (AdamW/Adam \xD7 cosine-annealing/flat-LR).\ + \ Defaults to AdamW with cosine annealing." + learning_rate: + type: number + title: Learning Rate + description: Peak learning rate. + default: 0.0001 + min_learning_rate: + title: Min Learning Rate + description: Minimum LR for cosine decay. + type: number + weight_decay: + type: number + title: Weight Decay + description: Weight decay coefficient. + default: 0.01 + adam_beta1: + type: number + title: Adam Beta1 + description: Adam beta1. + default: 0.9 + adam_beta2: + type: number + title: Adam Beta2 + description: Adam beta2. + default: 0.999 + adam_eps: + type: number + exclusiveMinimum: 0.0 + title: Adam Eps + description: Adam epsilon (numerical stability term). + default: 1.0e-05 + warmup_steps: + type: integer + minimum: 0.0 + title: Warmup Steps + description: Linear warmup steps. + default: 0 + epochs: + type: integer + exclusiveMinimum: 0.0 + title: Epochs + description: Number of passes through the dataset. + default: 1 + max_steps: + title: Max Steps + description: Max training steps (overrides epochs if set). + type: integer + exclusiveMinimum: 0.0 + val_check_interval: + title: Val Check Interval + description: Validation interval. Float <= 1.0 is fraction of epoch; > 1.0 + is step count. + type: number + val_at_end: + type: boolean + title: Val At End + description: Run a final validation pass after the last training step. Keep + enabled so the final checkpoint carries validation metrics and best-checkpoint + selection works; set False only to skip the extra eval. + default: true + keep_top_k: + type: integer + exclusiveMinimum: 0.0 + title: Keep Top K + description: Number of best checkpoints to retain (ranked by validation + loss). + default: 1 + batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Batch Size + description: Global batch size across all GPUs. + default: 32 + micro_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Micro Batch Size + description: Per-GPU micro batch size. + default: 1 + activation_checkpointing: + type: boolean + title: Activation Checkpointing + description: Recompute activations during the backward pass to reduce memory + at the cost of compute. Enable to fit larger models or longer sequences. + default: false + max_seq_length: + type: integer + exclusiveMinimum: 0.0 + title: Max Seq Length + description: Maximum token sequence length for training. + default: 2048 + seed: + title: Seed + description: Random seed for reproducibility. + type: integer + parallelism: + $ref: '#/components/schemas/ParallelismParams' + execution_profile: + title: Execution Profile + description: Execution profile for the GPU training step (operator-configured). + Falls back to the service default when omitted. + type: string + minLength: 1 + type: + type: string + const: dpo + title: Type + default: dpo + ref_policy_kl_penalty: + type: number + minimum: 0.0 + title: Ref Policy Kl Penalty + description: KL penalty coefficient (beta in the DPO paper). + default: 0.05 + preference_average_log_probs: + type: boolean + title: Preference Average Log Probs + description: Average log probabilities for preference loss calculation. + default: false + sft_average_log_probs: + type: boolean + title: Sft Average Log Probs + description: Average log probabilities for SFT regularization loss. + default: false + preference_loss_weight: + type: number + minimum: 0.0 + title: Preference Loss Weight + description: Weight for the preference (DPO) loss term. + default: 1.0 + sft_loss_weight: + type: number + minimum: 0.0 + title: Sft Loss Weight + description: Weight for SFT regularization loss (0 = disabled). + default: 0.0 + max_grad_norm: + type: number + minimum: 0.0 + title: Max Grad Norm + description: Maximum gradient norm for clipping. + default: 1.0 + additionalProperties: false + type: object + title: DPOTraining + description: "Direct Preference Optimization (full-weight only \u2014 PEFT unsupported)." + DatasetSpec: + properties: + training: + type: string + title: Training + description: Training fileset as 'name' or 'workspace/name'. + validation: + title: Validation + type: string + prompt_template: + title: Prompt Template + type: string + additionalProperties: false + type: object + required: + - training + title: DatasetSpec + DatetimeFilter: + additionalProperties: false + properties: + $gte: + description: Filter for results greater than or equal to this datetime. + title: $Gte + format: date-time + type: string + $lte: + description: Filter for results less than or equal to this datetime. + title: $Lte + format: date-time + type: string + title: DatetimeFilter + type: object + DeploymentParams: + properties: + gpu: + type: integer + title: Gpu + description: Number of GPUs required for the deployment. + default: 1 + additional_envs: + title: Additional Envs + description: Additional environment variables for the deployment. + additionalProperties: + type: string + type: object + disk_size: + title: Disk Size + description: Disk size for the deployment. + type: string + image_name: + title: Image Name + description: Container image name from NGC. If not specified, defaults to + multi-llm. + type: string + image_tag: + title: Image Tag + description: Container image tag from NGC. + type: string + lora_enabled: + type: boolean + title: Lora Enabled + description: When auto-deploying a full SFT training, setting this true + allows subsequent LoRA adapters to be deployed against it. + default: true + tool_call_config: + allOf: + - $ref: '#/components/schemas/ToolCallParams' + description: Tool calling configuration override for the NIM deployment. + additionalProperties: false + type: object + title: DeploymentParams + description: 'Inline deployment parameters for auto-deploying a trained model. + + + Used in :class:`UnslothJobInput.deployment_config` and passed through to + + the model_entity task at compile time. When unset, no deployment is launched.' + FileStorageType: + type: string + enum: + - fileset + title: FileStorageType + HTTPValidationError: + properties: + detail: + items: + $ref: '#/components/schemas/ValidationError' + type: array + title: Detail + type: object + title: HTTPValidationError + HardwareSpec: + properties: + gpus: + title: Gpus + description: Comma-separated GPU indices ('0' or '0,1') for CUDA_VISIBLE_DEVICES. + Selection, not reservation. + type: string + precision: + type: string + enum: + - bf16 + - fp16 + title: Precision + description: Mixed-precision dtype for training. bf16 recommended for Ampere+. + default: bf16 + additionalProperties: false + type: object + title: HardwareSpec + IntegrationsSpecInput: + properties: + wandb: + allOf: + - $ref: '#/components/schemas/WandbIntegration' + description: Weights & Biases integration configuration. + mlflow: + allOf: + - $ref: '#/components/schemas/MlflowIntegration' + description: MLflow integration configuration. + additionalProperties: false + type: object + title: IntegrationsSpecInput + description: 'Third-party experiment-tracking integrations for a job spec. + + + Each integration is requested by presence: omit or set a field to ``null`` + to + + disable it. Activation at training time still requires credentials/URIs + + (see compile-time warnings and runtime builders).' + IntegrationsSpecOutput: + properties: + wandb: + allOf: + - $ref: '#/components/schemas/WandbIntegration' + description: Weights & Biases integration configuration. + mlflow: + allOf: + - $ref: '#/components/schemas/MlflowIntegration' + description: MLflow integration configuration. + additionalProperties: false + type: object + title: IntegrationsSpecOutput + description: 'Third-party experiment-tracking integrations for a job spec. + + + Each integration is requested by presence: omit or set a field to ``null`` + to + + disable it. Activation at training time still requires credentials/URIs + + (see compile-time warnings and runtime builders).' + LoRAParams: + properties: + rank: + type: integer + exclusiveMinimum: 0.0 + title: Rank + default: 16 + alpha: + type: integer + exclusiveMinimum: 0.0 + title: Alpha + default: 32 + dropout: + type: number + maximum: 1.0 + minimum: 0.0 + title: Dropout + description: LoRA dropout probability for regularization. + default: 0.0 + merge: + type: boolean + title: Merge + default: false + target_modules: + title: Target Modules + items: + type: string + type: array + exclude_modules: + title: Exclude Modules + description: Module name patterns to exclude from LoRA (e.g. ['*.out_proj']). + items: + type: string + type: array + use_triton: + type: boolean + title: Use Triton + description: Use the optimized Triton LoRA kernel. + default: true + additionalProperties: false + type: object + title: LoRAParams + MlflowIntegration: + properties: + experiment_name: + title: Experiment Name + description: MLflow experiment name (groups related runs). Defaults to output.name + if not set. + type: string + name: + title: Name + description: MLflow run name. Defaults to job_id if not provided. + type: string + tags: + title: Tags + description: MLflow tags as key-value pairs for filtering runs. + additionalProperties: + type: string + type: object + description: + title: Description + description: MLflow run description. + type: string + tracking_uri: + title: Tracking Uri + description: MLflow tracking server URI (e.g., 'http://mlflow.mycompany.com:5000'). + Can also be set via MLFLOW_TRACKING_URI environment variable. + type: string + additionalProperties: false + type: object + title: MlflowIntegration + description: 'MLflow integration configuration. + + + To enable MLflow, provide a non-null ``mlflow`` object on :class:`IntegrationsSpec`.' + ModelLoadSpec: + properties: + name: + type: string + title: Name + description: Model entity reference. Accepts 'name' (uses the job's workspace) + or 'workspace/name'. The plugin's run resolves this to a local path before + training. + max_seq_length: + type: integer + exclusiveMinimum: 0.0 + title: Max Seq Length + default: 2048 + load_in_4bit: + type: boolean + title: Load In 4Bit + description: bitsandbytes 4-bit. Mutex with load_in_8bit. Default for Unsloth's + headline path. + default: true + load_in_8bit: + type: boolean + title: Load In 8Bit + default: false + dtype: + type: string + enum: + - auto + - bfloat16 + - float16 + - float32 + title: Dtype + default: auto + trust_remote_code: + type: boolean + title: Trust Remote Code + default: false + device_map: + anyOf: + - type: string + - type: integer + - additionalProperties: + type: integer + type: object + title: Device Map + description: "Device placement forwarded to FastLanguageModel.from_pretrained.\ + \ Omit (null) to pin the whole model to the single visible GPU ({'': 0})\ + \ \u2014 the right default for this single-GPU backend, and it avoids\ + \ accelerate's auto-placement under-sizing GPU memory on unified-memory\ + \ parts (e.g. GB10 / DGX Spark), which otherwise spills layers to CPU\ + \ and aborts 4-bit loads. Set 'auto', 'balanced', 'sequential', a device\ + \ index, or a custom map for multi-device experiments." + rope_scaling: + title: Rope Scaling + description: 'RoPE scaling config for long-context extension, passed to + FastLanguageModel.from_pretrained (e.g. {''type'': ''linear'', ''factor'': + 2.0}). None uses the model''s native context length.' + additionalProperties: true + type: object + additionalProperties: false + type: object + required: + - name + title: ModelLoadSpec + description: 'Args to ``FastLanguageModel.from_pretrained``. + + + ``name`` is a NeMo Platform model entity reference (``"name"`` or + + ``"workspace/name"``). The plugin''s run orchestration resolves the + + entity, downloads its fileset to a local path, and hands that path + + to :func:`train_sft`.' + OptimizerSpec: + properties: + learning_rate: + type: number + exclusiveMinimum: 0.0 + title: Learning Rate + default: 5.0e-06 + min_learning_rate: + title: Min Learning Rate + description: Minimum learning rate for the cosine decay schedule. + type: number + minimum: 0.0 + weight_decay: + type: number + minimum: 0.0 + title: Weight Decay + default: 0.01 + adam_beta1: + type: number + exclusiveMaximum: 1.0 + minimum: 0.0 + title: Adam Beta1 + description: Adam optimizer beta1. + default: 0.9 + adam_beta2: + type: number + exclusiveMaximum: 1.0 + minimum: 0.0 + title: Adam Beta2 + description: Adam optimizer beta2. + default: 0.999 + warmup_steps: + type: integer + minimum: 0.0 + title: Warmup Steps + default: 0 + adam_eps: + type: number + exclusiveMinimum: 0.0 + title: Adam Eps + description: Adam/AdamW epsilon for numerical stability. + default: 1.0e-08 + optimizer: + type: string + enum: + - Adam + - AdamW + title: Optimizer + description: Optimizer algorithm. + default: Adam + lr_decay_style: + type: string + enum: + - cosine + - linear + - constant + title: Lr Decay Style + description: Learning-rate decay schedule. + default: cosine + additionalProperties: false + type: object + title: OptimizerSpec + OptimizerType: + type: string + enum: + - adamw_with_cosine_annealing + - adam_with_cosine_annealing + - adamw_with_flat_lr + - adam_with_flat_lr + title: OptimizerType + description: Optimizer and scheduler combination types. + OutputRequest: + properties: + name: + type: string + title: Name + description: + title: Description + type: string + additionalProperties: false + type: object + required: + - name + title: OutputRequest + OutputResponse: + properties: + name: + type: string + title: Name + type: + type: string + enum: + - model + - adapter + title: Type + fileset: + type: string + title: Fileset + description: + title: Description + type: string + additionalProperties: false + type: object + required: + - name + - type + - fileset + title: OutputResponse + PaginationData: + properties: + page: + type: integer + title: Page + description: The current page number. + page_size: + type: integer + title: Page Size + description: The page size used for the query. + current_page_size: + type: integer + title: Current Page Size + description: The size for the current page. + total_pages: + type: integer + title: Total Pages + description: The total number of pages. + total_results: + type: integer + title: Total Results + description: The total number of results. + type: object + required: + - page + - page_size + - current_page_size + - total_pages + - total_results + title: PaginationData + ParallelismParams: + properties: + num_gpus_per_node: + type: integer + exclusiveMinimum: 0.0 + title: Num Gpus Per Node + description: Number of GPUs per node. + default: 1 + num_nodes: + type: integer + exclusiveMinimum: 0.0 + title: Num Nodes + description: "Number of nodes (>1 \u2192 multi-node Ray cluster)." + default: 1 + tensor_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Tensor Parallel Size + description: Tensor parallel size. + default: 1 + pipeline_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Pipeline Parallel Size + description: Pipeline parallel size. + default: 1 + context_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Context Parallel Size + description: Context parallel size. + default: 1 + sequence_parallel: + type: boolean + title: Sequence Parallel + description: Enable sequence parallelism. + default: false + additionalProperties: false + type: object + title: ParallelismParams + description: 'Distributed training parallelism configuration. + + + Single-node multi-GPU uses ``num_nodes=1`` with ``num_gpus_per_node>1``; + + multi-node sets ``num_nodes>1`` and the compiler emits a distributed-GPU + + executor (see :mod:`nmp.rl.app.jobs.compiler`).' + ParallelismSpec: + properties: + num_nodes: + type: integer + exclusiveMinimum: 0.0 + title: Num Nodes + default: 1 + num_gpus_per_node: + type: integer + exclusiveMinimum: 0.0 + title: Num Gpus Per Node + default: 1 + tensor_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Tensor Parallel Size + default: 1 + pipeline_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Pipeline Parallel Size + default: 1 + context_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Context Parallel Size + default: 1 + expert_parallel_size: + title: Expert Parallel Size + type: integer + exclusiveMinimum: 0.0 + sequence_parallel: + type: boolean + title: Sequence Parallel + description: Enable sequence parallelism. + default: false + additionalProperties: false + type: object + title: ParallelismSpec + PlatformJobListResultResponse: + properties: + data: + items: + $ref: '#/components/schemas/PlatformJobResultResponse' + type: array + title: Data + type: object + required: + - data + title: PlatformJobListResultResponse + PlatformJobLog: + properties: + timestamp: + type: string + format: date-time + title: Timestamp + job: + type: string + title: Job + job_step: + type: string + title: Job Step + job_task: + type: string + title: Job Task + message: + type: string + title: Message + type: object + required: + - timestamp + - job + - job_step + - job_task + - message + title: PlatformJobLog + PlatformJobLogPage: + properties: + data: + items: + $ref: '#/components/schemas/PlatformJobLog' + type: array + title: Data + total: + type: integer + title: Total + next_page: + title: Next Page + type: string + prev_page: + title: Prev Page + type: string + type: object + required: + - data + - total + - next_page + - prev_page + title: PlatformJobLogPage + PlatformJobResultResponse: + properties: + name: + type: string + title: Name + job: + type: string + title: Job + workspace: + type: string + title: Workspace + project: + title: Project + type: string + created_at: + type: string + format: date-time + title: Created At + updated_at: + type: string + format: date-time + title: Updated At + artifact_url: + type: string + title: Artifact Url + artifact_storage_type: + $ref: '#/components/schemas/FileStorageType' + download_url: + title: Download Url + type: string + type: object + required: + - name + - job + - workspace + - artifact_url + - artifact_storage_type + title: PlatformJobResultResponse + PlatformJobStatus: + type: string + enum: + - created + - pending + - active + - cancelled + - cancelling + - error + - completed + - paused + - pausing + - resuming + title: PlatformJobStatus + description: 'Enumeration of possible job statuses. + + + This enum represents the various states a job can be in during its lifecycle, + + from creation to a terminal state.' + PlatformJobStatusResponse: + properties: + id: + type: string + title: Id + name: + type: string + title: Name + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + steps: + items: + $ref: '#/components/schemas/PlatformJobStepStatusResponse' + type: array + title: Steps + created_at: + type: string + format: date-time + title: Created At + updated_at: + type: string + format: date-time + title: Updated At + type: object + required: + - id + - name + - status + - status_details + - error_details + - steps + - created_at + - updated_at + title: PlatformJobStatusResponse + PlatformJobStepStatusResponse: + properties: + id: + type: string + title: Id + name: + type: string + title: Name + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + tasks: + items: + $ref: '#/components/schemas/PlatformJobTaskStatusResponse' + type: array + title: Tasks + created_at: + type: string + format: date-time + title: Created At + updated_at: + type: string + format: date-time + title: Updated At + type: object + required: + - id + - name + - status + - status_details + - error_details + - tasks + - created_at + - updated_at + title: PlatformJobStepStatusResponse + PlatformJobTaskStatusResponse: + properties: + id: + type: string + title: Id + name: + type: string + title: Name + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + error_stack: + title: Error Stack + type: string + created_at: + type: string + format: date-time + title: Created At + updated_at: + type: string + format: date-time + title: Updated At + type: object + required: + - id + - name + - status + - status_details + - error_details + - error_stack + - created_at + - updated_at + title: PlatformJobTaskStatusResponse + RlJobInput: + properties: + name: + title: Name + type: string + model: + type: string + title: Model + description: Model entity reference ('name' or 'workspace/name'). + dataset: + type: string + title: Dataset + description: Preference dataset fileset reference. Must contain training.jsonl + + validation.jsonl. + training: + allOf: + - $ref: '#/components/schemas/DPOTraining' + description: DPO training method and hyperparameters. + integrations: + $ref: '#/components/schemas/IntegrationsSpecInput' + output: + $ref: '#/components/schemas/OutputRequest' + additionalProperties: false + type: object + required: + - model + - dataset + - training + title: RlJobInput + description: POST body / CLI JSON for ``nemo customization rl submit``. + RlJobOutput: + properties: + name: + title: Name + description: Optional job name; auto-generated when omitted. + type: string + model: + type: string + title: Model + description: Model entity reference ('name' or 'workspace/name'). + dataset: + type: string + title: Dataset + description: Preference dataset fileset reference ('name' or 'workspace/name'). + training: + allOf: + - $ref: '#/components/schemas/DPOTraining' + description: Training method and hyperparameters (DPO). + integrations: + allOf: + - $ref: '#/components/schemas/IntegrationsSpecOutput' + description: W&B / MLflow integrations. + output: + allOf: + - $ref: '#/components/schemas/OutputResponse' + description: Output artifact created by this job. + type: object + required: + - model + - dataset + - training + - output + title: RlJobOutput + description: 'Canonical NeMo-RL job spec (output of the plugin transform). + + + The ``dataset`` fileset must contain ``training.jsonl`` and ``validation.jsonl`` + + (any of the four supported preference formats); the dataset-preparation step + + splits/normalizes them at runtime.' + RlJobsJob: + properties: + id: + title: Id + type: string + name: + type: string + title: Name + description: + title: Description + type: string + project: + title: Project + type: string + workspace: + title: Workspace + type: string + created_at: + title: Created At + type: string + format: date-time + updated_at: + title: Updated At + type: string + format: date-time + spec: + $ref: '#/components/schemas/RlJobOutput' + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + title: Status Details + additionalProperties: true + type: object + error_details: + title: Error Details + additionalProperties: true + type: object + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - name + - spec + title: RlJobsJob + RlJobsJobRequest: + properties: + name: + title: Name + type: string + description: + title: Description + type: string + project: + title: Project + type: string + spec: + $ref: '#/components/schemas/RlJobInput' + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - spec + title: RlJobsJobRequest + RlJobsJobsListFilter: + additionalProperties: false + properties: + created_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs created at 'gte' datetime or 'lte' datetime. + name: + anyOf: + - $ref: '#/components/schemas/StringFilter' + - type: string + description: Name of the job. + title: Name + workspace: + description: Workspace of the job. + title: Workspace + type: string + project: + description: Project containing the job. + title: Project + type: string + status: + allOf: + - $ref: '#/components/schemas/PlatformJobStatus' + description: The current status. + updated_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs updated at 'gte' datetime or 'lte' datetime. + title: RlJobsJobsListFilter + type: object + RlJobsJobsPage: + properties: + data: + items: + $ref: '#/components/schemas/RlJobsJob' + type: array + title: Data + pagination: + allOf: + - $ref: '#/components/schemas/PaginationData' + description: Pagination information. + sort: + title: Sort + description: The field on which the results are sorted. + type: string + filter: + title: Filter + description: Filtering information. + additionalProperties: true + type: object + type: object + required: + - data + title: RlJobsJobsPage + RlJobsJobsSortField: + type: string + enum: + - created_at + - -created_at + - updated_at + - -updated_at + title: RlJobsJobsSortField + ScheduleSpec: + properties: + epochs: + type: integer + exclusiveMinimum: 0.0 + title: Epochs + default: 1 + max_steps: + title: Max Steps + type: integer + exclusiveMinimum: 0.0 + val_check_interval: + title: Val Check Interval + type: number + seed: + title: Seed + type: integer + additionalProperties: false + type: object + title: ScheduleSpec + SecretRef: + type: string + pattern: ^[a-z0-9_-]+(/[a-z0-9_-]+)?$ + title: SecretRef + description: 'Reference to a secret. Format: ''secret_name'' (uses request workspace) + or ''workspace/secret_name'' (explicit workspace).' + StringFilter: + additionalProperties: false + properties: + $eq: + description: Filter for results equal to this value. + title: $Eq + type: string + $like: + description: Filter for results matching this pattern. + title: $Like + type: string + $in: + description: Filter for results in this list of values. + title: $In + items: + type: string + type: array + $nin: + description: Filter for results not in this list of values. + title: $Nin + items: + type: string + type: array + title: StringFilter + type: object + ToolCallParams: + properties: + tool_call_parser: + title: Tool Call Parser + description: Name of the tool call parser to use (e.g., 'openai', 'hermes', + 'pythonic', 'llama3_json', 'mistral'). + type: string + tool_call_plugin: + title: Tool Call Plugin + description: 'Reference to a fileset containing the custom tool call plugin + Python file. Expected format: ''{workspace}/{fileset_name}''.' + type: string + pattern: ^[\w\-.]+/[\w\-.]+$ + auto_tool_choice: + title: Auto Tool Choice + description: Whether to enable automatic tool choice. + type: boolean + additionalProperties: false + type: object + title: ToolCallParams + description: Tool calling configuration for NIM deployments. + TrainingSpec: + properties: + training_type: + type: string + enum: + - sft + - distillation + title: Training Type + default: sft + finetuning_type: + type: string + enum: + - lora + - all_weights + - lora_merged + title: Finetuning Type + default: lora + lora: + $ref: '#/components/schemas/LoRAParams' + max_seq_length: + type: integer + exclusiveMinimum: 0.0 + title: Max Seq Length + default: 2048 + precision: + title: Precision + description: Model precision for training. Auto-detected from the checkpoint + when unset. + type: string + enum: + - bf16 + - fp16 + - fp32 + - fp8 + attn_implementation: + type: string + enum: + - sdpa + - flash_attention_2 + - eager + title: Attn Implementation + description: 'Attention backend: ''sdpa'' (PyTorch native), ''flash_attention_2'', + or ''eager''.' + default: sdpa + execution_profile: + title: Execution Profile + type: string + minLength: 1 + teacher_model: + title: Teacher Model + type: string + distillation_ratio: + type: number + maximum: 1.0 + minimum: 0.0 + title: Distillation Ratio + default: 0.5 + distillation_temperature: + type: number + exclusiveMinimum: 0.0 + title: Distillation Temperature + default: 1.0 + teacher_precision: + type: string + enum: + - bf16 + - fp16 + - fp32 + title: Teacher Precision + default: bf16 + offload_teacher: + type: boolean + title: Offload Teacher + default: false + additionalProperties: false + type: object + title: TrainingSpec + UnslothJobInput: + properties: + name: + title: Name + type: string + model: + $ref: '#/components/schemas/ModelLoadSpec' + dataset: + $ref: '#/components/schemas/DatasetSpec' + training: + $ref: '#/components/schemas/TrainingSpec' + schedule: + $ref: '#/components/schemas/ScheduleSpec' + batch: + $ref: '#/components/schemas/BatchSpec' + optimizer: + $ref: '#/components/schemas/OptimizerSpec' + hardware: + $ref: '#/components/schemas/HardwareSpec' + integrations: + $ref: '#/components/schemas/IntegrationsSpecInput' + output: + $ref: '#/components/schemas/OutputRequest' + deployment_config: + anyOf: + - type: string + title: Reference + description: A reference to DeploymentParams. + - $ref: '#/components/schemas/DeploymentParams' + title: Deployment Config + description: Deployment configuration for auto-deploying the model after + training. Pass a string to reference an existing ModelDeploymentConfig + by name ('my-config' or 'workspace/my-config'). An object provides inline + NIM deployment parameters. Omit to skip deployment. + additionalProperties: false + type: object + required: + - model + - dataset + title: UnslothJobInput + description: POST body / CLI JSON for ``nemo customization unsloth run``. + UnslothJobOutput: + properties: + name: + title: Name + type: string + model: + $ref: '#/components/schemas/ModelLoadSpec' + dataset: + $ref: '#/components/schemas/DatasetSpec' + training: + $ref: '#/components/schemas/TrainingSpec' + schedule: + $ref: '#/components/schemas/ScheduleSpec' + batch: + $ref: '#/components/schemas/BatchSpec' + optimizer: + $ref: '#/components/schemas/OptimizerSpec' + hardware: + $ref: '#/components/schemas/HardwareSpec' + integrations: + $ref: '#/components/schemas/IntegrationsSpecOutput' + output: + $ref: '#/components/schemas/OutputResponse' + deployment_config: + anyOf: + - type: string + title: Reference + description: A reference to DeploymentParams. + - $ref: '#/components/schemas/DeploymentParams' + title: Deployment Config + description: Deployment configuration for auto-deploying the model after + training. Pass a string to reference an existing ModelDeploymentConfig + by name ('my-config' or 'workspace/my-config'). An object provides inline + NIM deployment parameters. Omit to skip deployment. + additionalProperties: false + type: object + required: + - model + - dataset + - output + title: UnslothJobOutput + description: 'Canonical spec stored after the plugin''s ``to_spec()`` resolves + output naming. + + + Defaults match :class:`~nemo_unsloth_plugin.schema.UnslothJobInput` so SDK + + callers and tests can construct :class:`UnslothJobOutput` directly without + + restating every sub-section. The plugin''s ``to_spec`` always passes the + + resolved input values through, so these defaults never override real input.' + UnslothJobsJob: + properties: + id: + title: Id + type: string + name: + type: string + title: Name + description: + title: Description + type: string + project: + title: Project + type: string + workspace: + title: Workspace + type: string + created_at: + title: Created At + type: string + format: date-time + updated_at: + title: Updated At + type: string + format: date-time + spec: + $ref: '#/components/schemas/UnslothJobOutput' + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + title: Status Details + additionalProperties: true + type: object + error_details: + title: Error Details + additionalProperties: true + type: object + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - name + - spec + title: UnslothJobsJob + UnslothJobsJobRequest: + properties: + name: + title: Name + type: string + description: + title: Description + type: string + project: + title: Project + type: string + spec: + $ref: '#/components/schemas/UnslothJobInput' + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object + type: object + required: + - spec + title: UnslothJobsJobRequest + UnslothJobsJobsListFilter: + additionalProperties: false + properties: + created_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs created at 'gte' datetime or 'lte' datetime. + name: + anyOf: + - $ref: '#/components/schemas/StringFilter' + - type: string + description: Name of the job. + title: Name + workspace: + description: Workspace of the job. + title: Workspace + type: string + project: + description: Project containing the job. + title: Project + type: string + status: + allOf: + - $ref: '#/components/schemas/PlatformJobStatus' + description: The current status. + updated_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs updated at 'gte' datetime or 'lte' datetime. + title: UnslothJobsJobsListFilter + type: object + UnslothJobsJobsPage: + properties: + data: + items: + $ref: '#/components/schemas/UnslothJobsJob' + type: array + title: Data + pagination: + allOf: + - $ref: '#/components/schemas/PaginationData' + description: Pagination information. + sort: + title: Sort + description: The field on which the results are sorted. + type: string + filter: + title: Filter + description: Filtering information. + additionalProperties: true + type: object + type: object + required: + - data + title: UnslothJobsJobsPage + UnslothJobsJobsSortField: + type: string + enum: + - created_at + - -created_at + - updated_at + - -updated_at + title: UnslothJobsJobsSortField + ValidationError: + properties: + loc: + items: + anyOf: + - type: string + - type: integer + type: array + title: Location + msg: + type: string + title: Message + type: + type: string + title: Error Type + input: + title: Input + ctx: + type: object + title: Context + additionalProperties: true + type: object + required: + - loc + - msg + - type + title: ValidationError + WandbIntegration: + properties: + project: + title: Project + description: W&B project name (groups related runs). Defaults to output.name + if not set. + type: string + name: + title: Name + description: W&B run name. Defaults to job_id if not provided. + type: string + entity: + title: Entity + description: W&B entity (team or username). + type: string + tags: + title: Tags + description: W&B tags for filtering runs. + items: + type: string + type: array + notes: + title: Notes + description: W&B notes/description for the run. + type: string + base_url: + title: Base Url + description: Base URL for self-hosted W&B server (e.g., 'https://wandb.mycompany.com'). + If not provided, uses the default W&B cloud service. + type: string + api_key_secret: + allOf: + - $ref: '#/components/schemas/SecretRef' + description: 'Reference to a secret containing the WANDB_API_KEY. Format: + ''secret_name'' (uses request workspace) or ''workspace/secret_name'' + (explicit workspace).' + additionalProperties: false + type: object + title: WandbIntegration + description: 'Weights & Biases integration configuration. + + + To enable W&B, provide a non-null ``wandb`` object on :class:`IntegrationsSpec`. + + Provide ``api_key_secret`` referencing a secret that contains ``WANDB_API_KEY``. + + Optionally set ``base_url`` for self-hosted W&B servers.' diff --git a/plugins/nemo-customizer/pyproject.toml b/plugins/nemo-customizer/pyproject.toml index 79ed1224d0..70c23e582a 100644 --- a/plugins/nemo-customizer/pyproject.toml +++ b/plugins/nemo-customizer/pyproject.toml @@ -32,6 +32,8 @@ build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/nemo_customizer"] +[tool.nemo.openapi] + [tool.uv.sources] nemo-platform-plugin = { workspace = true } diff --git a/plugins/nemo-customizer/src/nemo_customizer/router.py b/plugins/nemo-customizer/src/nemo_customizer/router.py index d804c14c4a..c2b676c81f 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/router.py +++ b/plugins/nemo-customizer/src/nemo_customizer/router.py @@ -94,7 +94,7 @@ async def healthz() -> dict[str, object]: router=router, tag="Customization", description="Customization router health.", - prefix="", + prefix="/v2", ), ] diff --git a/plugins/nemo-customizer/src/nemo_customizer/sdk/resources.py b/plugins/nemo-customizer/src/nemo_customizer/sdk/resources.py index f14c0b7f91..b4ecabe9b9 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/sdk/resources.py +++ b/plugins/nemo-customizer/src/nemo_customizer/sdk/resources.py @@ -11,9 +11,18 @@ from nemo_platform_plugin.customization_contributor import CustomizationContributor from nemo_platform_plugin.discovery import discover_customization_contributors from nemo_platform_plugin.sdk import NemoPluginSDKResources +from nmp.customization_common.sdk.client import platform_default_headers, url logger = logging.getLogger(__name__) +_HEALTHZ_PATH = "v2/healthz" + + +def _coerce_health_payload(payload: object) -> dict[str, object]: + if not isinstance(payload, dict): + raise TypeError("customization health response must be a JSON object.") + return {str(key): value for key, value in payload.items()} + def _mount_contributor_sdk_resources( target: object, @@ -43,17 +52,37 @@ class Customization: """Sync SDK namespace mounted as ``client.customization``.""" def __init__(self, platform: NeMoPlatform) -> None: + self._platform = platform contributors = discover_customization_contributors() _mount_contributor_sdk_resources(self, platform, contributors, async_=False) + def plugin_status(self) -> dict[str, object]: + """Return customization router health, including the registered contributors.""" + response = self._platform._client.get( + url(self._platform, _HEALTHZ_PATH), + headers=platform_default_headers(self._platform), + ) + response.raise_for_status() + return _coerce_health_payload(response.json()) + class AsyncCustomization: """Async SDK namespace mounted as ``client.customization``.""" def __init__(self, platform: AsyncNeMoPlatform) -> None: + self._platform = platform contributors = discover_customization_contributors() _mount_contributor_sdk_resources(self, platform, contributors, async_=True) + async def plugin_status(self) -> dict[str, object]: + """Return customization router health, including the registered contributors.""" + response = await self._platform._client.get( + url(self._platform, _HEALTHZ_PATH), + headers=platform_default_headers(self._platform), + ) + response.raise_for_status() + return _coerce_health_payload(response.json()) + customization_sdk_resources = NemoPluginSDKResources( sync_resource=Customization, diff --git a/plugins/nemo-customizer/tests/test_router.py b/plugins/nemo-customizer/tests/test_router.py index 3c14ab01a0..09565448d9 100644 --- a/plugins/nemo-customizer/tests/test_router.py +++ b/plugins/nemo-customizer/tests/test_router.py @@ -84,7 +84,7 @@ def test_router_merges_contributor_routes(monkeypatch: pytest.MonkeyPatch) -> No app.include_router(spec.router) client = TestClient(app) - assert client.get("/healthz").json()["contributors"] == ["fake"] + assert client.get("/v2/healthz").json()["contributors"] == ["fake"] assert client.get("/v2/workspaces/ws-a/fake/ping").json() == {"backend": "fake"} @@ -192,6 +192,6 @@ def get_cli(self) -> typer.Typer: assert not any(spec.deny for methods in contribution.endpoints.values() for spec in methods.values()) assert "customization.automodel.jobs.create" in contribution.permissions assert "customization.unsloth.jobs.create" in contribution.permissions - # The hub's own /healthz is authenticated-but-permissionless (ruled, not denied). - hub_healthz = contribution.endpoints["/apis/customization/healthz"]["get"] + # The hub's own /v2/healthz is authenticated-but-permissionless (ruled, not denied). + hub_healthz = contribution.endpoints["/apis/customization/v2/healthz"]["get"] assert hub_healthz.permissions == [] and not hub_healthz.deny diff --git a/plugins/nemo-customizer/tests/test_sdk.py b/plugins/nemo-customizer/tests/test_sdk.py index 2120303311..147cfb0c28 100644 --- a/plugins/nemo-customizer/tests/test_sdk.py +++ b/plugins/nemo-customizer/tests/test_sdk.py @@ -3,8 +3,9 @@ from __future__ import annotations -from unittest.mock import MagicMock, patch +from unittest.mock import AsyncMock, MagicMock, patch +import pytest from nemo_automodel_plugin.sdk.resources import AutomodelCustomization from nemo_customizer.sdk.resources import ( AsyncCustomization, @@ -62,3 +63,60 @@ def test_customization_skips_contributors_without_sdk() -> None: customization = Customization(platform) assert not hasattr(customization, "noop") + + +def _health_platform() -> MagicMock: + platform = MagicMock() + platform.workspace = "default" + platform.base_url = "http://localhost:8000" + platform.default_headers = {} + return platform + + +def test_plugin_status_hits_versioned_hub_healthz() -> None: + platform = _health_platform() + response = MagicMock() + response.json.return_value = {"plugin": "customization", "status": "ok", "contributors": ["automodel"]} + platform._client.get.return_value = response + + with patch( + "nemo_customizer.sdk.resources.discover_customization_contributors", + return_value={}, + ): + status = Customization(platform).plugin_status() + + called_url = platform._client.get.call_args.args[0] + assert called_url == "http://localhost:8000/apis/customization/v2/healthz" + assert status["contributors"] == ["automodel"] + + +def test_plugin_status_rejects_non_object_payload() -> None: + platform = _health_platform() + response = MagicMock() + response.json.return_value = ["not", "an", "object"] + platform._client.get.return_value = response + + with patch( + "nemo_customizer.sdk.resources.discover_customization_contributors", + return_value={}, + ): + resource = Customization(platform) + with pytest.raises(TypeError): + resource.plugin_status() + + +async def test_async_plugin_status_hits_versioned_hub_healthz() -> None: + platform = _health_platform() + response = MagicMock() + response.json.return_value = {"plugin": "customization", "status": "ok", "contributors": []} + platform._client.get = AsyncMock(return_value=response) + + with patch( + "nemo_customizer.sdk.resources.discover_customization_contributors", + return_value={}, + ): + status = await AsyncCustomization(platform).plugin_status() + + called_url = platform._client.get.call_args.args[0] + assert called_url == "http://localhost:8000/apis/customization/v2/healthz" + assert status["status"] == "ok" diff --git a/plugins/nemo-unsloth/tests/test_contributor.py b/plugins/nemo-unsloth/tests/test_contributor.py index e7916f4f82..f930c513cb 100644 --- a/plugins/nemo-unsloth/tests/test_contributor.py +++ b/plugins/nemo-unsloth/tests/test_contributor.py @@ -6,8 +6,8 @@ Pin the contract the customization-router hub depends on: - ``name`` and ``dependencies`` (used by the hub's dep merger). -- ``get_routers`` returns the healthz + jobs routers under the right prefix, - with ``@path_rule`` authz stamped on the generated job routes (the platform +- ``get_routers`` returns the jobs router under the right prefix, with + ``@path_rule`` authz stamped on the generated job routes (the platform derives the policy from those rules — there is no ``get_authz_contribution``). - ``get_cli`` exposes ``run`` / ``submit`` / ``explain`` and the submit group accepts the ``JOB_JSON`` positional. ``run`` hard-fails. @@ -74,15 +74,16 @@ def test_job_routes_carry_unsloth_path_rules(self, contributor: object) -> None: class TestRouters: - def test_returns_two_router_specs(self, contributor: object) -> None: + def test_returns_jobs_router_spec(self, contributor: object) -> None: specs = () try: specs = contributor.get_routers() except ImportError as exc: pytest.skip(f"router deps unavailable in this env: {exc}") - assert len(specs) == 2 + # Only the jobs router — health lives on the customization router hub, + # not per contributor. + assert len(specs) == 1 prefixes = {s.prefix for s in specs} - assert "/v2/workspaces/{workspace}/unsloth" in prefixes # The jobs router is mounted at the workspace prefix; add_job_routes # adds the /unsloth/jobs suffix internally based on # UnslothJob.job_collection_path.