fix(api): honor async submit tool and agent availability

docs/source/integration/rest-api.md

@@ -26,7 +26,7 @@ Base path: `/v1/jobs/async`
 
 | Method | Path | Description |
 |--------|------|-------------|
-| `GET` | `/v1/jobs/async/agents` | List registered agent types |
+| `GET` | `/v1/jobs/async/agents` | List public agents configured in the active workflow |
 | `POST` | `/v1/jobs/async/submit` | Submit a new research job |
 | `GET` | `/v1/jobs/async/job/{job_id}` | Get job status |
 | `GET` | `/v1/jobs/async/job/{job_id}/stream` | SSE event stream from beginning |
@@ -43,11 +43,15 @@ Base path: `/v1/jobs/async`
 
 ### List Available Agents
 
-Returns all **public** registered agent types that can be used with the submit endpoint.
+Returns all **public** registered agent types configured in the active workflow. The
+`deep_researcher` and `shallow_researcher` entries are part of the default catalog, but
+each is returned only when its corresponding function config is present in the loaded
+workflow. Discovery reflects configuration availability, not credentials, tool health,
+source connectivity, Dask readiness, or other runtime health checks.
+
 Internal-only agents (registered with `public=False`, for example the `report_rewriter`
 used by report follow-up) are intentionally omitted from this list and are rejected by
-`POST /v1/jobs/async/submit` with `400` and a `detail` of `Agent type is internal-only: <agent_type>`
-(the requested agent type is interpolated into the message).
+`POST /v1/jobs/async/submit` with `400`.
 
 ```bash
 curl http://localhost:8000/v1/jobs/async/agents
@@ -85,7 +89,7 @@ curl -X POST http://localhost:8000/v1/jobs/async/submit \
 | `input` | `string` | Yes | Research query. Must be non-blank after trimming (whitespace-only is rejected with 422) |
 | `job_id` | `string` | No | Custom job ID. Auto-generated UUID if omitted. Pattern: `[a-zA-Z0-9_-]`, max 64 chars |
 | `expiry_seconds` | `integer` | No | Job expiry in seconds. Range: 600--604800 (10 min to 7 days). Default from config |
-| `data_sources` | `list[string]` | No | Optional data source IDs (from `/v1/data_sources`) to scope the job. Omit or `null` for all data-source tools; `[]` for no data-source tools. Unmapped utility tools remain available. Unknown IDs return 422 |
+| `data_sources` | `list[string]` | No | Optional data source IDs (from `/v1/data_sources`) to scope the job. Omit or `null` for all data-source tools; `[]` for no data-source tools. Unmapped utility tools remain available. Unknown IDs or sources unavailable to the selected agent return 422 |
 
 **Response (`JobStatusResponse`):**
 
@@ -101,10 +105,25 @@ curl -X POST http://localhost:8000/v1/jobs/async/submit \
 
 | Status | Reason |
 |--------|--------|
-| `400` | Unknown agent type, **internal-only agent type**, or invalid request |
-| `409` | A custom `job_id` was supplied that collides with an existing job |
-| `422` | Validation error: blank/whitespace-only `input`, invalid request fields, or one or more unknown data source IDs. Data source errors include `message`, `invalid_ids`, and `known_ids` for client-side recovery UX |
-| `503` | Dask scheduler not available |
+| `400` | Unknown, **internal-only**, or registered-but-unconfigured agent type, or invalid request |
+| `409` | A custom `job_id` collides with an existing job, or a selected protected source requires per-user authentication |
+| `422` | Validation error: blank/whitespace-only `input`, invalid request fields, unknown data source IDs, or sources unavailable to the selected agent. Data source errors include `message`, `invalid_ids`, `unavailable_for_agent`, and `known_ids` |
+| `500` | Content encryption configuration is invalid, authorization persistence fails, or agent/tool configuration fails unexpectedly |
+| `503` | Content encryption, Dask, or configured sandbox capacity is unavailable |
+
+A public catalog entry that is not configured in the active workflow is rejected before
+source validation or job creation:
+
+```json
+{
+  "detail": {
+    "code": "agent_not_configured",
+    "message": "Agent 'shallow_researcher' is not configured in the active workflow",
+    "agent_type": "shallow_researcher",
+    "config_name": "shallow_research_agent"
+  }
+}
+```
 
 ### Edit a Report (Report Follow-up)
 
@@ -454,7 +473,7 @@ register_agent(
 | `config_name` | Must match a function name in the NeMo Agent Toolkit YAML config (for example, `deep_research_agent`) |
 | `description` | Human-readable description shown in the agent list |
 
-The default agents (`deep_researcher` and `shallow_researcher`) are registered automatically when the `aiq_api` plugin loads.
+The default agents (`deep_researcher` and `shallow_researcher`) are registered automatically when the `aiq_api` plugin loads. Registration adds them to the catalog; each agent must also have its `config_name` defined in the active workflow to appear in `GET /v1/jobs/async/agents` or be accepted by `POST /v1/jobs/async/submit`.
 
 ## Knowledge API
 

frontends/aiq_api/README.md

@@ -67,7 +67,7 @@ Base path: `/v1/jobs/async`
 
 | Endpoint | Method | Description |
 |----------|--------|-------------|
-| `/v1/jobs/async/agents` | GET | List available agent types |
+| `/v1/jobs/async/agents` | GET | List public agents configured in the active workflow |
 | `/v1/jobs/async/submit` | POST | Submit a new job |
 | `/v1/jobs/async/job/{id}` | GET | Get job status |
 | `/v1/jobs/async/job/{id}/stream` | GET | SSE stream from beginning |
@@ -78,6 +78,11 @@ Base path: `/v1/jobs/async`
 
 ### List Available Agents
 
+The default catalog registers both `deep_researcher` and `shallow_researcher`, but this
+endpoint returns an agent only when its function config is present in the active
+workflow. It does not check credentials, source connectivity, tool health, or Dask
+readiness.
+
 ```bash
 curl http://localhost:8000/v1/jobs/async/agents
 ```
@@ -101,7 +106,21 @@ curl -X POST http://localhost:8000/v1/jobs/async/submit \
   -d '{"agent_type": "deep_researcher", "input": "Research quantum computing trends in 2026"}'
 ```
 
-Optional body fields: `job_id` (custom ID), `expiry_seconds` (600–604800), `data_sources` (subset of IDs from `/v1/data_sources`; `[]` disables data-source tools while leaving unmapped utility tools available; unknown IDs return 422).
+Optional body fields: `job_id` (custom ID), `expiry_seconds` (600–604800), `data_sources` (subset of IDs from `/v1/data_sources`; `[]` disables data-source tools while leaving unmapped utility tools available; unknown IDs and sources unavailable to the selected agent return 422).
+
+A public catalog agent that is absent from the active workflow returns `400` before a
+job is created:
+
+```json
+{
+  "detail": {
+    "code": "agent_not_configured",
+    "message": "Agent 'shallow_researcher' is not configured in the active workflow",
+    "agent_type": "shallow_researcher",
+    "config_name": "shallow_research_agent"
+  }
+}
+```
 
 Response:
 

frontends/aiq_api/src/aiq_api/routes/jobs.py

@@ -218,7 +218,17 @@ def _source_ids_by_lowercase() -> tuple[list[str], dict[str, str]]:
     return known_ids, {source_id.lower(): source_id for source_id in known_ids}
 
 
-async def _get_agent_available_source_ids(builder: WorkflowBuilder, agent_config_name: str) -> list[str]:
+def _get_configured_agent_function_config(builder: WorkflowBuilder, config_name: str) -> Any | None:
+    """Return an active function config, or None for NAT's missing-function error."""
+    try:
+        return builder.get_function_config(config_name)
+    except ValueError as exc:
+        if str(exc) == f"Function `{config_name}` not found":
+            return None
+        raise
+
+
+async def _get_agent_available_source_ids(builder: WorkflowBuilder, fn_config: Any) -> list[str]:
     """Return mapped source IDs with at least one effective tool for an agent config.
 
     This mirrors the async job runner's effective tool resolution: explicit
@@ -234,8 +244,7 @@ async def _get_agent_available_source_ids(builder: WorkflowBuilder, agent_config
     A registered agent without these fields is a registration-time bug, not a
     runtime concern.
     """
-    fn_config = builder.get_function_config(agent_config_name)
-    tool_refs = fn_config.tools if fn_config.tools is not None else get_all_tool_refs()
+    tool_refs = fn_config.tools or get_all_tool_refs()
     tools = await builder.get_tools(tool_names=tool_refs, wrapper_type=LLMFrameworkEnum.LANGCHAIN)
 
     excluded = set(fn_config.exclude_tools or [])
@@ -270,6 +279,7 @@ async def _validate_data_sources_for_agent(
     builder: WorkflowBuilder,
     agent_type: str,
     agent_config_name: str,
+    fn_config: Any,
     data_sources: list[str] | None,
 ) -> None:
     """Raise HTTP 422 if requested sources are unknown or unavailable to the selected agent."""
@@ -285,7 +295,7 @@ async def _validate_data_sources_for_agent(
     known_ids, known_by_lower = _source_ids_by_lowercase()
 
     try:
-        available_ids = await _get_agent_available_source_ids(builder, agent_config_name)
+        available_ids = await _get_agent_available_source_ids(builder, fn_config)
     except asyncio.CancelledError:
         raise
     except Exception as exc:
@@ -456,7 +466,7 @@ class AgentInfo(BaseModel):
 class AgentListResponse(BaseModel):
     """List of available agents."""
 
-    agents: list[AgentInfo] = Field(..., description="Registered agent types")
+    agents: list[AgentInfo] = Field(..., description="Public agent types configured in the active workflow")
 
 
 class DataSource(BaseModel):
@@ -617,15 +627,17 @@ async def health_check():
         response_model=AgentListResponse,
         tags=["async jobs"],
         summary="List available agents",
-        description="Returns all registered agent types that can be used with the submit endpoint.",
+        description="Returns public agent types configured in the active workflow.",
     )
     async def list_agents() -> AgentListResponse:
         """List available agent types for async job submission."""
-        agents = [
-            AgentInfo(agent_type=agent_type, description=config.description)
-            for agent_type, config in AGENT_REGISTRY.items()
-            if config.public
-        ]
+        agents = []
+        for agent_type, config in AGENT_REGISTRY.items():
+            if not config.public:
+                continue
+            if _get_configured_agent_function_config(builder, config.config_name) is None:
+                continue
+            agents.append(AgentInfo(agent_type=agent_type, description=config.description))
         return AgentListResponse(agents=agents)
 
     @app.get(
@@ -684,7 +696,7 @@ async def list_data_sources() -> list[DataSource]:
             "Submit a research query to a registered agent. Returns a job ID for tracking progress via SSE stream."
         ),
         responses={
-            400: {"description": "Unknown agent type or invalid request"},
+            400: {"description": "Unknown, internal-only, or unconfigured agent type, or invalid request"},
             409: {
                 "description": (
                     "A custom job_id was supplied that collides with an existing job, or a selected "
@@ -694,7 +706,8 @@ async def list_data_sources() -> list[DataSource]:
             422: {"description": "One or more unknown or agent-unavailable data source IDs"},
             500: {
                 "description": (
-                    "Content encryption configuration is invalid or async job authorization persistence failed"
+                    "Content encryption configuration is invalid, async job authorization persistence failed, "
+                    "or agent/tool configuration lookup failed unexpectedly"
                 )
             },
             503: {"description": "Content encryption, Dask scheduler, or sandbox capacity is unavailable"},
@@ -711,6 +724,18 @@ async def submit_job(
         if not agent_config.public:
             raise HTTPException(400, f"Agent type is internal-only and cannot be submitted directly: {req.agent_type}")
 
+        fn_config = _get_configured_agent_function_config(builder, agent_config.config_name)
+        if fn_config is None:
+            raise HTTPException(
+                status_code=400,
+                detail={
+                    "code": "agent_not_configured",
+                    "message": f"Agent '{req.agent_type}' is not configured in the active workflow",
+                    "agent_type": req.agent_type,
+                    "config_name": agent_config.config_name,
+                },
+            )
+
         expiry = req.expiry_seconds if req.expiry_seconds is not None else default_expiry_seconds
         # Authenticate the caller (raises 401/403 if unverified). The returned principal
         # is also forwarded to submit_authorized_job(...) below for ownership recording.
@@ -735,6 +760,7 @@ async def submit_job(
             builder=builder,
             agent_type=req.agent_type,
             agent_config_name=agent_config.config_name,
+            fn_config=fn_config,
             data_sources=req.data_sources,
         )
         logger.info(

frontends/aiq_api/tests/test_agent_registry_visibility.py

@@ -10,6 +10,17 @@
 from fastapi.testclient import TestClient
 
 
+def _builder_with_configs(*config_names: str) -> SimpleNamespace:
+    configured = set(config_names)
+
+    def _get_function_config(config_name: str):
+        if config_name not in configured:
+            raise ValueError(f"Function `{config_name}` not found")
+        return SimpleNamespace()
+
+    return SimpleNamespace(get_function_config=_get_function_config)
+
+
 def test_agent_config_defaults_to_public():
     from aiq_api.registry import AgentConfig
 
@@ -47,7 +58,7 @@ def test_default_registry_contains_internal_report_rewriter():
 
 
 @pytest.mark.asyncio
-async def test_list_agents_excludes_internal_agents(monkeypatch):
+async def test_list_agents_includes_only_configured_public_agents(monkeypatch):
     import aiq_api.routes.jobs as jobs_routes
     from aiq_api.registry import AgentConfig
 
@@ -60,6 +71,11 @@ async def test_list_agents_excludes_internal_agents(monkeypatch):
                 config_name="deep_research_agent",
                 description="Deep",
             ),
+            "shallow_researcher": AgentConfig(
+                class_path="aiq_agent.agents.shallow_researcher.agent.ShallowResearcherAgent",
+                config_name="shallow_research_agent",
+                description="Shallow",
+            ),
             "report_rewriter": AgentConfig(
                 class_path="aiq_agent.agents.report_rewriter.agent.ReportRewriterAgent",
                 config_name="deep_research_agent",
@@ -72,7 +88,7 @@ async def test_list_agents_excludes_internal_agents(monkeypatch):
     app = FastAPI()
     await jobs_routes.register_job_routes(
         app,
-        builder=SimpleNamespace(),
+        builder=_builder_with_configs("deep_research_agent"),
         worker=SimpleNamespace(_dask_available=False, _job_store=None),
     )
 
@@ -88,3 +104,56 @@ async def test_list_agents_excludes_internal_agents(monkeypatch):
             }
         ]
     }
+
+
+@pytest.mark.asyncio
+async def test_list_agents_includes_configured_shallow_researcher(monkeypatch):
+    import aiq_api.routes.jobs as jobs_routes
+    from aiq_api.registry import AgentConfig
+
+    monkeypatch.setattr(
+        jobs_routes,
+        "AGENT_REGISTRY",
+        {
+            "deep_researcher": AgentConfig(
+                class_path="aiq_agent.agents.deep_researcher.agent.DeepResearcherAgent",
+                config_name="deep_research_agent",
+                description="Deep",
+            ),
+            "shallow_researcher": AgentConfig(
+                class_path="aiq_agent.agents.shallow_researcher.agent.ShallowResearcherAgent",
+                config_name="shallow_research_agent",
+                description="Shallow",
+            ),
+            "report_rewriter": AgentConfig(
+                class_path="aiq_agent.agents.report_rewriter.agent.ReportRewriterAgent",
+                config_name="deep_research_agent",
+                description="Internal",
+                public=False,
+            ),
+        },
+    )
+
+    app = FastAPI()
+    await jobs_routes.register_job_routes(
+        app,
+        builder=_builder_with_configs("deep_research_agent", "shallow_research_agent"),
+        worker=SimpleNamespace(_dask_available=False, _job_store=None),
+    )
+
+    with TestClient(app) as client:
+        response = client.get("/v1/jobs/async/agents")
+
+    assert response.status_code == 200
+    assert response.json() == {
+        "agents": [
+            {
+                "agent_type": "deep_researcher",
+                "description": "Deep",
+            },
+            {
+                "agent_type": "shallow_researcher",
+                "description": "Shallow",
+            },
+        ]
+    }

frontends/aiq_api/tests/test_content_encryption_routes.py

@@ -459,10 +459,11 @@ async def test_submit_openapi_documents_encryption_failures(monkeypatch, tmp_pat
 
     responses = app.openapi()["paths"]["/v1/jobs/async/submit"]["post"]["responses"]
 
-    assert responses["400"]["description"] == "Unknown agent type or invalid request"
+    assert responses["400"]["description"] == ("Unknown, internal-only, or unconfigured agent type, or invalid request")
     assert responses["422"]["description"] == "One or more unknown or agent-unavailable data source IDs"
     assert responses["500"]["description"] == (
-        "Content encryption configuration is invalid or async job authorization persistence failed"
+        "Content encryption configuration is invalid, async job authorization persistence failed, "
+        "or agent/tool configuration lookup failed unexpectedly"
     )
     assert responses["503"]["description"] == ("Content encryption, Dask scheduler, or sandbox capacity is unavailable")