docs: add Python async gRPC client guide under gRPC remote execution

docs/cuopt/source/_static/large-rubric.css

@@ -0,0 +1,7 @@
+/* Rubric styled closer to an H2/H3 for in-page subheads that must stay out of the TOC. */
+p.rubric.large-rubric {
+  font-size: 1.4em;
+  font-weight: 600;
+  margin-top: 1.75rem;
+  margin-bottom: 0.75rem;
+}

docs/cuopt/source/conf.py

@@ -172,7 +172,11 @@
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
-html_css_files = ["swagger-nvidia.css", "install-selector.css"]
+html_css_files = [
+    "swagger-nvidia.css",
+    "install-selector.css",
+    "large-rubric.css",
+]
 html_js_files = ["cuopt-install-version.js", "install-selector.js"]
 html_extra_path = ["versions1.json"]
 

docs/cuopt/source/cuopt-grpc/advanced.rst

@@ -66,10 +66,18 @@ These variables apply when the container **entrypoint** builds a ``cuopt_grpc_se
 
 The REST server path in the same image still uses ``CUOPT_SERVER_PORT`` for HTTP in other docs; that is separate from the gRPC defaults above.
 
-Bundled Remote Client (Python, C API, ``cuopt_cli``)
-----------------------------------------------------
+Integrated Remote Client (Python, C API, ``cuopt_cli``)
+-------------------------------------------------------
 
-Remote mode is active when **both** ``CUOPT_REMOTE_HOST`` and ``CUOPT_REMOTE_PORT`` are set. A **custom** gRPC client does not read these automatically; it must configure the channel and protos itself (see :doc:`api`).
+These variables apply to **remote execution**: the client integrated into the
+Python solver APIs, the C API (``cuOptSolve``), and ``cuopt_cli``. Remote mode
+is active when **both** ``CUOPT_REMOTE_HOST`` and ``CUOPT_REMOTE_PORT`` are set.
+
+The :doc:`Python async gRPC client <python-async-client>` does **not** use
+``CUOPT_REMOTE_HOST`` / ``CUOPT_REMOTE_PORT``; you pass host and port to
+``Client(...)``. See *Python async gRPC client* below for variables that apply
+to that client. A **custom** gRPC client must configure the channel itself
+(see :doc:`api`).
 
 .. list-table::
    :header-rows: 1
@@ -82,7 +90,7 @@ Remote mode is active when **both** ``CUOPT_REMOTE_HOST`` and ``CUOPT_REMOTE_POR
    * - ``CUOPT_REMOTE_HOST``
      - For remote
      - —
-     - Server hostname or IP
+     - GPU server hostname or IP
    * - ``CUOPT_REMOTE_PORT``
      - For remote
      - —
@@ -116,6 +124,48 @@ Remote mode is active when **both** ``CUOPT_REMOTE_HOST`` and ``CUOPT_REMOTE_POR
      - ``0``
      - Non-zero: extra gRPC client logging
 
+Python Async gRPC Client (``cuopt.grpc``)
+-----------------------------------------
+
+``Client(host, port, tls=...)`` takes the server address in code. It does
+**not** read ``CUOPT_REMOTE_HOST`` or ``CUOPT_REMOTE_PORT``.
+
+When ``tls`` is omitted (``None``), the client honors the same ``CUOPT_TLS_*``
+variables as remote execution. Pass ``tls=False`` for plain TCP, or
+``tls=TlsConfig(...)`` for explicit PEM paths (see :doc:`python-async-client`).
+
+.. list-table::
+   :header-rows: 1
+   :widths: 26 14 18 42
+
+   * - Variable
+     - Required
+     - Default
+     - Description
+   * - ``CUOPT_TLS_ENABLED``
+     - No
+     - ``0``
+     - Used when ``tls=None``; non-zero enables TLS
+   * - ``CUOPT_TLS_ROOT_CERT``
+     - If TLS
+     - —
+     - PEM path to verify the **server** certificate
+   * - ``CUOPT_TLS_CLIENT_CERT``
+     - mTLS
+     - —
+     - Client certificate PEM
+   * - ``CUOPT_TLS_CLIENT_KEY``
+     - mTLS
+     - —
+     - Client private key PEM
+   * - ``CUOPT_GRPC_DEBUG``
+     - No
+     - ``0``
+     - Non-zero: extra gRPC client logging
+
+``CUOPT_CHUNK_SIZE`` and ``CUOPT_MAX_MESSAGE_BYTES`` also apply to this client
+when set (same defaults as the integrated remote client).
+
 Usage
 =====
 
@@ -259,14 +309,14 @@ Bypass the entrypoint:
 Client Environment (Examples)
 ------------------------------
 
-**Required** for remote (see *Bundled remote client* table for all variables):
+**Remote execution** — required host/port (see *Integrated remote client* table):
 
 .. code-block:: bash
 
    export CUOPT_REMOTE_HOST=<server-hostname>
    export CUOPT_REMOTE_PORT=5001
 
-**TLS** (optional):
+**TLS** (optional; also used by the Python async gRPC client when ``tls=None``):
 
 .. code-block:: bash
 
@@ -280,13 +330,43 @@ For mTLS, also:
    export CUOPT_TLS_CLIENT_CERT=client.crt
    export CUOPT_TLS_CLIENT_KEY=client.key
 
+**Python async gRPC client** — pass host and port to ``Client(...)`` (not
+``CUOPT_REMOTE_*``). With ``tls=None`` (default), the same ``CUOPT_TLS_*``
+variables above apply. For explicit PEM paths, use ``TlsConfig``:
+
+.. code-block:: python
+
+   from cuopt.grpc.linear_programming import Client, TlsConfig
+
+   # TLS: verify the server with a CA (or omit root_certs for the system trust store)
+   client = Client(
+       "server.example.com",
+       5001,
+       tls=TlsConfig(root_certs="ca.crt"),
+   )
+
+   # mTLS: also present a client certificate
+   client = Client(
+       "server.example.com",
+       5001,
+       tls=TlsConfig(
+           root_certs="ca.crt",
+           client_cert="client.crt",
+           client_key="client.key",
+       ),
+   )
+
+   # Plain TCP (ignore CUOPT_TLS_* even if set)
+   client = Client("localhost", 5001, tls=False)
+
+See :doc:`python-async-client` for the full job API.
 Limitations and Scope
 =====================
 
 * **Problem types** — **LP**, **MILP**, and **QP** are supported on the gRPC remote path. **Routing** (VRP, TSP, PDP) is **not** supported yet; use the :doc:`REST self-hosted server <../cuopt-server/index>` for remote routing until a future release adds routing over ``CuOptRemoteService``.
 * **Message size** — Large problems use chunking; very large models can still hit gRPC max message / timeout limits. Tune ``CUOPT_CHUNK_SIZE``, ``CUOPT_MAX_MESSAGE_BYTES``, server ``--max-message-mb``, and solver ``time_limit`` as needed.
 * **``CUOPT_GRPC_ARGS``** — Parsed on whitespace only; arguments containing spaces are awkward unless you invoke ``cuopt_grpc_server`` directly.
-* **CRL / OCSP** — Not handled by the bundled gRPC TLS stack; use a private CA rotation strategy or a TLS-terminating proxy if you need revocation workflows.
+* **CRL / OCSP** — Not handled by the integrated gRPC TLS stack; use a private CA rotation strategy or a TLS-terminating proxy if you need revocation workflows.
 
 Troubleshooting
 ===============
@@ -310,5 +390,6 @@ Further Reading
 ===============
 
 * :doc:`quick-start` — Plain TCP quick path.
-* :doc:`examples` — Links to Python, C, and CLI example sections (use with ``CUOPT_REMOTE_*`` on the client).
+* :doc:`examples` — Links to Python, C, and CLI example sections (use with ``CUOPT_REMOTE_*`` on the client for remote execution).
+* :doc:`python-async-client` — Explicit Python gRPC client.
 * :doc:`grpc-server-architecture` — Process model and job behavior (operator overview).

docs/cuopt/source/cuopt-grpc/api.rst

@@ -11,7 +11,16 @@ The **CuOptRemoteService** gRPC API is defined in Protocol Buffers under the ``c
 * ``cpp/src/grpc/cuopt_remote_service.proto`` — service and job/chunk/log RPCs
 * ``cpp/src/grpc/cuopt_remote.proto`` — LP/MIP problem, settings, and result messages
 
-Most users do **not** call these RPCs directly: the NVIDIA cuOpt **Python** API, **C API**, and **cuopt_cli** submit jobs using solver APIs plus :doc:`environment variables <advanced>`. **Custom** clients call ``CuOptRemoteService`` over gRPC using these definitions. This page summarizes the service for custom integrators and debugging.
+Most users do **not** call these RPCs directly:
+
+* **Remote execution** — Python, C (``cuOptSolve``), and ``cuopt_cli`` forward
+  solves when ``CUOPT_REMOTE_HOST`` and ``CUOPT_REMOTE_PORT`` are set
+  (:doc:`quick-start`, :doc:`advanced`).
+* **Python async gRPC client** — ``cuopt.grpc.linear_programming.Client``
+  (:doc:`python-async-client`).
+
+**Custom** clients call ``CuOptRemoteService`` over gRPC using these definitions.
+This page summarizes the service for custom integrators and debugging.
 
 Service: ``CuOptRemoteService``
 ================================
@@ -82,17 +91,24 @@ Streaming and Callbacks
    * - ``StreamLogs``
      - Server-streaming solver log lines for a job.
    * - ``GetIncumbents``
-     - MILP incumbent solutions since a given index.
+     - MILP incumbent solutions since a given index (only if the job was
+       submitted with ``enable_incumbents``; otherwise the list is empty).
 
 Messages and Constraints
 ========================
 
-* **Problem types** — LP and MILP in the enum; the problem payload can include quadratic objective data for **QP**-style solves where the client API supports it. **Routing** over this gRPC service is **not** available yet; it is planned for an **upcoming** release (use REST for remote routing today).
+* **Problem types** — Wire category is LP or MILP; QP uses the LP path with
+  quadratic objective data in the problem payload. **Routing** over this
+  gRPC service is **not** available yet (planned; use REST for remote routing today).
 * **Solver settings** — Carried as ``PDLPSolverSettings`` or ``MIPSolverSettings`` inside the request or chunked header, aligned with the NVIDIA cuOpt solver options documentation.
-* **Errors** — gRPC status codes carry failures (see comments at the end of ``cuopt_remote_service.proto``).
+* **Errors** — Transport and many hard failures use gRPC status codes. Some
+  outcomes (unknown job, solve failed, cancel/delete results) arrive as
+  ``Status::OK`` with fields in the response message (``job_status``,
+  ``status``, ``error_message``). See comments in ``cuopt_remote_service.proto``.
 
 Further Reading
 ===============
 
+* :doc:`python-async-client` / :doc:`python-async-client-api` — Python job client (``cuopt.grpc``) built on these RPCs.
 * :doc:`grpc-server-architecture` — Server process model and job lifecycle (overview); :doc:`advanced` for ``cuopt_grpc_server`` flags. Contributor details: ``cpp/docs/grpc-server-architecture.md``.
 * :doc:`advanced` — TLS, Docker, client environment variables, and limitations.

docs/cuopt/source/cuopt-grpc/examples.rst

@@ -6,13 +6,16 @@
 Examples
 ========
 
-gRPC remote execution uses the same **Python**, **C API**, and **cuopt_cli** entry points as a local solve. After you start ``cuopt_grpc_server`` on the GPU host (:doc:`quick-start`), set the client environment and run **any** of the examples below **unchanged** — no code edits are required.
+**Remote execution** uses the same **Python**, **C API**, and **cuopt_cli**
+entry points as a local solve. After you start ``cuopt_grpc_server`` on the
+GPU server (:doc:`quick-start`), set the client environment and run **any** of
+the examples below **unchanged** — no code edits are required.
 
-On the **client** host, before running the example commands or scripts:
+On the **client** machine, before running the example commands or scripts:
 
 .. code-block:: bash
 
-   export CUOPT_REMOTE_HOST=<gpu-hostname-or-ip>
+   export CUOPT_REMOTE_HOST=<gpu-server-hostname-or-ip>
    export CUOPT_REMOTE_PORT=5001
 
 Add TLS or tuning variables from :doc:`advanced` if your deployment uses them.
@@ -36,7 +39,9 @@ C API (LP / QP / MILP)
 * :doc:`../cuopt-c/convex/convex-examples` — LP and QP C examples.
 * :doc:`../cuopt-c/mip/mip-examples` — MILP C examples.
 
-  Compile and run these programs with the same exports in the shell; ``solve_lp`` / ``solve_mip`` use gRPC when both remote variables are set (see :doc:`../cuopt-c/convex/convex-c-api` for API reference).
+  Compile and run these programs with the same exports in the shell;
+  ``cuOptSolve`` uses gRPC when both remote variables are set (see
+  :doc:`../cuopt-c/convex/convex-c-api` for API reference).
 
 ``cuopt_cli``
 -------------
@@ -46,15 +51,28 @@ C API (LP / QP / MILP)
 Minimal Demos (This Section)
 ----------------------------
 
-Bundled with the gRPC docs source for a quick copy-paste path (also walked through in :doc:`quick-start`):
+Included with the gRPC docs source for a quick copy-paste path (also walked through in :doc:`quick-start`):
 
 * :download:`remote_lp_demo.py <examples/remote_lp_demo.py>`
 * :download:`remote_lp_demo.mps <examples/remote_lp_demo.mps>`
 
+Python Async gRPC Client
+------------------------
+
+For explicit job control (submit / wait / cancel / stream logs or incumbents)
+without ``CUOPT_REMOTE_*``, use ``cuopt.grpc.linear_programming.Client``:
+
+* :doc:`python-async-client` — overview
+* :doc:`python-async-client-examples` — log streaming and incumbent streaming
+* :doc:`python-async-client-api` — API reference
+
 Custom gRPC Client
 ------------------
 
-Integrations that do **not** use the bundled Python / C / CLI stack should speak ``CuOptRemoteService`` directly. See :doc:`api`, :doc:`grpc-server-architecture`, and ``cpp/docs/grpc-server-architecture.md`` in the repository for protos and server behavior.
+Integrations that do **not** use remote execution or the Python async gRPC
+client should speak ``CuOptRemoteService`` directly. See :doc:`api`,
+:doc:`grpc-server-architecture`, and ``cpp/docs/grpc-server-architecture.md``
+in the repository for protos and server behavior.
 
 More Samples
 ============

docs/cuopt/source/cuopt-grpc/examples/incumbent_stream_demo.py

@@ -0,0 +1,98 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""MIP incumbent streaming via the Python async gRPC client.
+
+Same callback registration as a local solve
+(:meth:`SolverSettings.set_mip_callback`), plus
+:meth:`Client.start_incumbent_stream` so those callbacks fire while the
+remote job runs.
+
+Start the server first::
+
+    cuopt_grpc_server --port 50051 --workers 1
+
+Then::
+
+    python incumbent_stream_demo.py --host localhost --port 50051
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from cuopt.grpc.linear_programming import Client, JobStatus
+from cuopt.linear_programming.internals import GetSolutionCallback
+from cuopt.linear_programming.problem import INTEGER, MAXIMIZE, Problem
+from cuopt.linear_programming.solver_settings import SolverSettings
+
+
+class IncumbentPrinter(GetSolutionCallback):
+    """Same callback type used for local ``problem.solve(settings)``."""
+
+    def __init__(self):
+        super().__init__()
+        self.entries = []
+
+    def get_solution(self, solution, solution_cost, solution_bound, user_data):
+        cost = float(solution_cost[0])
+        values = solution.tolist()
+        self.entries.append({"cost": cost, "solution": values})
+        print(f"incumbent cost={cost:.4f} values={values}", flush=True)
+
+
+def build_problem():
+    problem = Problem("incumbent_stream_demo")
+    x = problem.addVariable(lb=0, ub=10, vtype=INTEGER, name="x")
+    y = problem.addVariable(lb=0, ub=10, vtype=INTEGER, name="y")
+    problem.addConstraint(x + y <= 10, name="c1")
+    problem.addConstraint(x - y >= 0, name="c2")
+    problem.setObjective(x + 2 * y, sense=MAXIMIZE)
+    return problem
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--host", default="localhost")
+    parser.add_argument("--port", type=int, default=50051)
+    args = parser.parse_args(argv)
+
+    printer = IncumbentPrinter()
+    settings = SolverSettings()
+    # Local solves: register callback, then problem.solve(settings).
+    # Remote: same registration — submit enables server-side tracking, and
+    # start_incumbent_stream(..., settings=settings) delivers into printer.
+    settings.set_mip_callback(printer, None)
+    settings.set_parameter("time_limit", 30)
+
+    problem = build_problem()
+    client = Client(args.host, args.port)
+    job_id = client.submit(problem, settings)
+    client.start_incumbent_stream(job_id, settings=settings)
+
+    status = client.wait(job_id, timeout=120)
+    if status != JobStatus.COMPLETED:
+        print(f"unexpected status: {status}", file=sys.stderr)
+        return 1
+    client.join_incumbent_stream(job_id)
+
+    names = [v.getVariableName() for v in problem.getVariables()]
+    solution = client.result(job_id, variable_names=names)
+    print(
+        f"final reason={solution.get_termination_reason()} "
+        f"obj={solution.get_primal_objective()}"
+    )
+    client.delete(job_id)
+
+    if not printer.entries:
+        print(
+            "no incumbents delivered via settings callbacks", file=sys.stderr
+        )
+        return 1
+    print(f"OK ({len(printer.entries)} incumbent(s))")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docs/cuopt/source/cuopt-grpc/grpc-server-architecture.md

@@ -21,7 +21,7 @@ Implementation details (IPC layout, C++ source map, chunked transfer internals)
 | Topic | Detail |
 |-------|--------|
 | Log files | Per-job solver logs under `/tmp/cuopt_logs/job_<job_id>.log` (used by log streaming). |
-| Default caps | Up to **100** queued jobs and **100** stored results (server compile-time limits). |
+| Capacity | Up to **100** in-flight jobs (queued + processing). Completed results stay until `DeleteResult` (no fixed result cap). |
 | Workers | Recommended: **1 worker process per GPU**. Higher values are possible depending on the problems being solved but there is no specific guidance at this time. |
 
 ## Fault tolerance and cancellation