Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions projects/rccl/docs/api-reference/env-variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -783,6 +783,81 @@ in the following table.
``example`` loads ``librccl-rma-example.so``
| See :ref:`using-rccl-rma-plugin`

QP scheduling (CAST)
====================

CAST (Congestion Aware Sprayed Traffic) adds a dynamic QP (Queue Pair) scheduler
that balances RDMA traffic across multiple QPs per connection based on measured
round-trip time (RTT). The following variables tune the scheduler and are
accessible with either the ``RCCL_`` or ``NCCL_`` prefix; the ``RCCL_`` form is
shown below.

These variables only take effect when the IB-CAST net plugin is loaded
(``NCCL_NET=IB-CAST`` or ``ROCM-IB``, or an AINIC NIC with ``NCCL_NET`` unset)
*and* the CAST QP scheduler is enabled (see ``RCCL_IB_QP_SCHED_ENABLE`` below).

.. list-table::
:header-rows: 1
:widths: 40,60

* - **Environment variable**
- **Values**

* - | ``RCCL_IB_QP_SCHED_ENABLE``
| Enables the CAST QP scheduler.
- | ``-1``: Auto (default). Disabled unless ``NCCL_NET=IB-CAST`` is set,
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved
which forces it on. Can be explicitly overridden regardless of
``NCCL_NET``. Stays disabled, even with ``NCCL_NET=IB-CAST``, when
the connection's receiver matching scheme resolves to ``BY_ORDER``
— the default on AINIC, since CTS offload is enabled there by
default.
| ``0``: Force off.
| ``1``: Force on, except the scheduler is still disabled (with an
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved
``INFO`` log, not a warning) when port failover/recovery
(``NCCL_IB_RESILIENCY_PORT_FAILOVER``/``_PORT_RECOVERY``) or QP
sharing (``RCCL_IB_QP_SHARING_ENABLE``) is enabled. On a
``BY_ORDER`` comm it instead reports enabled (with a ``WARN``), but
WRR and split-data never run — an unsupported combination.

* - | ``RCCL_IB_QP_SCHED_RESET_INTERVAL``
| Interval at which accumulated RTT statistics are reset, to prevent
stale samples from permanently biasing the scheduler. Value is in
milliseconds.
- | Integer milliseconds. Default: ``60000`` (60 seconds).
| ``0``: Never reset.

* - | ``RCCL_IB_QP_SCHED_UPDATE_INTERVAL``
| Minimum interval between scheduler weight updates. Value is in
microseconds.
- | Integer microseconds. Default: ``50``.
| Clamped to the range 1 µs to 60 s; out-of-range values are ignored.
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved

* - | ``RCCL_IB_QP_SCHED_WEIGHT``
| Exponential moving average (EMA) weight applied to new RTT samples.
- | Floating-point value in the range ``0`` to ``1.0``. Default: ``0``.
| ``0``: Simple average.
| Values closer to ``1.0`` react faster to recent samples.

* - | ``RCCL_IB_QP_SCHED_SPLIT_DATA_MIN``
| Minimum chunk size when splitting a message across QPs (split-data
mode, enabled via ``NCCL_IB_SPLIT_DATA_ON_QPS``). Value is in bytes.
- | Integer bytes. Default: ``65536``.
| Only positive values are applied.

* - | ``RCCL_IB_QP_SCHED_LOG_PATH``
| Directory for a per-process scheduler log file (one line per QP:
device, min/current/max weight, and smoothed RTT). Log file is
named ``cast_log_<hostname>_<pid>``.
- | String directory path.
| Default: unset (logging disabled).

* - | ``RCCL_IB_QP_SCHED_LOG_INTERVAL``
| Interval at which scheduler statistics are written to the log file.
Value is in microseconds. Only used when
``RCCL_IB_QP_SCHED_LOG_PATH`` is set.
- | Integer microseconds. Default: ``1000000`` (1 second).
| Clamped to the range 1 µs to 60 s; out-of-range values are ignored.
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved

Development and testing (advanced)
==================================

Expand Down
Loading