Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions projects/rccl/docs/api-reference/env-variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -783,6 +783,90 @@ in the following table.
``example`` loads ``librccl-rma-example.so``
| See :ref:`using-rccl-rma-plugin`

QP scheduling (CAST)
====================

CAST (Congestion Aware Sprayed Traffic) adds a dynamic QP (Queue Pair) scheduler
that balances RDMA traffic across multiple QPs per connection based on measured
round-trip time (RTT). The following variables tune the scheduler and are
accessible with either the ``RCCL_`` or ``NCCL_`` prefix; the ``RCCL_`` form is
shown below.

These variables only take effect when the IB-CAST net plugin is loaded
(``NCCL_NET=IB-CAST`` or ``ROCM-IB``, or an AINIC NIC with ``NCCL_NET`` unset)
*and* the CAST QP scheduler is enabled (see ``RCCL_IB_QP_SCHED_ENABLE`` below).

.. list-table::
:header-rows: 1
:widths: 40,60

* - **Environment variable**
- **Values**

* - | ``RCCL_IB_QP_SCHED_ENABLE``
| Enables the CAST QP scheduler.
- | ``-1``: Auto (default). Enabled by default when
``NCCL_NET=IB-CAST`` is set explicitly. Disabled by default
otherwise — on AINIC with ``NCCL_NET`` unset, or with
``NCCL_NET=ROCM-IB`` — override either with
``RCCL_IB_QP_SCHED_ENABLE=1``. Independent of that rule, disabled
whenever the connection resolves to ``BY_ORDER`` receive matching.
``BY_ORDER`` is AINIC's default scheme for two reasons: CTS
offload (on by default for non-P2P AINIC connections) forces it,
and it's also the AINIC RoCE fallback even without CTS offload —
unless QP sharing, out-of-order receive queues, or port
failover/recovery force ``BY_ID`` instead, or
``NCCL_IB_RECEIVER_SIDE_MATCHING_SCHEME`` is set explicitly.
| ``0``: Force off.
| ``1``: Force on. On a ``BY_ORDER`` connection the scheduler still
reports "enabled", but WRR and split-data are no-ops there either
way, so it can't actually rebalance traffic — an unsupported
combination, flagged with a ``WARN`` instead of failing silently.
| Independent of the value above, the scheduler is always disabled
(``INFO`` log, not a ``WARN``) when port failover/recovery
(``NCCL_IB_RESILIENCY_PORT_FAILOVER``/``_PORT_RECOVERY``) or QP
sharing (``RCCL_IB_QP_SHARING_ENABLE``) is active — load-balancer
integration with those features is still pending.

* - | ``RCCL_IB_QP_SCHED_RESET_INTERVAL``
| Interval at which accumulated RTT statistics are reset, to prevent
stale samples from permanently biasing the scheduler. Value is in
milliseconds.
- | Integer milliseconds. Default: ``60000`` (60 seconds).
| ``0``: Never reset.

* - | ``RCCL_IB_QP_SCHED_UPDATE_INTERVAL``
| Minimum interval between scheduler weight updates. Value is in
microseconds.
- | Integer microseconds. Default: ``50``.
| Clamped to the range 1 µs to 60 s; out-of-range values are ignored.
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved

* - | ``RCCL_IB_QP_SCHED_WEIGHT``
| Exponential moving average (EMA) weight applied to new RTT samples.
- | Floating-point value in the range ``0`` to ``1.0``. Default: ``0``.
| ``0``: Simple average.
| Values closer to ``1.0`` react faster to recent samples.

* - | ``RCCL_IB_QP_SCHED_SPLIT_DATA_MIN``
| Minimum chunk size when splitting a message across QPs (split-data
mode, enabled via ``NCCL_IB_SPLIT_DATA_ON_QPS``). Value is in bytes.
- | Integer bytes. Default: ``65536``.
| Only positive values are applied.

* - | ``RCCL_IB_QP_SCHED_LOG_PATH``
| Directory for a per-process scheduler log file (one line per QP:
device, min/current/max weight, and smoothed RTT). Log file is
named ``cast_log_<hostname>_<pid>``.
- | String directory path.
| Default: unset (logging disabled).

* - | ``RCCL_IB_QP_SCHED_LOG_INTERVAL``
| Interval at which scheduler statistics are written to the log file.
Value is in microseconds. Only used when
``RCCL_IB_QP_SCHED_LOG_PATH`` is set.
- | Integer microseconds. Default: ``1000000`` (1 second).
| Clamped to the range 1 µs to 60 s; out-of-range values are ignored.
Comment thread
VadimKutovoi marked this conversation as resolved.
Resolved

Development and testing (advanced)
==================================

Expand Down
Loading