Skip to content

feat(redis): add Redis backend with async transfers and nixlbench support - #1791

Merged
aranadive merged 76 commits into
ai-dynamo:mainfrom
hxieustc:pr_nixl_kv_plugin_interface_redis
Oct 4, 2026
Merged

aranadive merged 76 commits into
ai-dynamo:mainfrom
hxieustc:pr_nixl_kv_plugin_interface_redis

Conversation

@gaowayne

@gaowayne gaowayne commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor
## What?

This PR adds a production `REDIS` storage backend to NIXL, together with build, test, documentation, and nixlbench integration.

The implementation is Redis-specific and follows the direct backend structure used by plugins such as INFINIA:

```text
nixlBackendEngine
└── nixlRedisKVEngine
    └── iRedisClient
        └── hiredisAsyncClient

There is intentionally no generic KV backend layer or installed KV plugin API.

Redis backend

  • Registers the dynamic or static plugin as backend name REDIS.
  • Supports DRAM_SEG and OBJ_SEG.
  • Implements asynchronous NIXL_WRITE and NIXL_READ using Redis SET and GET.
  • Implements synchronous queryMem() using Redis EXISTS.
  • Uses a dedicated hiredis/libevent async connection for transfers.
  • Uses a separate blocking hiredis connection for synchronous existence checks.
  • Maps descriptor metaInfo to the Redis key, falling back to the decimal devId.
  • Resolves all transfer keys before dispatching commands, preventing partially submitted transfers.
  • Exposes asynchronous completion through the normal NIXL postXfer() / checkXfer() request-handle lifecycle.

The internal iRedisClient interface isolates hiredis/libevent and provides a deterministic unit-test seam. It is not intended as a generic KV extension API.

Configuration

Connection settings are resolved once into an internal RedisConfig and passed to hiredisAsyncClient.

Configuration precedence is:

  1. Valid NIXL backend parameters
  2. Corresponding REDIS_* environment variables
  3. Built-in defaults
Backend parameter Environment fallback Default
host REDIS_HOST localhost
port REDIS_PORT 6379
username REDIS_USERNAME empty
password REDIS_PASSWORD empty
db none 0

Authentication behavior:

  • Empty password: do not send AUTH; intended for Redis deployments that permit unauthenticated access.
  • Password only: send AUTH password for the default/legacy user.
  • Username and password: send ACL-style AUTH username password.
  • Username without password: reject backend creation.

Explicit empty username or password parameters override environment fallbacks.

nixlbench integration

  • Adds REDIS as a nixlbench storage backend.
  • Uses Redis keys carried in descriptor metadata.
  • Seeds Redis keys before READ benchmarks.
  • Adds Redis dependency handling to the nixlbench build.
  • Adds --redis-only container build support for a static REDIS-focused image.

Build integration

  • Supports dynamic [libplugin_REDIS.so](http://libplugin_redis.so/) builds.
  • Supports static REDIS plugin registration.
  • Requires hiredis, libevent, and libevent-pthreads.
  • Skips the plugin with a warning when dependencies are unavailable during a default all-plugin build.
  • Fails configuration when REDIS is explicitly requested but its dependencies are unavailable.
  • Does not require a Redis-specific ASIO executor or thread pool.

Why?

Redis is widely deployed as a low-latency byte-oriented store and naturally fits NIXL’s storage descriptor model.

This backend enables applications and benchmarks to access Redis through the normal NIXL registration and transfer APIs without introducing a speculative generic KV framework. The direct nixlBackendEngine implementation keeps the plugin consistent with established NIXL backend architecture while retaining a small client boundary for hiredis isolation and testing.

Transfer flow

WRITE / READ

nixlAgent
  → nixlRedisKVEngine::prepXfer()
      validates operation, descriptor counts, and memory types
  → nixlRedisKVEngine::postXfer()
      resolves every Redis key
      creates promise/future state
      issues async SET or GET commands
      returns NIXL_IN_PROG
  → nixlRedisKVEngine::checkXfer()
      polls completion futures
      returns NIXL_IN_PROG, NIXL_SUCCESS, or backend error

queryMem

nixlAgent
  → nixlRedisKVEngine::queryMem()
  → iRedisClient::checkKeyExistsSync()
  → Redis EXISTS on the blocking connection

The synchronous connection keeps queryMem() independent from the asynchronous transfer event loop.

Testing

Redis unit coverage includes:

  • Backend capabilities and supported memory types
  • Default and parameter-based RedisConfig resolution
  • Backend-parameter precedence over environment variables
  • Unauthenticated and ACL credential validation
  • Registration and key selection
  • queryMem() found, missing, and error responses
  • Transfer validation
  • Asynchronous READ/WRITE polling and failures
  • Null request handles
  • Prevention of partial command dispatch

Validation performed:

  • Dynamic REDIS/full project build
  • Static REDIS/NIXL build
  • Redis unit tests: 11 passed
  • git diff --check
  • Live nixlbench READ/WRITE smoke test requires an available Redis server
# nixlbench result
build-nixlbench/nixlbench   --backend REDIS --op_type WRITE   --start_block_size 131072 --max_block_size 131072  --start_batch_size=1  --max_batch_size=64   --num_iter 1000   --warmup_iter 10   --num_threads 4
img_v3_0213k_5de10fce-fe1e-4ebe-8251-652898d2253g

@gaowayne
gaowayne requested review from a team, aranadive, brminich and ovidiusm as code owners June 17, 2026 08:18
@copy-pr-bot

copy-pr-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

👋 Hi gaowayne! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

@coderabbitai

coderabbitai Bot commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a Redis-backed KV plugin layer, wires it into the build and plugin system, extends nixlbench for Redis-backed storage checks, and adds documentation plus unit coverage for the new Redis path.

Changes

Redis KV Plugin and Benchmark Integration

Layer / File(s) Summary
KV abstraction
src/plugins/kv/kv_backend.h, src/plugins/kv/kv_backend.cpp
Defines the KV store interface, backend engine contract, and delegating KV engine wrapper.
Redis contracts
src/plugins/kv/redis/redis_engine.h, src/plugins/kv/redis/redis_executor.h, src/plugins/kv/redis/client.h, src/plugins/kv/redis/engine_impl.h
Declares the Redis client interface, thread-pool executor, Redis engine, and Redis engine implementation headers.
hiredis client implementation
src/plugins/kv/redis/client.cpp
Implements the async and sync Redis client paths, including connection setup, callbacks, async commands, existence checks, and fallback stubs.
Redis engine implementation
src/plugins/kv/redis/engine_impl.cpp, src/plugins/kv/redis/redis_engine.cpp
Implements Redis-backed memory registration, query, transfer preparation, async transfer scheduling, request polling, and engine construction.
Plugin wiring and build
src/plugins/kv/kv_plugin.cpp, src/plugins/kv/meson.build, src/plugins/kv/redis/meson.build, src/plugins/meson.build, meson.build, src/core/meson.build, src/core/nixl_plugin_manager.cpp, benchmark/nixlbench/src/utils/meson.build
Wires the Redis plugin into Meson, static registration, exported dependency handling, and plugin entry points.
nixlbench Redis support
benchmark/nixlbench/contrib/Dockerfile, benchmark/nixlbench/contrib/build.sh, benchmark/nixlbench/src/utils/utils.h, benchmark/nixlbench/src/utils/utils.cpp, benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
Adds Redis backend constants and seeding helpers, updates worker Redis handling, and extends Docker and benchmark build configuration.
Documentation and tests
src/plugins/kv/README.md, src/plugins/kv/redis/README.md, test/gtest/unit/meson.build, test/gtest/unit/redis/meson.build, test/gtest/unit/redis/redis.cpp
Adds KV and Redis plugin documentation and Redis unit test coverage.

Sequence Diagram(s)

sequenceDiagram
  participant nixlbench as nixlbench worker
  participant nixlRedisKVEngineImpl as Redis KV engine
  participant hiredisAsyncClient as Redis client
  participant RedisServer as Redis server

  nixlbench->>nixlRedisKVEngineImpl: prepXfer()
  nixlbench->>nixlRedisKVEngineImpl: postXfer()
  nixlRedisKVEngineImpl->>hiredisAsyncClient: putKeyAsync() / getKeyAsync()
  hiredisAsyncClient->>RedisServer: SET / GET
  RedisServer-->>hiredisAsyncClient: reply
  hiredisAsyncClient-->>nixlRedisKVEngineImpl: promise result
  nixlbench->>nixlRedisKVEngineImpl: checkXfer()
  nixlbench->>nixlRedisKVEngineImpl: queryMem()
  nixlRedisKVEngineImpl->>hiredisAsyncClient: checkKeyExistsSync()
  hiredisAsyncClient->>RedisServer: EXISTS
  RedisServer-->>hiredisAsyncClient: integer reply
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 13.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the Redis backend implementation and nixlbench integration, which are the main changes in the pull request.
Description check ✅ Passed The description covers What, Why, and How, and includes architecture, configuration, testing, and validation details.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/nixlbench/src/utils/utils.cpp`:
- Around line 1333-1342: The synchronous redisConnect call in the connection
block has no timeout mechanism and could block indefinitely if the Redis server
is unreachable or unresponsive. Replace the redisConnect function call with
redisConnectWithTimeout and provide an appropriate timeout value (in seconds and
microseconds as separate parameters) to ensure the connection attempt fails
gracefully within a reasonable time frame instead of blocking indefinitely.
- Around line 1319-1323: The port conversion using std::atoi silently returns 0
for invalid input strings, which would cause the code to attempt connecting to
port 0 instead of falling back to the default port 6379. Replace the std::atoi
call with std::stoi wrapped in a try-catch block to properly handle invalid port
strings, and ensure that if an exception is caught or the port value is invalid
(e.g., outside the valid port range 1-65535), the code either uses the default
port or logs an appropriate error message. This ensures that invalid REDIS_PORT
environment variable values are caught and handled gracefully rather than
silently defaulting to port 0.

In `@benchmark/nixlbench/src/utils/utils.h`:
- Line 97: Replace the preprocessor macro `XFERBENCH_BACKEND_REDIS` with a
`constexpr inline std::string_view` constant following the repository convention
for backend constants. Use lowercase naming as `xferbench_backend_redis` for the
new constant. Update all usages of `XFERBENCH_BACKEND_REDIS` throughout the
codebase to use the new constant, applying `.data()` or `std::string`
conversions where necessary for comparisons and string operations.

In `@benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp`:
- Around line 1259-1263: The assignment of redis_remote.metaInfo = iov.metaInfo
on line 1262 is redundant because redis_remote is already copy-constructed from
iov on line 1260 via the xferBenchIOV constructor, which automatically copies
all member variables including metaInfo. Remove the redundant metaInfo
assignment line to eliminate unnecessary duplication.
- Around line 1048-1049: The descriptor list creation at lines 1048-1049 uses a
two-step pattern (constructing an empty list with DRAM_SEG, then calling
iovListToNixlRegDlist) which differs from the consistent pattern used elsewhere
in the file at lines 990, 1021, and 1098. Replace the two-line pattern with the
single-line assignment pattern where nixl_reg_dlist_t desc_list is directly
assigned the result of calling iovListToNixlRegDlist with the iov_list and
appropriate segment type, ensuring consistency across all similar descriptor
list creations in the file.

In `@src/plugins/kv/kv_backend.cpp`:
- Around line 21-26: Replace the debug-only assert statement in the nixlKVEngine
constructor that validates impl_ is non-null with actual runtime validation
(such as throwing a std::invalid_argument exception or logging an error and
exiting the program). The current assert will be stripped from release builds,
allowing a null impl_ to be assigned to impl_ and then crash on the first method
call. Use a proper if-check followed by error handling to ensure the validation
persists in both debug and release builds.

In `@src/plugins/kv/kv_backend.h`:
- Around line 18-19: Replace the generic header guard macro `KV_BACKEND_H` with
a repository-scoped guard following the convention `NIXL_<PATH>_H` where PATH
represents the full repository-relative path. For the file at
src/plugins/kv/kv_backend.h, update both the `#ifndef` and `#define` directives
at lines 18-19, and ensure the corresponding closing `#endif` at line 161
references the same new guard macro name.

In `@src/plugins/kv/redis/client.cpp`:
- Around line 313-320: The Redis GET reply handling in the memcpy section marks
success even when the payload is truncated due to insufficient buffer capacity.
The current logic uses std::min to determine copy_len but sets success to true
regardless of whether copy_len is smaller than r->len, allowing truncated data
to be silently treated as successful. Fix this by only setting success to true
when the entire reply was copied without truncation, which means adding a check
to verify that copy_len equals r->len before marking the operation as
successful, and handle the case where copy_len is 0 or truncation occurred as a
failure condition.

In `@src/plugins/kv/redis/client.h`:
- Around line 27-28: The header guard KV_PLUGIN_REDIS_CLIENT_H does not follow
the repository-wide convention that uses the full repository-relative path
format with the NIXL_ prefix. Replace KV_PLUGIN_REDIS_CLIENT_H with the
full-path header guard format (NIXL_SRC_PLUGINS_KV_REDIS_CLIENT_H) in both the
opening `#ifndef` and `#define` directives at the top of the file, and update any
corresponding `#endif` closing guard to match this new convention consistently
throughout the file.

In `@src/plugins/kv/redis/engine_impl.cpp`:
- Around line 180-183: In the getSupportedMems() validation block where
NIXL_ERR_NOT_SUPPORTED is returned, the out parameter is not initialized before
the early return statement. This leaves the out parameter in an uninitialized
state when the error path is taken, potentially exposing stale data to callers.
Initialize the out parameter to a safe default value (typically a null pointer
or default-initialized value) immediately before the return
NIXL_ERR_NOT_SUPPORTED statement to ensure all code paths properly initialize
output parameters.
- Around line 298-333: The loop in the Redis engine implementation performs both
key validation and command dispatch within the same loop iteration. To avoid
partial side effects when validation fails on a later descriptor, refactor this
into two separate passes: first validate that all redis_key lookups from
devIdToRedisKey_ succeed for every descriptor in the loop, returning
NIXL_ERR_INVALID_PARAM immediately if any key is missing, and only after all
keys are validated should the second loop dispatch the putKeyAsync or
getKeyAsync commands.

In `@src/plugins/kv/redis/engine_impl.h`:
- Around line 23-24: Update the header guard to follow the repository convention
of using the full repository-relative path format prefixed with NIXL. Replace
the current guard `KV_PLUGIN_REDIS_ENGINE_IMPL_H` at lines 23-24 with the
appropriate guard name derived from the full repository-relative path of the
file (src/plugins/kv/redis/engine_impl.h). Also update the corresponding closing
endif guard at line 89 to match the new guard name convention.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1a791ca5-8787-4268-b14c-a6a0d9461302

📥 Commits

Reviewing files that changed from the base of the PR and between 1f4468f and dc629dc.

📒 Files selected for processing (24)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build

Comment thread benchmark/nixlbench/src/utils/utils.cpp Outdated
Comment thread benchmark/nixlbench/src/utils/utils.cpp Outdated
Comment thread benchmark/nixlbench/src/utils/utils.h
Comment thread benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp Outdated
Comment thread src/plugins/kv/kv_backend.cpp Outdated
Comment thread src/plugins/redis/redis_client.cpp
Comment thread src/plugins/kv/redis/client.h Outdated
Comment thread src/plugins/kv/redis/engine_impl.cpp Outdated
Comment thread src/plugins/kv/redis/engine_impl.cpp Outdated
Comment thread src/plugins/kv/redis/engine_impl.h Outdated
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from dc629dc to b3af297 Compare June 22, 2026 15:38

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
benchmark/nixlbench/contrib/Dockerfile (1)

29-53: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Harden apt install with --no-install-recommends.

This layer currently installs recommended packages implicitly, which unnecessarily expands image footprint and package exposure.

🔧 Suggested patch
 RUN apt-get update -y && \
-    DEBIAN_FRONTEND=noninteractive apt-get -y install \
+    DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
     ninja-build \
     pybind11-dev \
     libclang-dev \
@@
     build-essential \
     libhiredis-dev \
     libevent-dev \
-    redis-tools
+    redis-tools && \
+    rm -rf /var/lib/apt/lists/*
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/nixlbench/contrib/Dockerfile` around lines 29 - 53, The apt-get
install command in the RUN statement is missing the --no-install-recommends
flag, which causes unnecessary recommended packages to be installed, increasing
the Docker image footprint and security exposure. Add the
--no-install-recommends flag to the apt-get install command that installs
packages including ninja-build, pybind11-dev, libclang-dev, and the other build
and runtime dependencies. Place this flag after the -y option and before the
package list to ensure only explicitly requested packages are installed.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp`:
- Around line 1036-1041: The current code uses `continue` when
xferBenchUtils::putRedis fails during READ mode seeding, which skips the current
device descriptor but leaves downstream code expecting a complete per-device
`remote_regs_` list indexed by device position. This creates a mismatch between
the number of devices processed and the size of the `remote_regs_` list, causing
potential out-of-bounds access. Replace the `continue` statement with a
fail-fast mechanism (such as `return` or `break`) that exits the loop
immediately when Redis seeding fails, ensuring the device descriptor count
remains consistent with the indexing expectations in downstream code.

In `@src/plugins/kv/kv_backend.h`:
- Line 79: The destructor for the nixlKVEngine class is overriding a virtual
base destructor but is not explicitly marked with the override keyword. Add the
override keyword to the virtual destructor declaration of nixlKVEngine to
explicitly indicate that it overrides a base class virtual destructor, placing
it after the destructor signature and before the semicolon.

In `@src/plugins/kv/redis/client.cpp`:
- Line 221: The Redis client source file has formatting mismatches that
clang-format can automatically correct. Run clang-format on the client.cpp file
in the Redis plugin directory to fix the formatting inconsistencies that are
causing CI failures. This will ensure the code adheres to the project's coding
style guidelines.
- Around line 162-167: The AUTH and SELECT commands in the async initialization
are issued with nullptr callbacks, causing authentication and database selection
failures to be silently ignored. Create proper callback handler functions for
these commands that validate their responses and check for errors. Modify the
redisAsyncCommand calls for both "AUTH %s" and "SELECT %d" to pass these
callback handlers instead of nullptr, and ensure the client connection state is
only marked as ready after both AUTH (if password is provided) and SELECT (if db
is non-zero) commands complete successfully.

In `@src/plugins/kv/redis/client.h`:
- Around line 1-127: The header file src/plugins/kv/redis/client.h does not
conform to the project's code formatting standards as reported by
clang-format-diff-19 in CI. Run clang-format on the entire file to automatically
reformat it according to the project's style conventions. This should resolve
any formatting issues in the hiredisAsyncClient class definition and related
code.

In `@src/plugins/kv/redis/engine_impl.cpp`:
- Around line 91-95: The if statement checking `remote_agent != local_agent` is
missing braces around its body, which violates coding guidelines requiring
braces for all control statements. Add opening and closing braces around the
NIXL_WARN logging statement that follows the if condition to comply with the
style requirements.

In `@src/plugins/kv/redis/engine_impl.h`:
- Around line 45-55: The method declarations in the RedisEngine class
(specifically getSupportedMems, registerMem, deregisterMem, and queryMem) in
engine_impl.h do not conform to the repository's clang-format formatting
standards as verified by CI. Run clang-format on the entire engine_impl.h file
to automatically reformat these declarations according to the established coding
guidelines, which will resolve the CI formatting deltas reported in the
declaration block at lines 45-55 and also at lines 74-76.

In `@src/plugins/kv/redis/meson.build`:
- Around line 105-110: The redis_backend_lib shared_library call is missing
compile_flags that are present in the static build path, causing inconsistent
compilation between static and shared plugin artifacts. Locate the compile_flags
variable used in the static REDIS build definition in the same file, then add
these flags to the cpp_args parameter in the redis_backend_lib shared_library
call to ensure both static and shared builds use the same compiler flags.

In `@src/plugins/kv/redis/redis_engine.h`:
- Around line 1-103: The redis_engine.h file has formatting drift detected by
clang-format-diff-19, which is causing CI to fail. Run clang-format on the
redis_engine.h file using the project's clang-format configuration to
automatically fix all formatting issues in the file. This should resolve the
formatting drift in the copyright headers, includes, class definitions
iRedisClient and nixlRedisKVEngine, method declarations, and overall code style
to match the project standards.

In `@src/plugins/kv/redis/redis_executor.h`:
- Around line 1-53: The header file redis_executor.h containing the
redisThreadPoolExecutor class requires code formatting updates as flagged by the
project's clang-format CI checks. Run clang-format on this file to automatically
apply the required formatting changes and align it with the project's coding
style guidelines.
- Around line 34-37: The method name WaitUntilStopped uses PascalCase but the
repository convention requires lowerCamelCase for member methods. Rename the
method from WaitUntilStopped to waitUntilStopped in the redis_executor.h file,
and then search for and update all call sites throughout the codebase where
WaitUntilStopped is invoked to use the new lowercase name waitUntilStopped
instead.

---

Outside diff comments:
In `@benchmark/nixlbench/contrib/Dockerfile`:
- Around line 29-53: The apt-get install command in the RUN statement is missing
the --no-install-recommends flag, which causes unnecessary recommended packages
to be installed, increasing the Docker image footprint and security exposure.
Add the --no-install-recommends flag to the apt-get install command that
installs packages including ninja-build, pybind11-dev, libclang-dev, and the
other build and runtime dependencies. Place this flag after the -y option and
before the package list to ensure only explicitly requested packages are
installed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 23a86fa0-b1c2-421f-a0d3-56f4eae8784a

📥 Commits

Reviewing files that changed from the base of the PR and between dc629dc and b3af297.

📒 Files selected for processing (24)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build

Comment thread benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp Outdated
Comment thread src/plugins/kv/kv_backend.h Outdated
Comment thread src/plugins/kv/redis/client.cpp Outdated
Comment thread src/plugins/redis/redis_client.cpp
Comment thread src/plugins/kv/redis/client.h Outdated
Comment thread src/plugins/kv/redis/engine_impl.h Outdated
Comment thread src/plugins/redis/meson.build
Comment thread src/plugins/kv/redis/redis_engine.h Outdated
Comment thread src/plugins/kv/redis/redis_executor.h Outdated
Comment thread src/plugins/kv/redis/redis_executor.h Outdated
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from b3af297 to b1b1b03 Compare June 22, 2026 19:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
benchmark/nixlbench/contrib/Dockerfile (1)

29-53: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Use --no-install-recommends in the base apt install layer.

Line 30 installs many packages without --no-install-recommends, which increases image size/surface area unnecessarily.

♻️ Minimal fix
 RUN apt-get update -y && \
-    DEBIAN_FRONTEND=noninteractive apt-get -y install \
+    DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
     ninja-build \
     pybind11-dev \
@@
-    redis-tools
+    redis-tools && \
+    rm -rf /var/lib/apt/lists/*
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/nixlbench/contrib/Dockerfile` around lines 29 - 53, The apt-get
install command in the RUN layer that installs packages like ninja-build,
pybind11-dev, libclang-dev, cmake, and others is missing the
--no-install-recommends flag. Add --no-install-recommends to the apt-get install
command (after the -y flag and before the package list) to prevent the
installation of recommended packages that are not strictly required, which will
reduce the Docker image size and surface area.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/nixlbench/contrib/Dockerfile`:
- Around line 125-126: The Dockerfile ARG declarations for NIXL_ENABLE_PLUGINS
and NIXL_STATIC_PLUGINS are hardcoded to "REDIS", which restricts plugin
building and duplicates the --redis-only flag intent from the build script.
Either change both ARG assignments to use empty strings as values (allowing
Meson to build all plugins by default) or remove the --redis-only option from
the build script if Redis-only is the intended behavior. Clarify the intent and
ensure the Dockerfile defaults and build script flags are not redundant.

In `@benchmark/nixlbench/src/utils/utils.cpp`:
- Line 67: The backend option documentation at line 67 lists GUSLI as a valid
backend option along with REDIS and AZURE_BLOB, but the actual printed backend
list at lines 683-684 omits GUSLI. To fix this inconsistency, add GUSLI to the
backend list string that is printed at lines 683-684 so that it matches the
documented options shown in the help message at line 67, ensuring users see a
complete and accurate list of available backends when viewing configuration
output.

In `@src/plugins/kv/README.md`:
- Around line 49-52: The file map entry for the `iRedisClient` interface is
incorrectly documented as being in `redis_engine.h/.cpp` when it actually
resides in `src/plugins/kv/redis/client.h`. Update the mapping entry for
`iRedisClient` on line 49 to reflect the correct file location in
`client.h/.cpp` instead of `redis_engine.h/.cpp`.

In `@src/plugins/kv/redis/client.cpp`:
- Around line 176-183: The code is calling redisAsyncFree(asyncContext_)
immediately after redisAsyncDisconnect(asyncContext_), which causes a
double-free memory error. The hiredis async API automatically frees the context
when redisAsyncDisconnect() completes its callback, so the subsequent
redisAsyncFree() call is redundant and dangerous. Remove the
redisAsyncFree(asyncContext_) call that follows
redisAsyncDisconnect(asyncContext_) in the constructor timeout path where
connected_.load() is false. Apply this same fix pattern to any other locations
in the code where this redundant call pattern exists.

In `@src/plugins/kv/redis/meson.build`:
- Around line 62-68: The REDIS plugin build silently skips with a warning for
shared builds even when explicitly requested by the user, while static builds
properly fail with an error. Modify the condition on line 62 that currently only
checks if 'REDIS' is in static_plugins to also check if REDIS is explicitly
enabled for shared builds (check the get_option or similar mechanism for REDIS
explicit enablement). When REDIS is explicitly requested in any form and the
required dependencies (hiredis_async_dep or libevent_dep) are missing, throw an
error instead of just warning and skipping. Only allow the silent skip with a
warning when REDIS was not explicitly requested by the user.

In `@src/plugins/kv/redis/README.md`:
- Line 61: The documentation on line 61 of the README.md file incorrectly
implies that the `db` parameter can be configured via environment variables, but
the actual implementation in src/plugins/kv/redis/client.cpp only reads `db`
from custom params. Reword line 61 to clarify that `db` is configured through
NIXL custom params only, while keeping the distinction that `host`, `port`, and
`password` can come from either custom params or their respective environment
variables (REDIS_HOST, REDIS_PORT, REDIS_PASSWORD).

---

Outside diff comments:
In `@benchmark/nixlbench/contrib/Dockerfile`:
- Around line 29-53: The apt-get install command in the RUN layer that installs
packages like ninja-build, pybind11-dev, libclang-dev, cmake, and others is
missing the --no-install-recommends flag. Add --no-install-recommends to the
apt-get install command (after the -y flag and before the package list) to
prevent the installation of recommended packages that are not strictly required,
which will reduce the Docker image size and surface area.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 029f983b-216b-41e8-a2b6-96d6a47a757a

📥 Commits

Reviewing files that changed from the base of the PR and between b3af297 and b1b1b03.

📒 Files selected for processing (24)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build

Comment thread benchmark/nixlbench/contrib/Dockerfile Outdated
Comment thread benchmark/nixlbench/src/utils/utils.cpp
Comment thread src/plugins/kv/README.md Outdated
Comment thread src/plugins/kv/redis/client.cpp Outdated
Comment thread src/plugins/redis/meson.build Outdated
Comment thread src/plugins/kv/redis/README.md Outdated
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from b1b1b03 to 0eb91e0 Compare June 23, 2026 20:59

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

♻️ Duplicate comments (8)
src/plugins/kv/kv_backend.h (2)

79-79: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Mark the overriding destructor with override.

nixlKVEngine derives from nixlBackendEngine; the destructor declaration should be ~nixlKVEngine() override;.
As per path instructions, overridden virtual methods should explicitly use override.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/kv_backend.h` at line 79, The destructor declaration for the
nixlKVEngine class is missing the override keyword. Since nixlKVEngine derives
from nixlBackendEngine and overrides its virtual destructor, add the override
keyword to the destructor declaration. Modify the ~nixlKVEngine destructor to
include override after the empty parameter list to explicitly indicate it is
overriding a base class virtual method, following the codebase convention for
overridden virtual methods.

Source: Path instructions


18-19: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use the repository-relative header guard macro.

KV_BACKEND_H is too generic and can collide across headers. Use
NIXL_SRC_PLUGINS_KV_KV_BACKEND_H consistently in #ifndef/#define/#endif.
As per path instructions, headers should use traditional include guards; based on learnings, this repo uses full repository-relative NIXL_<PATH>_H guards.

Also applies to: 161-161

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/kv_backend.h` around lines 18 - 19, Replace the generic header
guard macro KV_BACKEND_H with the repository-relative macro
NIXL_SRC_PLUGINS_KV_KV_BACKEND_H to prevent collisions with other headers.
Update the `#ifndef` directive at the top of the file, the corresponding `#define`
directive, and the matching `#endif` at the end of the file to use the new
fully-qualified macro name NIXL_SRC_PLUGINS_KV_KV_BACKEND_H instead of the
generic KV_BACKEND_H.

Sources: Path instructions, Learnings

src/plugins/kv/redis/redis_executor.h (1)

34-37: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Normalize method naming to lower camel case.

WaitUntilStopped breaks the file’s own naming pattern (waitUntilIdle) and the repo convention. Rename it to waitUntilStopped and update call sites.
As per path instructions, functions/methods should use lower camel case.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/redis_executor.h` around lines 34 - 37, The method
`WaitUntilStopped` uses upper camel case naming which violates the repository
convention of lower camel case (as seen in the same file with `waitUntilIdle`).
Rename the method `WaitUntilStopped` to `waitUntilStopped` in its definition,
and update all call sites throughout the codebase that invoke this method to use
the new lower camel case name.

Source: Path instructions

src/plugins/kv/redis/client.h (1)

27-28: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Adopt the full repository-path header guard.

Replace KV_PLUGIN_REDIS_CLIENT_H with NIXL_SRC_PLUGINS_KV_REDIS_CLIENT_H in the opening and closing guard.
As per path instructions, use traditional include guards; based on learnings, this repo standard is full-path NIXL_<PATH>_H.

Also applies to: 127-127

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/client.h` around lines 27 - 28, The header guard macro
KV_PLUGIN_REDIS_CLIENT_H does not follow the repository's full-path naming
convention. Replace KV_PLUGIN_REDIS_CLIENT_H with
NIXL_SRC_PLUGINS_KV_REDIS_CLIENT_H in both locations: the opening `#ifndef` and
`#define` directives at the top of the file, and the closing `#endif` comment at the
bottom of the file. This ensures consistency with the repository standard of
using full-path prefixes in the format NIXL_<PATH>_H for all header guards.

Sources: Path instructions, Learnings

src/plugins/kv/redis/README.md (1)

61-61: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Clarify the db configuration source.

This line currently implies db may come from env vars, but only REDIS_HOST/PORT/PASSWORD are listed. Please explicitly separate: db from NIXL params, others from params or env fallback.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/README.md` at line 61, The documentation line describing
connection settings is ambiguous about where the `db` parameter is configured.
Currently it lists `db` alongside other parameters but then only provides
environment variable names for `host`, `port`, and `password`, leaving `db`'s
source unclear. Revise this line to explicitly separate the configuration
sources: clearly state which parameters come from NIXL custom params only, and
which can fall back to environment variables. If `db` has a corresponding
environment variable like `REDIS_DB`, include it; otherwise explicitly note that
`db` only comes from NIXL params to eliminate confusion.
src/plugins/kv/redis/engine_impl.h (2)

42-76: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Resolve the clang-format drift in declaration blocks.

CI is still reporting formatting deltas in this range (return-type/signature wrapping), which will keep the format check red until fixed.
As per path instructions, keep declaration formatting consistent (including return-type line breaks), and pipeline failures explicitly flag this range.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/engine_impl.h` around lines 42 - 76, The method
declarations in the nixlRedisKVEngineImpl class (including registerMem,
deregisterMem, queryMem, prepXfer, postXfer, checkXfer, and releaseReqH) have
inconsistent clang-format formatting with respect to return-type line breaks and
signature wrapping. Apply clang-format rules consistently across all
declarations in this range to ensure uniform formatting of return types,
parameter list alignment, and line breaks. Verify that the formatting matches
the project's clang-format configuration and that CI formatting checks pass
after the changes.

Sources: Path instructions, Pipeline failures


23-24: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use the repo-standard include guard macro.

KV_PLUGIN_REDIS_ENGINE_IMPL_H should be updated to
NIXL_SRC_PLUGINS_KV_REDIS_ENGINE_IMPL_H for consistency and collision safety.
As per path instructions, headers should use traditional path-derived guards; based on learnings, this repo enforces full-path NIXL_<PATH>_H.

Also applies to: 89-89

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/engine_impl.h` around lines 23 - 24, The include guard
macro KV_PLUGIN_REDIS_ENGINE_IMPL_H does not follow the repository's standard
naming convention. Replace all occurrences of this macro name with
NIXL_SRC_PLUGINS_KV_REDIS_ENGINE_IMPL_H throughout the file, including in the
`#ifndef` directive at the top, the `#define` directive on the following line, and
the corresponding `#endif` directive at the end of the file (around line 89). This
ensures consistency with the repo's full-path-derived include guard convention
for collision safety.

Sources: Path instructions, Learnings

src/plugins/kv/kv_backend.cpp (1)

21-26: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Replace debug-only assert with release-safe validation.

impl_ is only protected by assert, so release builds can construct with null and crash later on delegated calls.

Suggested fix
 `#include` "kv_backend.h"
 `#include` <cassert>
+#include <stdexcept>
@@
 nixlKVEngine::nixlKVEngine(const nixlBackendInitParams *init_params,
                            std::unique_ptr<nixlKVEngineImpl> impl)
     : nixlBackendEngine(init_params),
       impl_(std::move(impl)) {
+    if (!impl_) {
+        throw std::invalid_argument("nixlKVEngine requires a non-null nixlKVEngineImpl");
+    }
     assert(impl_ && "nixlKVEngine requires a non-null nixlKVEngineImpl");
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/kv_backend.cpp` around lines 21 - 26, The assert statement
protecting impl_ in the nixlKVEngine constructor only executes in debug builds
and is stripped in release builds, allowing null impl_ to pass through and cause
crashes on later delegated calls. Replace the assert in the nixlKVEngine
constructor with release-safe validation, such as throwing an exception if impl_
is null, to ensure the null check is enforced consistently across both debug and
release builds.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/nixlbench/src/utils/utils.cpp`:
- Line 827: The checkConsistency() function's storage WRITE path at line 827
does not properly handle the Redis backend before attempting to use pread() for
validation. Redis stores keys in metaInfo rather than file descriptors, which
causes the validation to fail when using --backend=REDIS with --op_type=WRITE
and --check_consistency=1. Add a Redis-specific validation branch within
checkConsistency() that handles Redis GET validation using the metaInfo keys, or
explicitly reject the Redis backend option in WRITE consistency validation mode
before reaching the pread() call.

In `@benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp`:
- Around line 1258-1262: In the Redis backend branch of
prepareTransferDescriptors(), the redis_remote descriptor is not applying the
block_size parameter like the file and GUSLI storage paths do. After setting
redis_remote.addr to 0 and redis_remote.metaInfo to iov.metaInfo, add a line to
set redis_remote.len to the block_size value to ensure the configured benchmark
block size is respected for Redis transfers instead of using the default iov
length.

In `@src/plugins/kv/redis/client.cpp`:
- Around line 334-387: The redisAsyncContext is not thread-safe and must only be
accessed from the event-loop thread (eventLoopThread_), but putKeyAsync and
getKeyAsync are calling redisAsyncCommand directly from caller threads without
synchronization. To fix this, you need to marshal the redisAsyncCommand calls
onto the event-loop thread instead of executing them directly. Use a mechanism
like event_base_once to schedule the command execution on the event loop, or
implement a thread-safe command queue with a wakeup mechanism. This ensures all
asyncContext_ accesses happen only within the event-loop thread that runs
event_base_dispatch.

---

Duplicate comments:
In `@src/plugins/kv/kv_backend.cpp`:
- Around line 21-26: The assert statement protecting impl_ in the nixlKVEngine
constructor only executes in debug builds and is stripped in release builds,
allowing null impl_ to pass through and cause crashes on later delegated calls.
Replace the assert in the nixlKVEngine constructor with release-safe validation,
such as throwing an exception if impl_ is null, to ensure the null check is
enforced consistently across both debug and release builds.

In `@src/plugins/kv/kv_backend.h`:
- Line 79: The destructor declaration for the nixlKVEngine class is missing the
override keyword. Since nixlKVEngine derives from nixlBackendEngine and
overrides its virtual destructor, add the override keyword to the destructor
declaration. Modify the ~nixlKVEngine destructor to include override after the
empty parameter list to explicitly indicate it is overriding a base class
virtual method, following the codebase convention for overridden virtual
methods.
- Around line 18-19: Replace the generic header guard macro KV_BACKEND_H with
the repository-relative macro NIXL_SRC_PLUGINS_KV_KV_BACKEND_H to prevent
collisions with other headers. Update the `#ifndef` directive at the top of the
file, the corresponding `#define` directive, and the matching `#endif` at the end of
the file to use the new fully-qualified macro name
NIXL_SRC_PLUGINS_KV_KV_BACKEND_H instead of the generic KV_BACKEND_H.

In `@src/plugins/kv/redis/client.h`:
- Around line 27-28: The header guard macro KV_PLUGIN_REDIS_CLIENT_H does not
follow the repository's full-path naming convention. Replace
KV_PLUGIN_REDIS_CLIENT_H with NIXL_SRC_PLUGINS_KV_REDIS_CLIENT_H in both
locations: the opening `#ifndef` and `#define` directives at the top of the file,
and the closing `#endif` comment at the bottom of the file. This ensures
consistency with the repository standard of using full-path prefixes in the
format NIXL_<PATH>_H for all header guards.

In `@src/plugins/kv/redis/engine_impl.h`:
- Around line 42-76: The method declarations in the nixlRedisKVEngineImpl class
(including registerMem, deregisterMem, queryMem, prepXfer, postXfer, checkXfer,
and releaseReqH) have inconsistent clang-format formatting with respect to
return-type line breaks and signature wrapping. Apply clang-format rules
consistently across all declarations in this range to ensure uniform formatting
of return types, parameter list alignment, and line breaks. Verify that the
formatting matches the project's clang-format configuration and that CI
formatting checks pass after the changes.
- Around line 23-24: The include guard macro KV_PLUGIN_REDIS_ENGINE_IMPL_H does
not follow the repository's standard naming convention. Replace all occurrences
of this macro name with NIXL_SRC_PLUGINS_KV_REDIS_ENGINE_IMPL_H throughout the
file, including in the `#ifndef` directive at the top, the `#define` directive on
the following line, and the corresponding `#endif` directive at the end of the
file (around line 89). This ensures consistency with the repo's
full-path-derived include guard convention for collision safety.

In `@src/plugins/kv/redis/README.md`:
- Line 61: The documentation line describing connection settings is ambiguous
about where the `db` parameter is configured. Currently it lists `db` alongside
other parameters but then only provides environment variable names for `host`,
`port`, and `password`, leaving `db`'s source unclear. Revise this line to
explicitly separate the configuration sources: clearly state which parameters
come from NIXL custom params only, and which can fall back to environment
variables. If `db` has a corresponding environment variable like `REDIS_DB`,
include it; otherwise explicitly note that `db` only comes from NIXL params to
eliminate confusion.

In `@src/plugins/kv/redis/redis_executor.h`:
- Around line 34-37: The method `WaitUntilStopped` uses upper camel case naming
which violates the repository convention of lower camel case (as seen in the
same file with `waitUntilIdle`). Rename the method `WaitUntilStopped` to
`waitUntilStopped` in its definition, and update all call sites throughout the
codebase that invoke this method to use the new lower camel case name.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1e6a6197-765e-4ba6-aab1-be92a9ec1875

📥 Commits

Reviewing files that changed from the base of the PR and between b1b1b03 and 0eb91e0.

📒 Files selected for processing (24)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build

Comment thread benchmark/nixlbench/src/utils/utils.cpp
Comment thread benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp Outdated
Comment thread src/plugins/redis/redis_client.cpp Outdated
@nv-nmailhot
nv-nmailhot requested review from a team as code owners June 24, 2026 06:28
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from 0eb91e0 to 8b203b2 Compare June 25, 2026 23:32

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
benchmark/nixlbench/contrib/Dockerfile (1)

29-53: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Trim recommended packages from this apt layer.

This install step still omits --no-install-recommends, which pulls extra packages into the benchmark image for no functional benefit.

♻️ Minimal fix
 RUN apt-get update -y && \
-    DEBIAN_FRONTEND=noninteractive apt-get -y install \
+    DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/nixlbench/contrib/Dockerfile` around lines 29 - 53, The apt install
step in the Dockerfile is still pulling recommended packages, which bloats the
benchmark image. Update the RUN apt-get install block to use the same
minimal-install approach by adding the no-recommends option to the package
installation command, keeping the existing package list in the nixlbench contrib
Dockerfile unchanged.

Source: Linters/SAST tools

♻️ Duplicate comments (3)
src/plugins/kv/redis/README.md (1)

61-61: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Clarify the db config source.

This still reads as if db can come from the env-var fallback list. Split db out explicitly so only host/port/password are env-backed and db stays custom-params only.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/README.md` at line 61, Clarify the Redis connection
settings documentation so `db` is not implied to be sourced from environment
variables. Update the README text around the Redis constructor settings to
explicitly separate `db` as a NIXL custom param only, while keeping `host`,
`port`, and `password` as the env-backed fields (`REDIS_HOST`, `REDIS_PORT`,
`REDIS_PASSWORD`).
src/plugins/kv/redis/meson.build (2)

105-110: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Keep compile_flags in the shared REDIS build.

The shared path still drops the repo-wide compile_flags used by the static path, so the two artifacts can compile under different warning/feature settings.

Suggested fix
     redis_backend_lib = shared_library(
         'REDIS',
         redis_sources,
         dependencies: redis_plugin_deps,
-        cpp_args: compile_defs_redis + ['-fPIC'],
+        cpp_args: compile_defs_redis + compile_flags + ['-fPIC'],
         include_directories: [nixl_inc_dirs, utils_inc_dirs],
         install: true,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/meson.build` around lines 105 - 110, The shared REDIS
build in meson.build is missing the repo-wide compile_flags that the static path
already uses, so the two artifacts can be built with different compiler
settings. Update the shared_library definition for redis_backend_lib to include
the same compile_flags used by the static REDIS target, keeping the shared and
static builds aligned under the same warning/feature configuration.

62-68: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Fail when REDIS is explicitly requested but its deps are missing.

This still turns -Denable_plugins=REDIS into a warning + skip because only the static-plugin path errors. The build can therefore succeed without the backend the caller asked for.

Suggested fix
 if not hiredis_async_dep.found() or not libevent_dep.found()
-    if 'REDIS' in static_plugins
+    if is_explicit_enable
+        error('REDIS plugin requested but hiredis and libevent are required. Install libhiredis-dev and libevent-dev (or equivalent), then reconfigure.')
+    elif 'REDIS' in static_plugins
         error('REDIS static plugin requested but hiredis and libevent are required. Install libhiredis-dev and libevent-dev (or equivalent), then reconfigure.')
     endif
     warning('hiredis-async or libevent not available, skipping REDIS plugin build')
     subdir_done()
 endif
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/kv/redis/meson.build` around lines 62 - 68, The REDIS plugin
check in meson.build only errors for the static_plugins path, so an explicit
REDIS request can still fall through to warning() and subdir_done(). Update the
dependency gate around the REDIS build logic to fail whenever REDIS is
explicitly enabled through enable_plugins or static_plugins and either
hiredis_async_dep or libevent_dep is missing, using the existing REDIS plugin
selection symbols to detect the request before skipping the subdir.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/plugins/kv/kv_backend.h`:
- Around line 29-159: Add Doxygen-style type documentation for the public KV API
in this header: `iKVStore`, `nixlKVEngineImpl`, and `nixlKVEngine` currently
lack class-level `/** ... */` comments. Document the intended ownership and
lifecycle semantics for `put`, `get`, `registerMem`, `deregisterMem`,
`prepXfer`, `postXfer`, `checkXfer`, and `releaseReqH`, and clarify how request
handles and the `impl_` ownership are managed so the exported interfaces are
properly described.

In `@src/plugins/kv/meson.build`:
- Line 1: The SPDX copyright header in this new meson.build file should use a
single year instead of a range. Update the top-of-file copyright text in
src/plugins/kv/meson.build to match the repo convention for new build-related
files, using 2026 only and keeping the SPDX header format consistent.

In `@src/plugins/kv/redis/engine_impl.cpp`:
- Around line 177-195: `nixlRedisKVEngineImpl::registerMem` currently accepts
`OBJ_SEG` and returns success even though `isValidPrepXferParams` only allows
`DRAM_SEG` for local transfers. Update the local registration path so `OBJ_SEG`
is rejected with a failure code while keeping any remote `OBJ_SEG` support in
`getSupportedMems` intact; use the existing `registerMem`, `getSupportedMems`,
and `isValidPrepXferParams` logic to keep the registration contract consistent
with transfer validation.
- Around line 257-263: Initialize the output handle on the failure path in
`prepXfer()` by clearing `handle` before the `isValidPrepXferParams()`
validation check, so a failed validation cannot leave a stale request pointer
behind. Update the `engine_impl.cpp` `prepXfer` logic to set `handle` to
null/empty up front, then keep the existing `nixlRedisBackendReqH` allocation
and `release()` assignment only on the success path.

In `@src/plugins/kv/redis/meson.build`:
- Line 1: The copyright header in the new meson.build file should use a
single-year notice instead of a range. Update the SPDX-FileCopyrightText header
at the top of the file to use 2026 only, following the repo’s convention for new
build-related files; locate it by the SPDX copyright line in this meson.build
file.
- Line 123: The redis_backend_interface declaration is missing the
redis_plugin_deps re-export, so the static REDIS path does not inherit the
hiredis/libevent/asio dependencies. Update redis_backend_interface in the
meson.build for the redis backend to include redis_plugin_deps via the
dependencies argument alongside link_with so downstream consumers in
src/core/meson.build get the full dependency set.

---

Outside diff comments:
In `@benchmark/nixlbench/contrib/Dockerfile`:
- Around line 29-53: The apt install step in the Dockerfile is still pulling
recommended packages, which bloats the benchmark image. Update the RUN apt-get
install block to use the same minimal-install approach by adding the
no-recommends option to the package installation command, keeping the existing
package list in the nixlbench contrib Dockerfile unchanged.

---

Duplicate comments:
In `@src/plugins/kv/redis/meson.build`:
- Around line 105-110: The shared REDIS build in meson.build is missing the
repo-wide compile_flags that the static path already uses, so the two artifacts
can be built with different compiler settings. Update the shared_library
definition for redis_backend_lib to include the same compile_flags used by the
static REDIS target, keeping the shared and static builds aligned under the same
warning/feature configuration.
- Around line 62-68: The REDIS plugin check in meson.build only errors for the
static_plugins path, so an explicit REDIS request can still fall through to
warning() and subdir_done(). Update the dependency gate around the REDIS build
logic to fail whenever REDIS is explicitly enabled through enable_plugins or
static_plugins and either hiredis_async_dep or libevent_dep is missing, using
the existing REDIS plugin selection symbols to detect the request before
skipping the subdir.

In `@src/plugins/kv/redis/README.md`:
- Line 61: Clarify the Redis connection settings documentation so `db` is not
implied to be sourced from environment variables. Update the README text around
the Redis constructor settings to explicitly separate `db` as a NIXL custom
param only, while keeping `host`, `port`, and `password` as the env-backed
fields (`REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD`).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: d231e727-e41d-470b-969f-7e95fb55b800

📥 Commits

Reviewing files that changed from the base of the PR and between 0eb91e0 and 8b203b2.

📒 Files selected for processing (24)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build

Comment thread src/plugins/kv/kv_backend.h Outdated
Comment thread src/plugins/kv/meson.build
Comment thread src/plugins/kv/redis/engine_impl.cpp Outdated
Comment thread src/plugins/kv/redis/engine_impl.cpp Outdated
Comment thread src/plugins/redis/meson.build Outdated
Comment thread src/plugins/kv/redis/meson.build Outdated
@hxieustc

Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/nixlbench/src/utils/meson.build`:
- Around line 28-29: The nixlbench Meson setup currently makes hiredis mandatory
through the utils dependency list, which breaks non-Redis builds. Update the
build so Redis support is optional by gating the Redis helpers behind a Meson
option or isolating them in a Redis-only translation unit, and adjust the setup
in the meson build for the utils target so only Redis-enabled builds require
hiredis while UCX/POSIX/OBJ-only builds still configure cleanly.

In `@src/plugins/kv/kv_plugin.cpp`:
- Around line 23-25: The file-local state in kv_plugin.cpp uses static for
internal linkage, but this codebase prefers an anonymous namespace in .cpp
files. Move supported_segments (and any other file-local symbols like the
redis_plugin_t alias if needed) into an anonymous namespace so the internal
linkage is explicit and consistent with the repo style.

In `@src/plugins/kv/redis/client.cpp`:
- Around line 95-104: The getRedisDB helper currently accepts partially parsed
or negative db values because it relies on std::stoi without verifying the whole
string or enforcing a non-negative range. Update getRedisDB in the Redis client
to validate the db parameter the same way as the port parsing: parse the full
custom_params->at("db") string, reject any trailing characters or malformed
input, and only return the value when it is an integer >= 0. Keep the existing
warning/fallback behavior for invalid input.
- Around line 155-157: The Redis client initialization in client.cpp should fail
fast if libevent threading setup does not succeed. In RedisClient’s constructor
or initialization path around evthread_use_pthreads() and event_base_new(),
check the return value from evthread_use_pthreads() and throw an exception
immediately when it indicates failure instead of proceeding to create
eventBase_. This keeps later cross-thread uses like event_base_once() and
event_base_loopbreak() from running on an unprotected base.

In `@src/plugins/kv/redis/engine_impl.cpp`:
- Around line 154-165: The Redis engine constructors in nixlRedisKVEngineImpl
have an initializer-list wrapping style that does not match clang-format. Update
the constructor definitions for nixlRedisKVEngineImpl so the executor_ and
redisClient_ initializer formatting follows the repository’s formatter output,
keeping the same symbols but adjusting only whitespace/wrapping to satisfy CI
format checks.

In `@src/plugins/kv/redis/engine_impl.h`:
- Around line 88-90: The registration cache in nixlKVEngineImpl is keyed only by
devId, so multiple registerMem calls on the same device can overwrite each other
and break later queryMem or transfer lookups. Update the state in
nixlKVEngineImpl to track each registration by a unique per-registration
identity from the full descriptor instead of just devId, and adjust
registerMem/queryMem/unregister logic to use that identity consistently. Keep
the Redis key mapping aligned with the specific registered region rather than
the device alone.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ab9462d4-7569-442f-b8a8-7d36c2a549e9

📥 Commits

Reviewing files that changed from the base of the PR and between 0eb91e0 and ace488f.

📒 Files selected for processing (27)
  • benchmark/nixlbench/contrib/Dockerfile
  • benchmark/nixlbench/contrib/build.sh
  • benchmark/nixlbench/src/utils/meson.build
  • benchmark/nixlbench/src/utils/utils.cpp
  • benchmark/nixlbench/src/utils/utils.h
  • benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp
  • meson.build
  • src/core/meson.build
  • src/core/nixl_plugin_manager.cpp
  • src/plugins/kv/README.md
  • src/plugins/kv/kv_backend.cpp
  • src/plugins/kv/kv_backend.h
  • src/plugins/kv/kv_plugin.cpp
  • src/plugins/kv/meson.build
  • src/plugins/kv/redis/README.md
  • src/plugins/kv/redis/client.cpp
  • src/plugins/kv/redis/client.h
  • src/plugins/kv/redis/engine_impl.cpp
  • src/plugins/kv/redis/engine_impl.h
  • src/plugins/kv/redis/meson.build
  • src/plugins/kv/redis/redis_engine.cpp
  • src/plugins/kv/redis/redis_engine.h
  • src/plugins/kv/redis/redis_executor.h
  • src/plugins/meson.build
  • test/gtest/unit/meson.build
  • test/gtest/unit/redis/meson.build
  • test/gtest/unit/redis/redis.cpp

Comment thread benchmark/nixlbench/src/utils/meson.build Outdated
Comment thread src/plugins/redis/redis_plugin.cpp
Comment thread src/plugins/redis/redis_client.cpp Outdated
Comment thread src/plugins/redis/redis_client.cpp Outdated
Comment thread src/plugins/kv/redis/engine_impl.cpp Outdated
Comment thread src/plugins/kv/redis/engine_impl.h Outdated
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from 7b40d51 to aca54c0 Compare July 4, 2026 02:07
@gaowayne gaowayne changed the title feat(kv): add shared KV plugin interfaces and REDIS backend with nixlbench support feat(redis): add Redis backend with async transfers and nixlbench support Jul 4, 2026
@hxieustc
hxieustc force-pushed the pr_nixl_kv_plugin_interface_redis branch from 80fd34a to 6e60c0c Compare July 6, 2026 16:31
gaowayne added 2 commits July 16, 2026 06:22
Signed-off-by: Wayne Gao <wayne.gao1@solidigm.com>
Signed-off-by: Wayne Gao <wayne.gao1@solidigm.com>
@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 2a18b64c

TL;DR: The build didn't fail on code — every Kubernetes agent pod for the post-build stages could not pull the freshly pushed CI base images from artifactory.nvidia.com with 401 Unauthorized, so all six matrix branches went ImagePullBackOff and the parallel stage was aborted; the namespace imagePullSecret/robot credential for sw-nbu-swx-nixl-docker-local needs to be refreshed (and the namespace CPU quota was simultaneously pinned at its limit, preventing retries from scheduling).

Full analysis

Summary: nixl-ci-non-gpu #3317 aborted in the "Parallel end" stage after all x86_64/aarch64 agent pods failed with ErrImagePull / ImagePullBackOff on .../ci/{x86_64,aarch64}/nixl-ci-non-gpu-base*:fe640dd8.

Root cause: Two infra conditions, neither in the PR's code:

  1. Registry auth failure on pull. From 06:37 onward, kubelet reported for every pod, e.g.:
    failed to pull and unpack image ".../ci/x86_64/nixl-ci-non-gpu-base:fe640dd8": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized.
    This is a pull-side credential problem, not a missing tag — the "Build Docker" stages (IDs 83/109/114/142/175/255) all succeeded minutes earlier and pushed those exact fe640dd8 tags; stage 255's log even shows a successful docker pull of nixl-ci-non-gpu-base-cuda12.9:fe640dd8 at 06:34:53 from inside the builder, which had working credentials. So the image exists and is pullable with valid creds; the nbu-swx-nixl namespace's image-pull secret is expired/invalid or wasn't projected into the agent pods. Jenkins' generic advice in the log ("Check if image tag name is spelled correctly") is a red herring here.
  2. Namespace CPU quota saturation, which blocked every pod-creation retry:
    pods "nixl-ci-non-gpu-...-3317-..." is forbidden: exceeded quota: resourcequota-nbu-swx-nixl, requested: cpu=8500m, used: cpu=799300m, limited: cpu=800.
    Used was 799300m of an 800-core limit, so the retry loop could not place replacement pods, and the stage ended with Queue task was cancelled → parallel task failed.

Implicated commit: none — commit [REDACTED:Hex High Entropy String] (PR #1791, pr_nixl_kv_plugin_interface_redis) was never compiled or tested; no repository change is implicated. Recent CI-infra commits in the area (e.g. e1fde65 "ci: check out the repository authenticated in every Jenkins job", NirWolfer) touch credential handling but the failing authorization is the kubelet's registry pull, not the pipeline's.

File: not a source defect — the failure is in the Kubernetes pod template / namespace configuration for nbu-swx-nixl (image-pull secret for artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local and resourcequota-nbu-swx-nixl), referenced only via the build log.

Suggested fix:

  1. Rotate/renew the Artifactory robot credential backing the nbu-swx-nixl namespace imagePullSecret, re-create the secret, and confirm the agent pod templates reference it via imagePullSecrets. Verify with a throwaway pod pulling .../ci/x86_64/nixl-ci-non-gpu-base:fe640dd8. (Treat the old credential as compromised/expired and rotate rather than reuse.)
  2. Reap the leaked pods holding 799300m CPU in nbu-swx-nixl — likely orphaned agents from earlier builds that never terminated — and/or raise resourcequota-nbu-swx-nixl, so the pipeline's retry logic can actually reschedule.
  3. Pipeline hardening: make the ImagePullBackOff handler distinguish 401 Unauthorized from a genuinely missing tag and fail fast with an explicit "registry authentication failed" message instead of the misleading "check if image tag name is spelled correctly".
  4. Once both are addressed, simply re-trigger #3317; no change to PR feat(redis): add Redis backend with async transfers and nixlbench support #1791 is required.

Related: none found — searches for the 401/oauth-token and image-pull-secret signatures returned no matching nixl issues or PRs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id d5e81815-5d40-4a98-821d-0a8ce1a55b39 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 2a18b64c

TL;DR: The aarch64 container build died at Dockerfile step 32 because git clone https://github.com/abseil/abseil-cpp.git got an HTTP auth challenge from github.com (could not read Username ... No such device or address) — an unauthenticated-clone rate-limit/credential issue in CI, not a code defect; fix by authenticating (or caching/mirroring + retrying) the third-party clones as proposed in PR #2310/#2263.

Full analysis

Summary: Stage "Build image" (node 191, the arm64 variant of the parallel matrix) failed after 354 s with Error: building at STEP "RUN git clone https://github.com/abseil/abseil-cpp.git ..." : exit status 128; the other three parallel image builds succeeded.

Root cause: At [2/2] STEP 32/73 the build log shows:

Cloning into 'abseil-cpp'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: the remote end hung up unexpectedly
subprocess exited with status 128

git only prompts for a username when github.com answers the anonymous fetch with a 401/auth challenge — i.e. the unauthenticated clone was rejected (rate-limited egress IP / credential-requiring proxy). Inside the build container there is no tty and no credential helper, so git aborts immediately. Nothing in the preceding steps failed: the DOCA 3.5 install and the rdma-core reinstall at steps 28–29 completed normally seconds earlier, and timestamps show continuous activity with no hang (the clone failed ~4 s after starting). This is environmental, not caused by the PR's change set — the abseil-cpp clone long predates this branch, and the branch's only Dockerfile commit (243ef84, Wayne Gao, "build: install Redis deps in contrib Dockerfiles") only adds apt packages (libhiredis-dev/redis-tools, visible installing successfully in the log).

Implicated commit: unknown — no code commit is responsible; the failure is in an unauthenticated github.com clone performed by the pre-existing Dockerfile step.

File: contrib/Dockerfile:170-ish — the RUN git clone https://github.com/abseil/abseil-cpp.git && ... ${ABSL_TAG} step (step 32/73); the same exposure exists at lines 181 (grpc), 204 (etcd-cpp-apiv3), 211 (aws-sdk-cpp), 216 (gusli) and 221 (azure-sdk-for-cpp).

Suggested fix: Short term, re-run the build — the sibling arm64/x86 builds on the same commit passed, so this reproduces only when GitHub rejects the anonymous fetch. Durable fix: land the already-open work that authenticates these clones (PR #2310 / #2263) by passing a CI credential into the build (e.g. a git config --global url."https://${GH_TOKEN}@github.com/".insteadOf "https://github.com/" driven by a BuildKit secret, never an ARG/ENV that bakes into a layer), and add bounded retries around each dependency clone as in PR #2291. Also prefer git clone --depth 1 --branch ${ABSL_TAG} here instead of full clone + git fetch --depth 1 + checkout, which halves the number of network round-trips that can be refused.

Related: PR #2310 (ci: authenticate the Dockerfiles' github.com clones in CI), PR #2263 (ci: authenticate third-party github.com clones), PR #2313 (same for ci-demo base images), PR #2291 (CI: retry git fetches in dependency builds), PR #2078 (nixlbench dockerfile fix for abseil failure).

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 2a18b64c

TL;DR: The build didn't fail on code — the aarch64 build_helper_vllm pod could never start because pulling …/ci/aarch64/build_helper_vllm:fe640dd8 from Artifactory returned 401 Unauthorized, and that aborted the pipeline, killing the still-running x86_64 Docker build with exit 143. Fix the image reference/pull credentials for the aarch64 helper image.

Full analysis

Summary: nixl-ci-build-wheel #1995 aborted because the aarch64 buildhelpervllm agent pod went into ImagePullBackOff on an unauthorized registry pull; the parallel x86_64 "Build Docker" branch was then cancelled mid-run (exit 143).

Root cause: Kubernetes could not pull the agent image for the aarch64 vLLM helper:

failed to pull and unpack image "artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/aarch64/build_helper_vllm:fe640dd8":
failed to resolve reference ...: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
→ ImagePullBackOff → "Unable to pull container image ... Check if image tag name is spelled correctly."

This happened on two successive pods (…-jrqxn, then the retry …-zsllw), so it is deterministic, not a flake. Jenkins exhausted the podTemplate retry, raised error, and that is what produced Calling Pipeline was cancelled → Sending interrupt signal to process → script returned exit code 143 in the x86_64 Build Docker stage (node 288).

The 143 is a symptom, not the cause, and it is not a hang: that stage logged continuously up to the kill — Downloaded torch at 06:59:12.927 and Installed 28 packages immediately before the interrupt at 06:59:21.256, with no gap larger than ~30 s anywhere in the stage. Raising the timeout would not help; the branch was externally cancelled by the sibling failure.

Two candidate reasons for the 401, both consistent with the log: (a) the tag fe640dd8 does not exist under the ci/aarch64/ path (Artifactory returns 401 rather than 404 for a path the pull secret can't read), or (b) the namespace imagePullSecret for nbu-swx-nixl is stale/missing for that repo. Note that (a) is made more likely by a Groovy warning emitted in the same build, immediately before the aarch64 podTemplate:

Did you forget the `def` keyword? Matrix seems to be setting a field named image
(to a value of type LinkedHashMap) which could lead to memory leaks or other issues.

An undeclared image variable shared across matrix branches can leak one architecture's resolved tag into another branch's pod spec, which would explain an x86-built digest being requested from the aarch64 repository path.

Implicated commit: unknown — the failing reference lives in the Jenkins matrix/podTemplate definition, which I could not read from the repo (.ci/jenkins/pipeline/* paths all returned not-found). Recent CI-image churn worth bisecting: [REDACTED:Hex High Entropy String] "build: bump CUDA and CI base images, stop restating them across CI" (NirWolfer) and [REDACTED:Hex High Entropy String] "nixl_ep: Test vLLM Elastic EP without patching vLLM" (Itay Alroy), which touched the vLLM helper Dockerfiles.

File: the Jenkins matrix podTemplate entry for container buildhelpervllm (image …/ci/aarch64/build_helper_vllm:fe640dd8); the unscoped image assignment flagged by the Groovy warning is in the same matrix block.

Suggested fix:

  1. Verify the tag actually exists: docker manifest inspect artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/aarch64/build_helper_vllm:fe640dd8. If absent, the aarch64 helper was never published for this tag — build/push it, or make the wheel matrix fall back to building the helper rather than assuming a prebuilt tag.
  2. Add def to the matrix-local image variable so per-architecture tags cannot leak between parallel branches (this is the likely mechanism for requesting an x86 tag from the aarch64 path).
  3. If the tag does exist, re-create/renew the imagePullSecret used by the nbu-swx-nixl namespace for sw-nbu-swx-nixl-docker-local and confirm the service account has read permission on the ci/aarch64/* path — the token endpoint is rejecting the credential outright. Rotate that registry credential if it has expired.
  4. Independently, make the matrix failFast/cancellation behaviour explicit so an agent-provisioning failure in one branch reports as an infra error instead of masquerading as an exit-143 build failure in a sibling branch.

Related: #2237 "ci: make wheel builds DR-safe" and #2310 "ci: authenticate the Dockerfiles' github.com clones in CI" both touch wheel-build image/credential handling and may already address or conflict with this.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b7e122ee-724d-4971-98c3-1187d0964962 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The build aborted because Kubernetes could not pull build_helper:fe640dd8 — kubelet got 401 Unauthorized from artifactory using the namespace's artifactory-pull-secret, even though the image exists and podman pulled it fine with the Jenkins credential. The fix is to refresh/recreate the artifactory-pull-secret imagePullSecret in the nbu-swx-nixl namespace; nothing in PR #1791 is implicated.

Full analysis

Summary: The build_helper/x86_64/1 test axis failed to start — its agent pod stayed in ErrImagePull/ImagePullBackOff for artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/x86_64/build_helper:fe640dd8 through two retries, the queue task was cancelled and the whole build finished ABORTED.

Root cause: Registry credential mismatch between the two auth paths, not a code defect and not a hang:

  • Inside the build pods, podman authenticated with the Jenkins credential svc-nixl-new-artifactory-token ("Login Succeeded!") and successfully pulled both nixl-ci-gpu-base:fe640dd8 and build_helper:fe640dd8 (Writing manifest to image destination / dd74bfcc43d7... at 07:01:22). The docker manifest inspect calls also returned the full OCI manifest for both tags, so the images unambiguously exist in artifactory with the correct fe640dd8 tag.
  • When Jenkins then created the test-stage pod nixl-ci-gpu-buildhelper-3845-*, kubelet pulled the same tag using the pod's imagePullSecrets: ['artifactory-pull-secret'] and got: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized.

Same image, same tag, same registry — succeeds with the Jenkins credential, fails with the k8s pull secret. That isolates the fault to the artifactory-pull-secret Secret in namespace nbu-swx-nixl holding stale, expired, or wrongly-scoped credentials. The repeated backoff then exhausted the retry wrapper ("Unable to pull container image ... Check if image tag name is spelled correctly" — a misleading message here, since the tag is correct). The run was killed by cancellation at 07:09, with no multi-minute silent gaps — it died on auth retries, not a timeout.

Note that the pipeline never reached Run CPP tests/Run Python tests, so none of the Redis-plugin changes in this PR were ever executed; this failure is unrelated to the commit under test and will reproduce on any branch.

Implicated commit: unknown — no code commit is implicated; the failure is in the Kubernetes registry pull secret, which lives outside the repo.

File: .ci/jenkins/lib/test-matrix.yaml:34 (imagePullSecrets: "['artifactory-pull-secret']") — the reference is correct; the referenced Secret's contents are the problem.

Suggested fix:

  1. Recreate the pull secret in the CI namespace with current service-account credentials (do not reuse any value copied out of a log):
    kubectl -n nbu-swx-nixl delete secret artifactory-pull-secret then recreate it as a docker-registry secret for server https://artifactory.nvidia.com using the same svc-nixl identity that the Jenkins credential svc-nixl-new-artifactory-token wraps.
  2. If that Jenkins credential was itself rotated recently, the old token is now live only in the k8s Secret — rotate the artifactory token for svc-nixl and update both places together so they cannot drift again.
  3. Hardening: have the pipeline create/refresh the namespace pull secret from the svc-nixl-new-artifactory-token Jenkins credential at the start of the run, so the pod-level and podman-level auth can never diverge. This class of failure has bitten the job before — the "Unable to pull container image ... Check if image tag name is spelled correctly" message should also be treated as a known red herring when the preceding docker pull in the same build succeeded.
  4. Re-run build #3845 once the secret is fixed to get real signal on PR feat(redis): add Redis backend with async transfers and nixlbench support #1791.

Related: none
`

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu-vr · commit 2a18b64c

TL;DR: The build aborted because Kubernetes could not pull the build_helper_dl:fe640dd8 agent image — the namespace's artifactory-pull-secret gets a 401 Unauthorized from artifactory.nvidia.com — not because of anything in PR #1791. Fix by refreshing/rotating the artifactory-pull-secret docker-registry secret in the nbu-swx-nixl namespace.

Full analysis

Summary: Stage "Parallel" (branch build_helper_dl/aarch64/1) aborted: every agent pod for the test steps went ErrImagePull → ImagePullBackOff pulling artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/aarch64/build_helper_dl:fe640dd8, and after retries Jenkins cancelled the queue task ("Queue task was cancelled", Finished: ABORTED).

Root cause: Kubelet-side registry authentication failure, not a missing image or a code defect:

  • Pod events repeat: failed to resolve reference ".../build_helper_dl:fe640dd8": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized (first at 07:07:30, repeated through 07:09:53 across three pods: ...fjx30-tc1qc, ...fjx30-hzhwk, ...29v4j-mkmtc/-cwtph).
  • The same tag demonstrably exists and is pullable with the pipeline's own credentials: in the earlier "Build Docker" stage docker manifest inspect .../build_helper_dl:fe640dd8 returned a full OCI manifest, podman search --list-tags | grep fe640dd8 matched, and docker pull of that exact tag succeeded at 07:01:23 after docker login -u svc-nixl to artifactory.nvidia.com ("Login Succeeded!").
  • The difference between the working pull and the failing one is who authenticates: the stage pull uses the Jenkins credential svc-nixl-new-artifactory-token passed explicitly, while the agent pod relies on imagePullSecrets: "['artifactory-pull-secret']" (test-vr-matrix.yaml:42). So the Kubernetes secret artifactory-pull-secret in namespace nbu-swx-nixl holds expired/invalid credentials. Jenkins' trailing message "Check if image tag name is spelled correctly" is misleading here — the tag is correct.
  • There is no sign of a hang: log timestamps are continuous (≤30 s gaps) from 07:00:42 to the abort at 07:09:54; the job ended on exhausted image-pull retries, not a wall-clock kill. The PR-changed files (redis plugin, nixlbench utils, gtest) were never compiled or executed.

Implicated commit: None in the PR. The job definition itself is from 2dd278bf (NirWolfer, 2026-09-30, "CI: add nixl-ci-gpu-vr pipeline on the Vera Rubin partition (#2182)"), but the defect is in the cluster-side pull secret, not in that commit.

File: .ci/jenkins/lib/test-vr-matrix.yaml:42 (imagePullSecrets: "['artifactory-pull-secret']"), with the working alternative path at lines 36 and 47-48 (registry_auth / svc-nixl-new-artifactory-token)

Suggested fix:

  1. Primary (infra): regenerate the artifactory-pull-secret docker-registry secret in the nbu-swx-nixl namespace with a current svc-nixl Artifactory identity token — the existing one is rejected with 401 at the oauth-token endpoint. Rotate the old token while doing so. Then re-run build ci: Added build and test scripts for CI. #64. The same secret is shared by nixl-ci-dl-gpu, so expect those jobs to be broken identically until it is refreshed.
  2. Hardening (optional): the Jenkins-side podman pull with --creds works, so consider having the pipeline verify artifactory-pull-secret validity (or pre-pull/warm the agent image using the Jenkins credential) before scheduling agent pods, so this surfaces as a clear auth error instead of a 2-minute ImagePullBackOff loop that ends in a generic ABORTED.
  3. Also worth filing: make the retry wrapper surface the underlying 401 Unauthorized in the failure message rather than "Check if image tag name is spelled correctly", which sends triage down the wrong path.

Related: none

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 2a18b64c

TL;DR: The build aborted because Kubernetes could not pull build_helper_dl:fe640dd8 — kubelet got 401 Unauthorized from artifactory while using the namespace imagePullSecret, even though podman had just pulled the same image successfully with the Jenkins credential. The artifactory-pull-secret in namespace nbu-swx-nixl is expired/invalid and needs to be refreshed; this is not a defect in PR #1791.

Full analysis

Summary: Stage "Parallel" → branch build_helper_dl/aarch64/1 failed with ImagePullBackOff / ErrImagePull on artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/aarch64/build_helper_dl:fe640dd8, both retries exhausted, pipeline finished ABORTED.

Root cause: Registry authentication at the kubelet level, not a missing image and not a code change.

  • At 07:00:49–07:01:06 the "Build Docker" stage pulled that exact tag successfully: docker pull …/build_helper_dl:fe640dd8 → Writing manifest to image destination / e5bcd63e…, after docker login -u svc-nixl … Login Succeeded! using the Jenkins credential svc-nixl-new-artifactory-token. So the image and tag both exist and are readable.
  • Starting 07:07:31, the next parallel block creates pod nixl-ci-dl-gpu-buildhelperdl-2544-*, where containerd pulls the same reference using the pod's imagePullSecrets: ['artifactory-pull-secret'] and gets: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized.
  • Jenkins retried on three fresh pods (…-jd982, …-j038v, …-czqx6/…-j9h5b); every attempt returned the identical 401, then Queue task was cancelled → FlowInterruptedException → ABORTED.

The divergence between the working in-stage credential and the failing pod-level secret isolates the fault to the Kubernetes pull secret artifactory-pull-secret in namespace nbu-swx-nixl. The trailing Jenkins message "Check if image tag name is spelled correctly" is a generic hint from the k8s plugin and is misleading here — the tag is correct.

Note this is also not a wall-clock/hang issue: the log shows continuous activity, with the failure arriving as an explicit auth rejection ~90s after each pod was created.

Implicated commit: unknown — no implicated code commit. The diff for this PR (src/plugins/redis/*, benchmark/nixlbench/*, test/gtest/unit/redis/*) is unrelated to image pulling, and CI_IMAGE_TAG was auto-derived to fe640dd8 and verified present in the registry. The most recent change to the matrix file is [REDACTED:Hex High Entropy String] (NirWolfer, 2026-09-10), which predates this failure and did not alter the pull secret.

File: .ci/jenkins/lib/test-dl-matrix.yaml:34 (imagePullSecrets: "['artifactory-pull-secret']")

Suggested fix: Have CI/infra rotate and re-apply the artifactory-pull-secret docker-registry secret in the nbu-swx-nixl namespace with a currently valid svc-nixl artifactory token (the same identity that succeeds for svc-nixl-new-artifactory-token), then re-run build #2544. Do not change the image tag or the matrix YAML. Verify with a throwaway pod pulling …/ci/aarch64/build_helper_dl:fe640dd8 using only that secret. Longer term, two hardening items are worth filing: (1) align the pod imagePullSecrets identity with the Jenkins credential so one expiring token cannot pass the build stage and fail the test stage, and (2) make the k8s plugin's "Unable to pull container image" path surface the underlying 401 as a credential error rather than suggesting a tag typo, since the current wording sends triage down the wrong path.

Related: none — the issue search for the pull/auth signature returned no matching reports, so this appears to be the first occurrence.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 80c4a0ae-dc30-4263-b27d-deb06e6c728d in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 2a18b64c

TL;DR: The build is not failing on the PR's code — every test pod died in ImagePullBackOff because the Kubernetes artifactory-pull-secret in namespace nbu-swx-nixl gets a 401 Unauthorized from artifactory.nvidia.com, so the credential behind that secret needs to be refreshed/rotated and the build re-run.

Full analysis

Summary: All six matrix branches of nixl-ci-non-gpu #3318 aborted during the "Parallel" test stage because the agent pods could not pull their base images (ErrImagePull → ImagePullBackOff → "Queue task was cancelled").

Root cause: Registry authentication failure at the kubelet level, not a missing or mis-tagged image. In the build log:

  • The "Build Docker" stages all succeeded: on the agents, docker login -u svc-nixl ... artifactory.nvidia.com returned Login Succeeded!, the manifests resolved, and docker pull ...:fe640dd8 completed for all six images (Writing manifest to image destination, e.g. 07:06:58, 07:10:38, 07:11:06). So tag fe640dd8 does exist in sw-nbu-swx-nixl-docker-local/ci/{x86_64,aarch64}/... for every variant.
  • Immediately afterwards, when Kubernetes pulled the same images for the test pods, every pod reported:
    failed to resolve reference ".../nixl-ci-non-gpu-base...:fe640dd8": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
  • This hit all 6 pods, both architectures, all three image variants, and persisted across all retries — a uniform auth failure, which rules out a per-image or per-arch problem.

The difference between the two paths is the credential: the pipeline shell steps use the Jenkins credential svc-nixl-new-artifactory-token (registry_auth in build-matrix.yaml:27), while the pods use the k8s imagePullSecrets: ['artifactory-pull-secret'] (build-matrix.yaml:38). The former works, the latter is rejected with 401 — i.e. the token stored in the artifactory-pull-secret docker-registry secret is expired/revoked or out of sync with the Jenkins credential. Jenkins' own message "Check if image tag name is spelled correctly" is misleading here; the underlying containerd error is explicitly an authorization failure.

Nothing in this PR (redis KV plugin: src/plugins/redis/*, test/gtest/unit/redis/*) is reachable from this failure — the pipeline never got as far as running .gitlab/build.sh.

Implicated commit: none in the repo — this is CI infrastructure credential state, not a code change. (Commit [REDACTED:Hex High Entropy String] / the redis plugin PR is unrelated to the failure.) The secret-vs-credential split dates to 64e83149 "CI: Replace Harbor with Artifactory for Docker images" (#1688, Noam Tsemah).

File: .ci/jenkins/lib/build-matrix.yaml:38 (imagePullSecrets: "['artifactory-pull-secret']"), paired with :27 (registry_auth: svc-nixl-new-artifactory-token)

Suggested fix:

  1. Have CI ops regenerate the artifactory-pull-secret docker-registry secret in the nbu-swx-nixl namespace on il-ipp-blossom-prod using a current, non-expired svc-nixl Artifactory token — the same identity as the Jenkins svc-nixl-new-artifactory-token credential — and rotate the old token since it is no longer accepted. Verify with a throwaway pod pulling artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/x86_64/nixl-ci-non-gpu-base:fe640dd8.
  2. Re-trigger build #3318 / the PR CI afterwards; no code change is required.
  3. Optional hardening: have the pipeline fail fast with a clear message when a pod's pull error contains 401 Unauthorized/failed to authorize, instead of emitting "Check if image tag name is spelled correctly", which sends triage down the wrong path. Also consider deriving the pull secret from the same Jenkins credential at pod-template creation time so the two can't drift apart.

Related: #1688 (Harbor → Artifactory migration, introduced this auth split); #2317 (ci: Move sanitizer base images from Harbor to Artifactory — same registry/credential area)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 66bed62b-fcea-4c41-8669-2e4af64d87ee in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 2a18b64c

TL;DR: The build aborted because Kubernetes could not pull build_helper_dl_ep:fe640dd8 — the kubelet's artifactory-pull-secret returned 401 Unauthorized — even though the same image had just been pulled successfully inside the pipeline using the svc-nixl-new-artifactory-token credential. The fix is to refresh/rotate the expired artifactory-pull-secret in the nbu-swx-nixl namespace; nothing in PR #1791 is at fault.

Full analysis

Summary: Stage "Parallel" (branch build_helper_dl_ep/aarch64/1) failed with ImagePullBackOff / Unable to pull container image artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/aarch64/build_helper_dl_ep:fe640dd8, which cancelled the queued task and aborted the build.

Root cause: Registry authentication failure at the Kubernetes layer, not a missing image or a code defect. Evidence from the build log:

  • 07:00:56 podman search --list-tags ... build_helper_dl_ep | grep fe640dd8 succeeded and 07:01:15 docker pull .../build_helper_dl_ep:fe640dd8 completed (Writing manifest to image destination, config sha256:821aaeb7…) — the tag demonstrably exists in Artifactory. That pull used the Jenkins credential svc-nixl-new-artifactory-token after Login Succeeded!.
  • 07:13:31 onward, when the matrix step asked k8s to schedule the test pod, the kubelet reported: failed to resolve reference ".../build_helper_dl_ep:fe640dd8": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized. That path uses imagePullSecrets: ['artifactory-pull-secret'] (test-dl-ep-matrix.yaml:40), a different, namespace-scoped credential.
  • All three retries (pods …8z4md-2msff, …8z4md-djs4l, …f644n-4bc1p) hit the identical 401, so this is deterministic, not a transient registry blip. The "Check if image tag name is spelled correctly" message is Jenkins' generic text and is misleading here — the tag is correct.

So the artifactory-pull-secret dockerconfigjson in namespace nbu-swx-nixl holds expired or revoked credentials, while the Jenkins-side token is still valid.

Implicated commit: unknown — no repo commit is implicated; the image tag fe640dd8 derived by .ci/cidemo-init.sh resolved correctly and the image was present in Artifactory.

File: .ci/jenkins/lib/test-dl-ep-matrix.yaml:40 (imagePullSecrets: "['artifactory-pull-secret']") — the config that selects the failing credential, not itself defective.

Suggested fix:

  1. Rotate/refresh the artifactory-pull-secret Kubernetes secret in namespace nbu-swx-nixl on cloud il-ipp-blossom-prod so it carries a current Artifactory token (the same service account whose Jenkins credential svc-nixl-new-artifactory-token is still working). Verify with kubectl -n nbu-swx-nixl get secret artifactory-pull-secret -o jsonpath='{.metadata.creationTimestamp}' and a test pod pulling that exact tag — do not print the secret contents.
  2. Because the token appears compromised/expired at the registry side, treat the old value as dead and revoke it rather than reusing it.
  3. Then re-run nixl-ci-dl-gpu-ep test: fix maybe-uninitialized warning in metadata_exchange test #1473 for PR feat(redis): add Redis backend with async transfers and nixlbench support #1791 — no code change is needed.
  4. Longer term, make this class of failure self-evident: the retry wrapper currently burns ~2.5 minutes and reports "check if image tag name is spelled correctly" for an auth error. Consider failing fast on 401 Unauthorized with an explicit "pull secret rejected" message, since repeated retries cannot fix a bad credential. The same artifactory-pull-secret is referenced by the sibling matrices, so other jobs (nixl-ci-dl-gpu, nixl-ci-gpu, etc.) are likely failing identically — worth checking before declaring the incident closed.

Related: none found — searches for the 401 / ImagePullBackOff signature in the nixl repo returned no matching issues or PRs.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The "Run CPP tests" stage failed because enroot/pyxis on compute node mizu01 could not pull the freshly-pushed test image — Artifactory returned 401 Unauthorized for the svc-nixl token request — so the container never started. This is a CI credential/infra problem, not a defect in PR #1791.

Full analysis

Summary: Stage Run CPP tests (node 188) of nixl-ci-gpu #3846 aborted ~6 s after launch: slurmstepd: error: pyxis: failed to import docker image / [ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized, then srun: error: mizu01: task 0: Exited with exit code 1.

Root cause: The image .../ci/pr/x86_64/nixl-ci-gpu-test:3846 was built and pushed successfully in the preceding "Compiling NIXL Docker Image" stage (the log ends with Writing manifest to image destination + break after podman push), so the registry and the image itself are fine. The failure happens on the pull side inside the slurm step: enroot authenticated as user svc-nixl using the credentials file at /labhome/svc-nixl/.config/enroot/.credentials on mizu01, and Artifactory rejected the token request with 401. In other words the enroot credential on the compute node is invalid/expired/out of sync with the credential Jenkins used to push. Everything after that is downstream noise: because the container was never created, the follow-up enroot export failed with No such file or directory: .../pyxis_nixl-ci-3846, and the allocation ended as CANCELLED when the pipeline stopped (slurm job_id 100446, name nixl-ci-gpu-3846, node mizu01, terminal row CANCELLED at 08:20:38 after starting 08:19:37 — consistent with pipeline teardown, not a node fault). No timestamp gap exceeds a few seconds, so nothing hung and no time limit was involved. Nothing in the diff for commit 2a18b64 (Redis KV plugin interface) is reachable from this failure — the build, link and install of all plugins including libplugin_REDIS.so completed cleanly.

Implicated commit: none — not a code regression. The PR head commit [REDACTED:Hex High Entropy String] is not implicated; the failing step is image pull, before any test binary ran.

File: not a source file — the failing operation is the pyxis/enroot image import in the slurm.run step invoked from the Run CPP tests stage (.gitlab/test_cpp.sh was never reached). The offending artifact is the enroot credentials file on mizu01 at /labhome/svc-nixl/.config/enroot/.credentials.

Suggested fix:

  1. Rotate/refresh the svc-nixl Artifactory credential and re-provision the enroot credentials file on all mizu* nodes (shared /labhome/svc-nixl/.config/enroot/.credentials). Verify with enroot import docker://artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/x86_64/nixl-ci-gpu-test:3846 as svc-nixl on mizu01. Treat the current credential as compromised/expired and rotate rather than reuse.
  2. Confirm the push-side credential (credentialsId: svc-nixl-scctl used by the Jenkins agent) and the pull-side enroot credential refer to the same, still-valid Artifactory identity — the asymmetry (push OK, pull 401) is the signature of them having drifted apart.
  3. CI hardening: make the pyxis import failure retry a couple of times and emit an explicit "infra: image pull failed" marker, so this class of failure isn't attributed to the PR under test. Also guard the exportEnroot step to skip (rather than log a confusing secondary error) when the container was never created.
  4. Re-run build #3846 once credentials are fixed to get a real verdict on PR feat(redis): add Redis backend with async transfers and nixlbench support #1791.

Related: none found — searches for the pyxis/401 signature in this repo's issues and PRs returned only unrelated matches, and recent history on .gitlab/test_cpp.sh (latest adf179a5, Lukas Humbel) is unrelated to image pulls.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b299fbd4-ad85-4efa-b1f7-e1b9a7510a1f in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 2a18b64c

TL;DR: Both the vLLM and SGLang sanity stages failed because enroot/pyxis on the DL cluster could not authenticate to artifactory.nvidia.com (/v2/token → 401 Unauthorized), so the just-pushed PR images could not be imported — this is an expired/invalid registry credential on the cluster side, not a code defect in PR #1791.

Full analysis

Summary: Run vLLM sanity (stage 638) and Run SGLang sanity (stage 576) both failed within ~70s of each other with pyxis: failed to import docker image / [ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized.

Root cause: Infrastructure/credential failure, not a build or test failure. The pipeline successfully built and pushed both images to Artifactory from the Jenkins agent (stage 511 ends with Writing manifest to image destination … + break, i.e. push succeeded), so the images exist. The failure happens later, on the slurm side, when srun --container-image=artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/{vllm,sglang}-nixl:1997 asks enroot to pull:

[INFO] Authenticating with user: svc-nixl
[INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
error: pyxis: couldn't start container
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
srun: error: gb-nvl-081-compute08: task 0: Exited with exit code 1   (job 2372455, sglang)
srun: error: gb-nvl-147-compute07: task 0: Exited with exit code 1   (job 2372476, vllm)

The svc-nixl enroot credentials file on the cluster head/compute nodes holds a stale or rotated Artifactory API token. Two independent slurm allocations on two different physical nodes (gb-nvl-081-compute08, gb-nvl-147-compute07) failed identically and immediately (~0.4s after the import started), which rules out a per-node or transient-network cause and points at the shared ~svc-nixl/.config/enroot/.credentials on the shared home filesystem. Nothing in the PR branch (pr_nixl_kv_plugin_interface_redis, Redis KV plugin interface) touches the registry, the Dockerfiles, or the sanity scripts — all build, wheel, and sanity-image stages passed.

Implicated commit: unknown — no repository commit is implicated; the failure is in the cluster-side Artifactory credential for the svc-nixl service account. (Commit [REDACTED:Hex High Entropy String] on PR #1791 is not the cause.)

File: /home/svc-nixl/.config/enroot/.credentials on dlcluster.nvidia.com (consumed via the slurm.run step; invoked from .ci/jenkins pipeline → .gitlab/test_vllm_sglang_sanity.sh)

Suggested fix:

  1. Regenerate the Artifactory identity token for the svc-nixl service account and refresh the enroot credentials file on the DL cluster (machine artifactory.nvidia.com login svc-nixl password <new-token>); verify with a standalone srun --container-image=.../ci/pr/aarch64/vllm-nixl:1997 true. Treat the old token as compromised/expired and rotate rather than reuse. Also confirm the svc-nixl account still has read permission on sw-nbu-swx-nixl-docker-local.
  2. Then re-run build ci: fix build-container push step auth and silent curl failure #1997 (or push a no-op to PR feat(redis): add Redis backend with async transfers and nixlbench support #1791) — no code change is needed.
  3. Hardening, so this is diagnosed in one line instead of per-PR triage: have the Allocate Environment / slurm.run wrapper do a cheap pre-flight registry auth check (e.g. enroot import -o /dev/null of a tiny image, or a curl -sf https://artifactory.nvidia.com/v2/token probe) and fail with an explicit "cluster Artifactory credentials invalid — infra issue" message, plus distinguish this exit path from a genuine sanity-test failure so the PR isn't blamed.

Related: PR #1791 (the branch under test — not the cause). No existing issue tracks this 401; worth filing an infra ticket for svc-nixl enroot credential rotation on dlcluster.nvidia.com. Nearest CI context: ad661a72 / b594eb0b (NirWolfer) which added and reshaped the vLLM/SGLang sanity allocation, and d6187c6e "CI: Suppress credentials in logs".

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 69b11bbb-90ee-4589-b9c4-95270909554c in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu-vr · commit 2a18b64c

TL;DR: The "Run Python tests" stage failed because enroot/pyxis on the allocated slurm node could not pull the test image — Artifactory returned 401 Unauthorized for the svc-nixl enroot credentials — so the container never started and no test ever ran. Fix is to refresh the enroot credentials file for svc-nixl on the DL cluster (the registry token/identity token it holds has expired), not anything in the PR.

Full analysis

Summary: Stage 188 "Run Python tests" (nixl-ci-gpu-vr #65) aborted after 5s: srun/pyxis failed to import nixl-ci-gpu-vr-test:65 from Artifactory with HTTP 401, so .gitlab/test_python.sh never executed.

Root cause: Registry authentication failure on the compute node, not a code defect. Build stages succeeded: the image compiled fully (stage 153 ends with Successfully tagged artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-gpu-vr-test:65 and a successful podman push using the Jenkins REPO_USER/REPO_PASS credentials), and the slurm allocation succeeded (stage 168: job 2372494 on vr-nvl72-ts3-l06-comp141). The test stage then ran srun --container-image="artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-gpu-vr-test:65", and enroot logged:

[INFO] Authenticating with user: svc-nixl
[INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
error: pyxis: failed to import docker image
srun: error: vr-nvl72-ts3-l06-comp141: task 0: Exited with exit code 1

So the push side credential works but the pull side credential — the separate enroot .credentials file in svc-nixl's home on the DL cluster — is rejected by Artifactory. This is a stale/expired Artifactory identity token (or a rotated svc-nixl password that was never propagated to that file), entirely independent of commit [REDACTED:Hex High Entropy String] / PR #1791. There is no hang: the whole stage is 5.4s with no gap, and the Parallel wrapper's 1032s is just the sibling image-build branch.

Implicated commit: unknown — no repository commit is implicated; the failure is in cluster-side registry credentials, not in the tree.

File: /home/svc-nixl/.config/enroot/.credentials on dlcluster.nvidia.com (consumed by the srun --container-image invocation issued from the slurm.run step; the stage's script entrypoint is .gitlab/test_python.sh)

Suggested fix:

  1. Regenerate the Artifactory identity token for the svc-nixl service account and rewrite the enroot credentials file on the DL cluster (machine artifactory.nvidia.com login svc-nixl password <new-token>), then verify with a bare srun --container-image=.../nixl-ci-gpu-vr-test:65 true. Rotate (do not reuse) the existing token, since the old one is invalid anyway.
  2. Longer-term, stop depending on a hand-maintained file on the cluster: have the pipeline inject the same Jenkins credential it already uses for podman push into the srun environment (ENROOT_CREDENTIALS / write a per-job credentials file in the job's scratch dir) so push and pull auth can never drift apart.
  3. Optionally make the slurm.run wrapper retry and surface pyxis/enroot auth errors as an explicit infra failure so these don't get triaged as PR regressions.

Related: none — searching the project for this error signature returned only unrelated PRs (#2230, #2211, #401); recent history on .gitlab/test_python.sh (latest aed5ef2e, 54a015fe) is unrelated to container image authentication.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 85858f2b-4726-41a8-a4bf-52b7c77d678a in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 2a18b64c

TL;DR: The "Run DL Python tests" stage failed because enroot/pyxis on the Slurm compute node could not authenticate to artifactory — the registry token endpoint returned 401 Unauthorized — so the freshly pushed test image could never be imported. This is a CI credential/infra failure unrelated to PR #1791; the enroot credentials for the svc-nixl account on dlcluster need to be refreshed.

Full analysis

Summary: Stage 188 "Run DL Python tests" failed after 5s: srun could not start the pyxis container because the docker image import was rejected with HTTP 401 from the artifactory token endpoint.

Root cause: The build pipeline succeeded end-to-end — the DL image compiled (stage 153, 936s) and podman push of artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-dl-gpu-test:2545 completed at 08:31:53 ("Writing manifest to image destination"). 45 seconds later the test stage ran srun ... --container-image="artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/.../nixl-ci-dl-gpu-test:2545" on gb-nvl-081-compute05, and enroot's log shows:

[INFO] Authenticating with user: svc-nixl
[INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
error: pyxis: failed to import docker image
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
srun: error: gb-nvl-081-compute05: task 0: Exited with exit code 1

A 401 at /v2/token means the credentials themselves were rejected, not that the image was missing (that would be a 404/manifest-unknown). The Jenkins agent's push credentials (REPO_USER/REPO_PASS) are clearly valid — the push worked — so the discrepancy is between those and the separate enroot credentials file in the svc-nixl home directory on the DL cluster head/compute nodes. There is no gap in the timestamps and no application code was ever reached, so this is not a hang, a timeout, or a test failure. Nothing in the changed source (redis KV plugin interface) is involved: libplugin_REDIS.so built and linked cleanly at step [102/296].

Implicated commit: unknown — not a source regression; the commit under test ([REDACTED:Hex High Entropy String], branch pr_nixl_kv_plugin_interface_redis) built and installed successfully.

File: CI infrastructure, not repo source — /home/svc-nixl/.config/enroot/.credentials on dlcluster.nvidia.com / gb-nvl-081-compute05; invocation site is the slurm.run step in the shared swx-jenkins-lib pipeline library (stage 188).

Suggested fix:

  1. Have CI ops refresh the artifactory credential stored in the enroot credentials file for svc-nixl on the DL cluster (head node plus compute node images). The token/identity token there has most likely expired or been rotated out of sync with the Jenkins svc-nixl registry credential. Treat the old value as compromised/stale and rotate it rather than reusing it.
  2. Verify both halves use the same secret: the Jenkins push uses ${REPO_USER}:${REPO_PASS}, while the srun side reads a file on the cluster. Sourcing both from the same Jenkins credential (e.g. writing the credentials file as part of the "Allocate DL Environment" stage) removes this drift permanently.
  3. Add a bounded retry with backoff around the srun --container-image import — the push loop already retries 6 times, but the pull side has none, so a transient registry 401/5xx fails the whole build immediately.
  4. Re-run build #2545 once credentials are refreshed; no code change is needed on PR feat(redis): add Redis backend with async transfers and nixlbench support #1791.

Related: none found — searches for the pyxis/enroot 401 signature returned no matching issues or PRs in this repository.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ab71224b-6f98-4590-8217-0665f870ae90 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 2a18b64c

TL;DR: This build wasn't a real failure — all four Build image branches were externally aborted mid-compile by Jenkins (Calling Pipeline was cancelled → SIGTERM → exit 143), almost certainly because the parent/triggering pipeline was superseded or cancelled while build #917 was still in flight. Re-run the build on commit 2a18b64; there is no code defect to fix.

Full analysis

Summary: nixl-ci-build-container-pr #917 ended in ABORTED — all 4 parallel Build image stages (nixl/nixlbench × x86_64/aarch64) were killed simultaneously with exit code 143.

Root cause: External cancellation, not a build error. The whole-build log shows Calling Pipeline was cancelled (a FlowInterruptedException raised by the calling pipeline) immediately followed by Sending interrupt signal to process / Killing processes on every branch, and the build result is Finished: ABORTED rather than FAILURE.

Evidence this is a symptom of an abort and not a hang or a genuine timeout:

  • Timestamps are continuous right up to the kill, with no gap larger than a few seconds. The x86_64 nixl branch logged Resolved 31 packages / Building nixl-cu13 @ file:///workspace/nixl at 08:34:11 and was still inside a normal ~90 s wheel build when killed at 08:35:14 (the sibling aarch64 branch completed that exact same step in 1 m 33 s successfully at 08:34:20). The aarch64 nixl branch was actively emitting ninja progress — [258/309] Compiling C++ object test/gtest/mocks/libplugin_MOCK_BACKEND.so.p/gmock_engine.cpp.o — one second before the interrupt.
  • All four independent branches died within the same ~1.5 s window (08:35:13–08:35:15). Four unrelated builds on different architectures cannot fail coincidentally; a single external signal killed them all.
  • No Cancelling nested steps due to timeout message appears, which is what the timeout wrapper would emit — so the pipeline's own timeout did not fire either.
  • Everything up to the abort was healthy: meson configure succeeded, all 309/191 ninja targets were compiling cleanly, the cuObject/libplugin_OBJ.so link check passed, and the Rust bindings built in 13.62 s.

The trigger for this build was poll (SCM polling). The most likely concrete cause is that a newer revision appeared on pr_nixl_kv_plugin_interface_redis (PR #1791) — or the upstream parent job was cancelled/aborted — and Jenkins killed the in-flight build for the superseded revision.

Implicated commit: none — no commit is implicated; [REDACTED:Hex High Entropy String] was building cleanly when aborted.

File: n/a — no source file is implicated.

Suggested fix: Treat #917 as a non-result and re-trigger the build on the current head of pr_nixl_kv_plugin_interface_redis. If this recurs often enough to be noisy, address it at the pipeline level rather than in code:

  • Confirm whether a newer build (updated headers (#903) #918+) exists for the same PR; if so, this abort is expected supersede behaviour and the only change worth making is to stop reporting superseded builds as CI failures (e.g. check currentBuild.result == 'ABORTED' plus the FlowInterruptedException cause and skip failure notification).
  • Add disableConcurrentBuilds(abortPrevious: true) (or the milestone step) explicitly to the PR job so supersede is deliberate and clearly labelled, instead of surfacing as a bare exit 143.
  • Note for cost: these builds burn ~37 min re-downloading and rebuilding every dependency from scratch (libfabric, gusli, liburing, taskflow, prometheus-cpp, asio, tomlplusplus, plus a 528 MiB torch download) before reaching NIXL itself. Caching those layers would make supersede-aborts far less expensive.

Related: PR #1791 (the PR under test). No existing issue matches this abort signature; searches for the cancellation/143 signature returned only unrelated CI PRs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c2f557cd-0f31-4f7b-99de-d46c3a083528 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 2a18b64c

TL;DR: Build #1998 wasn't a test or compile failure — the whole pipeline was cancelled from outside ("Calling Pipeline was cancelled"), which SIGTERM'd the two in-flight Setup Image branches and surfaced as script returned exit code 143. Re-run the build; nothing in the PR's code is implicated.

Full analysis

Summary: nixl-ci-build-wheel #1998 finished ABORTED after both Setup Image parallel branches (x86_64 and aarch64 nixl-wheel-base-cu13-manylinux_2_28) were interrupted mid-Dockerfile by an external pipeline cancellation.

Root cause: External abort, not a build defect. The whole-build log ends with:

Calling Pipeline was cancelled
[08:35:13] Sending interrupt signal to process
[08:35:14] script returned exit code 143
...
ERROR: parallel task failed with msg: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException: Calling Pipeline was cancelled
Finished: ABORTED

nixl-ci-build-wheel runs as a downstream build of a poller pipeline; when that calling run was cancelled, Jenkins raised FlowInterruptedException in every parallel branch, so exit 143 / SIGTERM here is purely the symptom of the kill.

I verified this is not a hang. Both branch logs show dense, continuous output right up to the moment of the kill — the largest gap in either branch is ~17 s:

  • x86_64 branch (node 104) was at [2/2] STEP 55/68 compiling libfabric, emitting CC prov/hook/dmabuf_peer_mem/.../hook_dmabuf_peer_mem.lo at 08:35:13.137Z, killed at 08:35:13.535Z.
  • aarch64 branch (node 111) was at [2/2] STEP 63/68, having just finished Installed 28 packages (torch 2.13.0+cu132) at 08:34:56.165Z, killed at 08:35:13.797Z.

No error, no assertion, no failed command appears in either branch. The modprobe: FATAL: Module gdrdrv not found and service: command not found lines during the gdrcopy RPM install are benign and expected inside a container (they appear identically in the successful image builds in stage 96/116 and in build #1999).

Secondary observation on fragility rather than cause: the Build Docker stage runs rm -f /etc/containers/storage.conf then podman system reset -f, wiping the local image store, so each run replays all 68 Dockerfile steps from scratch — an ~870 MB DOCA RPM download, a full gdrcopy RPM build, an autoreconf+compile of libfabric, and ~2 GB of torch/CUDA wheels. That makes Setup Image take 20+ minutes (branch durations 2,270 s and 2,260 s) and leaves a wide window in which any upstream cancellation lands mid-stage, as it did here.

Implicated commit: none — commit [REDACTED:Hex High Entropy String] (PR #1791, branch pr_nixl_kv_plugin_interface_redis) is not implicated; the checkout stage succeeded and resolved merge commit d08c4fb cleanly, and the abort originated in the calling pipeline.

File: n/a (no source file is at fault); the abort is recorded in the whole-build console log of nixl-ci-build-wheel #1998 and the Setup Image branch logs, stage IDs 104 and 111.

Suggested fix: Simply re-trigger nixl-ci-build-wheel for this PR — there is nothing to fix in the code. If the PR was re-pushed, this cancellation is the expected supersede behaviour and can be ignored. To stop superseded runs from being reported as failures, have the pipeline distinguish FlowInterruptedException from a genuine stage failure and mark the build ABORTED/NOT_BUILT without the hudson.AbortException: parallel task failed rethrow, and add a milestone() so older runs are retired cleanly. Separately, to shrink the exposure window, drop the unconditional podman system reset -f or pull the previous nixl-wheel-base-cu13-manylinux_2_28 tag as a cache source so the 68-step base image isn't rebuilt from zero on every run.

Related: none — the issue search returned only unrelated open PRs (#2231, #2248, #2211, #2290, #2204); recent .ci/jenkins commits ([REDACTED:Hex High Entropy String], e1fde65, [REDACTED:Hex High Entropy String]) do not touch the wheel-build abort path.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 26648fe2-3c79-4caf-8f3d-de8199fa5702 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 2a18b64c

TL;DR: Build #1474 wasn't a test or compile failure — the Allocate DL EP Environment stage was killed 19 s after it started by an external abort (FlowInterruptedException: Calling Pipeline was cancelled), so exit code 143 is just the SIGTERM symptom; re-run the PR check, and separately fix the slurm cleanup so the orphaned pending allocation 2372614 isn't leaked.

Full analysis

Summary: nixl-ci-dl-gpu-ep #1474 aborted in the Allocate DL EP Environment stage with script returned exit code 143 because the upstream pipeline that triggered it was cancelled mid-run.

Root cause: Everything the build actually did succeeded: Build Docker (852 s), Compiling NIXL EP Docker Image for DL (1 368 s) finished cleanly — the EP image was built, tagged nixl-ci-dl-gpu-ep-test:1474, and pushed at 08:34:49. The allocation stage then ran:

08:34:54  + ssh ... svc-nixl@dlcluster.nvidia.com salloc -N 1 -p gb200nvl72_cx8 \
            --job-name=nixl-ci-dl-gpu-ep-1474 --immediate=3600 --time=02:00:00 --no-shell
08:35:13  Calling Pipeline was cancelled
08:35:13  Sending interrupt signal to process / Killing processes
08:35:15  Terminated ... salloc: Pending job allocation 2372614
08:35:15  salloc: job 2372614 queued and waiting for resources
08:35:15  script returned exit code 143

There is no gap larger than a few seconds anywhere in the log — the build was making continuous progress right up to the kill, and the salloc had only been waiting 19 s out of its permitted --immediate=3600 (1 hour). So this is neither a hang nor a legitimate timeout. Jenkins' message Calling Pipeline was cancelled is raised when the upstream pipeline that invoked this job via the build step is itself aborted — on a poll-triggered PR job that normally means a newer commit on pr_nixl_kv_plugin_interface_redis superseded this run (or an operator aborted the parent). The final hudson.AbortException: parallel task failed with msg: ... FlowInterruptedException: Calling Pipeline was cancelled and Finished: ABORTED confirm the interrupt came from outside this job, not from its own timeout block (which closed normally).

Secondary, real defect exposed by the abort: salloc was SIGTERM'd before it could return the allocation ID to jobIdFile=/mnt/pvc/nixl-ci-dl-gpu-ep/job_id_ep_1474.txt, so the cleanup stage logged WARNING: No job ID files found in /mnt/pvc/nixl-ci-dl-gpu-ep/ and never cancelled anything. Pending allocation 2372614 on dlcluster.nvidia.com (partition gb200nvl72_cx8, account blackwell, --time=02:00:00) was therefore leaked and will sit in the queue — or run idle for up to 2 h — until someone scancels it.

Implicated commit: none — no code change is implicated; the PR commit [REDACTED:Hex High Entropy String] built and packaged successfully before the abort.

File: .ci/jenkins/pipeline (the slurm.allocation call with jobIdFile:/mnt/pvc/nixl-ci-dl-gpu-ep/job_id_ep_1474.txt) and the slurm.allocation / slurm.stopAllForBuild helpers in swx-jenkins-lib

Suggested fix:

  1. Re-trigger the PR check — this run carries no signal about the code. If a newer commit was pushed, the superseding build is the one to look at.
  2. Have an admin scancel 2372614 on dlcluster.nvidia.com (user svc-nixl) to reclaim the leaked allocation.
  3. Harden slurm.allocation against abort: parse salloc: Pending job allocation <id> from stdout and write it to jobIdFile as soon as it appears (e.g. tee/stdbuf the ssh output through an awk that writes the ID), rather than only after salloc returns. Additionally wrap the ssh in a shell trap 'scancel --name=$JOB_NAME' TERM INT so a SIGTERM'd stage cancels its own allocation. As a belt-and-braces measure, make slurm.stopAllForBuild fall back to scancel --name=<jobName> when no job-ID file exists instead of just warning.

Related: swx-jenkins-lib slurm.groovy; nixl PRs touching this pipeline: #2300, #2154, and #2311 ("stop the release poller dying on a stale in-flight reservation") which addresses the same class of stale-reservation bug in a different job.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7a7d780d-62a1-418f-8ec3-e15eed97e978 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 2a18b64c

TL;DR: The Test Python stage of one of six parallel variants failed because test_tcpstore_metadata_exchange could not bring up a PyTorch TCPStore on 127.0.0.1:10507 within its hard-coded 5-second timeout, after the container hit a transient name-resolution failure (err=-3, EAI_AGAIN); this is an environmental flake unrelated to the Redis PR, and the fix is to harden the test's timeout/retry rather than change the plugin.

Full analysis

Summary: pytest -s test/python failed on test/python/test_tcpstore_metadata.py::test_tcpstore_metadata_exchange with torch.distributed.DistNetworkError: The client socket has timed out after 5000ms while trying to connect to (127.0.0.1, 10507) — 1 failed, 24 passed, 3 skipped.

Root cause: Immediately before the failure the log shows the container's resolver failing:

  • [W socket.cpp:207] [c10d] The hostname of the client socket cannot be retrieved. err=-3 (EAI_AGAIN — temporary name-resolution failure)
  • then [E socket.cpp:1028] The client socket has timed out after 5000ms while trying to connect to (127.0.0.1, 10507)

dist.TCPStore(..., is_master=True) creates the server and then validates it with a client connect. Only the client connect failed (no "server socket failed to bind" error), and the test grants that step just timeout=timedelta(seconds=5). With the resolver stalling/retrying inside the pod, the 5-second budget was consumed before the loopback validation completed. Supporting evidence that this is environmental, not a code defect:

  • The port was verified free at 08:26:42 by get_next_tcp_port (ss -tuln | grep -q :10507 found nothing), so there was no listener conflict.
  • The same test passed in the other five parallel variants of build #3319 and in build #3320.
  • The whole stage took 7.4 s with no log gaps — nothing hung; the 5 s is an application-level socket timeout, not a wall-clock kill.
  • The PR under test adds a Redis plugin that isn't even built in this variant (Plugin file does not exist: .../libplugin_REDIS.so), and no Redis Python test is in the 28 collected items.

A secondary fragility contributes: the test depends on a CI-assigned fixed port handed over via NIXL_TCPSTORE_PORT, and get_next_tcp_port has a check-then-use (TOCTOU) window between allocation in .gitlab/test_python.sh:70 and the bind inside pytest two seconds later.

Implicated commit: aed5ef2 — aschwartz12, "test: add Python TCPStore metadata integration (#2148)" (introduced the test with the 5 s timeout and the fixed-port handoff). Not the PR under test ([REDACTED:Hex High Entropy String]).

File: test/python/test_tcpstore_metadata.py:21-28 (and .gitlab/test_python.sh:70, .ci/scripts/common.sh:38-61)

Suggested fix:

  1. Raise the TCPStore timeout from 5 s to ~30 s and wrap construction in a short retry loop on DistNetworkError, so a transient resolver hiccup doesn't fail the suite. Bump @pytest.mark.timeout(20) accordingly (e.g. 60) so the pytest timeout doesn't fire first.
  2. Prefer letting the kernel pick the port (port=0) and read back tcp_store.port — the test already uses tcp_store.port for NIXL_TCPSTORE_ENDPOINT, so the CI-assigned NIXL_TCPSTORE_PORT buys nothing and only adds the TOCTOU race. If the fixed port must stay, allocate it as late as possible.
  3. Re-run build #3319; no change to the Redis plugin branch is warranted for this failure.

Related: PR #2148 (added the test), PR #1791 (the branch under test, unaffected)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e4b2d501-35f3-4a6d-a8a2-b6c87cfc6704 in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 2a18b64c

TL;DR: The build compiled and pushed the test image fine, but the "Run DL Python tests" stage failed because pyxis/enroot on the slurm node could not pull that image — Artifactory returned 401 Unauthorized for the svc-nixl enroot credentials. This is a CI credential/infra issue, not a defect in PR #1791; the enroot credentials file on the DL cluster needs to be refreshed.

Full analysis

Summary: Stage 188 "Run DL Python tests" failed after 5s: srun on gb-nvl-059-compute05 aborted because pyxis could not import artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-dl-gpu-test:2548.

Root cause: Container image import on the slurm node failed at registry authentication, not in any test code. The enroot log embedded in the stage output shows:

  • [INFO] Authenticating with user: svc-nixl
  • [INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
  • [ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
  • error: pyxis: failed to import docker image → spank_pyxis.so: task_init() failed with rc=-1 → srun: error: gb-nvl-059-compute05: task 0: Exited with exit code 1

The stored credential used by enroot on the DL cluster (for user svc-nixl) is expired, revoked, or malformed. Notably the earlier "Compiling NIXL Docker Image for DL" stage built the image and pushed it to the same Artifactory repo successfully using the Jenkins-side REPO_USER/REPO_PASS credential, so the registry and the image both exist and are reachable — only the cluster-side enroot credential path is broken. There is no large time gap and no hang: the failure happened ~4 seconds after srun started, and the whole stage was 5s. The PR branch content (pr_nixl_kv_plugin_interface_redis, commit [REDACTED:Hex High Entropy String]) is irrelevant — the test binary never started.

Implicated commit: unknown — not a source regression; the failure is in CI container-pull credentials on the DL cluster, outside the repo.

File: /home/svc-nixl/.config/enroot/.credentials on dlcluster.nvidia.com / gb-nvl-059-compute05 (consumed by the srun invocation driven from .gitlab/test_python.sh via the slurm.run pipeline step; log line at stage 188, pyxis: importing docker image)

Suggested fix:

  1. Have the CI/infra owner regenerate the svc-nixl Artifactory access token and rewrite /home/svc-nixl/.config/enroot/.credentials on the DL cluster (it must be readable on the compute nodes, e.g. shared home or synced to each node). Rotate the old token since it may have been revoked rather than merely expired.
  2. Verify with a minimal srun --container-image=... pull of an existing tag before re-running the job.
  3. Harden the pipeline so this is diagnosed faster: have the slurm.run wrapper detect pyxis: failed to import docker image / HTTP 401 and fail with an explicit "registry authentication failed — rotate enroot credentials" message instead of a generic exit code 1, and consider deriving the enroot credential from the same Jenkins credential (svc-nixl repo user/pass) already proven to work in the build/push stage so the two cannot drift apart.
  4. Re-trigger build #2548 for PR feat(redis): add Redis backend with async transfers and nixlbench support #1791 once credentials are fixed; the PR itself has not been tested yet.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 42cc0639-fab0-4697-863a-aca4ac045abf in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu-vr · commit 2a18b64c

TL;DR: The build compiled and pushed its test image fine, but the Slurm/pyxis step on vr-nvl72-ts3-l06-comp141 could not pull it — enroot got 401 Unauthorized from https://artifactory.nvidia.com/v2/token using the svc-nixl enroot credentials file, so the container never started. This is an expired/invalid registry credential on the dlcluster node, not a code defect in PR #1791.

Full analysis

Summary: Stage "Run Python tests" (node 188) failed after 5s because srun --container-image=...nixl-ci-gpu-vr-test:68 could not import the image — pyxis/enroot registry authentication returned HTTP 401.

Root cause: Credential mismatch between the two halves of the pipeline. The "Compiling NIXL Docker Image" stage (node 153) built and pushed artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-gpu-vr-test:68 successfully at 09:00:38 using podman push --creds ${REPO_USER}:${REPO_PASS} ("Writing manifest to image destination"). 44 seconds later the test stage ran srun on the allocated node, where enroot authenticated from a different credential source:

[INFO] Authenticating with user: svc-nixl
[INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
error: pyxis: failed to import docker image
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
srun: error: vr-nvl72-ts3-l06-comp141: task 0: Exited with exit code 1

The image exists and the Jenkins-side push token is valid; the stale artifact is the svc-nixl enroot credentials file on the dlcluster node (expired or rotated artifactory token/password). No application code ran at all — the build, UCX install, meson configure, ninja build (296/296 targets) and install all succeeded, and the Slurm allocation (job 2372798, vrnvl72 partition, --account=rubin) was granted normally. Nothing in this log touches the Redis KV plugin code the PR changes.

Implicated commit: unknown — not a source regression; the failure is in CI credential/environment state, not in [REDACTED:Hex High Entropy String] or any commit touching the build.

File: /home/svc-nixl/.config/enroot/.credentials on vr-nvl72-ts3-l06-comp141 / dlcluster (infra, not in repo); triggered from the slurm.run step invoked for .gitlab/test_python.sh /opt/nixl

Suggested fix: Have CI/infra refresh the artifactory token in the svc-nixl enroot credentials file on the dlcluster nodes so it matches the credential Jenkins uses for podman push (rotate the token and redeploy the file — do not paste the token value into the PR). Longer term, make the pipeline inject the same REPO_USER/REPO_PASS credential into the srun environment (e.g. via ENROOT_CONFIG_PATH or --container-env using the existing svc-nixl-ssh_key-adjacent registry credential) so push and pull cannot drift apart, and add a fail-fast check that surfaces "registry auth failed" distinctly from a test failure. Then re-run build #68; no code change to PR #1791 is required.

Related: none — searches for the pyxis/enroot 401 signature and history on .gitlab/test_python.sh turned up no prior report of this failure mode.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id aa9fd2d8-a650-4f8e-846e-84a57b5ef13f in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The "Run CPP tests" stage never ran a single test — enroot/pyxis on mizu01 failed to pull the freshly-pushed CI image because https://artifactory.nvidia.com/v2/token returned 401 Unauthorized for user svc-nixl; the enroot credentials file on the slurm nodes needs to be refreshed/rotated.

Full analysis

Summary: nixl-ci-gpu #3849 failed in stage Run CPP tests (node 188, 10.8 s) when srun/pyxis could not import nixl-ci-gpu-test:3849 from Artifactory due to a registry authentication failure.

Root cause: Infrastructure/credential failure, not a code defect. The image was built and pushed successfully (Successfully tagged … nixl-ci-gpu-test:3849, podman push → Writing manifest to image destination, + break at 09:13:10) using the Jenkins artifactory-pull-secret/credentialed push. One second later, the test step ran on the slurm allocation (job 100472, node mizu01) and pyxis tried to pull the same tag using a different credential source:

pyxis: importing docker image: artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/x86_64/nixl-ci-gpu-test:3849
[INFO] Authenticating with user: svc-nixl
[INFO] Using credentials from file: /labhome/svc-nixl/.config/enroot/.credentials
[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
pyxis: couldn't start container
spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
srun: error: mizu01: task 0: Exited with exit code 1

The subsequent enroot export error (No such file or directory: …/pyxis_nixl-ci-3849) is a downstream artefact of the container never being created, and is reported as non-fatal. There is no compile error, no test output, and no timeout/hang anywhere — total stage duration was ~11 s with no gaps. Nothing in the PR branch (pr_nixl_kv_plugin_interface_redis, commit 2a18b64) is implicated: the REDIS plugin compiled and linked cleanly (libplugin_REDIS.so) and the whole ninja build plus install succeeded.

Implicated commit: none — not a source regression. The failing credential lives outside the repo (the enroot credentials file for svc-nixl on the mizu nodes).

File: not a repo file — /labhome/svc-nixl/.config/enroot/.credentials on mizu01 (consumed by spank_pyxis.so during the srun invoked from .gitlab/test_cpp.sh's caller, slurm.run in swx-jenkins-lib)

Suggested fix:

  1. Have CI/lab admins regenerate the Artifactory identity token for svc-nixl and rewrite the enroot credentials file on the mizu cluster nodes (shared /labhome home dir), then re-run build #3849. The token was almost certainly rotated or expired — do not quote or reuse the old value; rotate it.
  2. Verify that the credential used by the Jenkins push step (artifactory-pull-secret / svc-nixl-scctl) and the one in the enroot credentials file are driven from the same secret, so a rotation can't leave the pull path stale while the push path works — that split is exactly what this build exposes.
  3. Defensive CI improvement: make the pyxis import failure produce a clearly-labelled "image pull / auth failure" message rather than surfacing as a generic script returned exit code 1 from the parallel branch, so this class of failure isn't triaged as a test regression. Also consider gating the exportEnroot step on the container having actually started, to suppress the misleading follow-on No such file or directory error.

Related: none — searches for the pyxis/enroot 401 signature returned no matching issues or PRs in this repository.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 2a18b64c

TL;DR: The "Run DL NIXL EP tests" stage failed because pyxis/enroot on the DL cluster node gb-nvl-118-compute04 could not authenticate to artifactory.nvidia.com (HTTP 401 when fetching a registry token with the svc-nixl enroot credentials), so the test container image was never imported — nothing to do with the PR's code.

Full analysis

Summary: Build #1477 stage 188 ("Run DL NIXL EP tests", 7 s) aborted immediately: srun … --container-image=artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-dl-gpu-ep-test:1477 → pyxis: failed to import docker image, srun: error: gb-nvl-118-compute04: task 0: Exited with exit code 1.

Root cause: Credential failure on the slurm side, not a code or test defect. The enroot log embedded in the stage log shows:

  • [INFO] Authenticating with user: svc-nixl
  • [INFO] Using credentials from file: /home/svc-nixl/.config/enroot/.credentials
  • [ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
  • → pyxis: couldn't start container → spank_pyxis.so: task_init() failed with rc=-1

The image itself exists and is fine: the preceding stage (153, "Compiling NIXL EP Docker Image for DL") completed the full UCX + NIXL + nixlbench build, tagged …nixl-ci-dl-gpu-ep-test:1477, and podman push finished with "Writing manifest to image destination" using the Jenkins-side REPO_USER/REPO_PASS credentials. So the registry is reachable and the Jenkins credential is valid; only the separate enroot credentials file in the svc-nixl home directory on the DL cluster is rejected — i.e. an expired/rotated or malformed Artifactory token on the cluster side. There is no hang (the stage lasted ~7 s with no gaps) and no test ever executed.

Implicated commit: none — 2a18b64 (PR #1791, pr_nixl_kv_plugin_interface_redis) is not implicated; the failure precedes any test execution.

File: infrastructure, not repo code — /home/svc-nixl/.config/enroot/.credentials on dlcluster.nvidia.com / gb-nvl-118-compute04. The invoking call site is .gitlab/test_ep.sh launched via slurm.run from the Jenkins pipeline (stage 188).

Suggested fix:

  1. Have CI/infra refresh the Artifactory identity token in the svc-nixl enroot credentials file on the DL cluster (and verify it is consistent with the credential Jenkins uses for podman push, which still works). Treat the existing token as compromised/stale and rotate it rather than reusing it.
  2. Verify with enroot import docker://artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/aarch64/nixl-ci-dl-gpu-ep-test:1477 as svc-nixl on the head node before re-triggering.
  3. Hardening: have the pipeline's slurm.run wrapper detect pyxis: failed to import docker image / 401 and fail with an explicit "registry auth failure — infra issue" message (and optionally retry with backoff, as the podman push loop already does), so this doesn't present as a PR test failure. Re-run Add dockerfiles for inference images #1477 once the credential is refreshed.

Related: none found (searches for the pyxis/enroot 401 signature returned no matching nixl issues or PRs).

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The "Run CPP tests" stage never ran any test — pyxis/enroot on compute node mizu01 could not pull the freshly-pushed CI image because Artifactory returned 401 Unauthorized for the svc-nixl enroot credentials, so the container failed to start. This is a credentials/registry issue on the slurm node, not a defect in PR #1791.

Full analysis

Summary: Stage Run CPP tests (node 188) failed in ~10s: pyxis: failed to import docker image … URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized → srun: error: mizu01: task 0: Exited with exit code 1.

Root cause: The image .../ci/pr/x86_64/nixl-ci-gpu-test:3850 was built and pushed successfully from the Jenkins builder (stage 153 ends with podman push … Writing manifest to image destination / + break). When srun then started the container on mizu01, enroot authenticated as user svc-nixl using the credentials file under that user's enroot config directory and Artifactory rejected the token request with HTTP 401. Because the container was never created, the spank pyxis plugin aborted task_init (rc=-1) and the test script .gitlab/test_cpp.sh /opt/nixl run_uring never executed. The secondary error [ERROR] No such file or directory: /enroot-data/.../pyxis_nixl-ci-3850 and exportEnroot: export failed (non-fatal) are downstream consequences of the container not existing, not independent failures. No compile error, no test assertion, and no timeout/hang appears anywhere in the log — the failure is purely registry authentication on the compute node.

Implicated commit: unknown — not caused by commit [REDACTED:Hex High Entropy String] (PR #1791); the build and install stages all succeeded on this commit.

File: not a source file. Infra: the enroot credentials for svc-nixl on mizu01 (referenced by the pyxis log as the credentials file in that account's enroot config dir), consumed via slurm.run(credentialsId: svc-nixl-scctl) in the Jenkins pipeline's "Run CPP tests" step.

Suggested fix:

  1. Have CI/infra refresh the Artifactory credential used by enroot for the svc-nixl service account on the mizu nodes — the token/API key appears to be expired or revoked. Rotate it rather than copying the existing value, and verify with a manual enroot import of nixl-ci-gpu-test:3850 on mizu01.
  2. Confirm the credential has pull rights on sw-nbu-swx-nixl-docker-local; the push side (Jenkins builder, different credential) clearly works, so the two may have diverged.
  3. Hardening: make the pipeline fail fast with a clear "registry auth" message and retry the srun import a couple of times, so a transient 401 doesn't present as a PR test failure. Once credentials are fixed, simply re-run build #3850 — no code change is needed on PR feat(redis): add Redis backend with async transfers and nixlbench support #1791.

Related: none found — searches for the 401/pyxis/enroot signatures in this repo's issues and PRs returned no relevant matches, consistent with this being an infra-side credential problem rather than a tracked code bug.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 167c0e0f-7308-409f-a881-5ccbed66a1dd in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The "Run CPP tests" stage failed because pyxis/enroot could not pull the freshly-built test image on mizu01 — Artifactory returned 401 Unauthorized for the svc-nixl enroot credentials, so the container never started. This is a CI credential/infra problem, not a defect in PR #1791; the enroot credentials file for svc-nixl on the mizu nodes needs to be refreshed.

Full analysis

Summary: Stage 188 "Run CPP tests" (build #3854) failed in 10s: slurmstepd: error: pyxis: failed to import docker image … URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized.

Root cause: The build itself succeeded — stage 153 compiled NIXL, nixlbench and all plugins cleanly and podman push artifactory.nvidia.com/sw-nbu-swx-nixl-docker-local/ci/pr/x86_64/nixl-ci-gpu-test:3854 completed ("Writing manifest to image destination"), so the Jenkins agent's registry credentials are valid. The test stage then runs on slurm allocation 100508 / node mizu01 via srun --container-image=..., and there enroot authenticates separately using the credentials file under svc-nixl's home on the lab filesystem. That token request was rejected with 401, so pyxis aborted (spank: required plugin spank_pyxis.so: task_init() failed with rc=-1) and the container never launched. The subsequent [ERROR] No such file or directory: .../pyxis_nixl-ci-3854 is just a downstream consequence — there is no container to export.

This is not specific to this commit or branch. The identical 401 appears in build #3853 (allocation 100464, same node mizu01, same stage, same 10s duration), and slurm records show the neighbouring CI allocations nixl-ci-gpu-3850, -3851, -3853 on mizu01 all ending CANCELLED in the same window — a cluster-wide credential expiry, not a code regression.

Implicated commit: none — the failure is independent of [REDACTED:Hex High Entropy String]; the pulled image was built successfully from that commit.

File: not a source file — the enroot credentials file referenced by the runner on mizu01 ($HOME/.config/enroot/.credentials for the svc-nixl service account); orchestration is slurm.run in the shared swx-jenkins-lib, invoked for stage "Run CPP tests".

Suggested fix: Have CI/lab admins regenerate the Artifactory identity token for the svc-nixl service account and rewrite the enroot credentials file on the mizu nodes (it is shared via /labhome/svc-nixl, so one update covers the cluster) — then rotate/revoke the old token, since a 401 on a previously working token usually means it expired or was revoked. Do not quote or copy the credential contents when doing this. Secondarily, make the pipeline fail fast with a clear message on pyxis auth errors instead of continuing into the enroot export step, whose "No such file or directory" error masks the real cause; re-run #3854 once credentials are refreshed to confirm PR #1791 is green.

Related: Same signature in nixl-ci-gpu build #3853 (stage 188, allocation 100464, node mizu01); no matching tracking issue found in the nixl repo.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id dd471834-0218-4dfc-886c-eef85d7951af in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu-vr · commit 2a18b64c

TL;DR: The only failing test is test/python/test_nixl_api.py::test_prep_mem_view, which dies with nixlBackendError: NIXL_ERR_BACKEND because UCX on the Vera Rubin node cannot find a device-capable lane for the remote peer (ucp_device.c:700 lane not found for element 0) — an environment/skip-guard gap on the brand-new vrnvl72 partition, not a regression from this PR's Redis plugin changes.

Full analysis

Summary: Stage "Run Python tests" (node 188) failed: 1 failed, 25 passed, 2 skipped — test_prep_mem_view raised torch.multiprocessing.spawn.ProcessRaisedException wrapping nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND; srun on vr-nvl72-ts3-l06-comp141 exited 1.

Root cause: Both spawned ranks reached the remote overload of prep_mem_view and UCX rejected the device memory-list creation:

[vr-nvl72-ts3-l06-comp141:4190290] ucp_device.c:700  UCX ERROR lane not found for element 0
[vr-nvl72-ts3-l06-comp141:4190290] ucp_device.c:819  UCX ERROR failed to create handle: Invalid parameter
E ucx_backend.cpp:781] Failed to prepare remote memory view: Failed to create device memory list(remote): Invalid parameter

ucp_device_remote_mem_list_create() in src/plugins/ucx/mem_list.cpp:163 returned UCS_ERR_INVALID_PARAM (not UCS_ERR_NOT_CONNECTED, so the retry loop doesn't apply), so createMemList throws and nixlUcxEngine::prepMemView converts it to NIXL_ERR_BACKEND (src/plugins/ucx/ucx_backend.cpp:781-782). The local overload succeeded; only the remote one — which needs a real device-capable (cuda_ipc/GDA) lane to the peer endpoint — failed, meaning the endpoint selected on this VR node has no device lane despite the connection and notif round-trip completing.

The test's guards (test/python/test_nixl_api.py:327-336) only check bindings.HAVE_UCX_GPU_DEVICE_API and torch.cuda.device_count() >= 2; neither verifies that a device-capable transport lane actually exists to the peer, so the test runs and hard-fails on this hardware/UCX combination.

This is not caused by the PR commit: the branch's changes (243ef845 "build: install Redis deps in contrib Dockerfiles", 06247e29/[REDACTED:Hex High Entropy String] redis client formatting/copyright) touch nothing in the UCX device path. Build #72 is simply the first run on this pipeline where the container actually started — builds #65, #68, #69, #70, #71 all failed the same stage in ~5s on pyxis: failed to import docker image (a 401 from the registry token endpoint; the service account's enroot credentials need rotating/refreshing). The nixl-ci-gpu-vr job itself is new (2dd278bf, NirWolfer, 2026-09-30, "CI: add nixl-ci-gpu-vr pipeline on the Vera Rubin partition"), so this test has never passed on vrnvl72.

Implicated commit: Not the PR head. The test/guard originates in [REDACTED:Hex High Entropy String] (x41lakazam, "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads)"); exposure is from 2dd278bf (NirWolfer, "CI: add nixl-ci-gpu-vr pipeline on the Vera Rubin partition").

File: test/python/test_nixl_api.py:304 (and guards at test/python/test_nixl_api.py:327-336); backend error raised at src/plugins/ucx/mem_list.cpp:176-178 → src/plugins/ucx/ucx_backend.cpp:781.

Suggested fix:

  1. Tighten the skip guard for test_prep_mem_view so it is skipped when no device-capable lane is available, rather than failing: probe the remote overload (or UCX transport/UCX_TLS device support) and pytest.skip on NIXL_ERR_BACKEND/"lane not found", instead of relying solely on HAVE_UCX_GPU_DEVICE_API + device_count() >= 2. Alternatively mark the test xfail/skip for the vrnvl72 partition until the device API is validated there.
  2. In parallel, have the UCX owners confirm whether the GPU device API (cuda_ipc/GDA lane selection) is expected to work on Vera Rubin NVL72 with the bundled UCX v1.23.x on aarch64 — if it should, the real bug is lane selection and the test is correctly reporting it; consider setting the required UCX_TLS/device-channel config in the VR job env.
  3. Separately, fix the registry auth for the vrnvl72 runs: builds Mkhazraee/send recv fix #65–ucx/ucx_mo tests: fix for CUDA buffer case #71 died on enroot getting 401 Unauthorized from the Artifactory token endpoint. The service account's stored registry credential should be rotated/refreshed (do not paste it into tickets).
  4. Re-run build ci: Added ucx_mo_backend_test. #72 to confirm the failure is deterministic on this node before changing any code.

Related: PR #2194 "bugfix: test: make the device API tests actually run"; PR #1715 (introduced the test); PR #2182 (added the nixl-ci-gpu-vr pipeline); PR #2309 (recent Jenkins checkout/auth changes).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 2510e8b5-27da-4383-ae49-8a0278900f2d in the triage console for the audit trail.

@svc-nixl

svc-nixl commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — nixl-ci-gpu · commit 2a18b64c

TL;DR: The "Run CPP tests" stage failed because pyxis/enroot could not pull the freshly-built test image — https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized using the svc-nixl enroot credentials on mizu01. This is a CI credential problem, not a defect in PR #1791; refresh the enroot credentials file for svc-nixl on the mizu nodes and re-run.

Full analysis

Summary: Build #3856 stage 188 "Run CPP tests" (srun on slurm allocation 100516, node mizu01) aborted after 10 s because the container image import failed with HTTP 401 from the artifactory registry.

Root cause: The pipeline builds and pushes …/ci/pr/x86_64/nixl-ci-gpu-test:3856 successfully (stage 153: podman push completes, manifest written at 11:10:07), then asks slurm/pyxis to import that same image. Pyxis authenticated as user svc-nixl using the credentials file under that account's enroot config directory and got:

[ERROR] URL https://artifactory.nvidia.com/v2/token returned error code: 401 Unauthorized
pyxis: couldn't start container
spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
srun: error: mizu01: task 0: Exited with exit code 1

So the push-side credentials (used by podman on the Jenkins agent) are valid, while the pull-side enroot credentials on the compute node are expired/revoked/malformed. The subsequent enroot export … pyxis_nixl-ci-3856 error (No such file or directory: …/pyxis_nixl-ci-3856) is a downstream artifact of the container never being created — the pipeline itself labels it "non-fatal". There is no hang: the whole failing stage spans 11:10:32→11:10:42 with no gaps, and no compile or test error appears anywhere (stage 153 built and installed all 296 targets cleanly). Nothing in the changed code path (redis KV plugin) is reached. On the infra side, the only other slurm job on mizu01 in the window was nixl-ci-gpu-3855 (allocation 100515, RUNNING then CANCELLED at 11:11:40) — concurrent, but a neighbouring allocation cannot produce a registry 401.

Implicated commit: unknown — not a code regression; [REDACTED:Hex High Entropy String] (PR #1791) is unrelated to the failure. Last changes to the test entry point are adf179a5 (Lukas Humbel) and e286b8f7 (NirWolfer, which added the non-fatal enroot export), neither of which touches registry auth.

File: CI infrastructure — the enroot credentials file referenced in the stage log (/labhome/svc-nixl/.config/enroot/…) on the mizu nodes; invoked from .gitlab/test_cpp.sh via slurm.run in the Jenkins shared library (swx-jenkins-lib).

Suggested fix:

  1. Have CI/IT regenerate the artifactory token for the svc-nixl service account and rewrite the enroot credentials file on all mizu* nodes (it is read per-node from the shared labhome path). Rotate the old token if it may have leaked rather than merely expired.
  2. Verify with a standalone srun --container-image=artifactory.nvidia.com#/sw-nbu-swx-nixl-docker-local/ci/pr/x86_64/nixl-ci-gpu-test:3856 true before re-triggering the PR build.
  3. Hardening: make the pipeline fail fast with an explicit, distinguishable message on pyxis import/auth errors (and retry the import a couple of times), so a registry-auth outage is not reported as a "Run CPP tests" test failure against an innocent PR.
  4. Re-run build #3856 once credentials are refreshed; no change to PR feat(redis): add Redis backend with async transfers and nixlbench support #1791 is warranted.

Related: none found — issue/PR search for the pyxis/401 signature returned no matching reports in this repository.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4c8f2502-c80f-44ef-9a88-159f00b0048b in the triage console for the audit trail.

@aranadive
aranadive merged commit fc0eaab into ai-dynamo:main Oct 4, 2026
20 of 21 checks passed
riley-dixon added a commit to riley-dixon/nixl that referenced this pull request Oct 7, 2026
Resolve conflicts with REDIS (ai-dynamo#1791) and nixlbench CUDA locality-domain
VMM (ai-dynamo#2344) by keeping both sides: REDIS and AIS_MT in all_plugins and
the nixlbench backend lists, and the locality-domain block ahead of the
ROCm-aware VRAM warning.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Riley Dixon <riley.dixon@amd.com>
riley-dixon added a commit to riley-dixon/nixl that referenced this pull request Oct 7, 2026
Resolve conflicts with REDIS (ai-dynamo#1791) and nixlbench CUDA locality-domain
VMM (ai-dynamo#2344) by keeping both sides: REDIS and AIS_MT in all_plugins and
the nixlbench backend lists, and the locality-domain block ahead of the
ROCm-aware VRAM warning.

Signed-off-by: Riley Dixon <riley.dixon@amd.com>
aranadive added a commit that referenced this pull request Oct 22, 2026
install redis server into the CI workflow container, in order to make it
ready for testing CI workflow for redis plugin [PR
1791](#1791).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Redis benchmark connections can use `REDIS_*` environment variables or
defaults instead of per-parameter settings.
- **Bug Fixes**
- Corrected device ID handling in object-storage benchmark descriptors.
- **Tests**
- Added Redis-backed C++ unit tests and Nixlbench read/write tests with
consistency checks.
- **Chores**
- Updated build environments to support Redis testing and build hiredis
from source.
  - Ubuntu 22.04 builds now use Python 3.10 for wheel builds.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Harry Xie <harryx@nvidia.com>
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>

This branch was previously deployed

1 inactive deployment
SWX_AWS — 2a18b64c Deployed Oct 2, 2026 by copy-pr-bot[bot] via Run AWS Tests #9661
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants