Conversation
…-dynamo#612) * Plugins should set rpath so that they can load their dependencies at runtime without needing LD_LIBRARY_PATH Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Adjust rpath so that plugins load binary deps from parent dir only Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
…-dynamo#640) * EXAMPLES/PYTHON: partial md example - fix test logic and warnings Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: fixed formatting Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLE/PYTHON: reverted removal of print at the end Signed-off-by: Roie Danino <rdanino@nvidia.com> --------- Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Both to get_backend_options() and to the readme doc. Signed-off-by: Vlad Buslov <vladbu@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
…est scripts (ai-dynamo#568) * .CI/SCRIPTS: Avoid tcp port collisions, unify duplicated code in CI test scripts Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: fixed common.sh macros definitions Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: get_next_server_port returns the port, refactoring Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: fixed identations Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: using env variables from jenkins and gitlab ci Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: trying to pass gitlab CI_CONCURRENT_ID env to test scripts Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: reverted some changes Signed-off-by: Roie Danino <rdanino@nvidia.com> --------- Signed-off-by: Roie Danino <rdanino@nvidia.com>
This PR increase the number of builds we keep before retention policy removes them from Jenkins Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* CI: Support both NIXL and NIXLBench in container builder * CI: Limit parallel build processes to prevent resource exhaustion Add NPROC=10 to container builds to avoid consuming all available CPU cores on shared CI servers. * CI: Clarify impact of UCX_VERSION job argument * CI: Rework UCX handling per build target - nixlbench: Clone UCX source and get actual commit SHA - nixl: Extract UCX version from build script (no clone needed) * CI: Print commands for manual repro * CI: Add nightly builds for both targets and Ubuntu versions Schedule nixlbench/nixl builds for Ubuntu 24.04 and 22.04 at 3-6 AM with latest tag updates. * CI: make UCX_VERSION parameter work for both build targets - nixl: respect UCX_VERSION via UCX_REF environment variable - nixlbench: unchanged (already used UCX_VERSION for cloning) * CI: Refactor NIXL container build pipeline - Eliminate redundant variables and cross-step dependencies - Separate build steps by target (nixlbench vs nixl) with conditional execution - Remove build.env file passing, calculate variables locally in each step - Simplify Artifactory path construction using BUILD_TARGET directly - Use subshells for git operations to avoid directory pollution - Consolidate version info generation into unified template - Remove unnecessary default arguments that match script defaults - Standardize variable naming and quoting conventions throughout * Run yamllint * Fix path to ucx-src * Fix podman error on short-name resolution * The return of container selector * Commit out version info stage * Don't use ci-demo enable keyword * Set 'enable:' as env vars * CI: Simplify UCX ver handling * CI: un-simplify UCX ver handling * CI: Implement remarks * Add debug for repo settings * Troubleshoot null repo * Fix null repo * Change version-info to shell-sourceable format and reuse - Replace JSON template with shell-sourceable /etc/os-release style format - Calculate TAG_NAME only once and add to the version-info file - Simplify Push/Show Results stages to just source version-info file - Store version file as /opt/nixl-version instead of /opt/nixl-version.json * Lint a whitespace * Change email address for notifications * Remove support for empty UCX_VERSION value * Improve the reporting mail format * Linter * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Test mail reporting * Revert to simple mail format
* RUST: Implement queryMem binding Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Update src/bindings/rust/src/descriptors/query.rs Co-authored-by: Roie Danino <rdanino@nvidia.com> Signed-off-by: Vlad Buslov <vlad@buslov.dev> * Address code review comments Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Add missing stub Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Update src/bindings/rust/src/descriptors/reg.rs Co-authored-by: Roie Danino <rdanino@nvidia.com> Signed-off-by: Vlad Buslov <vlad@buslov.dev> * Fix compilation Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Address code review comments Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Remove code duplication in add_desc Signed-off-by: Vlad Buslov <vladbu@nvidia.com> * Address more code review comments Signed-off-by: Vlad Buslov <vladbu@nvidia.com> --------- Signed-off-by: Vlad Buslov <vladbu@nvidia.com> Signed-off-by: Vlad Buslov <vlad@buslov.dev> Co-authored-by: Roie Danino <rdanino@nvidia.com>
Move the nixl copy workspace command to after UCX build. This prevents UCX from building again if something has changed in NIXL dir. Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
* Factory method to create nixlUcxEngine * Minor refactoring + TODOs * Split nixlUcxThreadEngine into a separate class * Clang format * Clang format * Addressed PR comments * Clang format * Cosmetic changes * Added tests for UCX backend without progress thread * Clang format * nixl_b_params_get * Split nixlUcxThread into a separate class * Cosmetic change * Clang format * Moved helper nixl_b_params_get to utils, to not expose absl dep * Addressed PR comments * Reverted formatting * Fixed clang-tidy issues --------- Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com> Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
) * Allow UCX as backend in kvbench plan and profile commands Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix source/dest gpu/memory mapping Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Formatting Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
…i-dynamo#633) * PYTHON-PROJECT-WIDE: replaced print() calls with proper logging (ai-dynamo#512) * PYTHON-PROJECT-WIDE: replaced print() calls with proper logging Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed identations Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: added a configuration file and a centrelized nixl_logging script with support for NIXL_LOG_LEVEL env Signed-off-by: Roie Danino <rdanino@nvidia.com> * NIXL_LOGGING: remove trailing spaces, fix identations Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: fixed logging imports Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed logging imports Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH/COMMANDS: fixed blank lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: fixed blank lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: moved nixl_logging.py to api/python Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed identations and blank lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed identations and spaces Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed identations and spaces Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: fixed identations and spaces Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: install nixl python package in manylinux docker Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: moved up installing nixl package Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: added a CI step for checking there are no print() in the code and only logger is being used Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: added missing copyrights Signed-off-by: Roie Danino <rdanino@nvidia.com> * TEST/PYTHON: change debug logs to info, move script to .ci/scripts Signed-off-by: Roie Danino <rdanino@nvidia.com> * EAMPLES/PYTHON: fix pre-commit checks Signed-off-by: Roie Danino <rdanino@nvidia.com> * TEST/PYTHON: fixed '-' * 40 log lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: merge log lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB/MANYLINUX: removed unnecessary pip install Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: fixed formatting Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: added back the installtion of the nixl pack in manylinux docker Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: can't assume nixl package was installed in a script within contrib folder Signed-off-by: Roie Danino <rdanino@nvidia.com> * CONTRIB: moved import logging below hashlib Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: added port selection script as test_cpp and test_python might run in parallel Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: make server_port.sh executable Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/SCRIPTS: removed server_port.sh, lets keep it out of this PR Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMAKR/KVBENCH: log -> logger for consistency Signed-off-by: Roie Danino <rdanino@nvidia.com> * API/PYTHON: simplified logging.py code, and minor fixes Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: replaced format strings with %s formatting in logger lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * BENCHMARK/KVBENCH: replaced all logging line f-strings with % formatting (optimization/compatibility) Signed-off-by: Roie Danino <rdanino@nvidia.com> --------- Signed-off-by: Roie Danino <rdanino@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com> (cherry picked from commit 29ae0ad) * SRC/API/PYTHON: removed hostname from log format Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: removed unused import Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: checking if setting raiseExceptions = True will fail CI on tracebacks Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/API/PYTHON: changed file configuration to dict config, fixed log lines, added env for raising logger exceptions Signed-off-by: Roie Danino <rdanino@nvidia.com> * EXAMPLES/PYTHON: fixed formatting Signed-off-by: Roie Danino <rdanino@nvidia.com> --------- Signed-off-by: Roie Danino <rdanino@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
…st bindings tests to CI (ai-dynamo#290) * BINDINGS/RUST: added Cargo.toml in project root, rust bindings are being compiled automatically Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: rust bindings are built and installed with meson / ninja Signed-off-by: Roie Danino <rdanino@nvidia.com> * MESON_OPTIONS.TXT: added rust_bindings meson option, updated readme file Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: removed a redundant space Signed-off-by: Roie Danino <rdanino@nvidia.com> * Cargo.toml: added missing copyrights Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: include -> includes, added Cargo.lock, fixed aio detection issue Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: removed a blank line Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/PLUGINS/POSIX: added back compiler.find_library in addition to the dependency call Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB/TEST_RUST.SH: added rust bindings tests to CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: revert lib path to x86_64-linux-gnu for CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: build.rs will search in both ubuntu and redhat lib paths Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB/BUILD.SH: enable_rust_bindings=true on build Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: removed unnecessary inner Cargo.lock Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: added a flag to cargo build Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: lib64 path is based on OS dist Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: added /usr/include, for some reason CI cant find stdbool.h * SRC/BINDINGS/RUST: added use os_info; Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: checked linux type using os_info Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: cargo version should be 1.86 Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB/BUILD.SH: fixed rust installation on CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB/BUILD.SH added -y to rust installation Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: changed to CPP imports Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: tried to fix CI again Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: fixed rust CI scripts Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: still fixing CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: fixed rust testing in CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: fixed PKG_CONFIG_PATH Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: make sure cargo in PATH Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: [DEBUGGING] added prints for cargo Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: added ucx pkg-config path to to PKG_CONFIG_PATH Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: used pkg-config in build.rs Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: set NIXL_PREFIX to Signed-off-by: Roie Danino <rdanino@nvidia.com> * BINDINGS/PYTHON: changed installation path Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: fixing python CI - WIP Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/PYTHON: create a wheel file for python bindings Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/PYTHON: hopefull fixed python CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: uncomment -x -e Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: fixed cpu-specific paths Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: commented out some env exports Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: removed unnecessary modifications in test_python.sh * .GITLAB: set gds_path by CPU arch Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: trying to disable gds Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: commented out -e -x Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: adding back some missing APIs and tests (WIP) Signed-off-by: Roie Danino <rdanino@nvidia.com> * BINDINGS/RUST: added back send_local_md Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST/TESTS: adding back missing helper functions Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: fixed import issues Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: added back estimate_xfer_cost Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: reverted commenting set -e / set -x Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST/SRC: added back nixl_cap_estimate_xfer_cost bindings to lib.rs Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: req.inner.as_ptr() -> req.handle() Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUST: removed commented out lines Signed-off-by: Roie Danino <rdanino@nvidia.com> * BINDINGS/RUST: replaced error prints to error! instead of trace,, removed code duplication in build.rs Signed-off-by: Roie Danino <rdanino@nvidia.com> * .GITLAB: minor fixes Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/JENKINS/LIB: added rust tests to blossom CI Signed-off-by: Roie Danino <rdanino@nvidia.com> * .CI/JENKINS/LIB: removed a space Signed-off-by: Roie Danino <rdanino@nvidia.com> * SRC/BINDINGS/RUS: resolved CR comments, changed enable_rust_bindings to just 'rust' nixl -> NIXL, etc Signed-off-by: Roie Danino <rdanino@nvidia.com> * CARGO.TOML: fixed keywords list to fit into crates.io limitations Signed-off-by: Roie Danino <rdanino@nvidia.com> --------- Signed-off-by: Roie Danino <rdanino@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
* NIXLBENCH: Add support for GDS_MT backend Add thread_count backend parameter support as well. Update log messages. --------- Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
* Add option to set build type for nixlbench container, default to release Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add --build-type to help Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Cleanup build script Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Revert unrelated changes Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Nixlbench: also build ucx in debug mode if nixl and nixlbench are built in debug mode Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove valgrind Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* CI: Add 30 min timeout to test steps - Use ci-demo timeout feature to prevent test hangs. - Configurable via TEST_TIMEOUT environment variable. * CI: Add timeout to test scripts running in AWS * CI: Replace ci-demo timeout with per-command timeout wrapper * Set timeout per step assuming ci-demo support * Use ci-demo timeout property per step * CI: Add 30 min timeout to test steps - Use ci-demo timeout feature to prevent test hangs. - Configurable via TEST_TIMEOUT environment variable. * CI: Add timeout to test scripts running in AWS * CI: Replace ci-demo timeout with per-command timeout wrapper * Set timeout per step assuming ci-demo support * Use ci-demo timeout property per step * rm sneaky files from another branch * rm sneaky files from another branch * Add comment * Fix comment * CI: Unify TEST_TIMEOUT to numeric format * CI: add timeout to Rust tests
* Add nixlbench test to CI tests Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix build in AWS image Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add more tests Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Cleanup test script Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Cleanup test script Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Move nixlbench tests to their own script Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add nixlbench test to the CI config Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Simplify ucx path handling in meson.build Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Use dynamic etcd port and fix CI yaml Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Clean up build scripts Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Revert "Simplify ucx path handling in meson.build" This reverts commit 33fcaa2. Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove sudo fron nixl installation commands since tests may need to create files in the install dir Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Revert the other sudo Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove unused UCX dependency in nixlbench Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix nixlbench build with nvshmem Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix CI builds Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Cleanup test script Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add tests to the build matrix Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix merge Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com>
…smashing (ai-dynamo#681) * Bugfix: Replace select() with poll() in connectToIP to prevent stack smashing * return error value check for inet_ntop
Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
* NIXLBENCH: Set RW sync mode for UCX when num_threads > 1 * Reverted API change
* Add tail latency measurements to nixlbench Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Display stats Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix percentile calculation Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix thread safety and mem allocation issues Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove reduntant stats columns Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Adjust table size Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Use nixlTime instead of a different time source Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix nvshmem worker build Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Style fixes Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Change output to display latency with 0.1us precision Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* GTEST: add reconnect case to error handling suite Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * NIXL/AGENT: invalidate remote MD on failure Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * AGENT/DATA: remote data load refactoring + clang-format Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * CLANG: format Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * NIXL/AGENT: invalidate remote MD on failure, CR1 Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * NIXL/AGENT: invalidate remote MD on failure, CR2 Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * Revert "NIXL/AGENT: invalidate remote MD on failure, CR1" This reverts commit 93e7b47. Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * NIXL/AGENT: invalidate remote MD on failure, CR3 Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * GTEST: del TestErrorHandling::Agent::disconnect, Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> * NIXL/AGENT: invalidate remote MD on failure, CR4 - revert refactoring * CORE/AGENT: del leftover * CORE/AGENT: del leftover Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> --------- Signed-off-by: Evgeny Leksikov <evgenylek@nvidia.com> Co-authored-by: Yossi Itigin <yosefe@nvidia.com>
API/GPU/UCX: Implement GPU Device API Signed-off-by: Michal Shalev <mshalev@nvidia.com>
* CORE: Do not delete requests unless we own them Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Update usage Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* Add enable flag to telemetry task: This commit adds enable flag to the telemetry task to make it usage more clear. * Ignore telemetry run time when registering: Existing code reduced the interval each time which will cause a busy loop eventually. * Run pylint on telemetry_reader --------- Signed-off-by: Rafi Wiener <rwiener@nvidia.com> Co-authored-by: Rafi Wiener <rwiener@nvidia.com> Co-authored-by: ovidiusm <ovidium@nvidia.com>
Added captureTelemetry to nixlAgentConfig and added corresponding tests for it. --------- Authored-by: Moein Khazraee <moein@nvidia.com>
* Simplify copyright-check.ps1 support 2 line headers Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * testing... Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * testing 2 Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * testing 3 Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * use bash for copy-right check Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * use bash for copy-right check 2 Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * chmod +x Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * year range Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> * Added non-# comments. Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Enforcing bash to be used for the bash script Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Avoid early exit due to sed failure Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Improved date checks Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Revert last change Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Improved file filtering Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Missing file types, and temp .cu file Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Update copyright-check.sh Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * missing ones ... Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * Revert to see if other parts work Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> * regex doesnot account for leading whitespace Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Adding missing SPDX license header Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Update copyright-check.sh Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Delete copyright-check.ps1 Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Move spdx copyright check above the year check. Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Revert lines added for testing. Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> * Update copyright-check.sh Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> --------- Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> Signed-off-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com> Co-authored-by: Moein Khazraee <33970824+mkhazraee@users.noreply.github.com> Co-authored-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
…namo#784) - Implement new libfabric backend plugin for high-performance networking - Add automatic topology discovery using hwloc for GPU-to-EFA mapping - Include rail management system for load balancing across EFA devices - Update build system with libfabric and EFA installer dependencies - Configure CI/CD pipeline with EFA installer version 1.43.2 --------- Signed-off-by: Ye Xiang <yexiang@amazon.com> Signed-off-by: Arun Karthik <akkart@amazon.com>
Addressed the build failure issue when libfabric is not installed. Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
due to infra issues with our DGX server Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* Upgrade DOCA to 3.1 in nixlbench container Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Simplify scripts Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Do not hardcode distro name Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add download with retry Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
…ai-dynamo#809) - Implement direct memcpy for transfers within the same agent to avoid unnecessary network operations - Add debug logging for transfer path selection and agent comparison - Improve debug messaging for same-agent transfers --------- Signed-off-by: Arun Karthik <akkart@amazon.com>
…ynamo#817) - Refactor libfabric device discovery and connection management - Change getAvailableEfaDevices() to return provider-device mapping - Update connection state handling in disconnect logic - Modify rail manager to support provider-aware device selection - Enhance topology detection for multi-provider environments - Refactor libfabric memory registration to support device-specific operations - Add device ID parameter to memory registration functions - Update rail manager to handle device-aware memory registration - Improve code formatting and line wrapping consistency - Enhance topology management for multi-device scenarios - Remove numa node mapping with EFA, instead select all rails in case of DRAM - Clean up dead code to improve maintainability. - Remove large blocks of commented-out code in nixlLibfabricEngine::disconnect() and simplify libfabric common utilities. --------- Signed-off-by: Arun Karthik <akkart@amazon.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
ovidiusm
force-pushed
the
check-crash
branch
from
September 24, 2025 11:52
7d41ab1 to
99bc5cd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extra logs