[guy-ealey-morag]

What?

Build a device inventory from all of the discovered devices.
The inventory devices are grouped by type (net/acc device), sorted by bdf.
Devices under the same physical device (e.g. NIC ports, GPU sub-devices like with MPS MLOPart) are kept together.
Some filters may be also applied, such as keeping only CX9 NICs when the setup is detected as Vera Rubin.
This inventory will be used to group the devices into GPU-NIC groups, to later be used for GPU-NIC assignments.

How?

• Collect all devices into arrays by type
• Sort by BDF
• Filter out unwanted devices
• Group acc devices of the same physical GPU (BDF is equal)
• Group net devices of the same physical NIC together (BDF is the same except of the function)

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

pls document what sys_devs holds — the grouping rule differs per class (ucs_topo_groups_bus_id_match() uses full BDF for gpus, bdf-excluding-function for nics), so for a gpu element these are the sys_dev aliases of one BDF and for a nic element the functions/ports of one slot. Today that semantic is only visible in the .c file.

[svc-nvidia-pr-review]

minor: with this PR there are three identical ucs_qsort_r adapters around the comparator (ucp_proto_multi_sys_dev_cmp, topo_test_sys_device_cmp, and ucs_topo_groups_sys_dev_cmp). Can we also export a ucs_qsort_r-shaped ucs_topo_sys_device_qsort_cmp(const void*, const void*, void*) here, so the ucp and gtest wrappers can be removed?

[svc-nvidia-pr-review]

src/ucs/sys/topo/base/topo_groups.c

minor: the function builds group elements out of the collected devices, so ucs_topo_groups_elements_build would match <layer>_<object>_<verb> (docs/CodeStyle.md) better than devices_build.

src/ucs/sys/topo/base/topo_groups.c

minor: this static wrapper is a single ucs_array_init_dynamic() call used once — can we drop it and call ucs_array_init_dynamic(&groups) directly in ucs_topo_build_groups_inner()?

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, so the new inventory path (collect, sort, GPU alias filter, CX9 fw_ver probe, element grouping and the UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT limit) is not exercised by CI — in particular the NIC-by-slot vs GPU-by-BDF rule that was broken in the previous revision would not have been caught. The new sys_device_cmp test covers the comparator well for registered devices (BDF-primary order, user-value tie-break, EMPTY sorts last, resulting ucs_qsort_r order), but not an unregistered/UNKNOWN sys_dev.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/ucs/sys/topo/base/topo_groups.c

minor: this line is 81 columns and the continuation lines are indented one column short of the open paren (docs/CodeStyle.md: up to 80 columns, indent function arguments on column). The two call sites in ucs_topo_groups_inventory_build are off by one column too.

static ucs_status_t ucs_topo_groups_elements_build(
        const ucs_topo_sys_device_info_t *devices,
        const ucs_topo_groups_sys_dev_array_t *sys_devices,
        const ucs_topo_device_class_t device_class,
        ucs_topo_group_element_array_t *elements)

test/gtest/ucs/test_topo.cc

minor: nothing in the new test uses topo_groups.h — pls drop this include. The new sys_device_cmp test only uses ucs_qsort_r and ucs_topo_sys_device_cmp; nothing from topo_groups.h is referenced, and topo_groups.h pulls in the internal topo_int.h.

Not re-raised as a comment (already covered in earlier threads): ucs_topo_build_groups() still has no production caller and no gtest, so the collect/sort/GPU-alias-filter/CX9-fw_ver/element-grouping path added here is unexercised by CI. The new sys_device_cmp test is solid for the comparator with registered devices (BDF-primary order, user-value tie-break, EMPTY sorts last, resulting ucs_qsort_r order) but does not cover an unregistered/UNKNOWN sys_dev, which is exactly the input that triggers the previously reported out-of-bounds read.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

ucp_proto_multi_sys_dev_cmp here, ucs_topo_groups_sys_dev_cmp in topo_groups.c, and topo_test_sys_device_cmp in test_topo.cc are the same qsort_r trampoline. Can we export a qsort_r-compatible comparator from ucs/sys/topo/base/topo.h and drop this one and the gtest copy?

[svc-nvidia-pr-review]

minor: ucs_topo_group_array_t is now used only by this typedef — can we name the array type ucs_topo_groups_t directly in the UCS_ARRAY_DECLARE_TYPE above and drop this line?

[svc-nvidia-pr-review]

minor: inventory is a ucs_topo_group_t, but it holds every GPU/NIC in the system, while the type is documented as "Group of GPUs and NICs sharing a topology locality". Can we add a one-line comment that the inventory is the pre-split single-group form?

[svc-nvidia-pr-review]

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, so the collect/sort/alias-filter/CX9-fw_ver/element-grouping code added here is not exercised by CI, and ucs_topo_groups_build_groups() is still a stub that discards the inventory. The new sys_device_cmp test covers the comparator with registered devices (BDF-primary order, user-value tie-break, EMPTY sorts last, resulting ucs_qsort_r order) but not an unregistered/UNKNOWN sys_dev.

[tvegas1]

remove log indent's below, move this ucs_debug() at line of ucs_log_indent(-1) and add boolean as a string "added" / "skipped" or similar also add sys_dev and maybe index src

[guy-ealey-morag]

I don't think we want this, the goal of printing this before the indent is that we can see which device is related to the logs from ucs_topo_groups_is_nic_cx9

[guy-ealey-morag]

Updated the log message to include sys_dev

[tvegas1]

something like below would log only one line always with context/one line, could help when there is another is_cx9 user to avoid print logs without contenxt / make logs more concise.

static int
ucs_topo_groups_is_nic_cx9(const ucs_topo_sys_device_info_t *device,
                           ucs_sys_device_t sys_dev)
{
    const ucs_sys_pci_id_t *pci_id = &device->pci_id;
    char fw_ver[UCS_TOPO_GROUPS_FW_VER_MAX];
    const char *msg;
    ucs_status_t status;
    int is_cx9 = 0;
    if (pci_id->vendor != UCS_TOPO_GROUPS_MELLANOX_VENDOR_ID) {
        msg = "vendor mismatch, skipped";
    } else if (pci_id->device == UCS_TOPO_GROUPS_CX9_DEVICE_ID) {
        msg     = "CX9 device ID, kept";
        is_cx9 = 1;
    } else if (pci_id->device != UCS_TOPO_GROUPS_MLX5_VF_DEVICE_ID) {
        msg = "device ID mismatch, skipped";
    } else {
        status = ucs_topo_groups_read_ib_fw_ver(
                &device->bus_id, fw_ver, sizeof(fw_ver));
        if (status != UCS_OK) {
            msg = "firmware read failed, skipped";
        } else if (strncmp(fw_ver, "82.", 3) != 0) {
            msg = "firmware version mismatch, skipped";
        } else {
            msg     = "CX9 firmware version, kept";
            is_cx9 = 1;
        }
    }
    ucs_debug("cx9_filter: sys_dev=%u, bus_id=" UCS_SYS_BUS_ID_FMT
              ", pci_id=" UCS_SYS_PCI_ID_FMT ": %s",
              sys_dev, UCS_SYS_BUS_ID_ARG(&device->bus_id),
              UCS_SYS_PCI_ID_ARG(pci_id), msg);
    return is_cx9;
}

[guy-ealey-morag]

Applied

[tvegas1]

replace this third parameter by function pointer ucs_topo_groups_bus_id_same_slot and remove assert/dispatch function

[guy-ealey-morag]

I don't think it's needed, is there a clear advantage?

[tvegas1]

it will remove the dispatch function using it

[tvegas1]

replace this third parameter by function pointer ucs_topo_groups_bus_id_equal and remove assert/dispatch function

[guy-ealey-morag]

Related #11869 (comment)

[tvegas1]

it will remove the dispatch function using it

[tvegas1]

 src/ucs/sys/topo/base/topo_groups.c | 25 +++++--------------------
 1 file changed, 5 insertions(+), 20 deletions(-)

diff --git a/src/ucs/sys/topo/base/topo_groups.c b/src/ucs/sys/topo/base/topo_groups.c
--- a/src/ucs/sys/topo/base/topo_groups.c
+++ b/src/ucs/sys/topo/base/topo_groups.c
@@
-static int ucs_topo_groups_bus_id_match(ucs_topo_device_class_t device_class,
-                                        const ucs_sys_bus_id_t *bus_id1,
-                                        const ucs_sys_bus_id_t *bus_id2)
-{
-    if (device_class == UCS_TOPO_DEVICE_CLASS_ACC) {
-        /* Accelerator devices (GPUs) are grouped by full bus id equality. */
-        return ucs_topo_groups_bus_id_equal(bus_id1, bus_id2);
-    }
-
-    /* Network devices (NICs) are grouped by bdf equality excl. the function. */
-    ucs_assert(device_class == UCS_TOPO_DEVICE_CLASS_NET);
-    return ucs_topo_groups_bus_id_same_slot(bus_id1, bus_id2);
-}
-
 static ucs_status_t ucs_topo_groups_elements_build(
         const ucs_topo_sys_device_info_t *devices,
         const ucs_topo_groups_sys_dev_array_t *sys_devices,
-        const ucs_topo_device_class_t device_class,
+        int (*bus_id_match)(const ucs_sys_bus_id_t *bus_id1,
+                            const ucs_sys_bus_id_t *bus_id2),
         ucs_topo_group_element_array_t *elements)
 {
@@
-    ucs_assert((device_class == UCS_TOPO_DEVICE_CLASS_ACC) ||
-               (device_class == UCS_TOPO_DEVICE_CLASS_NET));
     ucs_assert(ucs_array_begin(sys_devices) != NULL);
@@
-        if ((prev_bus_id == NULL) ||
-            !ucs_topo_groups_bus_id_match(device_class, prev_bus_id, bus_id)) {
+        if ((prev_bus_id == NULL) || !bus_id_match(prev_bus_id, bus_id)) {
             /* No match, add a new element. */
             element = ucs_array_append(elements, return UCS_ERR_NO_MEMORY);
@@
     status = ucs_topo_groups_elements_build(devices, &acc_devices,
-                                            UCS_TOPO_DEVICE_CLASS_ACC,
+                                            ucs_topo_groups_bus_id_equal,
                                             &inventory.gpus);
@@
     status = ucs_topo_groups_elements_build(devices, &net_devices,
-                                            UCS_TOPO_DEVICE_CLASS_NET,
+                                            ucs_topo_groups_bus_id_same_slot,
                                             &inventory.nics);

[guy-ealey-morag]

Applied

[tvegas1]

lowercase N

[tvegas1]

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, so the collect/sort/alias-filter/CX9-fw_ver/element-grouping code added here is not exercised by CI, and ucs_topo_groups_build_groups() is still a stub that discards the inventory.

bot commented like above, could we add test to this pr addressing it and/or remove untested code? if it is from another pr could we maybe add some of them here?

[guy-ealey-morag]

Related #11869 (comment)

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Previously reported, still present: fw_ver sysfs I/O (opendir/readdir/ucs_read_file_str) plus ucs_malloc and ucs_debug run under ucs_topo_global_ctx.lock, held by ucs_topo_build_groups() → ..._inner → inventory_build → nics_cx9_filter → read_ib_fw_ver (see also topo.c L1417–1421 and topo_groups.c L201).

[svc-nvidia-pr-review]

Previously reported, still present: is_vera_rubin is hardcoded to 1 at its single call site.

[svc-nvidia-pr-review]

Previously reported, still present: element is declared uninitialized in ucs_topo_groups_elements_build and dereferenced at L325 outside the assigning branch (-Wmaybe-uninitialized risk with -Werror).

[svc-nvidia-pr-review]

src/ucs/sys/topo/base/topo_groups.h

minor: ucs_topo_group_array_t is now used only to define ucs_topo_groups_t — can we declare the array type as ucs_topo_groups_t directly and drop the alias? Two interchangeable names for one type invite inconsistent use in the follow-up that implements ucs_topo_groups_build_groups().

src/ucp/proto/proto_multi.c

Previously reported, still present: out-of-bounds read in ucs_topo_sys_device_cmp_nolock() on release builds — devices[] has UCS_SYS_DEVICE_ID_COUNT (255) entries and only ucs_assertv guards it, while ucp_proto_multi_sys_dev_cmp can be handed UCS_SYS_DEVICE_ID_UNKNOWN (255) from ucp_proto_common_get_sys_dev(). This change also drops the previous ucs_fatal in proto_multi.c.

src/ucs/sys/topo/base/topo_groups.c

Previously reported, still present: CX9 vendor/device-id and mlx5-VF fw_ver policy lives in UCS (src/AGENTS.md: "ucs: shared infrastructure only"), duplicating uct_ib_builtin_device_specs.

src/ucs/sys/topo/base/topo_groups.c

Previously reported, still present: ucs_topo_groups_sys_dev_cmp sorts using ucs_topo_global_ctx.devices while the rest of the file uses the devices parameter.

src/ucs/sys/topo/base/topo_groups.c

Previously reported, still present: returning UCS_ERR_EXCEEDS_LIMIT on > UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT fails the whole group build.

src/ucs/sys/topo/base/topo_groups.h

Previously reported, still present: ucs_topo_build_groups() returning UCS_ERR_UNSUPPORTED with *groups_p untouched is not documented in topo_groups.h.

src/ucs/sys/topo/base/topo_groups.c

Previously reported, still present: gpu_aliases_filter i += 2 stepping; redundant ucs_assert(... != NULL) / ucs_array_begin() != NULL asserts; single-use ucs_topo_init_groups() wrapper.

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, and ucs_topo_groups_build_groups() remains a stub that discards the inventory, so collect/sort/alias-filter/CX9-fw_ver/element-grouping are unexercised by CI — the NIC-by-slot vs GPU-by-BDF bug fixed in this revision would not have been caught by any test. The new sys_device_cmp test is solid for registered devices (BDF-primary order, user-value tie-break, EMPTY sorts last, resulting ucs_qsort_r order) but does not cover an unregistered/UNKNOWN sys_dev, which is the input behind the out-of-bounds read.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR · commit cab9de06

TL;DR: The "Basic compile" job failed with -Werror=maybe-uninitialized on reason in ucs_topo_groups_nics_cx9_filter(), because ucs_topo_groups_is_nic_cx9() takes const char *reason by value so the caller's variable is never written; change the parameter to const char **reason and pass &reason.

Full analysis

Summary: Azure Pipelines "Basic compile" stage of PR #11869 failed compiling src/ucs/sys/topo/base/topo_groups.c — GCC -Werror=maybe-uninitialized on variable reason.

Root cause: ucs_topo_groups_is_nic_cx9() is declared as static int ucs_topo_groups_is_nic_cx9(const ucs_topo_sys_device_info_t *device, const char *reason) (topo_groups.c:177-178). The function assigns reason = "..." in each branch, but since the pointer is passed by value, those assignments are local to the callee and never propagate back. In the caller ucs_topo_groups_nics_cx9_filter() (topo_groups.c:226), const char *reason; is declared uninitialized, passed at line 241, and then read at line 252 inside ucs_debug(...). GCC correctly detects the uninitialized read; with -Wall -Werror in the build flags this aborts the build:

topo_groups.c:241:18: error: 'reason' may be used uninitialized [-Werror=maybe-uninitialized]
  241 |         is_cx9 = ucs_topo_groups_is_nic_cx9(device, reason);
cc1: all warnings being treated as errors
make[3]: *** [Makefile:1967: sys/topo/base/libucs_la-topo_groups.lo] Error 1

This is a real logic bug, not just a warning — the debug message would print a garbage/stale pointer.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCS/TOPO: PR comments fix", Guy Ealey Morag (the head commit of branch topo-build-inventory; the reason out-param was introduced/reshaped in this series, cf. dcd7127e, 33a2ba7b).

File: src/ucs/sys/topo/base/topo_groups.c:178 (signature), :241 and :252 (call site / use)

Suggested fix: Make reason a genuine output parameter:

static int ucs_topo_groups_is_nic_cx9(const ucs_topo_sys_device_info_t *device,
                                      const char **reason_p)
{
    ...
    if (pci_id->vendor != UCS_TOPO_GROUPS_MELLANOX_VENDOR_ID) {
        *reason_p = "not a mellanox device";
        return 0;
    }
    /* ... same for every other branch ... */
    *reason_p = "cx9 device by firmware version";
    return 1;
}

and at the call site (line 241):

is_cx9 = ucs_topo_groups_is_nic_cx9(device, &reason);

Also initialize const char *reason = "unknown"; at line 226 as defensive hardening so any future branch that forgets to set it cannot reintroduce an uninitialized read. Verify locally with the same flags (-O0 -g -Wall -Werror) via contrib/test_jenkins.sh / a plain ./configure && make -j4 before pushing.

Related: PR #11869 (openucx/ucx) — this is a self-contained defect in the PR itself, no pre-existing issue found.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e672993d-8305-47b8-8210-c75f50acf477 in the triage console for the audit trail.

[svc-nvidia-pr-review]

src/ucs/sys/topo/base/topo_groups.c

blocker: in ucs_topo_groups_is_nic_cx9, reason is declared as const char *reason (passed by value) but the function treats it as an output — all five reason = "..." assignments write to the local copy and are discarded. The caller (ucs_topo_groups_nics_cx9_filter) declares const char *reason; uninitialized and then reads it in ucs_debug("cx9_filter: ... (%s)", ..., reason), which is undefined behavior / likely segfault whenever debug logging is enabled on a Vera Rubin node, and the log never carries the intended reason. It is also very likely to break the -Werror CI jobs via -Wmaybe-uninitialized on the caller and -Wunused-but-set-parameter on the callee. Please make it a real output parameter: const char **reason_p, assign *reason_p = "..." in each branch, and pass &reason from the caller (per docs/CodeStyle.md, output pointer args take a _p suffix).

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, and ucs_topo_groups_build_groups() remains a stub that discards the inventory — so none of the new code (collect, sort, GPU alias filter, CX9 fw_ver probe, element grouping) is exercised by CI. In particular, the blocker above sits on the CX9 path and no test would catch it.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/ucs/sys/topo/base/topo_groups.c

minor: pls drop the = from the log message (docs/LoggingStyle.md: prefer "device %s" over "device=%s", so the value can be selected by double-click and grepped); the last two arguments also fit on one line.

        ucs_debug("cx9_filter: network device sys_dev %u bus_id "
                  UCS_SYS_BUS_ID_FMT " pci_id " UCS_SYS_PCI_ID_FMT " %s (%s)",
                  sys_dev, UCS_SYS_BUS_ID_ARG(&device->bus_id),
                  UCS_SYS_PCI_ID_ARG(&device->pci_id),
                  is_cx9 ? "added" : "skipped", reason);

src/ucs/sys/topo/base/topo_groups.h

minor: after this move, both ucs_topo_init_group() and ucs_topo_release_group() are used only inside topo_groups.c (the public entry points are ucs_topo_build_groups/ucs_topo_release_groups). Can we make them static and drop both declarations, per docs/CodeStyle.md ("functions used only within one source file must be declared static")? Same rule that was applied to ucs_topo_init_groups.

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present in this revision): element declared without = NULL in ucs_topo_groups_elements_build() (-Wmaybe-uninitialized build risk in -Werror jobs).

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): fw_ver sysfs I/O (opendir/readdir/ucs_read_file_str) executed under the topo spinlock held by ucs_topo_build_groups().

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): CX9 vendor/device-id + mlx5-VF fw_ver policy living in UCS, duplicating uct_ib_builtin_device_specs.

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): ucs_topo_groups_sys_dev_cmp() sorting via ucs_topo_global_ctx.devices while the rest of the file uses the devices parameter.

src/ucs/sys/topo/base/topo.c

Not re-raised (already in open threads, still present): out-of-bounds read in ucs_topo_sys_device_cmp_nolock() on release builds — devices[] holds UCS_SYS_DEVICE_ID_COUNT (255) entries, only ucs_assertv guards the index, and ucp_proto_multi_sys_dev_cmp() can still be handed UCS_SYS_DEVICE_ID_UNKNOWN (255) from ucp_proto_common_get_sys_dev(); this also removes the previous ucs_fatal in proto_multi.c. Note the new doxygen in topo.h documents the "must be registered (not UNKNOWN)" contract that this caller can violate.

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): is_vera_rubin hardcoded to 1 at the only call site.

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): UCS_ERR_EXCEEDS_LIMIT failing the whole group build on > UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT.

src/ucs/sys/topo/base/topo_groups.h

Not re-raised (already in open threads, still present): UCS_ERR_UNSUPPORTED / untouched *groups_p not documented in topo_groups.h.

src/ucs/sys/topo/base/topo_groups.c

Not re-raised (already in open threads, still present): gpu_aliases_filter i += 2 stepping; redundant ucs_assert(... != NULL) / ucs_array_begin() != NULL asserts.

Residual coverage: ucs_topo_build_groups() still has no production caller and no gtest, and ucs_topo_groups_build_groups() remains a stub that receives the inventory and discards it, so the collect/sort/GPU-alias-filter/CX9-fw_ver/element-grouping code is unexercised by CI — the CX9 reason bug fixed in this revision sat exactly on that path and no test would have caught it. The sys_device_cmp gtest is solid for the comparator with registered devices (BDF-primary order, user-value tie-break, EMPTY sorts last, resulting ucs_qsort_r order), but not for an unregistered/UNKNOWN sys_dev, which is the input behind the out-of-bounds read.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Build build_x86_64 debian113) · commit 7a2e3fff

TL;DR: The build_x86_64 debian113 job was killed externally — "The agent has received a shutdown signal" — while make distclean was running between build variants; there is no compile/test error in the log, so this is CI infrastructure noise (agent/pod terminated or run superseded), and the fix is to re-run the job.

Full analysis

Summary: Azure job build_x86_64 debian113 (build 135410, step "Build on x86_64") ended with ##[error]The agent has received a shutdown signal / The operation was canceled during a make distclean cleanup pass — not from any build error.

Root cause: Infrastructure abort, not a code or build failure. Evidence from the log:

• The last ~12 seconds of output (11:10:09.36 → 11:10:21.07) are dense, sub-second-spaced rm -f / make[N]: Entering/Leaving directory lines with no gap larger than ~0.5 s. There is no stall anywhere in the visible window, so this is not a hang and not a job that "needed more time".
• The work in progress was make distclean under .../135410-20260910.14/build/..., i.e. the teardown/cleanup that buildlib/tools/builds.sh performs between successive configure/build variants. That means at least one configuration had already compiled successfully; no error:, undefined reference, or non-zero make exit appears.
• At 11:10:10.96 Azure injected ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled., immediately followed by The Operation will be canceled. The next steps may not contain expected logs. This message comes from the agent process itself receiving SIGTERM — the container/agent pod was stopped or the run was cancelled/superseded — not from the build script returning a failure. Note the remaining distclean output continues to interleave after the error, confirming the build process was still healthily running when the agent went down.
• Subsequent steps show Skipping step due to condition evaluation and Finalize Job … The operation was canceled, the standard cascade after an agent-level cancellation.

Since the pipeline is PR-triggered (buildlib/azure-pipelines-pr.yml, pr: branches: master, v*.*.x), the most likely trigger is Azure's default "automatically cancel in-progress runs" firing when a newer commit was pushed to PR #11869 (branch topo-build-inventory), with the alternative being agent-pod reclamation on the swx-azure-svc_azpcontainer host. Nothing in commit 7a2e3ff is implicated by the log.

Implicated commit: unknown — no build error attributable to [REDACTED:Hex High Entropy String] appears in the log

File: n/a (no source file implicated; failing step is "Build on x86_64" driven by buildlib/tools/builds.sh)

Suggested fix: Re-queue build 135410 (or push/re-trigger CI on PR #11869) and check whether the run was superseded by a later commit on topo-build-inventory — if a newer run for the same PR exists and passed, this build can be ignored. If the abort repeats on the same agent pool without a superseding commit, escalate to CI infra to check for agent pod evictions/OOM-kills on the debian113 container host; there is no source change to make here. Do not raise any job timeout — the log shows continuous progress up to the kill, so no timeout was involved.

Related: PR #11869 (this build). No existing issue found matching the agent-shutdown signature; the issue search returned only unrelated PRs (#11921, #11895, #11845, #11842, #11857).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 27139c7d-ac7e-4c63-b6e7-1a713eddf365 in the triage console for the audit trail.

[tvegas1]

should not be needed

[guy-ealey-morag]

It's for coverty, otherwise it fails because it's thinks the array pointer may be null