Skip to content

[Bugfix][TENT] Fail stop on terminal RDMA teardown errors - #4383

Merged
alogfans merged 1 commit into
kvcache-ai:mainfrom
OrbitInfra:fix/tent-rdma-terminal-teardown
Oct 27, 2026
Merged

alogfans merged 1 commit into
kvcache-ai:mainfrom
OrbitInfra:fix/tent-rdma-terminal-teardown

Conversation

@feichai0017

@feichai0017 feichai0017 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

If native QP destruction or notification MR deregistration fails, deconstruct() preserves endpoint resources for retry. disable() similarly preserves a context when its endpoint store cannot clear. Their destructors currently discard those errors and continue destroying the owners that were meant to be retained.

Check both terminal cleanup results and fail-stop before member destruction if cleanup still fails. Explicit deconstruct() / disable() callers retain the existing error-and-retry behavior. This trades a fatal process error for potentially releasing memory still owned by native resources; it does not add timeout-based reclamation or change ordinary successful teardown.

Fixes #4382

Related: #2664, #4062 and #4263. This targets terminal destructor error handling, independently of CQ retirement/drain and context-acquisition failure coverage.

Module

  • Transfer Engine (mooncake-transfer-engine)

Type of Change

  • Bug fix

How Has This Been Tested?

Based on 1ad008bc41c6de7c74e02fb1d48713309b880c0d, Linux x86_64, CPU-only Clang 20 build. The existing per-context verbs injection seam drives the real endpoint teardown and FIFO-store/context clear paths.

cmake --build build --target tent_endpoint_lifecycle_test tent_rdma_transport_test --parallel 4
build/tent/tests/tent_endpoint_lifecycle_test
build/tent/tests/tent_rdma_transport_test
# Separate build with -fsanitize=address,undefined -fno-omit-frame-pointer
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 UBSAN_OPTIONS=halt_on_error=1 \
  build-asan/tent/tests/tent_endpoint_lifecycle_test
  • Negative control: unmodified production code passes both explicit-retry cases but fails all three new terminal-destructor death tests.
  • Fixed endpoint suite: 29 passed, including failed QP destruction, failed MR deregistration, successful explicit retry and context-store teardown failure.
  • RDMA regression suite: 104 passed, 2 skipped because they require RDMA hardware.
  • Full native ASan/UBSan build: all 29 endpoint tests passed, with leak detection enabled.
  • Scoped pre-commit passed for the three changed files. No RNIC/provider fault-injection qualification is claimed.

Checklist

  • Human submitter has reviewed every changed line (draft)
  • Changed C++ lines formatted using the project hook
  • Pre-commit passed for the three changed files
  • Regression tests added and negative control executed
  • No architectural RFC required (small production change)

AI Assistance Disclosure

  • AI tools were used

Codex inspected current main and related issues/PRs, prepared the checks and regression tests, and ran baseline/fixed/sanitizer suites. This draft is pending human line-by-line review; no human-review completion is asserted.

@feichai0017
feichai0017 marked this pull request as ready for review September 29, 2026 04:42
Copilot AI balanced review requested due to automatic review settings September 29, 2026 04:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@alogfans alogfans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alogfans
alogfans merged commit 1dd37f5 into kvcache-ai:main Oct 27, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][TENT] Terminal destructors discard failed RDMA teardown and release retained owners

3 participants