[huangxiaobai-ydmy]

Description

Fixes #4694.

A classic WorkerPool that processes IBV_EVENT_PORT_ERR disables its
context and disconnects endpoints. The only thing that arms recovery
afterwards is the IBV_EVENT_PORT_ACTIVE branch. If that event is
never read, recovery_activate_after_ns_ stays 0, maybeActivateRecoveredContext()
is a no-op, and the rail stays inactive until the process restarts.

#3799 already added this safety net for TENT (Workers::resumePausedContexts(),
a 1 Hz ibv_query_port). Classic still drains the async fd, but it
never got the poll.

What changed

• Add RdmaContext::queryPortState() for the classic engine.
• On the 1 Hz monitor tick, if the context is inactive because of a
  fatal async event and no recovery is armed, query the port.
• On IBV_PORT_ACTIVE, call scheduleContextRecovery() so the
  existing 30 s delayed GID probe from [TE] Improve RDMA NIC failover recovery #2959 still decides when the
  RNIC is reused. Intermediate port states stay inactive.
• Do not steal a breaker-owned pause: local-WC TTL recovery still owns
  that path.
• Hardware-free unit tests via WorkerPoolTestPeer and --wrap of
  queryPortState. No IB device required.

@ironashram this follows the fix you outlined on #4694. Sorry I opened it without checking with you first. You wrote that you were happy to send the PR yourself — if you still want to, say so and I will close this one so you can take it.

Test plan

• worker_pool_rail_state_test covers missed PORT_ACTIVE,
  breaker-owned pauses, and an already-armed event recovery.
• CI worker_pool_rail_state_test on this PR.

[ironashram]

@huangxiaobai-ydmy no worries, thanks for working on it so quickly

[huangxiaobai-ydmy]

@ironashram thanks for letting me keep this.

[staryxchen]

resumePausedContext() is documented as the safety net for a missed PORT_ACTIVE, but it polls every inactive context with recovery_activate_after_ns_ == 0 and breaker_reactivate_after_ns_ == 0. handleContextEvent() leaves that same state for DEVICE_FATAL, CQ_ERR, WQ_FATAL, PORT_ERR, and LID_CHANGE, and resetContextBreaker(false) clears the breaker deadline, so the breaker check does not exclude the other four.

A port that is still IBV_PORT_ACTIVE then arms the 30 s GID probe and publishes the RNIC again. That probe does not rebuild the CQ, so a CQ_ERR context comes back on the CQ that failed, and a DEVICE_FATAL context comes back whenever ibv_query_port still returns ACTIVE.

Please remember the disabling event and run this poll only for PORT_ERR. A test for each of the other four, with the port probe returning IBV_PORT_ACTIVE, should leave recovery_activate_after_ns_ == 0 and active() == false.

[huangxiaobai-ydmy]

@staryxchen thanks. The poll now records the disabling event and runs only after PORT_ERR. DEVICE_FATAL, CQ_ERR, WQ_FATAL, and LID_CHANGE stay inactive when the port probe returns ACTIVE; the new rail-state test covers each of them.

[huangxiaobai-ydmy]

@staryxchen the latest CI failure is CTest in mooncake-store: master_service_evict_scenario_test and dfs_sync_client_test. worker_pool_rail_state_test passed. This PR does not touch those store tests, and I cannot rerun the job. Could someone with permission rerun CTest?

[huangxiaobai-ydmy]

@staryxchen the change from your review is on 2575747. The poll records the disabling event and only resumes after PORT_ERR. DEVICE_FATAL, CQ_ERR, WQ_FATAL, and LID_CHANGE stay inactive when the port is still ACTIVE. CI on that commit is green. Please Approve when you have a moment.

[staryxchen]

LGTM, thanks!