[he-yufeng]

What

Fix the SIGSEGV reported in #4761: with the local hot cache enabled
(MC_STORE_LOCAL_HOT_CACHE_SIZE), batch_get_into /
batch_get_into_multi_buffers crash the reading process when the destination
buffer is CUDA memory.

Root cause

After a successful remote read, Client::ProcessSlicesAsync submits every
destination slice to LocalHotCacheHandler::SubmitPutTask, which copies the
data into the cache block with a host std::memcpy(block->addr, slice.ptr, slice.size) (local_hot_cache.cpp). When the destination is a device pointer,
that host memcpy dereferences a device address and the process dies
(__memcpy_avx512_unaligned in the reporter's stack).

Single get_into into CUDA and batch reads with the hot cache disabled never
reach this path, which matches the reporter's matrix of what works and what
crashes.

Fix

In ProcessSlicesAsync, skip the writeback for slices whose destination is
device memory, using the existing accelerator registry probe
(RuntimeAccelerators().FindDeviceForPointer) — the same discriminator the
checksum path (ComputeObjectChecksumForSlices) and the DFS staging paths
already use. Device-resident read results are simply not promoted into the
host hot cache; the read itself is unaffected. On builds without any
accelerator backend the registry is empty and behavior is unchanged.

Caching device-destined reads by staging D2H into the cache block is a
possible follow-up, but silently dropping the writeback is the safe fix: it
restores the pre-hot-cache behavior for exactly the configurations that crash
today.

Test

New regression test
LocalHotCacheTest.BatchGetSkipsHotCacheWritebackForDeviceDestination:

• registers a fake AcceleratorDevice (same pattern as
  runtime_accelerator_test.cpp) that classifies one static buffer as device
  memory, so the guard is exercised on CPU-only runners;
• runs a real two-client TCP transfer (writer owns the segment, reader owns
  none) so the writeback path in ProcessSlicesAsync actually executes;
• asserts the device-destined key is not in the hot cache while the
  host-destined control key from the same batch is (waits for the async
  publish, so the skip cannot pass vacuously).

Verified on Linux (ubuntu:22.04 container, aarch64): the new test fails with
the guard removed (the fake-device key gets cached) and passes with it.

Fixes #4761

[Aionw]

Nit: maybe a logging here to show how many slice we skipped.

[he-yufeng]

Counted and logged per batch now (device_dest_slices -> one INFO line with key and skip count). 40/40 in the Linux container.

[he-yufeng]

CI note: the one CTest failure is dfs_sync_client_test, and it fails identically on main (run 39242136165, same "1 tests failed out of 258", same test #180), so it is pre-existing and not caused by this PR. The new client_local_hot_cache_test including the added BatchGetSkipsHotCacheWritebackForDeviceDestination case passes (17.1s in this run). Re-triggered once already and the failure reproduced on the same test, which also matches the main-branch behavior.

[he-yufeng]

Done — skipped slices are now counted and logged once per batch (Skipped hot-cache writeback for N device-resident slice(s), key=...). Rebuilt and re-ran client_local_hot_cache_test in the Linux container: 40/40 pass.

[he-yufeng]

Status note: the CTest failure blocking the CI Gate here is the DfsOnlyBatchFailureOnlyRevokesFailedKey main breakage from #4637; the fix is now #4781 (I closed my duplicate #4785 in its favor after verifying both in a container). Once either lands, a rebase here should turn the gate green.

[he-yufeng]

Rebased onto current main (5cf142e), which now carries #4781's DfsOnlyBatchFailureOnlyRevokesFailedKey fix — that was the failing gate here. Diff unchanged. Re-verified in the Linux container after the rebase: client_local_hot_cache_test builds clean and all 40 tests pass.