[Core][KV Connector] Remove use of num_cached_tokens in error handling

[markmc]

Refactor _update_requests_with_invalid_blocks() to avoid recomputation logic based on num_cached_tokens, simplifying the logic, and making the sync-shared-blocks case less special.

This also paves the way for refactoring prefill cache statistics in #37460 in an effort to stamp out reports of
Counters can only be incremented by non-negative amounts.

Based on Or's work in commit f02a5c80 of #35223

[gemini-code-assist]

Code Review

This pull request refactors the handling of invalid KV cache blocks within the scheduler. It introduces a num_scheduled_tokens parameter to _handle_invalid_blocks and _update_requests_with_invalid_blocks to refine how computed tokens are tracked, especially for newly scheduled tokens. The logic for truncating computed tokens and identifying blocks for eviction has been updated, incorporating a new cdiv utility. A review comment highlights a potential performance improvement: newly allocated, empty blocks that become invalid should be directly freed using kv_cache_manager.free() instead of being added to blocks_to_evict, as eviction might incur unnecessary overhead for empty blocks.

[orozery]

This change looks bigger than what it can be.
Can't we just replace:

                # Sync loading. num_computed_tokens includes new tokens
                req_num_computed_tokens = request.num_cached_tokens

with:

                # Sync loading. num_computed_tokens includes new tokens
                req_num_computed_tokens = (
                    request.num_computed_tokens - num_scheduled_tokens.get(req_id, 0)
                )

and replace the rest of request.num_cached_tokens with req_num_computed_tokens ?

[markmc]

This change looks bigger than what it can be.

Ok, fair. I took the refactoring a bit further, but that can come later if it is valuable

[markmc]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the KV cache invalidation logic within the scheduler by introducing num_scheduled_tokens to _handle_invalid_blocks and _update_requests_with_invalid_blocks. This new parameter allows for a more precise calculation of req_num_computed_tokens by subtracting scheduled but not yet computed tokens from the total, ensuring accurate token counts when handling invalid KV cache blocks. This change streamlines the logic for both asynchronous and synchronous loading scenarios. I have no feedback to provide.

[orozery]

LGTM