Avoid multimodal path allocation on text-only turns - #1789
Merged
Conversation
Contributor
ApprovabilityVerdict: Approved Performance optimization that avoids allocating multimodal path structures when processing text-only turns. The change is self-contained within You can customize Macroscope's approvability policy. Learn more. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Avoid materializing a full
(node_id, message)path for text-only V1 turns. New node IDs are tracked independently for routed-expert attribution, while multimodal turns continue to build the complete prompt path needed for cursor alignment.Why
_commit_turnpreviously built a tuple path for every reused prompt message before calling_attribute_mm. On text-only turns,multi_modal_datais absent and that helper returns immediately, leaving an O(P) container allocation and pass with no multimodal work to perform.Training responses still perform the required O(P) token-prefix validation. This change removes the separate, redundant path construction; it does not claim to make training commits sublinear.
Implementation
Performance
An isolated PEP 723 microbenchmark used 200,000 reused text messages, three repetitions,
time.perf_counter()for median wall time, andtracemallocfor Python allocations.The benchmark isolates the removed allocation. End-to-end training gains may be smaller because token-prefix validation remains, while the peak temporary-memory reduction directly reflects the eliminated list and tuple containers.
Note
Low Risk
Localized refactor in graph commit logic; multimodal and routed-expert behavior are preserved by construction, with no auth or data-model changes.
Overview
_commit_turnno longer builds a full(node_id, message)path on every turn. Text-only commits track new nodes innew_node_idsas they are created and skip the O(prefix) list/tuple work that_attribute_mmwould not use anyway.When
multi_modal_datais present, the reused prefix is materialized intomm_pathand extended for new input messages so image cursor attribution is unchanged. Routed-expert slicing still uses the samenew_node_idsordering (new inputs, then the assistant node).Reviewed by Cursor Bugbot for commit 8cf96d9. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Skip multimodal path allocation in
_commit_turnon text-only turnsIn
graph.py,_commit_turnpreviously built a unifiedpathlist and called_attribute_mmunconditionally, even when no multimodal data was present. This refactor splitspathintonew_node_ids(always tracked) andmm_path(only materialized whenmulti_modal_dataexists), and gates the_attribute_mmcall onmm_pathbeing non-None.Macroscope summarized 8cf96d9.