Repository navigation
refactor: adopt LineMap, remove position wrapper layers - #466
Merged
Merged
Conversation
Adopt kotatsu's decomposed position mapping API (build_line_starts, to_position, to_offset) to eliminate the PositionMapper class abstraction. - Store line_starts in Session, rebuilt on didOpen/didChange - Replace OpenFileIndex with lightweight FileOverlay view struct - Cache line_starts lazily in MergedIndexShard for index queries - Update all feature files, compiler, indexer, and agent client - Bump kotatsu to 980c178 (merged PR #171)
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughRemoves ChangesPositionMapper → line_starts Refactor
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e1922fa12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/feature/code_completion.cpp`:
- Around line 258-262: In the code_completion.cpp file around the Range
construction, replace the direct dereferencing of lsp::to_position calls with a
defensive pattern. First, assign the results of lsp::to_position for both
prefix.range.begin and prefix.range.end to separate variables (e.g., start_pos
and end_pos), then add null checks (if(!start_pos || !end_pos)) before
dereferencing them to initialize the protocol::Range fields. This matches the
defensive null-checking pattern already used in workspace.cpp and
tu_index_tests.cpp.
In `@src/feature/feature.h`:
- Around line 26-33: The to_range function dereferences the results of
lsp::to_position() calls without validating that the returned optional values
are valid, which can cause undefined behavior if the optional is empty. Fix this
by either adding assert statements to validate both optional results before
dereferencing them (documenting the precondition), or change the return type
from protocol::Range to std::optional<protocol::Range> and check both
lsp::to_position() results, returning an empty optional if either call fails.
In `@src/server/workspace/workspace.h`:
- Around line 98-117: The `line_starts()` const method in the `MergedIndexShard`
struct has a data race when modifying the `mutable` member `cached_line_starts`
without synchronization. Add a `mutable` mutex member variable to
`MergedIndexShard` and use it to synchronize access to `cached_line_starts` in
both the `line_starts()` method (where the check and assignment occur) and the
`invalidate()` method. This ensures that the check-then-assign operation on the
cache is atomic and prevents concurrent threads from simultaneously accessing or
modifying the vector.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1d3703fb-89a5-4812-a5c8-213002334cf7
📒 Files selected for processing (26)
cmake/package.cmakesrc/feature/code_completion.cppsrc/feature/diagnostics.cppsrc/feature/document_links.cppsrc/feature/document_symbols.cppsrc/feature/feature.hsrc/feature/folding_ranges.cppsrc/feature/formatting.cppsrc/feature/hover.cppsrc/feature/inlay_hints.cppsrc/feature/semantic_tokens.cppsrc/server/compiler/compiler.cppsrc/server/compiler/indexer.cppsrc/server/compiler/indexer.hsrc/server/service/agent_client.cppsrc/server/service/lsp_client.cppsrc/server/service/master_server.cppsrc/server/service/session.hsrc/server/workspace/workspace.cppsrc/server/workspace/workspace.htests/unit/feature/document_link_tests.cpptests/unit/feature/document_symbol_tests.cpptests/unit/feature/folding_range_tests.cpptests/unit/feature/inlay_hint_tests.cpptests/unit/feature/semantic_tokens_tests.cpptests/unit/index/tu_index_tests.cpp
Lazily compute and cache line start offsets in CompilationUnitRef, avoiding repeated scans of interested_content() across feature calls.
Remove the intermediate FileOverlay view type — Indexer now receives const Session& directly. Add to_range/to_position overloads taking CompilationUnitRef to simplify feature-layer call sites.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/server/compiler/indexer.h (1)
285-286: 💤 Low valueConsider renaming member variable to match new semantics.
The documentation now says "Iterates all open Sessions" but the member is still named
each_overlay. This could cause confusion for future maintainers.- /// Iterates all open Sessions with valid file indices. - std::function<void(OverlayVisitor)> each_overlay; + /// Iterates all open Sessions with valid file indices. + std::function<void(OverlayVisitor)> each_session;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/compiler/indexer.h` around lines 285 - 286, The member variable `each_overlay` has documentation that describes it as iterating all open Sessions, but the name no longer reflects this semantics. Rename the member variable from `each_overlay` to a name that better describes its purpose of iterating sessions (such as `each_session`), and update all references to this variable throughout the codebase to use the new name consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/feature/inlay_hints.cpp`:
- Around line 935-939: The call to `to_position` returns a
`std::optional<protocol::Position>`, but the code unconditionally dereferences
it with `*` in the loop over `collected` without checking if the optional
contains a value. Add a guard before dereferencing: check if the optional
returned by `to_position(unit, encoding, hint.offset)` has a value using a
conditional statement, and only construct the `protocol::InlayHint` out object
and proceed with the position assignment if the optional is valid. If the
conversion fails, skip adding that particular hint to the output.
---
Nitpick comments:
In `@src/server/compiler/indexer.h`:
- Around line 285-286: The member variable `each_overlay` has documentation that
describes it as iterating all open Sessions, but the name no longer reflects
this semantics. Rename the member variable from `each_overlay` to a name that
better describes its purpose of iterating sessions (such as `each_session`), and
update all references to this variable throughout the codebase to use the new
name consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 69385ca5-3ff8-4124-aafc-b0bd7c0715a4
📒 Files selected for processing (13)
src/feature/diagnostics.cppsrc/feature/document_links.cppsrc/feature/document_symbols.cppsrc/feature/feature.hsrc/feature/folding_ranges.cppsrc/feature/hover.cppsrc/feature/inlay_hints.cppsrc/feature/semantic_tokens.cppsrc/server/compiler/indexer.cppsrc/server/compiler/indexer.hsrc/server/service/agent_client.cppsrc/server/service/master_server.cppsrc/server/workspace/workspace.h
💤 Files with no reviewable changes (1)
- src/server/workspace/workspace.h
🚧 Files skipped from review as they are similar to previous changes (5)
- src/feature/hover.cpp
- src/feature/document_links.cpp
- src/server/service/master_server.cpp
- src/feature/feature.h
- src/feature/semantic_tokens.cpp
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
src/index/merged_index.h (1)
71-72:⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy liftMake the lazy
line_startscache safe under shared queries.
line_starts()is a const API backed by a mutable vector, and the implementation lazily writescached_line_startsbefore returning a span. Downstream query paths call this throughworkspace.merged_indices, so parallel requests can race on the cache initialization. Carry the previousMergedIndexShardcache-race fix over to the newMergedIndexowner; becauseMergedIndexis movable and returns spans into the cache, prefer a move-safe synchronized cache state or eager immutable cache refresh when content changes.Also applies to: 100-100
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/index/merged_index.h` around lines 71 - 72, The line_starts() const method in MergedIndex has a thread-safety issue where it lazily initializes the mutable cached_line_starts vector without synchronization, causing races when parallel queries call this method. Apply the same cache-race fix previously done in MergedIndexShard by either protecting cached_line_starts access with a mutex for move-safe synchronized state, or by eagerly computing and caching line_starts whenever the content changes rather than computing it lazily on first access.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/server/compiler/indexer.cpp`:
- Around line 57-88: Both the find_occurrence and find_relations functions
dereference session.file_index without first checking whether the optional has a
value. Add a guard at the beginning of each function to check if
session.file_index is valid before dereferencing it. In find_occurrence, return
std::nullopt if the file_index is not set. In find_relations, return early if
the file_index is not set. This prevents undefined behavior when these helpers
are called with sessions that haven't completed compilation yet.
- Line 676: The relation filtering at the condition checking `if(rel.kind !=
kind)` uses exact equality comparison but should use bitwise AND operation like
`rel.kind & kind` to properly handle cases where rel.kind has multiple flags
set. Change the comparison from `!=` (not equal) to check `if(!(rel.kind &
kind))` to ensure relations with multiple flags are correctly matched. This
makes the logic consistent with other relation filtering in the file such as at
line 523 where bitwise AND is already used.
---
Duplicate comments:
In `@src/index/merged_index.h`:
- Around line 71-72: The line_starts() const method in MergedIndex has a
thread-safety issue where it lazily initializes the mutable cached_line_starts
vector without synchronization, causing races when parallel queries call this
method. Apply the same cache-race fix previously done in MergedIndexShard by
either protecting cached_line_starts access with a mutex for move-safe
synchronized state, or by eagerly computing and caching line_starts whenever the
content changes rather than computing it lazily on first access.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: eb5bbf7c-b3d8-48e1-975c-5ac4fb0380b6
📒 Files selected for processing (6)
src/index/merged_index.cppsrc/index/merged_index.hsrc/server/compiler/indexer.cppsrc/server/service/agent_client.cppsrc/server/workspace/workspace.cppsrc/server/workspace/workspace.h
💤 Files with no reviewable changes (1)
- src/server/workspace/workspace.cpp
🚧 Files skipped from review as they are similar to previous changes (1)
- src/server/service/agent_client.cpp
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8e68e7394
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Replace all lsp::to_position/to_offset/build_line_starts free function calls with LineMap method calls. Add Session::line_map() helper. Remove parse_position_encoding. Bump kotatsu pin to 86d4b19.
16bit-ykiko
changed the title
- Remove all free-standing to_range/to_position/to_offset wrappers from feature.h and indexer.cpp; callers construct LineMap once and call methods directly. - Add FileIndex::lookup(offset) and FileIndex::lookup(symbol, kind, cb) to mirror MergedIndex's interface; delete the old free functions (lookup_occurrence, find_occurrence, find_relations) from indexer.cpp. - Fix test files to use lsp::LineMap instead of feature::lsp::LineMap.
- Add FileIndex::lookup unit tests (LookupOccurrence, LookupRelation) - Reuse build_line_starts() in CompilationUnitRef::line_starts() - Add FIXME for unchecked *map.to_range() optional dereferences - Rename OverlayVisitor→SessionVisitor, each_overlay→each_session, for_each_session→foreach_session - Inline find_relations helper in agent_client.cpp - Unify FileIndex::lookup(offset) to callback style matching MergedIndex
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87536b0a31
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The callback returned true (continue), overwriting hit with progressively wider containing ranges. Stop at the first valid match to preserve innermost-occurrence behavior, matching the MergedIndex path.
When ast_dirty is true the editor buffer has changed since file_index was built, so its byte offsets no longer match session.line_map(). Guard resolve_cursor and foreach_session to fall back to MergedIndex.
16bit-ykiko
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e774a0c1d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace
PositionMapperwith kotatsu'sLineMapacross all feature and server code.LineMapbundles content + line_starts + encoding and providesto_position(),to_offset(),to_range()— eliminating pairedto_positioncalls and theto_rangehelper that wrapped them.Remove
OpenFileIndexandMergedIndexShardwrapper types.OpenFileIndexbundled file_index + symbols + content + mapper into one struct; now Session ownsfile_index,symbols,text, andline_startsdirectly.MergedIndexShardwrappedMergedIndexwith a lazily-cachedPositionMapper; nowMergedIndexstoresline_startsnatively (serialized in the flatbuffer schema) and callers constructLineMapon demand.Move occurrence/relation lookup into
FileIndex::lookupmember methods with a callback interface matchingMergedIndex::lookup, replacing the scattered free functions in workspace.cpp and indexer.cpp.Rename Overlay → Session in Indexer API:
OverlayVisitor→SessionVisitor,for_each_overlay→foreach_session,get_overlay→with_session(path_id, callback).Guard stale session index against text drift. When
ast_dirtyis true (user edited but recompilation hasn't finished), session'sfile_indexoffsets no longer matchsession.text. Now we skip the session path and fall back toMergedIndex, which carries its own stored content.Details
The previous architecture had three position-conversion layers:
PositionMapper(low-level),OpenFileIndex::find_occurrence/find_relations(pre-converting positions), andMergedIndexShard::find_occurrence/find_relations(wrapping the merged index with a cached mapper). This PR collapses them into a single layer: callers construct aLineMapand convert positions themselves. This makes the data flow explicit and removes ~85 lines of wrapper code from workspace.h.The
line_startsarray is now part of theMergedIndexbinary format (FlatBuffer schema). On deserialization, if the field is missing (legacy shards),line_startsis rebuilt from the stored content viabuild_line_starts().kotatsu bumped to
0b38494for the newLineMapconstructors andbuild_line_starts()utility.Test plan
LookupOccurrenceandLookupRelationtests coverFileIndex::lookupcallback APIPositionMappertoLineMap