diff --git a/docs/en/design/incremental-parse.md b/docs/en/design/incremental-parse.md index b1f693b73..4a45af7e5 100644 --- a/docs/en/design/incremental-parse.md +++ b/docs/en/design/incremental-parse.md @@ -52,7 +52,7 @@ The preamble is compiled into a PCH file and cached on disk. Subsequent compilat - A hash of the preamble content - The preamble byte boundary (bound) - A dependency snapshot (`DepsSnapshot`, see below) -- DocumentLink information extracted from the PCH (`#include` directive positions and targets, for the editor to display clickable links) +- A handle to the paired preamble-state blob, stored next to the PCH and sharing its lifecycle: the preamble's symbol index plus feature state extracted at build time (document links, inactive regions, the open conditional stack), opened as a memory-mapped FlatBuffer and queried lazily (see [symbol index](symbol-index.md)) ### Two-Layer Invalidation Detection diff --git a/docs/en/design/symbol-index.md b/docs/en/design/symbol-index.md index f4e94dced..da1347406 100644 --- a/docs/en/design/symbol-index.md +++ b/docs/en/design/symbol-index.md @@ -239,8 +239,6 @@ Background indexing scheduling must balance index timeliness against interferenc The improvement direction is to build a dedicated search index over symbol names. A tokenizer is needed to split symbol names by naming conventions (`getSymbolHash` → `[get, Symbol, Hash]`, `push_back` → `[push, back]`), then build an inverted index over the tokens. For example, trigrams (three-character groups) can be used as index keys, and at query time trigram intersections produce a candidate set that is then scored precisely. clangd's Dex index uses this trigram posting list approach and serves as a useful reference implementation. Another direction is to adopt a mature full-text search library, though the cost of introducing an external dependency needs to be evaluated. -- **PCH-induced index split**. When using PCH (precompiled header) optimization, a file's compilation is effectively split into two phases: first the preamble (the `#include` directives at the top of the file) is compiled to produce the PCH, then the PCH is used to compile the rest of the file. The PCH itself is a compilation unit and produces its own index data. +- ~~**PCH-induced index split**~~ (resolved). When using PCH optimization, a file's compilation is split into two phases: the preamble is compiled into the PCH, then the PCH compiles the rest of the file. The PCH swallows everything before the preamble bound — the main file's compilation cannot see the headers' contents or the preamble region's own directives, and an open buffer's preamble may describe a compilation context that no disk translation unit was ever indexed with. - This split affects the index. Take document links (clickable `#include` directives in the editor) as an example: `#include` directives in the preamble belong to the PCH compilation phase, and the main file's compilation cannot see them. Since the index system does not currently store document link information, it cannot reconstruct the PCH portion's results through the index. The current workaround is to pre-serialize the PCH's document links as JSON during PCH construction and store it in the PCH metadata, then manually splice it into the main file's results at query time. This works but is not clean. - - A better approach would be to incorporate document links into the PCH metadata system (which already stores dependency file lists and other information), or to leverage the include relationship information already present in the index to reconstruct document links. The latter has the problem that index construction takes time, and after PCH compilation completes it should be put to use as quickly as possible — waiting for indexing to complete would add latency. Neither approach is fully implemented yet. + This is now addressed by pairing each PCH with a preamble-state blob, produced by the same worker build while the freshly parsed preamble AST is still in memory (the only moment its index is obtainable without deserializing the whole PCH). The blob carries the preamble's full symbol index — every covered header plus the main file's preamble region — together with per-file content and line tables for position mapping, document links, inactive regions and the open conditional stack. It is stored, hit and evicted together with the PCH, opened as a memory-mapped FlatBuffer and queried without deserialization. Open files overlay these blobs onto index queries: set queries take the union with disk shards (identical rows collapse by location), single-answer queries prefer the overlay, and the buffer's preamble region resolves through the blob's main-file entry. This keeps navigation working for open files whose translation unit the background indexer has not (or cannot) index, and keeps results faithful to unsaved preamble edits. diff --git a/docs/zh/design/incremental-parse.md b/docs/zh/design/incremental-parse.md index 6ac396768..90436cb41 100644 --- a/docs/zh/design/incremental-parse.md +++ b/docs/zh/design/incremental-parse.md @@ -52,7 +52,7 @@ Preamble 编译为 PCH 文件后缓存在磁盘上。后续编译加载 PCH 后 - preamble 内容的哈希值 - preamble 的字节边界（bound） - 依赖快照（`DepsSnapshot`，见下文） -- PCH 中提取的 DocumentLink 信息（`#include` 指令的位置和目标，供编辑器显示可点击链接） +- 与 PCH 配对存储、共享生命周期的 preamble 状态 blob 的句柄：构建时提取的 preamble 符号索引和功能状态（document links、非活跃区域、未闭合条件栈），以内存映射 FlatBuffer 形式打开、按需懒查询（见[符号索引](symbol-index.md)） ### 两层失效检测 diff --git a/docs/zh/design/symbol-index.md b/docs/zh/design/symbol-index.md index c2b5b3d5e..690d2e5cc 100644 --- a/docs/zh/design/symbol-index.md +++ b/docs/zh/design/symbol-index.md @@ -239,8 +239,6 @@ FileIndex 打开文件的实时覆盖层（来自内存中的 AST） 改进方向是为符号名建立专门的搜索索引。需要一个分词器将符号名按命名约定拆分为词元（`getSymbolHash` → `[get, Symbol, Hash]`，`push_back` → `[push, back]`），然后基于词元建立倒排索引。例如使用 trigram（三字符组）作为索引键，查询时取 trigram 交集得到候选集，再精确评分排序。clangd 的 Dex 索引采用了这种 trigram posting list 方案，是一个可参考的实现。另一个方向是引入成熟的全文搜索库，但需要评估引入外部依赖的代价。 -- **PCH 导致的索引分裂**。使用 PCH（预编译头）优化时，一个文件的编译实际上被分成两个阶段：先编译 preamble 部分（文件顶部的 `#include` 指令）生成 PCH，再用 PCH 编译文件的其余部分。PCH 本身也是一个编译单元，会产生独立的索引数据。 +- ~~**PCH 导致的索引分裂**~~（已解决）。使用 PCH 优化时，一个文件的编译被分成两个阶段：先把 preamble 编译成 PCH，再用 PCH 编译文件的其余部分。PCH 吞掉了 bound 之前的一切——主文件的编译既看不到头文件的内容，也看不到 preamble 区自身的指令；而且打开缓冲区的 preamble 可能描述一个任何磁盘翻译单元都未曾以之索引过的编译上下文。 - 这种分裂对索引产生了影响。以 document links（编辑器中可点击的 `#include` 指令）为例：preamble 中的 `#include` 指令属于 PCH 编译阶段，主文件的编译看不到它们。而索引系统目前不存储 document links 信息，因此无法通过索引来补全 PCH 部分的结果。当前的做法是在 PCH 构建时将其 document links 预序列化为 JSON 存储在 PCH 元数据中，查询时手动拼接到主文件的结果中。这种做法可以工作但不够干净。 - - 更合理的方案是将 document links 纳入 PCH 的元数据体系（目前 PCH 元数据已经存储了依赖文件列表等信息），或者利用索引中已有的 include 关系信息来重建 document links。后者的问题在于索引构建需要时间，而 PCH 编译完成后应尽快投入使用，等待索引完成会增加延迟。目前两种方案都尚未完整实现。 + 现在的方案是为每个 PCH 配对一个 preamble 状态 blob，由同一次 worker 构建在刚解析完的 preamble AST 仍在内存时产出（这是唯一无需完整反序列化 PCH 就能获得其索引的时机）。blob 携带 preamble 的完整符号索引——覆盖的每个头文件加上主文件的 preamble 区——以及每文件的内容与行表（用于位置换算）、document links、非活跃区域和未闭合条件栈。它与 PCH 一同存储、一同命中、一同淘汰，以内存映射 FlatBuffer 形式打开，查询无需反序列化。打开文件把这些 blob 作为 overlay 叠加到索引查询上：集合查询与磁盘分片取并集（相同行按位置坍缩），单答案查询优先 overlay，缓冲区 preamble 区的游标经 blob 的主文件条目解析。这使得后台索引尚未（或无法）覆盖其翻译单元的打开文件仍能正常导航，查询结果也忠实于未保存的 preamble 编辑。 diff --git a/src/feature/document_link.h b/src/feature/document_link.h index a46bde9f6..098023fd9 100644 --- a/src/feature/document_link.h +++ b/src/feature/document_link.h @@ -2,16 +2,17 @@ #include -#include "kota/ipc/lsp/protocol.h" +#include "syntax/token.h" namespace clice::feature { /// A resolved document link: the argument range of an include-like -/// directive and the absolute path of the target file. Plain data — it -/// serializes over the worker RPC as-is and becomes an LSP DocumentLink -/// only at the reply edge. +/// directive (byte offsets in the containing file) and the absolute path +/// of the target file. Plain data — it serializes over the worker RPC and +/// the PCH's PreambleState blob as-is and becomes an LSP DocumentLink only +/// at the reply edge, where the session's line map does the conversion. struct DocumentLink { - kota::ipc::protocol::Range range; + LocalSourceRange range; std::string target; }; diff --git a/src/feature/document_links.cpp b/src/feature/document_links.cpp index 8d974e92d..f338e9961 100644 --- a/src/feature/document_links.cpp +++ b/src/feature/document_links.cpp @@ -7,8 +7,7 @@ namespace clice::feature { -auto document_links(CompilationUnitRef unit, PositionEncoding encoding) - -> std::vector { +auto document_links(CompilationUnitRef unit) -> std::vector { std::vector links; auto interested = unit.interested_file(); @@ -20,7 +19,6 @@ auto document_links(CompilationUnitRef unit, PositionEncoding encoding) auto content = unit.interested_content(); auto& directives = directives_it->second; auto* lang_opts = &unit.lang_options(); - LineMap map(content, unit.line_starts(), encoding); auto add_link = [&](clang::SourceLocation loc, llvm::StringRef target) { auto [fid, offset] = unit.decompose_location(loc); @@ -29,10 +27,7 @@ auto document_links(CompilationUnitRef unit, PositionEncoding encoding) auto range = find_directive_argument(content, offset, lang_opts); if(!range) return; - auto protocol_range = to_range(map, *range); - if(!protocol_range) - return; - links.push_back(DocumentLink{.range = *protocol_range, .target = target.str()}); + links.push_back(DocumentLink{.range = *range, .target = target.str()}); }; for(const auto& include: directives.includes) { diff --git a/src/feature/feature.h b/src/feature/feature.h index 031370d06..536f48209 100644 --- a/src/feature/feature.h +++ b/src/feature/feature.h @@ -289,8 +289,9 @@ auto inlay_hints(CompilationUnitRef unit, const InlayHintsOptions& options, PositionEncoding encoding) -> std::vector; -auto document_links(CompilationUnitRef unit, PositionEncoding encoding = PositionEncoding::UTF16) - -> std::vector; +/// Include-directive links of the interested file, in byte offsets; the +/// reply edge converts them with the session's line map. +auto document_links(CompilationUnitRef unit) -> std::vector; /// Go-to-definition on an include directive: when `offset` falls on the /// argument of an #include or __has_include in the interested file, the diff --git a/src/index/preamble_state.cpp b/src/index/preamble_state.cpp new file mode 100644 index 000000000..81c885394 --- /dev/null +++ b/src/index/preamble_state.cpp @@ -0,0 +1,326 @@ +#include "index/preamble_state.h" + +#include +#include +#include +#include + +#include "compile/compilation_unit.h" +#include "index/serialization.h" + +#include "kota/ipc/lsp/text.h" +#include "llvm/ADT/SmallVector.h" + +namespace clice::index { + +namespace { + +/// Serialize one FileIndex into a PreambleFileEntry, with relations sorted +/// by symbol hash so the read path can binary-search them. +fbs::Offset serialize_entry(fbs::FlatBufferBuilder& builder, + std::uint32_t path_id, + const FileIndex& index, + llvm::StringRef content, + llvm::ArrayRef line_starts) { + auto occs = CreateStructVector(builder, index.occurrences); + + llvm::SmallVector*>, 0> sorted; + sorted.reserve(index.relations.size()); + for(auto& [symbol_id, relations]: index.relations) { + sorted.emplace_back(symbol_id, &relations); + } + std::ranges::sort(sorted, {}, [](const auto& entry) { return entry.first; }); + + auto rels = transform(sorted, [&](const auto& entry) { + return binary::CreateTUFileRelationsEntry( + builder, + entry.first, + CreateStructVector(builder, *entry.second)); + }); + + return binary::CreatePreambleFileEntry( + builder, + path_id, + occs, + CreateVector(builder, rels), + content.empty() ? 0 : CreateString(builder, content), + line_starts.empty() ? 0 : CreateVector(builder, line_starts)); +} + +} // namespace + +void PreambleState::serialize(CompilationUnitRef unit, + const TUIndex& index, + llvm::ArrayRef links, + llvm::ArrayRef inactive_regions, + llvm::ArrayRef open_conditionals, + llvm::raw_ostream& os) { + fbs::FlatBufferBuilder builder(4096); + + auto paths = + transform(index.graph.paths, [&](const std::string& p) { return builder.CreateString(p); }); + + Offsets files; + files.reserve(index.file_indices.size()); + for(auto& [fid, file_index]: index.file_indices) { + // A file with no include edge is a synthetic buffer (predefines, + // ): it has no real path to attribute rows to, and + // path_id() would misfile them under the source file. Real files + // forced in via -include are not affected — clang records their + // include edge in the predefines buffer, which is a valid + // location (covered by ForcedIncludeServed). + if(index.graph.include_location_id(fid) == static_cast(-1)) { + continue; + } + auto content = unit.file_content(fid); + auto line_starts = + kota::ipc::lsp::build_line_starts(std::string_view(content.data(), content.size())); + files.push_back( + serialize_entry(builder, index.graph.path_id(fid), file_index, content, line_starts)); + } + + // The source file is the last path in graph.paths (convention from + // IncludeGraph). The preamble compile remaps the buffer truncated at + // the bound, so interested_content() is exactly the preamble text the + // PCH was built from — stored so consumers can compare it against the + // live buffer's prefix before serving these rows. + auto preamble_text = unit.interested_content(); + auto preamble_starts = kota::ipc::lsp::build_line_starts( + std::string_view(preamble_text.data(), preamble_text.size())); + auto preamble_entry = serialize_entry(builder, + static_cast(index.graph.paths.size() - 1), + index.main_file_index, + preamble_text, + preamble_starts); + + llvm::SmallVector, 0> sorted_symbols; + sorted_symbols.reserve(index.symbols.size()); + for(auto& [symbol_id, symbol]: index.symbols) { + sorted_symbols.emplace_back(symbol_id, &symbol); + } + std::ranges::sort(sorted_symbols, {}, [](const auto& entry) { return entry.first; }); + + auto syms = transform(sorted_symbols, [&](const auto& entry) { + return binary::CreatePreambleSymbolEntry(builder, + entry.first, + CreateString(builder, entry.second->name), + entry.second->kind.value()); + }); + + auto link_entries = transform(links, [&](const feature::DocumentLink& link) { + binary::Range range(link.range.begin, link.range.end); + return binary::CreatePreambleDocumentLink(builder, + &range, + CreateString(builder, link.target)); + }); + + auto root = binary::CreatePreambleState(builder, + preamble_format_version, + CreateVector(builder, paths), + CreateVector(builder, files), + preamble_entry, + CreateVector(builder, syms), + CreateVector(builder, link_entries), + CreateVector(builder, inactive_regions), + CreateVector(builder, open_conditionals)); + builder.Finish(root); + os.write(safe_cast(builder.GetBufferPointer()), builder.GetSize()); +} + +PreambleState::PreambleState(std::unique_ptr buffer) : + buffer(std::move(buffer)) {} + +std::shared_ptr PreambleState::load(llvm::StringRef path) { + auto buffer = llvm::MemoryBuffer::getFile(path); + if(!buffer) { + return nullptr; + } + + // A stale or truncated blob must never crash the server. Verify it is + // a structurally valid flatbuffer, then discard any blob whose format + // version differs (version-less blobs read back 0). The table budget + // is far above the default: a large preamble's symbol table alone can + // exceed a million entries, and a verification failure here would + // otherwise send every compile into a rebuild loop. + auto data = reinterpret_cast((*buffer)->getBufferStart()); + fbs::Verifier verifier(data, + (*buffer)->getBufferSize(), + /*max_depth=*/64, + /*max_tables=*/1u << 26); + if(!verifier.VerifyBuffer(nullptr)) { + return nullptr; + } + + auto root = fbs::GetRoot((*buffer)->getBufferStart()); + if(root->format_version() != preamble_format_version) { + return nullptr; + } + + return std::shared_ptr(new PreambleState(std::move(*buffer))); +} + +void PreambleState::lookup(SymbolHash symbol, + RelationKind kind, + llvm::function_ref callback) const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto paths = root->paths(); + + for(auto entry: *root->files()) { + auto rels = entry->relations(); + if(!rels) { + continue; + } + + auto it = std::ranges::lower_bound(*rels, symbol, {}, [](auto e) { return e->symbol(); }); + if(it == rels->end() || it->symbol() != symbol || !it->relations()) { + continue; + } + + // The verifier checks structure, not cross-references: a corrupt + // path_id would index out of bounds on the mapped file. + if(entry->path_id() >= paths->size()) { + continue; + } + auto path = paths->Get(entry->path_id()); + File file{ + .path = llvm::StringRef(path->c_str(), path->size()), + .content = entry->content() + ? llvm::StringRef(entry->content()->c_str(), entry->content()->size()) + : llvm::StringRef(), + .line_starts = entry->line_starts() ? std::span(entry->line_starts()->data(), + entry->line_starts()->size()) + : std::span(), + }; + + for(auto rel: *it->relations()) { + auto r = safe_cast(rel); + if(r->kind & kind) { + if(!callback(file, *r)) { + return; + } + } + } + } +} + +llvm::StringRef PreambleState::source_path() const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto paths = root->paths(); + if(paths->size() == 0) { + return {}; + } + // The source file is the last path, by IncludeGraph convention. + auto path = paths->Get(paths->size() - 1); + return llvm::StringRef(path->c_str(), path->size()); +} + +llvm::StringRef PreambleState::preamble_content() const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto preamble = root->preamble(); + if(!preamble || !preamble->content()) { + return {}; + } + return llvm::StringRef(preamble->content()->c_str(), preamble->content()->size()); +} + +void PreambleState::lookup_preamble(std::uint32_t offset, + llvm::function_ref callback) const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto preamble = root->preamble(); + if(!preamble || !preamble->occurrences()) { + return; + } + + auto& occurrences = *preamble->occurrences(); + auto it = + std::ranges::lower_bound(occurrences, offset, {}, [](auto o) { return o->range().end(); }); + + while(it != occurrences.end()) { + auto o = safe_cast(*it); + if(!o->range.contains(offset)) { + break; + } + if(!callback(*o)) { + break; + } + ++it; + } +} + +void PreambleState::lookup_preamble(SymbolHash symbol, + RelationKind kind, + llvm::function_ref callback) const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto preamble = root->preamble(); + if(!preamble || !preamble->relations()) { + return; + } + + auto& rels = *preamble->relations(); + auto it = std::ranges::lower_bound(rels, symbol, {}, [](auto e) { return e->symbol(); }); + if(it == rels.end() || it->symbol() != symbol || !it->relations()) { + return; + } + + for(auto rel: *it->relations()) { + auto r = safe_cast(rel); + if(r->kind & kind) { + if(!callback(*r)) { + return; + } + } + } +} + +bool PreambleState::find_symbol(SymbolHash hash, std::string& name, SymbolKind& kind) const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto& syms = *root->symbols(); + + auto it = std::ranges::lower_bound(syms, hash, {}, [](auto e) { return e->symbol_id(); }); + if(it == syms.end() || it->symbol_id() != hash) { + return false; + } + + name = it->name() ? it->name()->str() : std::string(); + kind = SymbolKind(static_cast(it->kind())); + return true; +} + +std::vector PreambleState::links() const { + std::vector links; + auto root = fbs::GetRoot(buffer->getBufferStart()); + if(auto ls = root->links()) { + links.reserve(ls->size()); + for(auto entry: *ls) { + feature::DocumentLink link; + if(auto range = entry->range()) { + link.range = *safe_cast(range); + } + if(auto target = entry->target()) { + link.target = target->str(); + } + links.push_back(std::move(link)); + } + } + return links; +} + +llvm::ArrayRef PreambleState::inactive_regions() const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto regions = root->inactive_regions(); + if(!regions) { + return {}; + } + return llvm::ArrayRef(regions->data(), regions->size()); +} + +llvm::ArrayRef PreambleState::open_conditionals() const { + auto root = fbs::GetRoot(buffer->getBufferStart()); + auto conditionals = root->open_conditionals(); + if(!conditionals) { + return {}; + } + return llvm::ArrayRef(conditionals->data(), conditionals->size()); +} + +} // namespace clice::index diff --git a/src/index/preamble_state.h b/src/index/preamble_state.h new file mode 100644 index 000000000..97afc213e --- /dev/null +++ b/src/index/preamble_state.h @@ -0,0 +1,117 @@ +#pragma once + +#include +#include +#include +#include + +#include "feature/document_link.h" +#include "index/tu_index.h" + +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/Support/raw_ostream.h" + +namespace clice::index { + +/// On-disk PreambleState blob schema version (the PCH's `.pch.idx` pair). +/// Bump whenever `schema.fbs` changes the PreambleState layout; a blob +/// carrying a different value loads as "missing" and the PCH pair is +/// rebuilt. cache.json records it so a version change is caught at load +/// time instead of on the first overlay query. +constexpr inline std::uint32_t preamble_format_version = 3; + +/// All master-visible state derived from one PCH build. +/// +/// The stateless worker serializes it next to the PCH blob (the store's +/// `.pch.idx` pair) and the master opens it as a memory-mapped FlatBuffer: +/// queries run directly on the serialized data, nothing is deserialized up +/// front. It carries the preamble's full symbol index — every header the +/// PCH covers plus the main file's preamble region — with per-file content +/// and line starts for position mapping (mirroring MergedIndex shards), +/// and the PCH-derived feature state that is spliced into main-file +/// results: document links, inactive regions and the open conditional +/// stack at the preamble bound. +/// +/// Lifecycle equals the PCH's: the pair is committed, hit and evicted +/// together, so no separate invalidation is needed — a preamble change or +/// a stale dependency rebuilds both. +class PreambleState { +public: + /// A file entry handed to lookup callbacks: everything needed to turn + /// a byte-offset hit into an LSP location. Views borrow the mapped + /// blob; keep the PreambleState alive while using them. + struct File { + llvm::StringRef path; + llvm::StringRef content; + std::span line_starts; + }; + + /// Serialize a preamble compilation's state. `index` must be built + /// over the preamble unit with interested_only=false and its + /// main_file_index intact (it holds the preamble region's own + /// occurrences — macro definitions and references before the bound). + static void serialize(CompilationUnitRef unit, + const TUIndex& index, + llvm::ArrayRef links, + llvm::ArrayRef inactive_regions, + llvm::ArrayRef open_conditionals, + llvm::raw_ostream& os); + + /// Open a blob from disk (memory-mapped). Returns nullptr when the + /// file is unreadable, structurally invalid or written by a different + /// format version — callers treat all of these as a PCH cache miss. + static std::shared_ptr load(llvm::StringRef path); + + /// Iterate relations of `symbol` matching `kind` across all header + /// entries. Return false from the callback to stop. This is the only + /// query shape overlays serve: hash-anchored answering. Discovery + /// inputs (by name, by path and line) are the disk index's job. + void lookup(SymbolHash symbol, + RelationKind kind, + llvm::function_ref callback) const; + + /// Path of the file whose preamble built this blob. Files with + /// identical preambles share one PCH (the key excludes the source + /// path), but the preamble entry carries file-local symbol identities + /// — macro USRs embed the source path — so its lookups must be scoped + /// to this file. Borrows the mapped blob. + llvm::StringRef source_path() const; + + /// The exact preamble text this blob was built from. Consumers serve + /// preamble-entry rows only while the live buffer still starts with + /// it — the rows are buffer offsets into this prefix. Borrows the + /// mapped blob. + llvm::StringRef preamble_content() const; + + /// Occurrence lookup in the source file's preamble region (buffer + /// offsets below the preamble bound). + void lookup_preamble(std::uint32_t offset, + llvm::function_ref callback) const; + + /// Relations of `symbol` in the source file's preamble region. + void lookup_preamble(SymbolHash symbol, + RelationKind kind, + llvm::function_ref callback) const; + + /// Look up a symbol's name and kind in the blob's symbol table. + bool find_symbol(SymbolHash hash, std::string& name, SymbolKind& kind) const; + + /// Document links of the preamble region, materialized from the blob. + std::vector links() const; + + /// Inactive regions within the preamble (flat begin/end offset pairs). + /// Borrows the mapped blob. + llvm::ArrayRef inactive_regions() const; + + /// Conditional stack still open at the preamble bound. Borrows the + /// mapped blob. + llvm::ArrayRef open_conditionals() const; + +private: + explicit PreambleState(std::unique_ptr buffer); + + std::unique_ptr buffer; +}; + +} // namespace clice::index diff --git a/src/index/schema.fbs b/src/index/schema.fbs index 2262ebad4..dacbcd36a 100644 --- a/src/index/schema.fbs +++ b/src/index/schema.fbs @@ -151,6 +151,84 @@ relations: [Relation]; } +table PreambleDocumentLink { +range: + Range; +target: + string; +} + +// One file covered by a preamble compilation. Content and line starts are +// stored (like MergedIndex shards) so the master converts byte offsets to +// LSP positions without touching the filesystem. +table PreambleFileEntry { +path_id: + uint; + +// Sorted by (range.begin, range.end, target). +occurrences: + [Occurrence]; + +// Sorted by symbol hash for binary search. +relations: + [TUFileRelationsEntry]; + +content: + string; + +line_starts: + [uint]; +} + +table PreambleSymbolEntry { +symbol_id: + ulong; +name: + string; +kind: + ubyte; +} + +// Everything the master needs from one PCH build, written by the stateless +// worker next to the PCH blob and opened as a memory-mapped FlatBuffer: +// the preamble's symbol index plus the PCH-derived feature state that the +// master splices into main-file results (document links, inactive regions, +// open conditional stack). +table PreambleState { +format_version: + uint; + +// Blob-local path table; every path_id indexes into it. +paths: + [string] (required); + +// Header entries covered by the preamble. +files: + [PreambleFileEntry] (required); + +// The source file's own preamble region, in buffer offsets. Its content +// is the exact preamble text the PCH was built from — consumers compare +// it against the live buffer's prefix before serving these rows. +preamble: + PreambleFileEntry; + +// Sorted by symbol_id for binary search. +symbols: + [PreambleSymbolEntry] (required); + +links: + [PreambleDocumentLink]; + +// Flat begin/end offset pairs within the preamble region. +inactive_regions: + [uint]; + +// Conditional stack still open at the preamble bound (see +// feature::InactiveScan::open_stack encoding). +open_conditionals: + [ubyte]; +} + table TUFileIndexEntry { file_id: uint; diff --git a/src/index/tu_index.cpp b/src/index/tu_index.cpp index b0943c6ed..b11dc8f61 100644 --- a/src/index/tu_index.cpp +++ b/src/index/tu_index.cpp @@ -77,6 +77,17 @@ class Builder : public SemanticVisitor { auto& index = result.file_indices[fid]; auto symbol_id = unit.getSymbolID(def); + // Macros get a symbol-table entry like declarations do; without it + // build() would default-construct a nameless entry when recording + // reference files, and every name lookup for the macro would come + // back empty. + auto [it, success] = result.symbols.try_emplace(symbol_id.hash); + if(success) { + auto& symbol = it->second; + symbol.name = unit.token_spelling(location).str(); + symbol.kind = SymbolKind::Macro; + symbol.scope = SymbolScope::External; + } index.occurrences.emplace_back(range, symbol_id.hash); Relation relation{ @@ -85,6 +96,17 @@ class Builder : public SemanticVisitor { .target_symbol = 0, }; + // Definition relations carry the macro's full extent (name through + // last body token), like declarations do — definition-text + // consumers decode it out of target_symbol. + if(kind.isDeclOrDef() && def) { + auto [def_fid, def_range] = unit.decompose_range( + clang::SourceRange(def->getDefinitionLoc(), def->getDefinitionEndLoc())); + if(def_fid == fid) { + relation.set_definition_range(def_range); + } + } + index.relations[symbol_id.hash].emplace_back(relation); } diff --git a/src/server/compiler/compiler.cpp b/src/server/compiler/compiler.cpp index c9b43923c..bd3b62073 100644 --- a/src/server/compiler/compiler.cpp +++ b/src/server/compiler/compiler.cpp @@ -8,6 +8,7 @@ #include #include "command/argument_parser.h" +#include "index/preamble_state.h" #include "index/tu_index.h" #include "server/compiler/context_resolver.h" #include "server/protocol/extension.h" @@ -283,14 +284,14 @@ std::string uri_to_path(const std::string& uri) { return uri; } -/// The pch_ref write license: a round may (re)write the session's PCH +/// The pch_key write license: a round may (re)write the session's PCH /// reference only while BOTH staleness tokens still hold their takeoff /// values. A supersede bumps generation; a Lost-type invalidation (disk or /// CDB change behind an in-flight round) bumps only dirty_epoch — either /// way the round's resolved directory/arguments may describe a command /// that no longer exists, and writing its PCH key back would hand later /// incomplete-preamble edits a stale-flag PCH. -static bool may_write_pch_ref(const Session& session, +static bool may_write_pch_key(const Session& session, std::uint64_t launch_generation, std::uint64_t launch_epoch) { return session.generation == launch_generation && session.dirty_epoch == launch_epoch; @@ -302,9 +303,9 @@ kota::task Compiler::ensure_pch(Session& session, const std::string& directory, const std::vector& arguments) { // A round invalidated during the caller's earlier awaits (module - // dependencies) must not touch pch_ref at all: the reset and cache-hit + // dependencies) must not touch pch_key at all: the reset and cache-hit // branches below write it before the first suspension point. - if(!may_write_pch_ref(session, launch_generation, launch_epoch)) { + if(!may_write_pch_key(session, launch_generation, launch_epoch)) { co_return false; } @@ -318,7 +319,7 @@ kota::task Compiler::ensure_pch(Session& session, // No preamble directives and no injected -include — PCH would be // empty. Self-contained header contexts land here too: they borrow // a command but inject nothing. - session.pch_ref.reset(); + session.pch_key.reset(); co_return true; } @@ -350,15 +351,23 @@ kota::task Compiler::ensure_pch(Session& session, llvm::StringRef pch_miss = "no_entry"; if(auto it = workspace.pch_cache.find(pch_key); it != workspace.pch_cache.end()) { auto& st = it->second; - bool in_store = workspace.store && workspace.store->lookup("pch", pch_key); + // Both halves of the pair must be present: a PCH whose + // PreambleState blob is gone (crash between commits, failed aux + // commit) rebuilds whole. + bool in_store = workspace.store && workspace.store->lookup("pch", pch_key) && + workspace.store->lookup_aux("pch", pch_key); if(st.path.empty()) { pch_miss = "incomplete_entry"; } else if(!in_store) { pch_miss = "evicted"; + } else if(st.index_path.empty()) { + // load_state() found the blob unreadable earlier; republish the + // pair rather than serving a PCH with no index forever. + pch_miss = "idx_unreadable"; } else if(deps_changed(workspace.path_pool, st.deps)) { pch_miss = "deps_changed"; } else { - session.pch_ref = Session::PCHRef{pch_key, bound}; + session.pch_key = pch_key; LOG_PERF("cache", "ns=pch event=hit key={} file={}", pch_key, path); co_return true; } @@ -374,8 +383,8 @@ kota::task Compiler::ensure_pch(Session& session, // the session's previous PCH if it is still available. if(!is_preamble_complete(text, bound)) { LOG_DEBUG("Preamble incomplete for {}, deferring PCH rebuild", path); - if(session.pch_ref.has_value()) { - auto it = workspace.pch_cache.find(session.pch_ref->key); + if(session.pch_key.has_value()) { + auto it = workspace.pch_cache.find(*session.pch_key); co_return it != workspace.pch_cache.end() && !it->second.path.empty(); } co_return false; @@ -386,15 +395,15 @@ kota::task Compiler::ensure_pch(Session& session, if(auto it = workspace.pch_cache.find(pch_key); it != workspace.pch_cache.end() && it->second.building) { co_await it->second.building->wait(); - // Guard the pch_ref write below against an invalidated round's + // Guard the pch_key write below against an invalidated round's // continuation: a newer round (or a context switch) may have // established the session's PCH identity while we waited. - if(!may_write_pch_ref(session, launch_generation, launch_epoch)) { + if(!may_write_pch_key(session, launch_generation, launch_epoch)) { co_return false; } if(auto it2 = workspace.pch_cache.find(pch_key); it2 != workspace.pch_cache.end() && !it2->second.path.empty()) { - session.pch_ref = Session::PCHRef{pch_key, it2->second.bound}; + session.pch_key = pch_key; co_return true; } co_return false; @@ -411,9 +420,11 @@ kota::task Compiler::ensure_pch(Session& session, co_return false; } - // Build a new PCH via stateless worker: it writes the blob to the tmp - // path allocated here; the store commits (fsync + rename) on success. + // Build a new PCH pair via stateless worker: it writes the PCH and its + // PreambleState blob to the tmp paths allocated here; the store + // commits (fsync + rename) both on success, primary first. auto pending = workspace.store->begin_store("pch", pch_key); + auto pending_idx = workspace.store->begin_store_aux("pch", pch_key); worker::BuildParams bp; bp.priority = worker::Priority::High; @@ -424,6 +435,7 @@ kota::task Compiler::ensure_pch(Session& session, bp.text = text; bp.preamble_bound = bound; bp.output_path = pending.tmp_path; + bp.index_output_path = pending_idx.tmp_path; LOG_DEBUG("Building PCH for {}, bound={}, key={}", path, bound, pch_key); @@ -431,6 +443,7 @@ kota::task Compiler::ensure_pch(Session& session, if(!result.has_value() || !result.value().success) { workspace.store->abort(pending); + workspace.store->abort(pending_idx); if(expected_build_failure(result)) { LOG_WARN("PCH build failed for {}: {}", path, build_failure_message(result)); } else { @@ -442,21 +455,61 @@ kota::task Compiler::ensure_pch(Session& session, co_return false; } - // Commit on the thread pool: it fsyncs the freshly written PCH. - auto committed = - co_await kota::queue([&] { return workspace.store->commit(std::move(pending)); }); - if(!committed.has_value() || !committed.value().has_value()) { + // Commit the pair on the thread pool as one job: the fsyncs stay off + // the event loop, and no cancellation can land between the two commits + // — the store either publishes the whole pair or retracts it (a half + // pair would let waiters adopt a PCH whose blob is gone). Opening the + // freshly committed blob (mmap + flatbuffer verification, which walks + // the whole file) also happens here so no later consumer pays that + // walk on the event loop. + struct PairCommit { + std::optional pch_path; + std::optional index_path; + std::shared_ptr state; + }; + + auto committed = co_await kota::queue([&]() -> PairCommit { + PairCommit outcome; + auto pch_path = workspace.store->commit(std::move(pending)); + if(!pch_path) { + workspace.store->abort(pending_idx); + return outcome; + } + outcome.pch_path = std::move(*pch_path); + + // The pair is only usable complete: when the index blob cannot be + // published, retract the PCH too — the next compile rebuilds both. + auto index_path = workspace.store->commit(std::move(pending_idx)); + if(!index_path) { + workspace.store->invalidate("pch", pch_key); + return outcome; + } + outcome.index_path = std::move(*index_path); + outcome.state = index::PreambleState::load(*outcome.index_path); + return outcome; + }); + if(!committed.has_value() || !committed.value().pch_path.has_value()) { LOG_WARN("Failed to commit PCH for {}", path); co_return false; } + if(!committed.value().index_path.has_value()) { + LOG_WARN("Failed to commit PreambleState blob for {}", path); + // A rebuild of an existing key just had its blobs retracted from + // the store; the entry's paths now dangle and waiters checking + // `!path.empty()` would hand the compile a deleted PCH. Drop it — + // the guard tolerates a missing entry. + workspace.pch_cache.erase(pch_key); + co_return false; + } auto& st = workspace.pch_cache[pch_key]; - st.path = committed.value().value(); + st.path = *committed.value().pch_path; st.bound = bound; st.deps = capture_deps_snapshot(workspace.path_pool, result.value().deps); - st.preamble_links = std::move(result.value().preamble_links); - st.inactive_regions = std::move(result.value().inactive_regions); - st.open_conditionals = std::move(result.value().open_conditionals); + st.index_path = *committed.value().index_path; + // Replace the previous blob's mapping (same key, rebuilt content); + // in-flight holders of the old shared_ptr stay valid. + st.state = committed.value().state; LOG_INFO("PCH built for {}: {}", path, st.path); @@ -465,10 +518,10 @@ kota::task Compiler::ensure_pch(Session& session, // The cache entry above is content-keyed and correct regardless; only // the session pointer must not be written by an invalidated round. - if(!may_write_pch_ref(session, launch_generation, launch_epoch)) { + if(!may_write_pch_key(session, launch_generation, launch_epoch)) { co_return false; } - session.pch_ref = Session::PCHRef{pch_key, bound}; + session.pch_key = pch_key; co_return true; } @@ -577,8 +630,8 @@ kota::task Compiler::ensure_deps(Session& session, // Build or reuse PCH. auto pch_ok = co_await ensure_pch(session, launch_generation, launch_epoch, directory, arguments); - if(pch_ok && session.pch_ref.has_value()) { - if(auto pch_it = workspace.pch_cache.find(session.pch_ref->key); + if(pch_ok && session.pch_key.has_value()) { + if(auto pch_it = workspace.pch_cache.find(*session.pch_key); pch_it != workspace.pch_cache.end()) { pch = {pch_it->second.path, pch_it->second.bound}; } @@ -601,9 +654,9 @@ bool Compiler::is_stale(const Session& session) { header_context && deps_changed(workspace.path_pool, header_context->deps)) return true; - // Check PCH staleness via the session's pch_ref. - if(session.pch_ref.has_value()) { - auto pch_it = workspace.pch_cache.find(session.pch_ref->key); + // Check PCH staleness via the session's pch_key. + if(session.pch_key.has_value()) { + auto pch_it = workspace.pch_cache.find(*session.pch_key); if(pch_it != workspace.pch_cache.end() && deps_changed(workspace.path_pool, pch_it->second.deps)) return true; @@ -706,12 +759,16 @@ kota::task<> Compiler::run_compile(std::shared_ptr session) { // out: concurrent compiles can insert into pch_cache across the // await below and rehash the map from under a held pointer. std::vector pch_inactive; - if(session->pch_ref.has_value()) { - if(auto it = workspace.pch_cache.find(session->pch_ref->key); - it != workspace.pch_cache.end()) { - pch_inactive = it->second.inactive_regions; - params.open_conditionals = it->second.open_conditionals; - } + std::shared_ptr preamble_state; + if(session->pch_key.has_value()) { + auto it = workspace.pch_cache.find(*session->pch_key); + preamble_state = it != workspace.pch_cache.end() ? it->second.load_state() : nullptr; + } + if(preamble_state) { + auto regions = preamble_state->inactive_regions(); + pch_inactive.assign(regions.begin(), regions.end()); + auto conditionals = preamble_state->open_conditionals(); + params.open_conditionals.assign(conditionals.begin(), conditionals.end()); } auto result = co_await pool.send_stateful(pid, params); @@ -784,7 +841,7 @@ kota::task<> Compiler::run_compile(std::shared_ptr session) { contexts.record_header_mode(pid, HeaderMode::NeedsContext, hash_file(file_path)); workspace.save_cache(contexts); contexts.drop_header_context(pid); - session->pch_ref.reset(); + session->pch_key.reset(); continue; } } @@ -1009,6 +1066,13 @@ kota::task, kota::ipc::Error> } co_return kota::outcome_error(std::move(result.error())); } + // The result carries byte offsets against the compiled buffer; a + // didChange that landed during the await makes them describe text the + // session no longer holds — the reply edge would map them onto the + // edited buffer at wrong positions. + if(session->generation != gen) { + co_return std::vector{}; + } LOG_PERF("request", "kind=DocumentLink file={} wait_ms={} total_ms={}", path, @@ -1023,8 +1087,8 @@ Compiler::RawResult Compiler::forward_build(worker::BuildKind kind, auto path_id = session->path_id; auto path = std::string(workspace.path_pool.resolve(path_id)); auto gen = session->generation; - // Takeoff snapshot for the pch_ref write license (see - // may_write_pch_ref): this request runs concurrently with compiles and + // Takeoff snapshot for the pch_key write license (see + // may_write_pch_key): this request runs concurrently with compiles and // holds no compiling token, so it is the easiest continuation to come // back stale after a disk/CDB change. auto epoch = session->dirty_epoch; diff --git a/src/server/compiler/compiler.h b/src/server/compiler/compiler.h index eb105afc3..00673e2ef 100644 --- a/src/server/compiler/compiler.h +++ b/src/server/compiler/compiler.h @@ -40,7 +40,7 @@ std::string uri_to_path(const std::string& uri); /// /// Compiler holds no persistent state of its own. All project-wide data /// lives in Workspace; per-file data lives in Session. Compiler reads from -/// both and writes compilation results back to Session (file_index, pch_ref, +/// both and writes compilation results back to Session (file_index, pch_key, /// ast_deps, diagnostics). /// /// Responsibilities: @@ -65,7 +65,7 @@ class Compiler { void init_compile_graph(); /// Compile an open file's AST if dirty. On success, updates session's - /// file_index, pch_ref, ast_deps, and publishes diagnostics. + /// file_index, pch_key, ast_deps, and publishes diagnostics. kota::task ensure_compiled(std::shared_ptr session); using RawResult = kota::task; @@ -87,7 +87,7 @@ class Compiler { /// Forward a document-link query to the stateful worker holding this /// file's AST. Covers the main-file region only: the preamble's links - /// are cached master-side with the PCH (see PCHState::preamble_links). + /// live in the PCH's PreambleState blob (see PCHState::load_state). kota::task, kota::ipc::Error> forward_document_links(std::shared_ptr session); @@ -121,12 +121,12 @@ class Compiler { /// snapshots from the moment its round took off, NOT ones /// taken on entry: a round invalidated during the /// dependency phase would otherwise re-snapshot the new - /// values here and slip a stale pch_ref past the write + /// values here and slip a stale pch_key past the write /// guards. Both tokens are needed — a supersede bumps /// generation, but a Lost-type invalidation (disk or CDB /// change behind an in-flight round) bumps only /// dirty_epoch, and a round that resolved its command - /// before the event must not write pch_ref back either. + /// before the event must not write pch_key back either. /// @param scope When set, cancels the module-dependency wait if this /// compile round is superseded by a newer one. kota::task ensure_deps(Session& session, diff --git a/src/server/compiler/indexer.cpp b/src/server/compiler/indexer.cpp index f2488dcba..e78069d46 100644 --- a/src/server/compiler/indexer.cpp +++ b/src/server/compiler/indexer.cpp @@ -413,7 +413,12 @@ kota::task<> Indexer::index_one(std::uint32_t server_path_id, std::size_t total) { auto file_path = std::string(workspace.path_pool.resolve(server_path_id)); - if(sessions.find(server_path_id) != nullptr) + // Open files are skipped until an agent shows up: the LSP side never + // reads their shards (sessions serve them), so indexing them is pure + // waste — but agents read disk truth and need the shards, snapshot + // taken from disk regardless of the live buffer. Skipping loses no + // debt: BufferClosed re-checks the shard against the disk on close. + if(!index_open_files && sessions.find(server_path_id) != nullptr) co_return; // The engine's own observation is authoritative for content changes: diff --git a/src/server/compiler/indexer.h b/src/server/compiler/indexer.h index f16a3425e..6306f0918 100644 --- a/src/server/compiler/indexer.h +++ b/src/server/compiler/indexer.h @@ -17,8 +17,8 @@ namespace clice { class ContextResolver; -class WorkerPool; struct SessionStore; +class WorkerPool; /// Why a file awaits re-indexing. The invalidation engine knows the cause /// at enqueue time, so queries can decide in O(1) whether a pending file's @@ -61,6 +61,15 @@ class Indexer { loop(loop), bg_tasks(loop), workspace(workspace), pool(pool), contexts(contexts), sessions(sessions) {} + /// Whether open files' disk snapshots are indexed like closed ones. + /// Off by default: the LSP side never reads an open file's shard (its + /// session serves it), so the work would be pure waste — until an + /// agent shows up, whose disk-truth queries need those shards. Turned + /// on (sticky) by the first agentic index query; files closed before + /// that are already covered, because BufferClosed re-enqueues a file + /// whose shard does not match the disk. + bool index_open_files = false; + /// Temporarily pause background indexing to give priority to user /// requests. Indexing tasks already dispatched to workers continue, /// but no new tasks will be sent until resume_indexing() is called. @@ -179,10 +188,10 @@ class Indexer { Workspace& workspace; WorkerPool& pool; ContextResolver& contexts; + const SessionStore& sessions; /// Open documents, read-only. A file with an open Session is skipped by /// background indexing (its buffer index is authoritative). - const SessionStore& sessions; /// Background indexing queue and scheduling state. pending_ids mirrors /// the un-consumed tail of index_queue so enqueue can dedupe; the queue diff --git a/src/server/protocol/worker.h b/src/server/protocol/worker.h index a8c0e174b..f9acc3c8d 100644 --- a/src/server/protocol/worker.h +++ b/src/server/protocol/worker.h @@ -129,7 +129,14 @@ struct BuildParams { std::pair pch; std::unordered_map pcms; - std::string output_path; ///< BuildPCH, BuildPCM + std::string output_path; ///< BuildPCH, BuildPCM + + /// BuildPCH: tmp path for the PreambleState blob (the PCH's paired + /// `.pch.idx`), allocated by the master's store alongside output_path. + /// The worker serializes the preamble's index and feature state into + /// it; the master commits both blobs together. + std::string index_output_path; + std::string module_name; ///< BuildPCM uint32_t preamble_bound = UINT32_MAX; ///< BuildPCH LocalSourceRange format_range; ///< Format (default = full document) @@ -146,21 +153,13 @@ struct BuildResult { bool has_user_errors = false; std::string output_path; ///< PCH or PCM path std::vector deps; - std::string tu_index_data; - /// Include directives of the PCH preamble. Structured so the master can - /// serve both document links and go-to-definition on preamble lines. - std::vector preamble_links; - - /// Inactive regions within the preamble region (flat offset pairs) - /// and the conditional stack still open at the bound. - std::vector inactive_regions; - std::vector open_conditionals; + std::string tu_index_data; ///< Index: serialized TUIndex, merged by the master kota::codec::RawValue result_json; ///< Completion/SignatureHelp result }; /// Request the document links of an open file's AST. Only the main-file /// region is covered: the preamble is compiled into the PCH, and its links -/// travel in BuildResult::preamble_links. +/// live in the PCH's PreambleState blob (spliced in by the master). struct DocumentLinkParams { std::string path; }; diff --git a/src/server/service/feature_router.cpp b/src/server/service/feature_router.cpp index 6cf521b37..388858a44 100644 --- a/src/server/service/feature_router.cpp +++ b/src/server/service/feature_router.cpp @@ -37,14 +37,21 @@ static kota::ipc::Error item_not_resolved(llvm::StringRef kind) { std::format("Failed to resolve {} item", kind)}; } -const std::vector* - FeatureRouter::find_preamble_links(const Session& session) { - if(!session.pch_ref) - return nullptr; - auto it = workspace.pch_cache.find(session.pch_ref->key); - if(it == workspace.pch_cache.end() || it->second.preamble_links.empty()) - return nullptr; - return &it->second.preamble_links; +std::vector FeatureRouter::find_preamble_links(const Session& session) { + if(!session.pch_key) + return {}; + auto it = workspace.pch_cache.find(*session.pch_key); + if(it == workspace.pch_cache.end()) + return {}; + auto& state = it->second.load_state(); + if(!state) + return {}; + // Link offsets are buffer coordinates as of the PCH build; serve them + // only while the buffer still starts with that exact preamble text + // (a deferred rebuild mid-edit keeps an old blob for a moved buffer). + if(!llvm::StringRef(session.text).starts_with(state->preamble_content())) + return {}; + return state->links(); } std::vector @@ -53,22 +60,24 @@ std::vector std::vector locations; // Preamble include lines: compiled into the PCH, invisible to the - // worker's AST — the PCH's cached links carry the targets. - if(auto* links = find_preamble_links(session)) { - for(auto& link: *links) { - if(link.range.start.line != position.line) - continue; - if(position.character < link.range.start.character || - position.character > link.range.end.character) - continue; + // worker's AST — the PCH's stored links carry the targets. + auto links = find_preamble_links(session); + if(links.empty()) + return locations; + + auto offset = session.line_map().to_offset(position); + if(!offset) + return locations; + + for(auto& link: links) { + if(link.range.contains(*offset)) { locations.push_back(protocol::Location{ .uri = feature::to_uri(link.target), .range = protocol::Range{}, }); - return locations; + break; } } - return locations; } @@ -80,19 +89,18 @@ kota::task, kota::ipc::Error> // The preamble is compiled into the PCH, so the worker's AST only // covers the rest of the file — merge the preamble's links in front. + // Links carry byte offsets; this reply edge converts them. std::vector links; + auto map = session->line_map(); auto append = [&](const feature::DocumentLink& link) { - protocol::DocumentLink out{.range = link.range}; + auto range = map.to_range(link.range.begin, link.range.end); + if(!range) + return; + protocol::DocumentLink out{.range = *range}; out.target = link.target; links.push_back(std::move(out)); }; - // Skipped while dirty: a failed or superseded compile leaves - // the cached links describing the pre-edit preamble. - if(!session->ast_dirty) { - if(auto* pch_links = find_preamble_links(*session)) { - std::ranges::for_each(*pch_links, append); - } - } + std::ranges::for_each(find_preamble_links(*session), append); std::ranges::for_each(result.value(), append); co_return links; } diff --git a/src/server/service/feature_router.h b/src/server/service/feature_router.h index 9b713565e..102baf912 100644 --- a/src/server/service/feature_router.h +++ b/src/server/service/feature_router.h @@ -139,8 +139,9 @@ class FeatureRouter { RawResult workspace_symbol(llvm::StringRef query); private: - /// The preamble include links of a session's active PCH, or nullptr. - const std::vector* find_preamble_links(const Session& session); + /// The preamble include links of a session's active PCH; empty when + /// there is no PCH or its preamble no longer matches the buffer. + std::vector find_preamble_links(const Session& session); /// Resolve go-to-definition on a preamble include line that the worker /// AST cannot see: the include is compiled into the PCH, so the target diff --git a/src/server/service/query.cpp b/src/server/service/query.cpp index 29a18fe44..c7a2cf4f4 100644 --- a/src/server/service/query.cpp +++ b/src/server/service/query.cpp @@ -2,10 +2,13 @@ #include #include +#include #include +#include #include #include +#include "index/preamble_state.h" #include "index/tu_index.h" #include "server/compiler/compiler.h" #include "server/compiler/indexer.h" @@ -17,6 +20,7 @@ #include "kota/ipc/lsp/protocol.h" #include "kota/ipc/lsp/uri.h" #include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/StringSet.h" #include "llvm/Support/Path.h" namespace clice { @@ -24,6 +28,9 @@ namespace clice { namespace lsp = kota::ipc::lsp; void IndexQuery::visit_sessions(SessionVisitor visitor) const { + if(options.disk_only) { + return; + } sessions.for_each([&](std::uint32_t path_id, const Session& session) -> bool { // Freshness contract, clause 3: a dirty session's file index may // describe a buffer that no longer exists — skip it. @@ -38,6 +45,102 @@ bool IndexQuery::is_path_open(std::uint32_t path_id) const { return sessions.find(path_id) != nullptr; } +std::shared_ptr IndexQuery::overlay_of(const Session& session) const { + if(!session.pch_key) { + return nullptr; + } + auto it = workspace.pch_cache.find(*session.pch_key); + if(it == workspace.pch_cache.end()) { + return nullptr; + } + // Return the shared_ptr by value: consumers run synchronously, but a + // reference into the map value would not survive a rehash. + return it->second.load_state(); +} + +void IndexQuery::visit_overlays( + llvm::function_ref visitor) const { + if(options.disk_only) { + return; + } + // Sessions with identical preambles share one blob; visit it once. + llvm::StringSet<> seen; + sessions.for_each([&](std::uint32_t, const Session& session) -> bool { + if(!session.pch_key || !seen.insert(*session.pch_key).second) { + return true; + } + auto state = overlay_of(session); + return state ? visitor(*state) : true; + }); +} + +void IndexQuery::visit_preambles( + llvm::function_ref visitor) + const { + if(options.disk_only) { + return; + } + sessions.for_each([&](std::uint32_t path_id, const Session& session) -> bool { + auto state = overlay_of(session); + if(!state || !serves_preamble(session, *state)) { + return true; + } + return visitor(path_id, session, *state); + }); +} + +bool IndexQuery::serves_preamble(const Session& session, const index::PreambleState& state) const { + // The preamble entry's rows are buffer offsets of the file that built + // the blob: serve them only for that very file (identical preambles + // share a PCH, but macro USRs embed the source path) and only while + // the buffer still starts with the exact preamble text the blob was + // built from. The prefix comparison validates the described region + // directly — body edits never move preamble rows — so no dirty-flag + // gating is needed on top. The blob stores clang's native path + // (backslashes on Windows) while the pool normalizes separators, so + // compare through the pool's lookup, not raw strings. + return workspace.path_pool.find(state.source_path()) == session.path_id && + llvm::StringRef(session.text).starts_with(state.preamble_content()); +} + +bool IndexQuery::should_serve_overlay_file(llvm::StringRef path) const { + // An open file serves its own buffer-true rows (its session, plus the + // is_path_open shard skip); overlay rows for it were computed from the + // disk snapshot and would map onto the edited buffer at wrong lines — + // and dedup cannot collapse them, since the positions differ. + // Freshness contract, clause 2, same as shards: a file whose own + // content changed on disk has its rows suppressed until an up-to-date + // view lands — the blob snapshot describes text that no longer exists. + if(auto path_id = workspace.path_pool.find(path)) { + if(is_path_open(*path_id) || skip_stale_contribution(*path_id)) { + return false; + } + } + return !workspace.is_synthesized_artifact(path); +} + +/// Cross-source dedup: a row present in both a disk shard and a PCH +/// overlay (or in two overlays sharing a preamble) comes out identical. +static void dedup_locations(std::vector& locations) { + auto key = [](const protocol::Location& location) { + return std::tie(location.uri, + location.range.start.line, + location.range.start.character, + location.range.end.line, + location.range.end.character); + }; + std::ranges::sort(locations, + [&](const auto& lhs, const auto& rhs) { return key(lhs) < key(rhs); }); + auto dup = std::ranges::unique(locations, [&](const auto& lhs, const auto& rhs) { + return key(lhs) == key(rhs); + }); + locations.erase(dup.begin(), dup.end()); +} + +bool IndexQuery::skip_shard(std::uint32_t path_id) const { + return (!options.disk_only && is_path_open(path_id)) || skip_stale_contribution(path_id); +} + bool IndexQuery::skip_stale_contribution(std::uint32_t path_id) const { // With background indexing disabled nothing ever catches up: serving // the last-known rows beats a permanent hole. @@ -75,6 +178,16 @@ bool IndexQuery::find_symbol_info(index::SymbolHash hash, return true; } + // Check PCH overlays: a symbol that exists only under an open buffer's + // context (or in headers no disk TU has been indexed with) is in no + // disk table. + visit_overlays([&](const index::PreambleState& state) { + found = state.find_symbol(hash, name, kind); + return !found; + }); + if(found) + return true; + // Check per-file MergedIndex shards (TU-local + file-local symbols). // Each shard stores exactly the local symbols its occurrences reference, // so the symbol will be in the shard that produced the occurrence. @@ -113,6 +226,22 @@ IndexQuery::CursorHit IndexQuery::resolve_cursor(llvm::StringRef path, } return true; }); + // The preamble region is compiled into the PCH and invisible to + // the per-edit index; its occurrences (macro definitions and + // references before the bound) live in the PCH's overlay, in the + // same buffer coordinates — served only under the main-entry gate + // (preamble drift, shared-PCH identity). + auto overlay = hit.hash == 0 ? overlay_of(*session) : nullptr; + if(overlay && serves_preamble(*session, *overlay)) { + overlay->lookup_preamble(*offset, [&](const index::Occurrence& occ) { + auto range = map.to_range(occ.range.begin, occ.range.end); + if(range) { + hit = {occ.target, *range}; + return false; + } + return true; + }); + } return hit; } @@ -162,7 +291,7 @@ std::vector IndexQuery::query_relations(llvm::StringRef path auto sym_it = workspace.project_index.symbols.find(hit.hash); if(sym_it != workspace.project_index.symbols.end()) { for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -196,6 +325,42 @@ std::vector IndexQuery::query_relations(llvm::StringRef path return true; }); + // PCH overlays: header rows under each open buffer's live context. + // Rows a disk shard also holds come out identical and collapse in the + // dedup below. + visit_overlays([&](const index::PreambleState& state) { + state.lookup(hit.hash, + kind, + [&](const index::PreambleState::File& file, const index::Relation& r) { + if(!should_serve_overlay_file(file.path) || file.line_starts.empty()) + return true; + auto uri = lsp::URI::from_file_path(file.path); + if(!uri) + return true; + lsp::LineMap map(file.content, file.line_starts); + if(auto range = map.to_range(r.range.begin, r.range.end)) + locations.push_back({uri->str(), *range}); + return true; + }); + return true; + }); + + // Preamble entries: the buffers' own preamble regions. + visit_preambles( + [&](std::uint32_t id, const Session& session, const index::PreambleState& state) { + auto uri = lsp::URI::from_file_path(std::string(workspace.path_pool.resolve(id))); + if(!uri) + return true; + auto map = session.line_map(); + state.lookup_preamble(hit.hash, kind, [&](const index::Relation& r) { + if(auto range = map.to_range(r.range.begin, r.range.end)) + locations.push_back({uri->str(), *range}); + return true; + }); + return true; + }); + + dedup_locations(locations); return locations; } @@ -254,13 +419,57 @@ std::optional IndexQuery::find_definition_location(index::Sy if(session_result) return session_result; + // PCH overlays outrank disk shards: they carry the definition as seen + // under the live buffer's context, and exist even when no disk TU has + // been indexed — the in-memory-file case behind empty go-to-definition. + // First the buffers' own preamble regions, then the header entries. + std::optional overlay_result; + visit_preambles( + [&](std::uint32_t id, const Session& session, const index::PreambleState& state) { + auto uri = lsp::URI::from_file_path(std::string(workspace.path_pool.resolve(id))); + if(!uri) + return true; + auto map = session.line_map(); + state.lookup_preamble(hash, RelationKind::Definition, [&](const index::Relation& r) { + if(auto range = map.to_range(r.range.begin, r.range.end)) { + overlay_result = protocol::Location{uri->str(), *range}; + return false; + } + return true; + }); + return !overlay_result.has_value(); + }); + if(overlay_result) + return overlay_result; + + visit_overlays([&](const index::PreambleState& state) { + state.lookup(hash, + RelationKind::Definition, + [&](const index::PreambleState::File& file, const index::Relation& r) { + if(!should_serve_overlay_file(file.path) || file.line_starts.empty()) + return true; + auto uri = lsp::URI::from_file_path(file.path); + if(!uri) + return true; + lsp::LineMap map(file.content, file.line_starts); + if(auto range = map.to_range(r.range.begin, r.range.end)) { + overlay_result = protocol::Location{uri->str(), *range}; + return false; + } + return true; + }); + return !overlay_result.has_value(); + }); + if(overlay_result) + return overlay_result; + // Fall back to ProjectIndex reference files. auto sym_it = workspace.project_index.symbols.find(hash); if(sym_it == workspace.project_index.symbols.end()) return std::nullopt; for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -314,7 +523,7 @@ void IndexQuery::collect_grouped_relations( auto sym_it = workspace.project_index.symbols.find(hash); if(sym_it != workspace.project_index.symbols.end()) { for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -340,6 +549,40 @@ void IndexQuery::collect_grouped_relations( }); return true; }); + + // PCH overlays: call/type relations inside headers under an open + // buffer's context. The main-file entry cannot contribute — the + // preamble region holds only preprocessor directives. + visit_overlays([&](const index::PreambleState& state) { + state.lookup(hash, + kind, + [&](const index::PreambleState::File& file, const index::Relation& r) { + if(!should_serve_overlay_file(file.path) || file.line_starts.empty()) + return true; + lsp::LineMap map(file.content, file.line_starts); + if(auto range = map.to_range(r.range.begin, r.range.end)) + target_ranges[r.target_symbol].push_back(*range); + return true; + }); + return true; + }); + + // A row present in both a shard and an overlay lands twice; hierarchy + // items must not repeat call sites. + auto key = [](const protocol::Range& range) { + return std::tie(range.start.line, + range.start.character, + range.end.line, + range.end.character); + }; + for(auto& [target, ranges]: target_ranges) { + std::ranges::sort(ranges, + [&](const auto& lhs, const auto& rhs) { return key(lhs) < key(rhs); }); + auto dup = std::ranges::unique(ranges, [&](const auto& lhs, const auto& rhs) { + return key(lhs) == key(rhs); + }); + ranges.erase(dup.begin(), dup.end()); + } } void IndexQuery::collect_unique_targets(index::SymbolHash hash, @@ -349,7 +592,7 @@ void IndexQuery::collect_unique_targets(index::SymbolHash hash, auto sym_it = workspace.project_index.symbols.find(hash); if(sym_it != workspace.project_index.symbols.end()) { for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -375,6 +618,24 @@ void IndexQuery::collect_unique_targets(index::SymbolHash hash, } return true; }); + + // PCH overlays follow the same file rules as every other consumer: an + // open header's session is authoritative for its relations (an edited + // `struct D : NewBase` must not resurface the disk snapshot's OldBase + // through another file's overlay). + visit_overlays([&](const index::PreambleState& state) { + state.lookup(hash, + kind, + [&](const index::PreambleState::File& file, const index::Relation& r) { + if(!should_serve_overlay_file(file.path)) + return true; + if(seen.insert(r.target_symbol).second) { + targets.push_back(r.target_symbol); + } + return true; + }); + return true; + }); } /// Resolve a symbol hash into a SymbolInfo with definition location. @@ -406,36 +667,12 @@ static std::string extract_line(llvm::StringRef content, std::uint32_t offset) { } std::optional IndexQuery::get_definition_text(index::SymbolHash hash) { - std::optional session_result; - visit_sessions([&](std::uint32_t id, const Session& session) -> bool { - auto map = session.line_map(); - session.file_index->lookup(hash, RelationKind::Definition, [&](const index::Relation& rel) { - auto def_range = std::bit_cast(rel.target_symbol); - if(def_range.begin >= def_range.end || def_range.end > session.text.size()) - return true; - auto range = map.to_range(def_range.begin, def_range.end); - if(!range) - return true; - session_result = DefinitionText{ - .file = std::string(workspace.path_pool.resolve(id)), - .start_line = static_cast(range->start.line) + 1, - .end_line = static_cast(range->end.line) + 1, - .text = std::string( - session.text.substr(def_range.begin, def_range.end - def_range.begin)), - }; - return false; - }); - return !session_result.has_value(); - }); - if(session_result) - return session_result; - auto sym_it = workspace.project_index.symbols.find(hash); if(sym_it == workspace.project_index.symbols.end()) return std::nullopt; for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -478,7 +715,7 @@ std::vector IndexQuery::collect_references(ind auto sym_it = workspace.project_index.symbols.find(hash); if(sym_it != workspace.project_index.symbols.end()) { for(auto file_id: sym_it->second.reference_files) { - if(is_path_open(file_id) || skip_stale_contribution(file_id)) + if(skip_shard(file_id)) continue; auto shard_it = workspace.merged_indices.find(file_id); if(shard_it == workspace.merged_indices.end()) @@ -505,23 +742,6 @@ std::vector IndexQuery::collect_references(ind } } - visit_sessions([&](std::uint32_t id, const Session& session) -> bool { - auto map = session.line_map(); - auto file_path = workspace.path_pool.resolve(id); - session.file_index->lookup(hash, kind, [&](const index::Relation& rel) { - auto pos = map.to_position(rel.range.begin); - if(!pos) - return true; - results.push_back(ReferenceWithContext{ - .file = file_path.str(), - .line = static_cast(pos->line) + 1, - .context = extract_line(session.text, rel.range.begin), - }); - return true; - }); - return true; - }); - return results; } @@ -714,11 +934,6 @@ std::vector IndexQuery::locate_symbols(const agentic::ReadSymbol for(auto& [hash, symbol]: workspace.project_index.symbols) try_symbol(hash, symbol); - visit_sessions([&](std::uint32_t, const Session& session) -> bool { - for(auto& [hash, symbol]: *session.symbols) - try_symbol(hash, symbol); - return true; - }); if(!exact_matches.empty()) return exact_matches; @@ -729,34 +944,6 @@ std::vector IndexQuery::locate_symbols(const agentic::ReadSymbol auto path_str = *loc.path; auto target_line = static_cast(*loc.line - 1); - auto pool_it = workspace.path_pool.cache.find(path_str); - auto server_id = pool_it != workspace.path_pool.cache.end() ? pool_it->second : ~0u; - if(server_id != ~0u) { - std::vector session_result; - with_session(server_id, [&](const Session& session) { - auto map = session.line_map(); - for(auto& [hash, rels]: session.file_index->relations) { - for(auto& rel: rels) { - if(rel.kind.value() != RelationKind::Definition) - continue; - auto start = map.to_position(rel.range.begin); - if(start && start->line == target_line) { - std::string name; - SymbolKind kind; - if(!find_symbol_info(hash, name, kind)) - continue; - if(kind == SymbolKind::Parameter || kind == SymbolKind::Label) - continue; - session_result.push_back( - {hash, std::move(name), kind, path_str, *loc.line}); - } - } - } - }); - if(!session_result.empty()) - return session_result; - } - auto path_id = workspace.path_pool.find(path_str); if(!path_id) return {}; diff --git a/src/server/service/query.h b/src/server/service/query.h index 68b4f6e6c..e62eb6d47 100644 --- a/src/server/service/query.h +++ b/src/server/service/query.h @@ -90,14 +90,28 @@ struct ResolvedSymbol { /// TODO: a dedicated "is the index ready?" request so agent consumers /// can distinguish "no references" from "not indexed yet". Not /// implemented — needs protocol design. + +/// Which index sources an IndexQuery instance serves from. +struct IndexQueryOptions { + /// Disk truth only: buffer state — open sessions' file indexes and + /// their PCH overlays — never participates, and open files answer + /// from their shards exactly like closed ones. This is the agentic + /// transport's mode: agents read files from disk, so positions from + /// unsaved buffers would not match what they read. + bool disk_only = false; +}; + class IndexQuery { public: /// Visitor for iterating open Sessions. Returns false to stop early. using SessionVisitor = std::function; - IndexQuery(Workspace& workspace, const SessionStore& sessions, const Indexer& indexer) : - workspace(workspace), sessions(sessions), indexer(indexer) {} + IndexQuery(Workspace& workspace, + const SessionStore& sessions, + const Indexer& indexer, + IndexQueryOptions options = {}) : + workspace(workspace), sessions(sessions), indexer(indexer), options(options) {} /// Query relations (Definition, Reference, etc.) for a symbol at cursor. /// @param session Active Session for this file, or nullptr to use MergedIndex only. @@ -153,6 +167,10 @@ class IndexQuery { std::vector search_symbols(llvm::StringRef query, std::size_t max_results = 100); + /// The three queries below serve the agentic tools and are only ever + /// called on the disk_only instance: their bodies read shards alone, + /// with no session or overlay passes to disable. + struct DefinitionText { std::string file; int start_line; @@ -177,22 +195,14 @@ class IndexQuery { /// read/definition/references tools share. std::vector locate_symbols(const agentic::ReadSymbolParams& loc); + /// Whether a file's shard sits this query out: its own disk content + /// changed and awaits reindexing (clause 2), or — unless disk_only — + /// the file is open and its session serves it instead. + bool skip_shard(std::uint32_t path_id) const; + /// Iterate all open Sessions with valid, up-to-date file indices. void visit_sessions(SessionVisitor visitor) const; - /// Invoke a callback with the Session for a specific server-level path_id. - /// The callback is not invoked if no Session exists for that path_id. - template - void with_session(std::uint32_t server_path_id, Fn&& fn) const { - visit_sessions([&](std::uint32_t id, const Session& session) -> bool { - if(id == server_path_id) { - fn(session); - return false; - } - return true; - }); - } - /// Convert internal SymbolKind to LSP SymbolKind. static protocol::SymbolKind to_lsp_symbol_kind(SymbolKind kind); @@ -213,6 +223,43 @@ class IndexQuery { const protocol::Position& position, Session* session); + /// Visit each distinct PCH overlay blob once (sessions sharing a + /// preamble share one blob). Overlays are the only index source for + /// headers as seen under a live buffer's context (novel unsaved + /// preamble edits, headers not reachable from any indexed disk TU); + /// their header entries hold disk-derived coordinates that buffer + /// edits cannot move, so no session gating applies — the blob's own + /// staleness follows the PCH's dependency discipline. Identical rows + /// also present in disk shards are collapsed by per-location dedup at + /// result assembly. Return false from the visitor to stop. + void visit_overlays(llvm::function_ref visitor) const; + + /// Visit each open session whose overlay preamble entry may serve + /// (see serves_preamble), paired with that blob. + void visit_preambles(llvm::function_ref visitor) const; + + /// The PCH overlay of a session, or nullptr when it has no PCH or the + /// blob is unreadable. + std::shared_ptr overlay_of(const Session& session) const; + + /// Whether a session's overlay preamble entry may serve: the blob was + /// built from this very file (identical preambles share one PCH, but + /// macro USRs embed the source path) and the buffer still starts with + /// the blob's stored preamble text. + bool serves_preamble(const Session& session, const index::PreambleState& state) const; + + /// Whether an overlay file entry may contribute results. Filters + /// synthesized context artifacts (their positions live in + /// cache-directory files the user should never be sent to), files + /// that are themselves open — their sessions serve buffer-true rows, + /// while overlay rows describe the disk snapshot and would map onto + /// the edited buffer at the wrong lines — and files whose own disk + /// content changed and awaits reindexing (freshness contract, clause + /// 2, same as shard contributions). + bool should_serve_overlay_file(llvm::StringRef path) const; + /// Collect relations grouped by target symbol, across all index sources. void collect_grouped_relations( index::SymbolHash hash, @@ -238,6 +285,7 @@ class IndexQuery { Workspace& workspace; const SessionStore& sessions; const Indexer& indexer; + IndexQueryOptions options; }; } // namespace clice diff --git a/src/server/state/invalidator.cpp b/src/server/state/invalidator.cpp index 5eb367065..1ed3cf918 100644 --- a/src/server/state/invalidator.cpp +++ b/src/server/state/invalidator.cpp @@ -139,6 +139,16 @@ DirtySet Invalidator::apply(llvm::ArrayRef events) { // disk-content cascade covers everything a save invalidates. cascade_disk_content_change(path_id, dirty); + // The file's own shard describes the pre-save disk. With + // open-file indexing off the queued slot is skipped and + // BufferClosed repairs on close; with it on (an agent is + // around), the reindex lands promptly. Saves only come from + // open buffers — the session check just drops synthetic + // events for files nobody has open. + if(store.find(path_id)) { + dirty.add_reindex_content_changed(path_id); + } + // ... unless a save hook or formatter rewrote the file as it // landed, leaving the disk ahead of the buffer. Dependents // already read the rewritten disk through the cascade above; @@ -199,8 +209,11 @@ DirtySet Invalidator::apply(llvm::ArrayRef events) { // Open file: the buffer is the truth, so no disk rescan — // what the disk change means for this file is decided by // the next compile's deps validation. Recompile so that - // validation actually runs. + // validation actually runs. The shard describes the old + // disk regardless of the buffer; queue its reindex like + // a save (skipped-and-repaired-on-close without agents). dirty.mark_ast_dirty.push_back(path_id); + dirty.add_reindex_content_changed(path_id); break; } // Closed file: disk is the truth. Run the same cascade a @@ -299,8 +312,14 @@ DirtySet Invalidator::apply(llvm::ArrayRef events) { if(store.find(path_id)) { // The next compile re-resolves the command (added: // first real entry replaces the guessed one; - // changed: new flags; removed: fall back). + // changed: new flags; removed: fall back). The + // shard was indexed under the old command either + // way — same eviction as the closed branch. dirty.mark_ast_dirty.push_back(path_id); + if(!keep_shard) { + workspace.merged_indices.erase(path_id); + dirty.add_reindex_content_changed(path_id); + } } else if(!keep_shard) { // The shard was indexed under the old command, and // the indexer's freshness gate validates content diff --git a/src/server/state/session.h b/src/server/state/session.h index f8d56d327..5f12a9f6f 100644 --- a/src/server/state/session.h +++ b/src/server/state/session.h @@ -126,15 +126,12 @@ struct Session { std::shared_ptr compiling; - /// Reference to the PCH entry in Workspace.pch_cache, if any. - /// The PCH itself is owned by Workspace (shared, content-addressed); - /// Session only stores enough to locate and validate it. - struct PCHRef { - std::string key; ///< Content key into Workspace.pch_cache. - std::uint32_t bound = 0; ///< Preamble byte boundary. - }; - - std::optional pch_ref; + /// Content key into Workspace.pch_cache for this session's PCH, if + /// any. The PCH itself is owned by Workspace (shared, + /// content-addressed); whether its preamble-derived state still + /// describes this buffer is checked against the blob's stored + /// preamble text at the point of use. + std::optional pch_key; /// Dependency snapshot from the last successful AST compilation. /// Used for two-layer staleness detection (mtime + content hash). diff --git a/src/server/state/session_store.cpp b/src/server/state/session_store.cpp index cf090af93..2733fe632 100644 --- a/src/server/state/session_store.cpp +++ b/src/server/state/session_store.cpp @@ -97,7 +97,7 @@ void SessionStore::reset_compile_state(Session& session, ResetDepth depth) { session.ast_dirty = true; switch(depth) { case ResetDepth::Superseded: { - session.pch_ref.reset(); + session.pch_key.reset(); session.ast_deps.reset(); session.trial_done = false; // Invalidate any in-flight compile: without the bump it would diff --git a/src/server/state/workspace.cpp b/src/server/state/workspace.cpp index a72acdc34..aab7592d4 100644 --- a/src/server/state/workspace.cpp +++ b/src/server/state/workspace.cpp @@ -279,10 +279,6 @@ struct CachePCHEntry { std::uint32_t bound; std::int64_t build_at; std::vector deps; - - // Preamble share of the inactive-region scan; consumed on PCH reuse. - std::vector inactive_regions; - std::vector open_conditionals; }; struct CachePCMEntry { @@ -295,6 +291,14 @@ struct CachePCMEntry { struct CacheData { std::vector paths; + + // preamble_format_version the .pch.idx blobs were written with (one + // binary writes them all). A mismatch drops every PCH entry at load so + // the pairs rebuild immediately, instead of the mismatch surfacing + // lazily on the first overlay query — which cannot trigger a rebuild. + // Old cache.json files read back 0 and are dropped the same way. + std::uint32_t pch_index_format = 0; + std::vector pch; std::vector pcm; std::vector header_modes; @@ -304,6 +308,20 @@ struct CacheData { } // namespace +const std::shared_ptr& PCHState::load_state() { + if(!state && !index_path.empty()) { + state = index::PreambleState::load(index_path); + if(!state) { + // Unreadable blob: clear the path so queries don't retry the + // mmap + verification on every call. The pair now looks + // incomplete and ensure_pch rebuilds it on the next compile. + LOG_WARN("Failed to open PreambleState blob {}", index_path); + index_path.clear(); + } + } + return state; +} + void Workspace::load_cache(ContextResolver& contexts) { if(!store) return; @@ -339,17 +357,28 @@ void Workspace::load_cache(ContextResolver& contexts) { return deps; }; + bool pch_format_ok = data.pch_index_format == index::preamble_format_version; for(auto& entry: data.pch) { + if(!pch_format_ok) { + break; + } + auto pch_path = store->lookup("pch", entry.key); if(!pch_path) continue; + // A PCH without its PreambleState blob is an incomplete pair + // (crash between the two commits): treat it as absent so the next + // compile rebuilds both. + auto index_path = store->lookup_aux("pch", entry.key); + if(!index_path) + continue; + auto& st = pch_cache[entry.key]; st.path = *pch_path; st.bound = entry.bound; st.deps = load_deps(entry.build_at, entry.deps); - st.inactive_regions = entry.inactive_regions; - st.open_conditionals = entry.open_conditionals; + st.index_path = *index_path; LOG_DEBUG("Loaded cached PCH: {} -> {}", entry.key, *pch_path); } @@ -380,6 +409,7 @@ void Workspace::save_cache(const ContextResolver& contexts) { return; CacheData data; + data.pch_index_format = index::preamble_format_version; std::unordered_map index_map; auto intern = [&](std::uint32_t runtime_path_id) -> std::uint32_t { @@ -401,8 +431,6 @@ void Workspace::save_cache(const ContextResolver& contexts) { entry.key = e.getKey().str(); entry.bound = st.bound; entry.build_at = st.deps.build_at; - entry.inactive_regions = st.inactive_regions; - entry.open_conditionals = st.open_conditionals; for(std::size_t i = 0; i < st.deps.path_ids.size(); ++i) { entry.deps.push_back({intern(st.deps.path_ids[i]), st.deps.hashes[i]}); } diff --git a/src/server/state/workspace.h b/src/server/state/workspace.h index 188918bad..fd5061cc1 100644 --- a/src/server/state/workspace.h +++ b/src/server/state/workspace.h @@ -12,6 +12,7 @@ #include "command/toolchain.h" #include "feature/document_link.h" #include "index/merged_index.h" +#include "index/preamble_state.h" #include "index/project_index.h" #include "semantic/relation_kind.h" #include "server/compiler/compile_graph.h" @@ -32,7 +33,7 @@ class ContextResolver; /// On-disk cache layout version (CacheStore root `cache/v{N}`). /// Bump to discard all cached artifacts after incompatible format changes. -constexpr inline std::uint32_t cache_format_version = 3; +constexpr inline std::uint32_t cache_format_version = 4; /// Two-layer staleness snapshot for compilation artifacts (PCH, AST, etc.). /// @@ -103,19 +104,28 @@ struct SavedContext { /// Cached PCH state. Stored in Workspace.pch_cache keyed by the content /// key (hex of xxh3_128bits over preamble text + directories + canonical /// flags), so files with identical preambles share one PCH. +/// +/// Everything derived from the PCH build beyond validity metadata — the +/// preamble's symbol index, document links, inactive regions, the open +/// conditional stack — lives in the paired PreambleState blob (the store's +/// `.pch.idx` aux file), committed and evicted together with the PCH. struct PCHState { std::string path; std::uint32_t bound = 0; DepsSnapshot deps; - /// Include directives of the preamble, used for document links and - /// go-to-definition on preamble include lines. - std::vector preamble_links; - - /// Inactive regions within the preamble (flat offset pairs) and the - /// conditional stack still open at the bound — a #if cut by the bound - /// resumes in the AST compile's scan. - std::vector inactive_regions; - std::vector open_conditionals; + + /// Path of the paired PreambleState blob. + std::string index_path; + + /// Lazily opened blob; shared so a consumer holding it across an await + /// survives concurrent entry replacement or eviction. + std::shared_ptr state; + + /// Open the blob on first use (memory-mapped, no deserialization). + /// Returns nullptr when the blob is missing or unreadable — consumers + /// degrade (no overlay, no preamble links) and the next ensure_pch + /// treats the incomplete pair as a cache miss. + const std::shared_ptr& load_state(); std::shared_ptr building; }; diff --git a/src/server/transport/agent_client.cpp b/src/server/transport/agent_client.cpp index c14d8ff0d..f90ee5cf0 100644 --- a/src/server/transport/agent_client.cpp +++ b/src/server/transport/agent_client.cpp @@ -262,6 +262,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request([&srv](RequestContext&, const SymbolSearchParams& params) -> RequestResult { + srv.on_agentic_query(); auto max = params.max_results.value_or(100); std::string query_lower = llvm::StringRef(params.query).lower(); @@ -282,7 +283,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : if(std::ranges::find(filter, kind_name) == filter.end()) return; } - auto def_loc = srv.index_query.find_definition_location(hash); + auto def_loc = srv.agent_query.find_definition_location(hash); if(!def_loc) return; if(!seen.insert(hash).second) @@ -299,23 +300,19 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : for(auto& [hash, symbol]: srv.workspace.project_index.symbols) try_symbol(hash, symbol); - srv.index_query.visit_sessions([&](std::uint32_t, const Session& session) -> bool { - for(auto& [hash, symbol]: *session.symbols) - try_symbol(hash, symbol); - return true; - }); co_return result; }); peer.on_request( [&srv](RequestContext&, const ReadSymbolParams& params) -> RequestResult { - auto resolved = resolve_unique(params, srv.index_query); + srv.on_agentic_query(); + auto resolved = resolve_unique(params, srv.agent_query); if(!resolved) co_return kota::outcome_error(std::move(resolved.error())); auto& rs = *resolved; - auto def_text = srv.index_query.get_definition_text(rs.hash); + auto def_text = srv.agent_query.get_definition_text(rs.hash); if(!def_text) co_return kota::outcome_error(kota::ipc::Error{"definition not found"}); @@ -333,6 +330,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request( [&srv](RequestContext&, const DocumentSymbolsParams& params) -> RequestResult { + srv.on_agentic_query(); auto is_document_level = [](SymbolKind kind) { return kind == SymbolKind::Namespace || kind == SymbolKind::Class || kind == SymbolKind::Struct || kind == SymbolKind::Union || @@ -346,44 +344,16 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : DocumentSymbolsResult result; - auto pool_it = srv.workspace.path_pool.cache.find(params.path); - if(pool_it == srv.workspace.path_pool.cache.end()) - co_return result; - auto server_id = pool_it->second; - bool found_session = false; - srv.index_query.with_session(server_id, [&](const Session& session) { - found_session = true; - for(auto& [hash, rels]: session.file_index->relations) { - for(auto& rel: rels) { - if(rel.kind.value() != RelationKind::Definition) - continue; - std::string name; - SymbolKind kind; - if(!srv.index_query.find_symbol_info(hash, name, kind)) - continue; - if(!is_document_level(kind)) - continue; - auto range = session.line_map().to_range(rel.range.begin, rel.range.end); - if(range) { - result.symbols.push_back(DocumentSymbolEntry{ - .name = std::move(name), - .kind = std::string(symbol_kind_name(kind)), - .start_line = static_cast(range->start.line) + 1, - .end_line = static_cast(range->end.line) + 1, - .symbol_id = hash, - }); - break; - } - } - } - }); - if(found_session) - co_return result; - auto path_id = srv.workspace.path_pool.find(params.path); if(!path_id) co_return result; + // The same shard gate every other agentic lookup applies + // (freshness contract, clause 2): a shard whose file changed + // on disk serves nothing until its reindex lands. + if(srv.agent_query.skip_shard(*path_id)) + co_return result; + auto shard_it = srv.workspace.merged_indices.find(*path_id); if(shard_it == srv.workspace.merged_indices.end()) co_return result; @@ -423,9 +393,10 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request( [&srv](RequestContext&, const DefinitionParams& params) -> RequestResult { + srv.on_agentic_query(); auto resolved = resolve_unique( ReadSymbolParams{params.name, params.path, params.line, params.symbol_id}, - srv.index_query); + srv.agent_query); if(!resolved) co_return kota::outcome_error(std::move(resolved.error())); @@ -436,7 +407,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : result.kind = std::string(symbol_kind_name(rs.kind)); result.symbol_id = rs.hash; - if(auto def_text = srv.index_query.get_definition_text(rs.hash)) { + if(auto def_text = srv.agent_query.get_definition_text(rs.hash)) { result.definition = LocationEntry{ .file = std::move(def_text->file), .start_line = def_text->start_line, @@ -450,9 +421,10 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request([&srv](RequestContext&, const ReferencesParams& params) -> RequestResult { + srv.on_agentic_query(); auto resolved = resolve_unique( ReadSymbolParams{params.name, params.path, params.line, params.symbol_id}, - srv.index_query); + srv.agent_query); if(!resolved) co_return kota::outcome_error(std::move(resolved.error())); @@ -463,7 +435,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : result.kind = std::string(symbol_kind_name(rs.kind)); result.symbol_id = rs.hash; - for(auto& ref: srv.index_query.collect_references(rs.hash, RelationKind::Reference)) { + for(auto& ref: srv.agent_query.collect_references(rs.hash, RelationKind::Reference)) { result.references.push_back(ReferenceEntry{ .file = std::move(ref.file), .line = ref.line, @@ -471,7 +443,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : }); } if(params.include_declaration.value_or(false)) { - for(auto& ref: srv.index_query.collect_references(rs.hash, RelationKind::Definition)) { + for(auto& ref: srv.agent_query.collect_references(rs.hash, RelationKind::Definition)) { result.references.push_back(ReferenceEntry{ .file = std::move(ref.file), .line = ref.line, @@ -486,9 +458,10 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request( [&srv](RequestContext&, const CallGraphParams& params) -> RequestResult { + srv.on_agentic_query(); auto resolved = resolve_unique( ReadSymbolParams{params.name, params.path, params.line, params.symbol_id}, - srv.index_query); + srv.agent_query); if(!resolved) co_return kota::outcome_error(std::move(resolved.error())); @@ -509,13 +482,13 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : return "Function"; std::string name; SymbolKind kind; - if(srv.index_query.find_symbol_info(sym_id, name, kind)) + if(srv.agent_query.find_symbol_info(sym_id, name, kind)) return std::string(symbol_kind_name(kind)); return "Function"; }; if(direction == "callers" || direction == "both") { - auto incoming = srv.index_query.find_incoming_calls(rs.hash); + auto incoming = srv.agent_query.find_incoming_calls(rs.hash); for(auto& call: incoming) { auto sid = extract_symbol_id(call.from.data); result.callers.push_back(CallGraphEntry{ @@ -529,7 +502,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : } if(direction == "callees" || direction == "both") { - auto outgoing = srv.index_query.find_outgoing_calls(rs.hash); + auto outgoing = srv.agent_query.find_outgoing_calls(rs.hash); for(auto& call: outgoing) { auto sid = extract_symbol_id(call.to.data); result.callees.push_back(CallGraphEntry{ @@ -548,9 +521,10 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : peer.on_request( [&srv](RequestContext&, const TypeHierarchyParams& params) -> RequestResult { + srv.on_agentic_query(); auto resolved = resolve_unique( ReadSymbolParams{params.name, params.path, params.line, params.symbol_id}, - srv.index_query); + srv.agent_query); if(!resolved) co_return kota::outcome_error(std::move(resolved.error())); @@ -571,13 +545,13 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : return "Class"; std::string name; SymbolKind kind; - if(srv.index_query.find_symbol_info(sym_id, name, kind)) + if(srv.agent_query.find_symbol_info(sym_id, name, kind)) return std::string(symbol_kind_name(kind)); return "Class"; }; if(direction == "supertypes" || direction == "both") { - for(auto& item: srv.index_query.find_supertypes(rs.hash)) { + for(auto& item: srv.agent_query.find_supertypes(rs.hash)) { auto sid = extract_symbol_id(item.data); result.supertypes.push_back(TypeHierarchyEntry{ .name = item.name, @@ -590,7 +564,7 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : } if(direction == "subtypes" || direction == "both") { - for(auto& item: srv.index_query.find_subtypes(rs.hash)) { + for(auto& item: srv.agent_query.find_subtypes(rs.hash)) { auto sid = extract_symbol_id(item.data); result.subtypes.push_back(TypeHierarchyEntry{ .name = item.name, @@ -606,11 +580,15 @@ AgentClient::AgentClient(MasterServer& server, kota::ipc::JsonPeer& peer) : }); peer.on_request([&srv](RequestContext&, const StatusParams&) -> RequestResult { + // The progress numbers describe the current round — or the last + // one, retained after it ends; the live queue is compacted between + // rounds and would read as "nothing was ever indexed". + auto& progress = srv.indexer.progress(); StatusResult result; result.idle = srv.indexer.is_idle(); result.pending = static_cast(srv.indexer.pending_files()); - result.total = static_cast(srv.indexer.total_queued()); - result.indexed = std::max(0, result.total - result.pending); + result.total = static_cast(progress.total); + result.indexed = static_cast(progress.completed); co_return result; }); diff --git a/src/server/transport/master_server.cpp b/src/server/transport/master_server.cpp index 0b0e66c18..19f409a08 100644 --- a/src/server/transport/master_server.cpp +++ b/src/server/transport/master_server.cpp @@ -35,6 +35,7 @@ constexpr static std::size_t notify_log_limit = 128; MasterServer::MasterServer(kota::event_loop& loop, std::string self_path) : loop(loop), pool(loop), contexts(workspace), compiler(loop, workspace, contexts, pool), indexer(loop, workspace, pool, contexts, sessions), index_query(workspace, sessions, indexer), + agent_query(workspace, sessions, indexer, {.disk_only = true}), features(compiler, index_query, workspace, contexts, indexer), invalidator(workspace, sessions, contexts), bg_tasks(loop), self_path(std::move(self_path)) { // The notify hook is process-wide because the logging layer cannot @@ -238,6 +239,36 @@ void MasterServer::close_session(std::uint32_t path_id) { LOG_DEBUG("didClose: {}", path); } +void MasterServer::on_agentic_query() { + if(indexer.index_open_files) { + return; + } + // First agentic index query: agents read disk truth, so open files' + // disk snapshots must be indexed too — background indexing skips + // them otherwise, since the LSP side is fully served by their + // sessions. Sticky for the server's lifetime. + indexer.index_open_files = true; + for(auto& [path_id, session]: sessions.sessions) { + if(!session) { + continue; + } + // Same disk-vs-shard arbitration as BufferClosed: a current shard + // keeps serving through the catch-up, while a stale or missing one + // (a save that landed while its reindex slot was still skipped) + // must not answer agents with the pre-save rows. + auto disk = fs::read(workspace.path_pool.resolve(path_id)); + if(!disk) { + continue; + } + auto shard_it = workspace.merged_indices.find(path_id); + bool shard_current = + shard_it != workspace.merged_indices.end() && *disk == shard_it->second.content(); + indexer.enqueue(path_id, + shard_current ? ReindexReason::DepsOnly : ReindexReason::ContentChanged); + } + indexer.schedule(); +} + void MasterServer::dispatch(llvm::ArrayRef events) { auto dirty = invalidator.apply(events); @@ -373,8 +404,11 @@ void MasterServer::open_cache_store() { // Size budgets are deliberately generous: eviction exists to bound // disk usage, not to keep the working set tight. constexpr std::uint64_t GiB = 1ull << 30; - store->register_namespace( - {.name = "pch", .extension = ".pch", .policy = CachePolicy::LRU, .max_bytes = 8 * GiB}); + store->register_namespace({.name = "pch", + .extension = ".pch", + .aux_extension = ".pch.idx", + .policy = CachePolicy::LRU, + .max_bytes = 8 * GiB}); store->register_namespace( {.name = "pcm", .extension = ".pcm", .policy = CachePolicy::LRU, .max_bytes = 8 * GiB}); store->register_namespace( diff --git a/src/server/transport/master_server.h b/src/server/transport/master_server.h index 480b4e1f8..af7813817 100644 --- a/src/server/transport/master_server.h +++ b/src/server/transport/master_server.h @@ -115,6 +115,12 @@ class MasterServer { /// services (sessions, context resolver, background indexer). void dispatch(llvm::ArrayRef events); + /// Called by the agentic index-query handlers before answering. The + /// first call turns on open-file indexing (sticky) and enqueues the + /// currently open files, so agents get shards for files whose + /// sessions otherwise satisfied every consumer. + void on_agentic_query(); + void schedule_shutdown(); kota::cancellation_token shutdown_token() const { @@ -137,6 +143,12 @@ class MasterServer { Compiler compiler; Indexer indexer; IndexQuery index_query; + + /// The agentic transport's view of the index: disk truth only. + /// Agents read files from disk, so buffer state must not leak into + /// their answers — see IndexQueryOptions::disk_only. + IndexQuery agent_query; + FeatureRouter features; Invalidator invalidator; diff --git a/src/server/worker/stateful_worker.cpp b/src/server/worker/stateful_worker.cpp index 8bf2092a7..aca6f579d 100644 --- a/src/server/worker/stateful_worker.cpp +++ b/src/server/worker/stateful_worker.cpp @@ -214,9 +214,7 @@ void StatefulWorker::register_handlers() { co_return co_await with_ast_or( params.path, std::vector{}, - [&](DocumentEntry& doc) { - return feature::document_links(doc.unit, feature::PositionEncoding::UTF16); - }); + [&](DocumentEntry& doc) { return feature::document_links(doc.unit); }); }); // === Evict === diff --git a/src/server/worker/stateless_worker.cpp b/src/server/worker/stateless_worker.cpp index 3bf75de33..09e1d19b1 100644 --- a/src/server/worker/stateless_worker.cpp +++ b/src/server/worker/stateless_worker.cpp @@ -1,10 +1,13 @@ #include "server/worker/stateless_worker.h" #include +#include +#include #include "compile/compilation.h" #include "feature/feature.h" #include "feature/inactive_regions.h" +#include "index/preamble_state.h" #include "index/tu_index.h" #include "server/protocol/worker.h" #include "server/worker/worker_common.h" @@ -50,16 +53,50 @@ static std::string collect_errors(CompilationUnit& unit) { return errors; } -/// Build a TUIndex, serialize it, and return as a string. -static std::string serialize_tu_index(CompilationUnit& unit, bool interested_only = false) { - auto tu_index = index::TUIndex::build(unit, interested_only); - if(!interested_only) { - tu_index.main_file_index = index::FileIndex(); +/// Serialize the preamble's PreambleState blob (full index + document +/// links + inactive regions) into a string. Runs while the freshly +/// parsed AST is still in memory — the only moment the preamble's index +/// is obtainable without deserializing the whole PCH. The file write +/// happens separately, after the PCH itself is flushed. +static std::string serialize_preamble_state(CompilationUnit& unit, std::uint32_t preamble_bound) { + auto tu_index = index::TUIndex::build(unit); + auto links = feature::document_links(unit); + auto inactive = feature::inactive_regions(unit, {}, 0, preamble_bound); + + std::string blob; + llvm::raw_string_ostream os(blob); + index::PreambleState::serialize(unit, + tu_index, + links, + inactive.regions, + inactive.open_stack, + os); + return blob; +} + +/// Write the serialized blob next to the PCH. Returns an error description +/// on failure so the master's anomaly carries the cause. +static std::optional write_preamble_state(llvm::StringRef blob, + llvm::StringRef output_path) { + std::error_code ec; + llvm::raw_fd_ostream os(output_path, ec); + if(ec) { + auto message = + std::format("cannot open PreambleState blob {}: {}", output_path, ec.message()); + LOG_ERROR("BuildPCH: {}", message); + return message; } - std::string serialized; - llvm::raw_string_ostream os(serialized); - tu_index.serialize(os); - return serialized; + os << blob; + os.flush(); + if(os.has_error()) { + auto message = std::format("failed writing PreambleState blob {}: {}", + output_path, + os.error().message()); + os.clear_error(); + LOG_ERROR("BuildPCH: {}", message); + return message; + } + return std::nullopt; } static worker::BuildResult handle_build_pch(const worker::BuildParams& params) { @@ -94,28 +131,36 @@ static worker::BuildResult handle_build_pch(const worker::BuildParams& params) { if(!success) errors = collect_errors(unit); - std::string tu_index_data; - std::vector preamble_links; - feature::InactiveScan inactive; + std::string blob; if(success) { - tu_index_data = serialize_tu_index(unit); - preamble_links = feature::document_links(unit); - inactive = feature::inactive_regions(unit, {}, 0, params.preamble_bound); + blob = serialize_preamble_state(unit, params.preamble_bound); } // Destroy CompilationUnit to flush PCH to disk. unit = CompilationUnit(nullptr); + // Write the blob strictly after the PCH flush: the CacheStore's + // restart adoption validates a pair by "aux not older than primary" + // (renames preserve mtimes), so the on-disk order must match the + // logical one. The PCH is only served together with its blob, so a + // blob write failure fails the whole build. It is an internal I/O + // failure, never a user-code problem — must not be downgraded to an + // expected build failure. + bool internal_error = false; + if(success) { + if(auto error = write_preamble_state(blob, params.index_output_path)) { + success = false; + internal_error = true; + errors = std::move(*error); + } + } + if(success) { LOG_INFO("BuildPCH done: file={}, output={}, {}ms", params.file, tmp_path, timer.ms()); worker::BuildResult result; result.success = true; result.output_path = tmp_path; result.deps = pch_info.deps; - result.tu_index_data = std::move(tu_index_data); - result.preamble_links = std::move(preamble_links); - result.inactive_regions = std::move(inactive.regions); - result.open_conditionals = std::move(inactive.open_stack); return result; } else { LOG_WARN("BuildPCH failed: file={}, {}ms, errors=[{}]", params.file, timer.ms(), errors); @@ -123,7 +168,7 @@ static worker::BuildResult handle_build_pch(const worker::BuildParams& params) { worker::BuildResult result; result.success = false; result.error = errors.empty() ? "PCH compilation failed" : errors; - result.has_user_errors = !errors.empty(); + result.has_user_errors = !internal_error && !errors.empty(); return result; } } @@ -160,10 +205,10 @@ static worker::BuildResult handle_build_pcm(const worker::BuildParams& params) { if(!success) errors = collect_errors(unit); - std::string tu_index_data; - if(success) - tu_index_data = serialize_tu_index(unit, true); - + // TODO: PCM indexing. Unlike the PCH, a PCM is not a transient + // buffer-derived artifact — module units are ordinary disk files with + // CDB entries, so their symbols should flow through the normal + // background-indexing path (no per-blob pair needed). unit = CompilationUnit(nullptr); if(success) { @@ -172,7 +217,6 @@ static worker::BuildResult handle_build_pcm(const worker::BuildParams& params) { result.success = true; result.output_path = tmp_path; result.deps = pcm_info.deps; - result.tu_index_data = std::move(tu_index_data); return result; } else { LOG_WARN("BuildPCM failed: module={}, {}ms, errors=[{}]", diff --git a/src/support/cache_store.cpp b/src/support/cache_store.cpp index 6704d2834..ca3d28c96 100644 --- a/src/support/cache_store.cpp +++ b/src/support/cache_store.cpp @@ -146,7 +146,11 @@ std::int64_t now_ms() { struct CacheStore::State { struct Entry { + /// Primary blob size. total_size accounts size + aux_size. std::uint64_t size = 0; + /// Aux blob size; non-zero means the aux blob is committed (blobs + /// are never empty, so zero doubles as "absent"). + std::uint64_t aux_size = 0; std::int64_t atime = 0; }; @@ -193,6 +197,26 @@ struct CacheStore::State { return path::join(ns.dir, key.str() + ns.config.extension); } + std::string aux_blob_path(const Namespace& ns, llvm::StringRef key) const { + return path::join(ns.dir, key.str() + ns.config.aux_extension); + } + + /// Drop a key's committed aux blob (file + accounting). Called when + /// the primary is republished: the old aux describes the old primary, + /// and an incomplete pair is strictly better than a mismatched one. + /// The accounting is zeroed even when the delete fails (file open on + /// Windows) — lookup_aux must stop serving the stale blob immediately; + /// the leftover file is overwritten by the following aux commit or + /// swept when the entry is eventually evicted. + void reset_aux_locked(Namespace& ns, llvm::StringRef key, Entry& entry) { + if(ns.config.aux_extension.empty() || entry.aux_size == 0) { + return; + } + llvm::sys::fs::remove(aux_blob_path(ns, key)); + ns.total_size -= entry.aux_size; + entry.aux_size = 0; + } + Namespace* find_namespace(llvm::StringRef name) { auto it = namespaces.find(name); return it != namespaces.end() ? &it->second : nullptr; @@ -302,16 +326,21 @@ void CacheStore::register_namespace(CacheNamespace ns) { // Adopt blobs already on disk. The directory scan, not the manifest, // decides existence: this also picks up blobs committed after the last - // checkpoint of a crashed instance. + // checkpoint of a crashed instance. Aux blobs are collected aside and + // attached after the scan — directory order is arbitrary, so an aux + // file may be seen before its primary. + struct AuxBlob { + std::uint64_t size; + llvm::sys::TimePoint<> mtime; + }; + + llvm::StringMap aux_blobs; + llvm::StringMap> primary_mtimes; std::error_code ec; for(auto iter = llvm::sys::fs::directory_iterator(ns_state.dir, ec); !ec && iter != llvm::sys::fs::directory_iterator(); iter.increment(ec)) { auto filename = path::filename(iter->path()); - auto& ext = ns_state.config.extension; - if(!ext.empty() && !filename.consume_back(ext)) { - continue; - } llvm::sys::fs::file_status status; if(llvm::sys::fs::status(iter->path(), status) || @@ -319,6 +348,18 @@ void CacheStore::register_namespace(CacheNamespace ns) { continue; } + auto& aux_ext = ns_state.config.aux_extension; + if(!aux_ext.empty() && filename.ends_with(aux_ext)) { + aux_blobs[filename.drop_back(aux_ext.size())] = {status.getSize(), + status.getLastModificationTime()}; + continue; + } + + auto& ext = ns_state.config.extension; + if(!ext.empty() && !filename.consume_back(ext)) { + continue; + } + auto atime_it = state->manifest_atimes.find(ns_state.config.name + "/" + filename.str()); auto atime = atime_it != state->manifest_atimes.end() ? atime_it->second @@ -326,8 +367,29 @@ void CacheStore::register_namespace(CacheNamespace ns) { status.getLastModificationTime().time_since_epoch()) .count(); - ns_state.entries[filename] = {status.getSize(), atime}; + ns_state.entries[filename] = {status.getSize(), 0, atime}; ns_state.total_size += status.getSize(); + primary_mtimes[filename] = status.getLastModificationTime(); + } + + // Attach aux blobs to their entries. An aux without a primary is crash + // residue (a pair evicted halfway); an aux OLDER than its primary is + // residue of a removal that failed while the file was held open (see + // reset_aux_locked) — writers put the aux file on disk only after the + // primary is fully flushed (see handle_build_pch) and pairs commit + // primary-first, so a legitimate aux is never older, and adopting one + // would pair a fresh primary with stale aux data. Remove both kinds, + // nothing references them. + for(auto& entry: aux_blobs) { + auto key = entry.getKey(); + auto it = ns_state.entries.find(key); + if(it != ns_state.entries.end() && entry.getValue().mtime >= primary_mtimes[key]) { + it->second.aux_size = entry.getValue().size; + ns_state.total_size += entry.getValue().size; + } else { + llvm::sys::fs::remove(state->aux_blob_path(ns_state, key)); + LOG_DEBUG("CacheStore: removed stale aux blob {} in {}", key, ns_state.config.name); + } } // Enforce the budget immediately in case it shrank since the last run. @@ -352,6 +414,24 @@ std::optional CacheStore::lookup(llvm::StringRef ns, llvm::StringRe return state->blob_path(*ns_state, key); } +std::optional CacheStore::lookup_aux(llvm::StringRef ns, llvm::StringRef key) { + std::lock_guard guard(state->mutex); + + auto* ns_state = state->find_namespace(ns); + if(!ns_state || ns_state->config.aux_extension.empty()) { + return std::nullopt; + } + + auto it = ns_state->entries.find(key); + if(it == ns_state->entries.end() || it->second.aux_size == 0) { + return std::nullopt; + } + + it->second.atime = state->next_stamp(); + state->dirty = true; + return state->aux_blob_path(*ns_state, key); +} + CacheStore::PendingEntry CacheStore::begin_store(llvm::StringRef ns, llvm::StringRef key) { std::lock_guard guard(state->mutex); @@ -373,6 +453,28 @@ CacheStore::PendingEntry CacheStore::begin_store(llvm::StringRef ns, llvm::Strin return PendingEntry{ns.str(), key.str(), path::join(state->tmp_dir, tmp_name)}; } +CacheStore::PendingEntry CacheStore::begin_store_aux(llvm::StringRef ns, llvm::StringRef key) { + std::lock_guard guard(state->mutex); + + auto* ns_state = state->find_namespace(ns); + assert(ns_state && "begin_store_aux on unregistered namespace"); + if(!ns_state) { + LOG_ERROR("CacheStore: begin_store_aux on unregistered namespace {}", ns); + return {}; + } + if(ns_state->config.aux_extension.empty()) { + LOG_ERROR("CacheStore: begin_store_aux on namespace {} without aux extension", ns); + return {}; + } + + if(auto ec = llvm::sys::fs::create_directories(state->tmp_dir)) { + LOG_WARN("CacheStore: cannot re-create tmp dir {}: {}", state->tmp_dir, ec.message()); + } + + auto tmp_name = std::format("{}{}", state->next_tmp_id++, ns_state->config.aux_extension); + return PendingEntry{ns.str(), key.str(), path::join(state->tmp_dir, tmp_name), /*aux=*/true}; +} + std::expected CacheStore::commit(PendingEntry pending) { if(pending.tmp_path.empty()) { return std::unexpected(std::make_error_code(std::errc::invalid_argument)); @@ -412,7 +514,16 @@ std::expected CacheStore::commit(PendingEntry pend return std::unexpected(std::make_error_code(std::errc::invalid_argument)); } - final_path = state->blob_path(*ns_state, pending.key); + // An aux blob attaches to a live entry; without one (primary commit + // failed, or the entry was evicted in between) it would be an + // orphan — refuse, the caller rebuilds the pair. + if(pending.aux && !ns_state->entries.contains(pending.key)) { + llvm::sys::fs::remove(pending.tmp_path); + return std::unexpected(std::make_error_code(std::errc::no_such_file_or_directory)); + } + + final_path = pending.aux ? state->aux_blob_path(*ns_state, pending.key) + : state->blob_path(*ns_state, pending.key); // The namespace dir can be wiped externally while the server runs; // re-create it so the rename below doesn't fail forever. if(auto ec = llvm::sys::fs::create_directories(ns_state->dir)) { @@ -439,10 +550,20 @@ std::expected CacheStore::commit(PendingEntry pend llvm::sys::fs::remove(final_path); if(auto retry = fs::rename(pending.tmp_path, final_path); !retry) { llvm::sys::fs::remove(pending.tmp_path); - if(auto it = ns_state->entries.find(pending.key); - it != ns_state->entries.end() && llvm::sys::fs::status(final_path, status)) { + auto it = ns_state->entries.find(pending.key); + bool entry_alive = it != ns_state->entries.end(); + if(pending.aux && entry_alive) { + // The removal above may have deleted a committed + // aux blob; stop serving it. The primary is + // intact, the pair is merely incomplete. + ns_state->total_size -= it->second.aux_size; + it->second.aux_size = 0; + state->dirty = true; + } else if(!pending.aux && entry_alive && + llvm::sys::fs::status(final_path, status)) { // The old blob is gone as well: drop its entry so // lookups don't hand out a dangling path. + state->reset_aux_locked(*ns_state, pending.key, it->second); ns_state->total_size -= it->second.size; ns_state->entries.erase(it); state->dirty = true; @@ -453,11 +574,20 @@ std::expected CacheStore::commit(PendingEntry pend } auto& entry = ns_state->entries[pending.key]; - // Unsigned wraparound is intentional and exact here: entry.size is - // already included in total_size, so total + new - old stays correct - // even when the replacement blob is smaller. - ns_state->total_size += status.getSize() - entry.size; - entry.size = status.getSize(); + if(pending.aux) { + ns_state->total_size += status.getSize() - entry.aux_size; + entry.aux_size = status.getSize(); + } else { + // A republished primary invalidates the old aux blob: serving + // yesterday's aux next to today's primary would be a silent + // mismatch, while an incomplete pair is a plain cache miss. + state->reset_aux_locked(*ns_state, pending.key, entry); + // Unsigned wraparound is intentional and exact here: entry.size + // is already included in total_size, so total + new - old stays + // correct even when the replacement blob is smaller. + ns_state->total_size += status.getSize() - entry.size; + entry.size = status.getSize(); + } entry.atime = state->next_stamp(); if(ns_state->config.policy == CachePolicy::LRU) { @@ -494,8 +624,11 @@ void CacheStore::invalidate(llvm::StringRef ns, llvm::StringRef key) { return; } + if(!ns_state->config.aux_extension.empty()) { + llvm::sys::fs::remove(state->aux_blob_path(*ns_state, key)); + } llvm::sys::fs::remove(state->blob_path(*ns_state, key)); - ns_state->total_size -= it->second.size; + ns_state->total_size -= it->second.size + it->second.aux_size; ns_state->entries.erase(it); if(ns_state->config.policy != CachePolicy::Scratch) { @@ -546,7 +679,8 @@ void CacheStore::State::evict_locked(Namespace& ns, llvm::StringRef keep_key) { candidates.reserve(ns.entries.size()); for(auto& entry: ns.entries) { if(entry.first() != keep_key) { - candidates.push_back({entry.first(), entry.second.atime, entry.second.size}); + candidates.push_back( + {entry.first(), entry.second.atime, entry.second.size + entry.second.aux_size}); } } std::ranges::sort(candidates, {}, &Candidate::atime); @@ -555,8 +689,23 @@ void CacheStore::State::evict_locked(Namespace& ns, llvm::StringRef keep_key) { if(ns.total_size <= ns.config.max_bytes) { break; } - // A failed delete (e.g. the file is open on Windows) keeps the - // entry so the next eviction round retries it. + // Aux first, then primary: whatever a failure leaves behind is a + // served primary with a missing aux (a plain pair miss), never an + // orphan aux. A failed delete (e.g. the file is open on Windows) + // keeps the entry so the next eviction round retries it. + if(!ns.config.aux_extension.empty()) { + auto it = ns.entries.find(candidate.key); + if(it->second.aux_size != 0) { + if(llvm::sys::fs::remove(aux_blob_path(ns, candidate.key))) { + LOG_DEBUG("CacheStore: cannot evict aux {} from {}, retrying later", + candidate.key, + ns.config.name); + continue; + } + ns.total_size -= it->second.aux_size; + it->second.aux_size = 0; + } + } if(llvm::sys::fs::remove(blob_path(ns, candidate.key))) { LOG_DEBUG("CacheStore: cannot evict {} from {}, retrying later", candidate.key, @@ -568,8 +717,9 @@ void CacheStore::State::evict_locked(Namespace& ns, llvm::StringRef keep_key) { ns.config.name, candidate.key, candidate.size); - ns.total_size -= candidate.size; - ns.entries.erase(candidate.key); + auto it = ns.entries.find(candidate.key); + ns.total_size -= it->second.size; + ns.entries.erase(it); dirty = true; } } diff --git a/src/support/cache_store.h b/src/support/cache_store.h index 659c80a55..39634ca39 100644 --- a/src/support/cache_store.h +++ b/src/support/cache_store.h @@ -36,6 +36,14 @@ struct CacheNamespace { /// File extension appended to keys, including the dot, e.g. ".pch". std::string extension; + /// Optional paired-blob extension, e.g. ".pch.idx". When set, a key + /// owns two files — `{key}{extension}` plus `{key}{aux_extension}` — + /// forming a single entry: sized, aged and evicted together. The + /// primary is committed first; committing it resets any stale aux + /// blob, so a pair is only served complete (see lookup_aux). Must not + /// be a suffix collision with `extension`. + std::string aux_extension; + CachePolicy policy = CachePolicy::LRU; /// Size budget for LRU namespaces; 0 means unlimited. @@ -85,6 +93,9 @@ class CacheStore { std::string ns; std::string key; std::string tmp_path; + + /// Whether this write targets the key's aux blob (begin_store_aux). + bool aux = false; }; /// Open (creating if necessary) the store under `root`. Any sibling @@ -107,10 +118,21 @@ class CacheStore { /// last-accessed time (persisted on the next checkpoint). No disk IO. std::optional lookup(llvm::StringRef ns, llvm::StringRef key); + /// Return the key's aux blob path when the entry exists and its aux + /// blob was committed. Also refreshes last-accessed. A miss with a + /// present primary means the pair is incomplete (crash between the two + /// commits, failed aux commit) — callers treat it as a cache miss and + /// rebuild the pair. + std::optional lookup_aux(llvm::StringRef ns, llvm::StringRef key); + /// Begin a two-phase write: returns a unique tmp path the blob must be /// written to (safe to hand to a worker process). PendingEntry begin_store(llvm::StringRef ns, llvm::StringRef key); + /// Begin a two-phase write of the key's aux blob. Commit the primary + /// first: committing an aux blob for a key with no live entry fails. + PendingEntry begin_store_aux(llvm::StringRef ns, llvm::StringRef key); + /// Finish a two-phase write: fsync the tmp file and atomically rename /// it to its final path. Triggers LRU eviction when the namespace /// exceeds its budget. Returns the final blob path. diff --git a/tests/integration/agentic/test_agentic.py b/tests/integration/agentic/test_agentic.py index a1db68117..b1dbb052a 100644 --- a/tests/integration/agentic/test_agentic.py +++ b/tests/integration/agentic/test_agentic.py @@ -156,18 +156,22 @@ async def indexed_agentic(request, executable, workspace): c = CliceClient() await c.start_io(*cmd) - await c.initialize(workspace) + # The first agentic query triggers a catch-up indexing round for open + # files; the default 3s idle timer would dominate every fixture setup. + await c.initialize( + workspace, initialization_options={"project": {"idle_timeout_ms": 10}} + ) uri, _ = await c.open_and_wait(workspace / "main.cpp") assert await wait_for_index(c, uri, "add"), "Index not ready" rpc = AgenticRpcClient(host, port) - for _ in range(30): + for _ in range(300): resp = rpc.request("agentic/symbolSearch", {"query": "add"}) if "result" in resp and resp["result"]["symbols"]: break - await asyncio.sleep(1) + await asyncio.sleep(0.1) else: pytest.fail("agentic/symbolSearch never returned indexed symbols") diff --git a/tests/integration/agentic/test_cli.py b/tests/integration/agentic/test_cli.py index 1449d630a..d0476429a 100644 --- a/tests/integration/agentic/test_cli.py +++ b/tests/integration/agentic/test_cli.py @@ -44,7 +44,11 @@ async def indexed_server(request, executable, workspace): c = CliceClient() await c.start_io(*cmd) - await c.initialize(workspace) + # The first agentic query triggers a catch-up indexing round for open + # files; the default 3s idle timer would dominate every fixture setup. + await c.initialize( + workspace, initialization_options={"project": {"idle_timeout_ms": 10}} + ) uri, _ = await c.open_and_wait(workspace / "main.cpp") assert await wait_for_index(c, uri, "add"), "Index not ready" @@ -52,11 +56,11 @@ async def indexed_server(request, executable, workspace): from tests.integration.agentic.test_agentic import AgenticRpcClient rpc = AgenticRpcClient(host, port) - for _ in range(30): + for _ in range(300): resp = rpc.request("agentic/symbolSearch", {"query": "add"}) if "result" in resp and resp["result"]["symbols"]: break - await asyncio.sleep(1) + await asyncio.sleep(0.1) rpc.close() yield executable, host, port, workspace diff --git a/tests/integration/compilation/test_persistent_cache.py b/tests/integration/compilation/test_persistent_cache.py index de63a0244..526f43a75 100644 --- a/tests/integration/compilation/test_persistent_cache.py +++ b/tests/integration/compilation/test_persistent_cache.py @@ -1,7 +1,7 @@ """Integration tests for persistent PCH/PCM cache. Verifies that PCH/PCM artifacts are written to the unified cache store -(.clice/cache/v2/{pch,pcm}/) with content-addressed filenames, survive +(.clice/cache/v4/{pch,pcm}/) with content-addressed filenames, survive server restarts via cache.json, and are properly reused across sessions. """ @@ -267,12 +267,14 @@ async def test_no_tmp_files_after_build(client, tmp_path): uri, _ = await client.open_and_wait(tmp_path / "main.cpp") assert_clean_compile(client, uri) - # No in-flight tmp files should linger after the build settles. + # No in-flight tmp files should linger after the build settles. The + # pch namespace legitimately holds the paired .pch.idx blobs. assert list_tmp_files(tmp_path) == [], "Stale tmp files found" - for subdir in ("pch", "pcm"): + expected = {"pch": (".pch", ".pch.idx"), "pcm": (".pcm",)} + for subdir, extensions in expected.items(): blob_dir = cache_root(tmp_path) / subdir if blob_dir.exists(): - stray = [p for p in blob_dir.iterdir() if not p.name.endswith(f".{subdir}")] + stray = [p for p in blob_dir.iterdir() if not p.name.endswith(extensions)] assert stray == [], f"Stray files in {subdir}/: {stray}" @@ -377,8 +379,13 @@ async def test_kill9_recovery(executable, tmp_path): assert_clean_compile(c2, uri) pch_files = list_pch_files(tmp_path) assert len(pch_files) >= 1, "PCH should be (re)built after crash" - # Blob directories contain only committed blobs, never partial writes. - stray = [p for p in (cache_root(tmp_path) / "pch").iterdir() if p.suffix != ".pch"] + # Blob directories contain only committed blobs (the PCH and its + # paired index), never partial writes. + stray = [ + p + for p in (cache_root(tmp_path) / "pch").iterdir() + if not p.name.endswith((".pch", ".pch.idx")) + ] assert stray == [], f"Crash residue in pch/: {stray}" assert_no_anomaly(c2, tmp_path) await shutdown_client(c2) diff --git a/tests/integration/features/test_index.py b/tests/integration/features/test_index.py index 808d355eb..f2ae20211 100644 --- a/tests/integration/features/test_index.py +++ b/tests/integration/features/test_index.py @@ -17,7 +17,7 @@ ) from tests.tools.workspace import doc -from tests.tools.checks import wait_for_index +from tests.tools.checks import locations_of, wait_for_index @pytest.mark.workspace("index_features") @@ -249,14 +249,6 @@ async def test_workspace_symbol_class(client, workspace): client.close(uri) -def locations_of(result): - if result is None: - return [] - if isinstance(result, (list, tuple)): - return list(result) - return [result] - - @pytest.mark.workspace("index_features") async def test_goto_declaration_cross_file(client, workspace): # Query a closed file: background indexing skips open files, so nav.h's diff --git a/tests/integration/features/test_pch_overlay.py b/tests/integration/features/test_pch_overlay.py new file mode 100644 index 000000000..cae5c186e --- /dev/null +++ b/tests/integration/features/test_pch_overlay.py @@ -0,0 +1,223 @@ +"""Integration tests for the PCH overlay: header symbols of open in-memory +files resolve through the PCH's paired index blob, independent of the disk +index and faithful to the live buffer's preprocessor context.""" + +import asyncio + +from tests.tools.lifecycle import make_client, shutdown_client +from tests.tools.compile_commands import write_cdb +from tests.tools.checks import ( + MTIME_GRANULARITY, + locations_of, + wait_for_index, + wait_for_recompile, +) +from tests.tools.workspace import ( + cache_root, + did_change, + list_pch_files, + pin_cache_to_workspace, +) + +NO_INDEXING = {"project": {"enable_indexing": False}} + + +async def test_definition_into_unindexed_header(client, tmp_path): + (tmp_path / "foo.h").write_text("inline void foo() {}\n") + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + write_cdb(tmp_path, ["main.cpp"]) + await client.initialize(tmp_path, initialization_options=NO_INDEXING) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + # With background indexing off, only the PCH overlay knows the header. + locs = locations_of(await client.definition_at(uri, 1, 13)) + assert any( + loc.uri.endswith("foo.h") and loc.range.start.line == 0 for loc in locs + ), ( + f"expected definition in foo.h line 0, got {[(loc.uri, loc.range.start.line) for loc in locs]}" + ) + + +async def test_references_include_header_rows(client, tmp_path): + (tmp_path / "foo.h").write_text( + "inline void foo() {}\ninline void bar() { foo(); }\n" + ) + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + write_cdb(tmp_path, ["main.cpp"]) + await client.initialize(tmp_path, initialization_options=NO_INDEXING) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + refs = locations_of(await client.references_at(uri, 1, 13)) + assert any(r.uri.endswith("foo.h") and r.range.start.line == 1 for r in refs), ( + f"expected header-internal reference, got {[(r.uri, r.range.start.line) for r in refs]}" + ) + assert any(r.uri.endswith("main.cpp") for r in refs) + + +async def test_buffer_context_overrides_disk(client, tmp_path): + (tmp_path / "crypto.h").write_text( + "#ifdef USE_A\ninline void only_a() {}\n#else\ninline void only_b() {}\n#endif\n" + ) + disk_text = '#include "crypto.h"\nint main() { only_b(); return 0; }\n' + (tmp_path / "main.cpp").write_text(disk_text) + write_cdb(tmp_path, ["main.cpp"]) + await client.initialize(tmp_path) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + assert await wait_for_index(client, uri, "only_b"), "Index not ready after 30s" + + # The buffer's preamble now activates the branch no disk context has + # ever seen; only the rebuilt PCH's overlay can resolve only_a. + did_change( + client, + uri, + 2, + '#define USE_A 1\n#include "crypto.h"\nint main() { only_a(); return 0; }\n', + ) + await wait_for_recompile(client, uri) + + locs = locations_of(await client.definition_at(uri, 2, 13)) + assert any( + loc.uri.endswith("crypto.h") and loc.range.start.line == 1 for loc in locs + ), ( + f"expected only_a in crypto.h line 1, got {[(loc.uri, loc.range.start.line) for loc in locs]}" + ) + + +async def test_no_duplicate_reference_rows(client, tmp_path): + (tmp_path / "foo.h").write_text( + "inline void foo() {}\ninline void bar() { foo(); }\n" + ) + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + (tmp_path / "other.cpp").write_text('#include "foo.h"\nint other() { return 0; }\n') + write_cdb(tmp_path, ["main.cpp", "other.cpp"]) + await client.initialize(tmp_path) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + # Open files are skipped by background indexing; the closed other.cpp + # is what carries foo.h's rows into the disk index. + assert await wait_for_index(client, uri, "bar"), "Index not ready after 30s" + + # The header's rows exist in both its disk shard and the overlay; the + # union must collapse them. + refs = locations_of(await client.references_at(uri, 1, 13)) + keys = [(r.uri, r.range.start.line, r.range.start.character) for r in refs] + assert len(keys) == len(set(keys)), f"duplicate reference rows: {keys}" + assert any(key[0].endswith("foo.h") for key in keys) + + +async def test_preamble_macro_definition(client, tmp_path): + (tmp_path / "main.cpp").write_text( + "#define ANSWER 42\nint main() { return ANSWER; }\n" + ) + write_cdb(tmp_path, ["main.cpp"]) + await client.initialize(tmp_path, initialization_options=NO_INDEXING) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + # The #define lives in the preamble region swallowed by the PCH; its + # definition is served from the overlay's main-file entry. + locs = locations_of(await client.definition_at(uri, 1, 20)) + assert any( + loc.uri.endswith("main.cpp") and loc.range.start.line == 0 for loc in locs + ), ( + f"expected #define on line 0, got {[(loc.uri, loc.range.start.line) for loc in locs]}" + ) + + +async def test_preamble_links_survive_restart(executable, tmp_path): + pin_cache_to_workspace(tmp_path) + (tmp_path / "foo.h").write_text("inline void foo() {}\n") + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + write_cdb(tmp_path, ["main.cpp"]) + + c1 = await make_client(executable, tmp_path) + uri, _ = await c1.open_and_wait(tmp_path / "main.cpp") + links = await c1.document_links(uri) + assert any(link.target.endswith("foo.h") for link in links) + pch_mtime = list_pch_files(tmp_path)[0].stat().st_mtime + await shutdown_client(c1) + + # Session 2 hits the persisted PCH pair: the preamble's links must be + # served from the reloaded blob, not lost with the process. + c2 = await make_client(executable, tmp_path) + uri2, _ = await c2.open_and_wait(tmp_path / "main.cpp") + links2 = await c2.document_links(uri2) + assert any(link.target.endswith("foo.h") for link in links2), ( + "preamble document links lost across restart" + ) + assert list_pch_files(tmp_path)[0].stat().st_mtime == pch_mtime, ( + "PCH was rebuilt instead of reused" + ) + await shutdown_client(c2) + + +async def test_missing_idx_rebuilds_pair(executable, tmp_path): + pin_cache_to_workspace(tmp_path) + (tmp_path / "foo.h").write_text("inline void foo() {}\n") + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + write_cdb(tmp_path, ["main.cpp"]) + + c1 = await make_client(executable, tmp_path) + uri, _ = await c1.open_and_wait(tmp_path / "main.cpp") + locs = locations_of(await c1.definition_at(uri, 1, 13)) + assert any(loc.uri.endswith("foo.h") for loc in locs) + pch_mtime = list_pch_files(tmp_path)[0].stat().st_mtime + await shutdown_client(c1) + + # Half the pair vanishes (crash residue, external cleanup): the next + # session must treat the PCH as a miss and rebuild both blobs. + idx_files = list((cache_root(tmp_path) / "pch").glob("*.pch.idx")) + assert idx_files, "expected a committed .pch.idx next to the PCH" + for idx in idx_files: + idx.unlink() + await asyncio.sleep(MTIME_GRANULARITY) + + c2 = await make_client(executable, tmp_path) + uri2, _ = await c2.open_and_wait(tmp_path / "main.cpp") + locs2 = locations_of(await c2.definition_at(uri2, 1, 13)) + assert any(loc.uri.endswith("foo.h") for loc in locs2), ( + "overlay dead after losing the idx half of the pair" + ) + assert list_pch_files(tmp_path)[0].stat().st_mtime != pch_mtime, ( + "PCH pair should have been rebuilt" + ) + assert list((cache_root(tmp_path) / "pch").glob("*.pch.idx")), ( + "rebuilt pair is missing its idx blob" + ) + await shutdown_client(c2) + + +async def test_header_edit_refreshes_overlay(client, tmp_path): + (tmp_path / "foo.h").write_text("inline void foo() {}\n") + (tmp_path / "main.cpp").write_text( + '#include "foo.h"\nint main() { foo(); return 0; }\n' + ) + write_cdb(tmp_path, ["main.cpp"]) + await client.initialize(tmp_path, initialization_options=NO_INDEXING) + + uri, _ = await client.open_and_wait(tmp_path / "main.cpp") + locs = locations_of(await client.definition_at(uri, 1, 13)) + assert any(loc.range.start.line == 0 for loc in locs) + + # Same preamble text, so the PCH key is unchanged; the header edit must + # still refresh the pair (deps_changed) and the served overlay with it. + await asyncio.sleep(MTIME_GRANULARITY) + (tmp_path / "foo.h").write_text("// moved\ninline void foo() {}\n") + await wait_for_recompile(client, uri) + + locs = locations_of(await client.definition_at(uri, 1, 13)) + assert any( + loc.uri.endswith("foo.h") and loc.range.start.line == 1 for loc in locs + ), ( + f"overlay still serves pre-edit rows: {[(loc.uri, loc.range.start.line) for loc in locs]}" + ) diff --git a/tests/tools/checks.py b/tests/tools/checks.py index 384aef84c..faa365c87 100644 --- a/tests/tools/checks.py +++ b/tests/tools/checks.py @@ -19,6 +19,15 @@ IDLE_TIMEOUT = 5.0 # Idle soak time in lifecycle tests +def locations_of(result): + """Normalize a definition/references response to a list of Locations.""" + if result is None: + return [] + if isinstance(result, (list, tuple)): + return list(result) + return [result] + + async def wait_for_recompile(client, uri: str, *, timeout: float = 60.0) -> None: """Trigger recompilation via hover and wait for fresh diagnostics. @@ -115,6 +124,25 @@ def anomalies_in_log_files(workspace: Path | None) -> list[str]: return found +CRASH_TRACE_MARKER = "=== CRASH STACK TRACE ===" + + +def crash_traces_in_log_files(workspace: Path | None) -> list[str]: + """Crash stack traces recorded by crashed processes in their log files.""" + if workspace is None: + return [] + logs_dir = Path(workspace) / ".clice" / "logs" + if not logs_dir.exists(): + return [] + traces = [] + for log_file in sorted(logs_dir.rglob("*.log")): + text = log_file.read_text(errors="replace") + pos = text.find(CRASH_TRACE_MARKER) + if pos != -1: + traces.append(f"--- {log_file.name} ---\n{text[pos:]}") + return traces + + def assert_no_anomaly(client, workspace: Path | None = None) -> None: """Assert the session produced zero anomalies (client messages + logs). @@ -123,7 +151,11 @@ def assert_no_anomaly(client, workspace: Path | None = None) -> None: """ found = anomalies_in_log_messages(client) found += anomalies_in_log_files(workspace) - assert not found, f"clice reported internal anomalies: {found}" + # A crashed worker leaves its stack trace in its own log file; surface + # it here so a one-off CI crash is diagnosable from the test output. + traces = crash_traces_in_log_files(workspace) if found else [] + detail = "\n" + "\n".join(traces) if traces else "" + assert not found, f"clice reported internal anomalies: {found}{detail}" def guidance_messages(client) -> list[str]: diff --git a/tests/tools/workspace.py b/tests/tools/workspace.py index 9fb78a640..350b3d8e7 100644 --- a/tests/tools/workspace.py +++ b/tests/tools/workspace.py @@ -43,7 +43,7 @@ def get_field(obj, key, default=None): # Versioned root of the unified cache store; bump together with # cache_format_version in src/server/state/workspace.h. -CACHE_ROOT = Path(".clice") / "cache" / "v3" +CACHE_ROOT = Path(".clice") / "cache" / "v4" def cache_root(workspace: Path) -> Path: diff --git a/tests/unit/feature/document_link_tests.cpp b/tests/unit/feature/document_link_tests.cpp index 42d11b243..5af5f54fa 100644 --- a/tests/unit/feature/document_link_tests.cpp +++ b/tests/unit/feature/document_link_tests.cpp @@ -9,9 +9,6 @@ namespace clice::testing { namespace { -namespace lsp = kota::ipc::lsp; -namespace protocol = kota::ipc::protocol; - TEST_SUITE(document_link, Tester) { std::vector links; @@ -19,23 +16,15 @@ std::vector links; void run(llvm::StringRef source, llvm::StringRef standard = "-std=c++17") { add_files("main.cpp", source); ASSERT_TRUE(compile(standard)); - links = feature::document_links(*unit, feature::PositionEncoding::UTF8); -} - -auto to_local_range(const protocol::Range& range) -> LocalSourceRange { - auto content = unit->interested_content(); - auto line_starts = unit->line_starts(); - lsp::LineMap map(content, line_starts, feature::PositionEncoding::UTF8); - return LocalSourceRange(*map.to_offset(range.start), *map.to_offset(range.end)); + links = feature::document_links(*unit); } void EXPECT_LINK(std::size_t index, llvm::StringRef name, llvm::StringRef path) { auto& link = links[index]; auto expected = range(name, "main.cpp"); - auto actual = to_local_range(link.range); - ASSERT_EQ(actual.begin, expected.begin); - ASSERT_EQ(actual.end, expected.end); + ASSERT_EQ(link.range.begin, expected.begin); + ASSERT_EQ(link.range.end, expected.end); llvm::SmallString<128> target(link.target.begin(), link.target.end()); path::remove_dots(target); diff --git a/tests/unit/index/preamble_state_tests.cpp b/tests/unit/index/preamble_state_tests.cpp new file mode 100644 index 000000000..f21699a07 --- /dev/null +++ b/tests/unit/index/preamble_state_tests.cpp @@ -0,0 +1,248 @@ +#include "schema_generated.h" +#include "test/temp_dir.h" +#include "test/test.h" +#include "test/tester.h" +#include "index/preamble_state.h" + +#include "llvm/Support/raw_ostream.h" + +namespace clice::testing { + +namespace { + +TEST_SUITE(PreambleState, Tester) { + +index::TUIndex tu_index; +TempDir dir; +std::shared_ptr state; + +std::vector links; +std::vector inactive; +std::vector conditionals; + +/// Compile, build a full TUIndex, serialize a PreambleState blob to disk +/// and load it back. +void build_state(std::source_location location = std::source_location::current()) { + ASSERT_TRUE(compile()); + tu_index = index::TUIndex::build(*unit); + + links.resize(1); + links[0].range = {12, 20}; + links[0].target = "/include/foo.h"; + inactive = {4, 9, 30, 42}; + conditionals = {1, 0, 2}; + + auto blob_path = dir.path("state.pch.idx"); + std::error_code ec; + llvm::raw_fd_ostream os(blob_path, ec); + ASSERT_FALSE(bool(ec)); + index::PreambleState::serialize(*unit, tu_index, links, inactive, conditionals, os); + os.close(); + + state = index::PreambleState::load(blob_path); + ASSERT_TRUE(state != nullptr); +} + +index::SymbolHash hash_of(llvm::StringRef name, + std::source_location location = std::source_location::current()) { + index::SymbolHash hash = 0; + std::uint32_t count = 0; + for(auto& [symbol_id, symbol]: tu_index.symbols) { + if(symbol.name == name) { + hash = symbol_id; + count += 1; + } + } + EXPECT_EQ(count, 1); + return hash; +} + +TEST_CASE(ForcedIncludeServed) { + add_file("forced.h", R"(int @def[forced_value] = 1;)"); + add_main("main.cpp", R"(int x = forced_value;)"); + + // A compile-command forced include: clang records its include edge in + // the predefines buffer, which is a valid location — so unlike the + // synthetic buffers themselves, the file must stay in the blob under + // its own path. + prepare(); + owned_args.insert(owned_args.end() - 1, "-include"); + owned_args.insert(owned_args.end() - 1, TestVFS::path("forced.h")); + params.arguments.clear(); + for(auto& arg: owned_args) { + params.arguments.push_back(arg.c_str()); + } + ASSERT_TRUE(try_compile()); + tu_index = index::TUIndex::build(*unit); + + auto blob_path = dir.path("state.pch.idx"); + std::error_code ec; + llvm::raw_fd_ostream os(blob_path, ec); + ASSERT_FALSE(bool(ec)); + index::PreambleState::serialize(*unit, tu_index, {}, {}, {}, os); + os.close(); + + state = index::PreambleState::load(blob_path); + ASSERT_TRUE(state != nullptr); + + bool found = false; + state->lookup(hash_of("forced_value"), + RelationKind::Definition, + [&](const index::PreambleState::File& file, const index::Relation& r) { + EXPECT_TRUE(file.path.ends_with("forced.h")); + EXPECT_EQ(dump(r.range), dump(range("def", "forced.h"))); + found = true; + return false; + }); + EXPECT_TRUE(found); +} + +TEST_CASE(HeaderRelationLookup) { + add_file("foo.h", R"( +inline void @def[foo]() {} +inline void bar() { @href[foo](); } +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[foo](); return 0; } +)"); + build_state(); + + auto foo = hash_of("foo"); + + // The definition inside the header is served from the blob, together + // with everything needed to map it to an LSP location. + bool found_def = false; + state->lookup(foo, + RelationKind::Definition, + [&](const index::PreambleState::File& file, const index::Relation& r) { + EXPECT_TRUE(file.path.ends_with("foo.h")); + EXPECT_FALSE(file.content.empty()); + EXPECT_FALSE(file.line_starts.empty()); + EXPECT_EQ(dump(r.range), dump(range("def", "foo.h"))); + found_def = true; + return false; + }); + EXPECT_TRUE(found_def); + + // Header-internal references are in the blob too. + bool found_ref = false; + state->lookup(foo, + RelationKind::Reference, + [&](const index::PreambleState::File& file, const index::Relation& r) { + if(r.range == range("href", "foo.h")) { + found_ref = true; + return false; + } + return true; + }); + EXPECT_TRUE(found_ref); +} + +TEST_CASE(PreambleLookup) { + add_file("foo.h", R"( +inline void @def[foo]() {} +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[$(ref)foo](); return 0; } +)"); + build_state(); + + auto foo = hash_of("foo"); + + // Occurrence lookup by offset in the preamble entry. + bool found_occurrence = false; + state->lookup_preamble(point("ref"), [&](const index::Occurrence& occurrence) { + EXPECT_EQ(occurrence.target, foo); + EXPECT_EQ(dump(occurrence.range), dump(range("ref"))); + found_occurrence = true; + return false; + }); + EXPECT_TRUE(found_occurrence); + + // Relation lookup by symbol in the preamble entry. + bool found_relation = false; + state->lookup_preamble(foo, RelationKind::Reference, [&](const index::Relation& r) { + EXPECT_EQ(dump(r.range), dump(range("ref"))); + found_relation = true; + return false; + }); + EXPECT_TRUE(found_relation); +} + +TEST_CASE(SymbolTableLookup) { + add_file("foo.h", R"( +inline void @def[foo]() {} +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[foo](); return 0; } +)"); + build_state(); + + auto foo = hash_of("foo"); + + std::string name; + SymbolKind kind; + ASSERT_TRUE(state->find_symbol(foo, name, kind)); + EXPECT_EQ(name, "foo"); + EXPECT_EQ(kind.value(), SymbolKind(SymbolKind::Function).value()); + + EXPECT_FALSE(state->find_symbol(foo + 1, name, kind)); +} + +TEST_CASE(FeatureStateRoundtrip) { + add_main("main.cpp", R"( +int main() { return 0; } +)"); + build_state(); + + auto loaded_links = state->links(); + ASSERT_EQ(loaded_links.size(), 1); + EXPECT_EQ(loaded_links[0].range, LocalSourceRange(12, 20)); + EXPECT_EQ(loaded_links[0].target, "/include/foo.h"); + + EXPECT_EQ(state->inactive_regions(), llvm::ArrayRef(inactive)); + EXPECT_EQ(state->open_conditionals(), llvm::ArrayRef(conditionals)); + + // A blob with no header entries answers lookups with silence, not UB. + bool visited = false; + state->lookup(42, RelationKind::Reference, [&](auto&, auto&) { + visited = true; + return true; + }); + EXPECT_FALSE(visited); +} + +TEST_CASE(RejectBadBlob) { + EXPECT_TRUE(index::PreambleState::load(dir.path("missing.pch.idx")) == nullptr); + + dir.touch("garbage.pch.idx", "not a flatbuffer at all"); + EXPECT_TRUE(index::PreambleState::load(dir.path("garbage.pch.idx")) == nullptr); +} + +TEST_CASE(RejectVersionMismatch) { + // A structurally valid blob written by a different format version (0 is + // what a version-less blob reads back) must load as missing, so the + // PCH pair rebuilds instead of serving a stale layout. + flatbuffers::FlatBufferBuilder builder(64); + auto paths = builder.CreateVector(std::vector>{}); + auto files = + builder.CreateVector(std::vector>{}); + auto symbols = builder.CreateVector( + std::vector>{}); + builder.Finish(index::binary::CreatePreambleState(builder, 0, paths, files, 0, symbols)); + + auto blob_path = dir.path("stale.pch.idx"); + dir.touch("stale.pch.idx", + llvm::StringRef(reinterpret_cast(builder.GetBufferPointer()), + builder.GetSize())); + EXPECT_TRUE(index::PreambleState::load(blob_path) == nullptr); +} + +}; // TEST_SUITE(PreambleState) + +} // namespace + +} // namespace clice::testing diff --git a/tests/unit/server/compiler_tests.cpp b/tests/unit/server/compiler_tests.cpp index 5b87ec7fa..52afaa106 100644 --- a/tests/unit/server/compiler_tests.cpp +++ b/tests/unit/server/compiler_tests.cpp @@ -32,10 +32,10 @@ TEST_CASE(EpochGuardsPchWrite) { Session session; session.path_id = workspace.path_pool.intern("/proj/a.cpp"); - // No preamble directives: a current round would take the pch_ref reset + // No preamble directives: a current round would take the pch_key reset // branch; an invalidated continuation must not touch it. session.text = "int x;"; - session.pch_ref = Session::PCHRef{"key", 0}; + session.pch_key = "key"; auto gen = session.generation; auto epoch = session.dirty_epoch; @@ -60,8 +60,8 @@ TEST_CASE(EpochGuardsPchWrite) { EXPECT_FALSE(wrote); // The stale continuation left the session's PCH reference untouched. - ASSERT_TRUE(session.pch_ref.has_value()); - EXPECT_EQ(session.pch_ref->key, std::string("key")); + ASSERT_TRUE(session.pch_key.has_value()); + EXPECT_EQ(*session.pch_key, std::string("key")); } }; // TEST_SUITE(CompilerGuards) diff --git a/tests/unit/server/invalidator_tests.cpp b/tests/unit/server/invalidator_tests.cpp index a0cdf6170..c46d4642e 100644 --- a/tests/unit/server/invalidator_tests.cpp +++ b/tests/unit/server/invalidator_tests.cpp @@ -358,9 +358,10 @@ TEST_CASE(DiskChangeOpenMarksDirty) { // The buffer is the truth for an open file: recompile so the next // compile's deps validation judges the disk change, but no rescan and - // no cascade. + // no cascade. The file's shard describes the old disk, so its reindex + // queues alongside (skipped while open-file indexing is off). ASSERT_EQ(dirty.mark_ast_dirty, llvm::SmallVector{open_file}); - ASSERT_TRUE(dirty.reindex_content_changed.empty()); + ASSERT_EQ(dirty.reindex_content_changed, llvm::SmallVector{open_file}); ASSERT_TRUE(dirty.reindex_deps_only.empty()); ASSERT_TRUE(dirty.reset_trial.empty()); ASSERT_FALSE(dirty.recheck_contexts); @@ -529,16 +530,20 @@ TEST_CASE(CDBChangedSplitsOpenClosed) { // Flag changes recompile open files and reindex closed ones; the // pull-side cache keys (canonical flags) miss on their own. ASSERT_EQ(dirty.mark_ast_dirty, llvm::SmallVector{open_id}); - ASSERT_EQ(dirty.reindex_content_changed, llvm::SmallVector{closed_id}); + llvm::SmallVector reindexed{open_id, closed_id}; + llvm::sort(reindexed); + auto content_changed = dirty.reindex_content_changed; + llvm::sort(content_changed); + ASSERT_EQ(content_changed, reindexed); ASSERT_TRUE(dirty.reindex_deps_only.empty()); ASSERT_TRUE(dirty.recheck_contexts); - // The closed file's shard was built under the old command and looks - // fresh to content-only validation: it must be evicted so the queued - // reindex is not filtered out. The open file's shard stays (its next - // compile owns the refresh). + // Both shards were built under the old command and look fresh to + // content-only validation: evict them so the queued reindexes are not + // filtered out. The open file's slot is skipped while open-file + // indexing is off; its next compile owns the session-side refresh. ASSERT_EQ(workspace.merged_indices.count(closed_id), 0u); - ASSERT_EQ(workspace.merged_indices.count(open_id), 1u); + ASSERT_EQ(workspace.merged_indices.count(open_id), 0u); } TEST_CASE(CDBAddedOpenMarksDirty) { @@ -554,9 +559,10 @@ TEST_CASE(CDBAddedOpenMarksDirty) { auto dirty = invalidator.apply(FileEvent::cdb_changed(std::move(delta))); // The open file gained its first real entry: drop the guessed command - // it was compiled with instead of queueing a background reindex. + // it was compiled with, and queue the reindex that builds its shard + // under the real command once open-file indexing is on. ASSERT_EQ(dirty.mark_ast_dirty, llvm::SmallVector{file}); - ASSERT_TRUE(dirty.reindex_content_changed.empty()); + ASSERT_EQ(dirty.reindex_content_changed, llvm::SmallVector{file}); ASSERT_TRUE(dirty.reindex_deps_only.empty()); } diff --git a/tests/unit/server/pch_worker_tests.cpp b/tests/unit/server/pch_worker_tests.cpp index 80f61840c..8dd5ec364 100644 --- a/tests/unit/server/pch_worker_tests.cpp +++ b/tests/unit/server/pch_worker_tests.cpp @@ -1,7 +1,9 @@ +#include #include #include #include "test/test.h" +#include "index/preamble_state.h" #include "server/protocol/worker.h" #include "server/worker_test_helpers.h" #include "syntax/scan.h" @@ -50,7 +52,9 @@ TEST_CASE(BuildPCHThenCompile) { dir, main_file}; params.text = main_text; + params.preamble_bound = compute_preamble_bound(main_text); params.output_path = tmp.path("preamble.pch"); + params.index_output_path = tmp.path("preamble.pch.idx"); auto result = co_await sl.peer->send_request(params); CO_ASSERT_TRUE(result.has_value()); @@ -68,6 +72,15 @@ TEST_CASE(BuildPCHThenCompile) { // Verify the PCH file exists on disk. ASSERT_TRUE(llvm::sys::fs::exists(pch_path)); + // The worker wrote the paired PreambleState blob: it must load and + // carry the preamble's document links (the #include of common.h). + auto state = index::PreambleState::load(tmp.path("preamble.pch.idx")); + ASSERT_TRUE(state != nullptr); + bool has_common_link = std::ranges::any_of(state->links(), [&](auto& link) { + return llvm::StringRef(link.target).ends_with("common.h"); + }); + EXPECT_TRUE(has_common_link); + WorkerHandle sf; ASSERT_TRUE(sf.spawn(4ULL * 1024 * 1024 * 1024)); @@ -104,6 +117,51 @@ TEST_CASE(BuildPCHThenCompile) { std::remove(pch_path.c_str()); } +TEST_CASE(BlobWriteFailure) { + TempDir tmp; + + tmp.touch("common.h", R"cpp(struct Point { int x, y; };)cpp" "\n"); + std::string main_text = "#include \"common.h\"\nPoint p{1,2};\n"; + tmp.touch("main.cpp", main_text); + auto main_file = tmp.path("main.cpp"); + auto dir = std::string(tmp.root); + + WorkerHandle sl; + ASSERT_TRUE(sl.spawn()); + + bool done = false; + sl.run([&]() -> kota::task<> { + worker::BuildParams params; + params.kind = worker::BuildKind::BuildPCH; + params.file = main_file; + params.directory = dir; + params.arguments = {"clang++", + "-resource-dir", + std::string(resource_dir()), + "-x", + "c++-header", + "-I", + dir, + main_file}; + params.text = main_text; + params.preamble_bound = compute_preamble_bound(main_text); + params.output_path = tmp.path("preamble.pch"); + // Unwritable blob path: the whole build must fail (the PCH is only + // served together with its blob), classified as an internal error. + params.index_output_path = tmp.path("no_such_dir/preamble.pch.idx"); + + auto result = co_await sl.peer->send_request(params); + CO_ASSERT_TRUE(result.has_value()); + EXPECT_FALSE(result.value().success); + EXPECT_FALSE(result.value().has_user_errors); + + done = true; + sl.peer->close_output(); + }); + + ASSERT_TRUE(done); +} + TEST_CASE(CompileWithoutPCHStillWorks) { TempDir tmp; diff --git a/tests/unit/server/query_freshness_tests.cpp b/tests/unit/server/query_freshness_tests.cpp index 0c8ec1604..92d817348 100644 --- a/tests/unit/server/query_freshness_tests.cpp +++ b/tests/unit/server/query_freshness_tests.cpp @@ -26,6 +26,7 @@ WorkerPool pool{loop}; ContextResolver resolver{workspace}; Indexer indexer{loop, workspace, pool, resolver, store}; IndexQuery index_query{workspace, store, indexer}; +IndexQuery agent_query{workspace, store, indexer, {.disk_only = true}}; std::uint32_t main_id = 0; std::uint32_t header_id = 0; @@ -83,7 +84,7 @@ index::SymbolHash symbol_at(std::uint32_t path_id, std::uint32_t offset) { /// Files contributing reference rows for a symbol, by basename. std::vector reference_files(index::SymbolHash hash) { std::vector files; - for(auto& ref: index_query.collect_references(hash, RelationKind::Reference)) { + for(auto& ref: agent_query.collect_references(hash, RelationKind::Reference)) { files.push_back(llvm::sys::path::filename(ref.file).str()); } return files; @@ -138,7 +139,7 @@ TEST_CASE(PendingGateSplitsRows) { agentic::ReadSymbolParams by_line; by_line.path = std::string(workspace.path_pool.resolve(main_id)); by_line.line = 3; - ASSERT_FALSE(index_query.locate_symbols(by_line).empty()); + ASSERT_FALSE(agent_query.locate_symbols(by_line).empty()); // The file's own content changed: its contribution is skipped until the // reindex lands; other files' rows are unaffected. @@ -148,7 +149,7 @@ TEST_CASE(PendingGateSplitsRows) { // Cursor-style resolution against the stale rows is unresolvable: the // line numbers describe text that no longer exists. - ASSERT_TRUE(index_query.locate_symbols(by_line).empty()); + ASSERT_TRUE(agent_query.locate_symbols(by_line).empty()); // A content-changed definition file drops out of definition lookups. indexer.enqueue(header_id, ReindexReason::ContentChanged); diff --git a/tests/unit/server/query_overlay_tests.cpp b/tests/unit/server/query_overlay_tests.cpp new file mode 100644 index 000000000..bf4b28502 --- /dev/null +++ b/tests/unit/server/query_overlay_tests.cpp @@ -0,0 +1,464 @@ +#include +#include + +#include "test/temp_dir.h" +#include "test/test.h" +#include "test/tester.h" +#include "index/preamble_state.h" +#include "server/compiler/context_resolver.h" +#include "server/compiler/indexer.h" +#include "server/service/query.h" +#include "server/state/session_store.h" +#include "server/worker/worker_pool.h" + +#include "kota/ipc/lsp/text.h" +#include "llvm/Support/Path.h" +#include "llvm/Support/raw_ostream.h" + +namespace clice::testing { +namespace { + +TEST_SUITE(QueryOverlay, Tester) { + +kota::event_loop loop; +Workspace workspace; +SessionStore session_store; +WorkerPool pool{loop}; +ContextResolver resolver{workspace}; +Indexer indexer{loop, workspace, pool, resolver, session_store}; +IndexQuery index_query{workspace, session_store, indexer}; +IndexQuery agent_query{workspace, session_store, indexer, {.disk_only = true}}; + +TempDir dir; +index::TUIndex full_index; +std::shared_ptr session; +std::string main_path; + +/// Compile the added sources, serialize the full TUIndex as a +/// PreambleState blob (exactly what the PCH build produces), and open a +/// session whose pch_key points at it. The session's own file index is +/// the interested-only index, mirroring the production per-edit index. +void open_with_overlay(std::source_location location = std::source_location::current()) { + ASSERT_TRUE(compile()); + + full_index = index::TUIndex::build(*unit); + auto blob_path = dir.path("overlay.pch.idx"); + { + std::error_code ec; + llvm::raw_fd_ostream os(blob_path, ec); + ASSERT_FALSE(bool(ec)); + index::PreambleState::serialize(*unit, full_index, {}, {}, {}, os); + } + + auto& st = workspace.pch_cache["key"]; + st.path = "unused.pch"; + st.index_path = blob_path; + st.state = nullptr; + + main_path = full_index.graph.paths.back(); + auto path_id = workspace.path_pool.intern(main_path); + session = session_store.open(path_id); + + auto it = sources.all_files.find(llvm::sys::path::filename(main_path)); + ASSERT_TRUE(it != sources.all_files.end()); + session->text = it->second.content; + session->line_starts = kota::ipc::lsp::build_line_starts(session->text); + + auto session_index = index::TUIndex::build(*unit, true); + session->file_index = std::move(session_index.main_file_index); + session->symbols = std::move(session_index.symbols); + session->ast_dirty = false; + session->pch_key = "key"; +} + +index::SymbolHash hash_of(llvm::StringRef name, + std::source_location location = std::source_location::current()) { + index::SymbolHash hash = 0; + std::uint32_t count = 0; + for(auto& [symbol_id, symbol]: full_index.symbols) { + if(symbol.name == name) { + hash = symbol_id; + count += 1; + } + } + EXPECT_EQ(count, 1); + return hash; +} + +std::string header_path(llvm::StringRef basename) { + for(auto& path: full_index.graph.paths) { + if(llvm::sys::path::filename(path) == basename) + return path; + } + return {}; +} + +/// Merge the full TUIndex into the workspace's disk index with real +/// contents, as background indexing would. +void merge_disk_index() { + auto file_ids_map = workspace.project_index.merge(full_index, workspace.path_pool); + + auto content_of = [&](llvm::StringRef path) -> llvm::StringRef { + auto it = sources.all_files.find(llvm::sys::path::filename(path)); + return it != sources.all_files.end() ? llvm::StringRef(it->second.content) + : llvm::StringRef(); + }; + + auto main_tu_path_id = static_cast(full_index.graph.paths.size() - 1); + llvm::StringRef main_tu_path = full_index.graph.paths[main_tu_path_id]; + + llvm::SmallVector deps; + for(auto& loc: full_index.graph.locations) { + deps.push_back({full_index.graph.paths[loc.path_id], loc.line, loc.include}); + } + workspace.merged_indices[file_ids_map[main_tu_path_id]].merge(main_tu_path, + full_index.built_at, + deps, + full_index.main_file_index, + content_of(main_tu_path)); + + for(auto& [fid, file_idx]: full_index.file_indices) { + auto tu_pid = full_index.graph.path_id(fid); + workspace.merged_indices[file_ids_map[tu_pid]].merge( + main_tu_path, + full_index.graph.include_location_id(fid), + file_idx, + content_of(full_index.graph.paths[tu_pid])); + } +} + +void setup() { + // skip_stale_contribution dereferences this optional; a default + // Workspace leaves it empty (apply_defaults never runs in tests). + workspace.config.project.enable_indexing = true; +} + +protocol::Position position_of(llvm::StringRef name) { + auto pos = session->line_map().to_position(point(name)); + return pos ? *pos : protocol::Position{}; +} + +TEST_CASE(DefinitionFromOverlayOnly) { + add_file("foo.h", R"( +inline void @def[foo]() {} +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[$(ref)foo](); return 0; } +)"); + open_with_overlay(); + + // No disk index at all — the in-memory-file case: the overlay is the + // only source that knows where foo is defined. + auto locations = index_query.query_relations(main_path, + position_of("ref"), + RelationKind::Definition, + session.get()); + ASSERT_EQ(locations.size(), 1); + EXPECT_TRUE(llvm::StringRef(locations[0].uri).ends_with("foo.h")); +} + +TEST_CASE(ReferencesUnionWithDedup) { + add_file("foo.h", R"( +inline void @def[foo]() {} +inline void bar() { @href[$(href)foo](); } +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[$(ref)foo](); return 0; } +)"); + open_with_overlay(); + // The header's disk shard and the overlay now both carry the + // header-internal reference; results must contain it exactly once. + merge_disk_index(); + + auto locations = index_query.query_relations(main_path, + position_of("ref"), + RelationKind::Reference, + session.get()); + ASSERT_EQ(locations.size(), 2); + + std::size_t header_rows = 0; + std::size_t main_rows = 0; + for(auto& location: locations) { + if(llvm::StringRef(location.uri).ends_with("foo.h")) + header_rows += 1; + if(llvm::StringRef(location.uri).ends_with("main.cpp")) + main_rows += 1; + } + EXPECT_EQ(header_rows, 1); + EXPECT_EQ(main_rows, 1); +} + +TEST_CASE(PreambleMacroCursor) { + add_main("main.cpp", R"(#define @macro[$(macro)FOO] 1 +int main() { return 0; } +)"); + open_with_overlay(); + + // Production per-edit indexes never see the preamble region (the PCH + // swallows it); emulate that by emptying the session's own index so + // the cursor can only resolve through the overlay's main-file entry. + session->file_index = index::FileIndex(); + session->symbols = index::SymbolTable(); + + auto uri = std::string("file://") + main_path; + auto info = index_query.lookup_symbol(uri, main_path, position_of("macro"), session.get()); + ASSERT_TRUE(info.has_value()); + EXPECT_EQ(info->name, "FOO"); +} + +TEST_CASE(OverlaySymbolInfo) { + add_file("foo.h", R"( +inline void @def[foo]() {} +inline void @bardef[bar]() { foo(); } +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[foo](); return 0; } +)"); + open_with_overlay(); + + // bar is never referenced by the buffer, so neither the session's + // symbol table nor the (empty) project index knows it — only the + // overlay's symbol table does. + auto bar_hash = hash_of("bar"); + + std::string name; + SymbolKind kind; + ASSERT_TRUE(index_query.find_symbol_info(bar_hash, name, kind)); + EXPECT_EQ(name, "bar"); + + auto def_loc = index_query.find_definition_location(bar_hash); + ASSERT_TRUE(def_loc.has_value()); + EXPECT_TRUE(llvm::StringRef(def_loc->uri).ends_with("foo.h")); +} + +TEST_CASE(OpenHeaderExcluded) { + add_file("foo.h", R"( +inline void @def[foo]() {} +inline void bar() { @href[foo](); } +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[$(ref)foo](); return 0; } +)"); + open_with_overlay(); + + // Opening the header makes its session authoritative: overlay rows + // for it describe the disk snapshot and would map onto the edited + // buffer at wrong lines, so they must vanish from results. + session_store.open(workspace.path_pool.intern(header_path("foo.h"))); + + auto locations = index_query.query_relations(main_path, + position_of("ref"), + RelationKind::Reference, + session.get()); + ASSERT_EQ(locations.size(), 1); + EXPECT_TRUE(llvm::StringRef(locations[0].uri).ends_with("main.cpp")); +} + +TEST_CASE(IncomingCallsDedup) { + add_file("foo.h", R"( +inline void @def[callee]() {} +inline void caller() { @call[callee](); } +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @mcall[$(mcall)callee](); return 0; } +)"); + open_with_overlay(); + // The header call site now exists in both its disk shard and the + // overlay; each caller must report it exactly once. + merge_disk_index(); + + auto calls = index_query.find_incoming_calls(hash_of("callee")); + ASSERT_EQ(calls.size(), 2); + for(auto& call: calls) { + EXPECT_EQ(call.from_ranges.size(), 1); + } +} + +TEST_CASE(OpenHeaderTargetsExcluded) { + add_file("base.h", R"( +struct @b[Base] {}; +)"); + add_file("derived.h", R"( +#include "base.h" +struct @d[Derived] : Base {}; +)"); + add_main("main.cpp", R"( +#include "derived.h" +Derived instance; +)"); + open_with_overlay(); + + auto derived = hash_of("Derived"); + auto supertypes = index_query.find_supertypes(derived); + ASSERT_EQ(supertypes.size(), 1); + EXPECT_EQ(supertypes[0].name, "Base"); + + // Once derived.h is open, its session owns the type relations spelled + // there; the overlay's disk-snapshot rows must stop contributing. + session_store.open(workspace.path_pool.intern(header_path("derived.h"))); + supertypes = index_query.find_supertypes(derived); + EXPECT_EQ(supertypes.size(), 0); +} + +TEST_CASE(StaleHeaderSuppressed) { + add_file("foo.h", R"( +inline void @def[foo]() {} +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[foo](); return 0; } +)"); + open_with_overlay(); + + ASSERT_TRUE(index_query.find_definition_location(hash_of("foo")).has_value()); + + // The header's own disk content changed and awaits reindexing: its + // overlay rows describe text that no longer exists (freshness + // contract, clause 2), exactly like a shard contribution. + indexer.enqueue(workspace.path_pool.intern(header_path("foo.h")), + ReindexReason::ContentChanged); + EXPECT_FALSE(index_query.find_definition_location(hash_of("foo")).has_value()); +} + +TEST_CASE(MacroDefinitionText) { + add_main("main.cpp", R"(#define @macro[FOO] 1 +int main() { return 0; } +)"); + ASSERT_TRUE(compile()); + full_index = index::TUIndex::build(*unit); + merge_disk_index(); + + // Macro Definition relations carry the full #define extent, so the + // agentic text path works for macros through the disk index. + auto text = agent_query.get_definition_text(hash_of("FOO")); + ASSERT_TRUE(text.has_value()); + EXPECT_TRUE(llvm::StringRef(text->text).contains("FOO")); +} + +TEST_CASE(SharedPreambleScoped) { + add_main("main.cpp", R"(#define @macro[$(macro)FOO] 1 +#if FOO +#endif +int main() { return 0; } +)"); + open_with_overlay(); + session->file_index = index::FileIndex(); + session->symbols = index::SymbolTable(); + + // A second file with a byte-identical preamble shares the PCH (the + // key excludes the source path), but the preamble entry carries + // file-local macro identities — its rows must stay scoped to the + // file that built the blob. + auto other_path = std::string(llvm::sys::path::parent_path(main_path)) + "/other.cpp"; + auto other = session_store.open(workspace.path_pool.intern(other_path)); + other->text = session->text; + other->line_starts = session->line_starts; + other->ast_dirty = false; + other->pch_key = session->pch_key; + + auto locations = index_query.query_relations(main_path, + position_of("macro"), + RelationKind::Reference, + session.get()); + ASSERT_EQ(locations.size(), 1); + EXPECT_TRUE(llvm::StringRef(locations[0].uri).ends_with("main.cpp")); +} + +TEST_CASE(DirtyPreambleServed) { + add_main("main.cpp", R"(#define @macro[FOO] 1 +int main() { return 0; } +)"); + open_with_overlay(); + session->file_index = index::FileIndex(); + session->symbols = index::SymbolTable(); + + // Body edits dirty the session but never move preamble rows: as long + // as the buffer still starts with the blob's preamble text, the + // entry keeps serving — the prefix comparison is the freshness check. + session->ast_dirty = true; + session->text += "int more;\n"; + session->line_starts = kota::ipc::lsp::build_line_starts(session->text); + EXPECT_TRUE(index_query.find_definition_location(hash_of("FOO")).has_value()); +} + +TEST_CASE(PreambleDriftSkipped) { + add_main("main.cpp", R"(#define @macro[FOO] 1 +int main() { return 0; } +)"); + open_with_overlay(); + session->file_index = index::FileIndex(); + session->symbols = index::SymbolTable(); + + // A deferred PCH rebuild keeps an old blob while the buffer's + // preamble moved on; once the buffer no longer starts with the blob's + // stored preamble text, its rows must not be served. + session->text = "// drift\n" + session->text; + session->line_starts = kota::ipc::lsp::build_line_starts(session->text); + EXPECT_FALSE(index_query.find_definition_location(hash_of("FOO")).has_value()); +} + +TEST_CASE(OverlayOutranksDisk) { + add_file("foo.h", R"( +inline void @def[foo]() {} +)"); + add_main("main.cpp", R"( +#include "foo.h" +int main() { @ref[foo](); return 0; } +)"); + open_with_overlay(); + + // Fabricate a divergent disk row: another context's shard claims the + // definition sits on line 0. The overlay (live context) must win. + auto foo = hash_of("foo"); + index::FileIndex fake; + index::Relation relation{ + .kind = RelationKind::Definition, + .range = {0, 3} + }; + relation.set_definition_range({0, 3}); + fake.relations[foo].push_back(relation); + auto header_id = workspace.path_pool.intern(header_path("foo.h")); + workspace.merged_indices[header_id].merge("other_tu", 0, fake, "xxx\n"); + workspace.project_index.symbols[foo].reference_files.add(header_id); + + auto def_loc = index_query.find_definition_location(foo); + ASSERT_TRUE(def_loc.has_value()); + EXPECT_EQ(def_loc->range.start.line, 1); +} + +TEST_CASE(SynthesizedArtifactSkipped) { + workspace.config.project.cache_dir = TestVFS::root(); + add_file("header_context/gen.h", R"( +inline void @def[gen]() {} +)"); + add_main("main.cpp", R"( +#include "header_context/gen.h" +int main() { @ref[gen](); return 0; } +)"); + open_with_overlay(); + + // The header lives inside the synthesized-artifact directory: its + // overlay rows must never send the user into the cache. + EXPECT_FALSE(index_query.find_definition_location(hash_of("gen")).has_value()); +} + +TEST_CASE(UnreadableBlobCleared) { + dir.touch("junk.pch.idx", "not a flatbuffer"); + + PCHState st; + st.index_path = dir.path("junk.pch.idx"); + EXPECT_TRUE(st.load_state() == nullptr); + // The cleared path makes the pair look incomplete, so the next + // ensure_pch round rebuilds it instead of retrying the mmap forever. + EXPECT_TRUE(st.index_path.empty()); +} + +}; // TEST_SUITE(QueryOverlay) + +} // namespace +} // namespace clice::testing diff --git a/tests/unit/server/session_store_tests.cpp b/tests/unit/server/session_store_tests.cpp index e247ba81a..8f3ee7036 100644 --- a/tests/unit/server/session_store_tests.cpp +++ b/tests/unit/server/session_store_tests.cpp @@ -126,7 +126,7 @@ TEST_CASE(ResetSupersededBumpsGeneration) { store.apply_open(*session, "int x;", 1); session->ast_dirty = false; session->trial_done = true; - session->pch_ref = Session::PCHRef{"key", 4}; + session->pch_key = "key"; session->ast_deps.emplace(); auto gen = session->generation; auto epoch = session->dirty_epoch; @@ -135,7 +135,7 @@ TEST_CASE(ResetSupersededBumpsGeneration) { ASSERT_TRUE(session->ast_dirty); ASSERT_FALSE(session->trial_done); - ASSERT_FALSE(session->pch_ref.has_value()); + ASSERT_FALSE(session->pch_key.has_value()); ASSERT_FALSE(session->ast_deps.has_value()); ASSERT_EQ(session->generation, gen + 1); ASSERT_EQ(session->dirty_epoch, epoch); @@ -147,7 +147,7 @@ TEST_CASE(ResetLostBumpsEpoch) { store.apply_open(*session, "int x;", 1); session->ast_dirty = false; session->trial_done = true; - session->pch_ref = Session::PCHRef{"key", 4}; + session->pch_key = "key"; auto gen = session->generation; auto epoch = session->dirty_epoch; @@ -157,7 +157,7 @@ TEST_CASE(ResetLostBumpsEpoch) { // only the freshness claim is revoked. ASSERT_TRUE(session->ast_dirty); ASSERT_TRUE(session->trial_done); - ASSERT_TRUE(session->pch_ref.has_value()); + ASSERT_TRUE(session->pch_key.has_value()); ASSERT_EQ(session->generation, gen); ASSERT_EQ(session->dirty_epoch, epoch + 1); } diff --git a/tests/unit/server/stateless_worker_tests.cpp b/tests/unit/server/stateless_worker_tests.cpp index fc060e4dc..0083b1533 100644 --- a/tests/unit/server/stateless_worker_tests.cpp +++ b/tests/unit/server/stateless_worker_tests.cpp @@ -99,6 +99,8 @@ TEST_CASE(BuildPCHRequest) { {"clang++", "-resource-dir", std::string(resource_dir()), "-x", "c++-header", hdr}; params.text = "#pragma once\nint pch_global = 42;\n"; params.output_path = tmp.path("test_pch.pch"); + // The pair is mandatory: a PCH is only served with its blob. + params.index_output_path = tmp.path("test_pch.pch.idx"); auto result = co_await w.peer->send_request(params); EXPECT_TRUE(result.has_value()); diff --git a/tests/unit/support/cache_store_tests.cpp b/tests/unit/support/cache_store_tests.cpp index f2ad33915..ea5f92510 100644 --- a/tests/unit/support/cache_store_tests.cpp +++ b/tests/unit/support/cache_store_tests.cpp @@ -44,6 +44,25 @@ void register_lru(CacheStore& store, std::uint64_t max_bytes = 0) { {.name = "pch", .extension = ".pch", .policy = CachePolicy::LRU, .max_bytes = max_bytes}); } +void register_paired(CacheStore& store, std::uint64_t max_bytes = 0) { + store.register_namespace({.name = "pch", + .extension = ".pch", + .aux_extension = ".pch.idx", + .policy = CachePolicy::LRU, + .max_bytes = max_bytes}); +} + +/// Run a full two-phase aux write with the given content. +std::string + put_aux(CacheStore& store, llvm::StringRef ns, llvm::StringRef key, llvm::StringRef content) { + auto pending = store.begin_store_aux(ns, key); + require(!pending.tmp_path.empty(), "begin_store_aux returned no tmp path"); + require(fs::write(pending.tmp_path, content).has_value(), "tmp write failed"); + auto committed = store.commit(std::move(pending)); + require(committed.has_value(), "aux commit failed"); + return *committed; +} + /// Run a full two-phase write with the given content. std::string put(CacheStore& store, llvm::StringRef ns, llvm::StringRef key, llvm::StringRef content) { @@ -449,6 +468,157 @@ TEST_CASE(CheckpointAutoTriggers) { ASSERT_TRUE(llvm::StringRef(*manifest).contains("k0")); } +TEST_CASE(PairStoreAndLookup) { + TempDir tmp; + auto store = open_store(tmp); + register_paired(store); + + // A primary alone is an incomplete pair: lookup serves it, lookup_aux + // must miss. + put(store, "pch", "k1", "primary blob"); + ASSERT_TRUE(store.lookup("pch", "k1").has_value()); + ASSERT_FALSE(store.lookup_aux("pch", "k1").has_value()); + + auto aux_path = put_aux(store, "pch", "k1", "aux blob"); + ASSERT_TRUE(llvm::StringRef(aux_path).ends_with(".pch.idx")); + + auto hit = store.lookup_aux("pch", "k1"); + ASSERT_TRUE(hit.has_value()); + ASSERT_EQ(fs::read(*hit).value_or(""), "aux blob"); +} + +TEST_CASE(AuxWithoutPrimaryFails) { + TempDir tmp; + auto store = open_store(tmp); + register_paired(store); + + auto pending = store.begin_store_aux("pch", "ghost"); + require(fs::write(pending.tmp_path, "orphan").has_value(), "tmp write failed"); + ASSERT_FALSE(store.commit(std::move(pending)).has_value()); + ASSERT_FALSE(store.lookup_aux("pch", "ghost").has_value()); +} + +TEST_CASE(PrimaryRecommitResetsAux) { + TempDir tmp; + auto store = open_store(tmp); + register_paired(store); + + put(store, "pch", "k1", "old primary"); + auto aux_path = put_aux(store, "pch", "k1", "old aux"); + + // Republishing the primary must drop the stale aux: yesterday's aux + // next to today's primary would be a silent mismatch. + put(store, "pch", "k1", "new primary"); + ASSERT_FALSE(store.lookup_aux("pch", "k1").has_value()); + ASSERT_FALSE(fs::read(aux_path).has_value()); + + put_aux(store, "pch", "k1", "new aux"); + ASSERT_TRUE(store.lookup_aux("pch", "k1").has_value()); +} + +TEST_CASE(PairEvictedTogether) { + TempDir tmp; + auto store = open_store(tmp); + register_paired(store, 25); + + put(store, "pch", "k1", "aaaaaaaaaa"); + auto aux_path = put_aux(store, "pch", "k1", "aaaaaaaaaa"); + + // The pair counts as one 20-byte entry; the next 10-byte pair pushes + // the total over budget and k1 must vanish whole — both files. + put(store, "pch", "k2", "bbbbb"); + put_aux(store, "pch", "k2", "bbbbb"); + + ASSERT_FALSE(store.lookup("pch", "k1").has_value()); + ASSERT_FALSE(store.lookup_aux("pch", "k1").has_value()); + ASSERT_FALSE(fs::read(aux_path).has_value()); + ASSERT_TRUE(store.lookup("pch", "k2").has_value()); + ASSERT_TRUE(store.lookup_aux("pch", "k2").has_value()); +} + +TEST_CASE(PairSurvivesReopen) { + TempDir tmp; + { + auto store = open_store(tmp); + register_paired(store); + put(store, "pch", "k1", "primary"); + put_aux(store, "pch", "k1", "aux"); + store.shutdown(); + } + + auto store = open_store(tmp); + register_paired(store); + ASSERT_TRUE(store.lookup("pch", "k1").has_value()); + auto hit = store.lookup_aux("pch", "k1"); + ASSERT_TRUE(hit.has_value()); + ASSERT_EQ(fs::read(*hit).value_or(""), "aux"); +} + +TEST_CASE(StaleAuxDropped) { + TempDir tmp; + { + auto store = open_store(tmp); + register_paired(store); + put(store, "pch", "k1", "primary"); + put_aux(store, "pch", "k1", "aux"); + store.shutdown(); + } + + // Residue of an aux removal that failed while the file was held open + // (see reset_aux_locked): the primary was republished, the old aux + // stayed behind. Pairs commit primary-first, so a legitimate aux is + // never older than its primary — registration must not adopt this one. + // int-FD flavor: setLastAccessAndModificationTime has no overload for + // the native handle type on Windows. + auto aux_path = tmp.path("root/cache/v1/pch/k1.pch.idx"); + int fd = 0; + ASSERT_FALSE(bool(llvm::sys::fs::openFileForWrite(aux_path, + fd, + llvm::sys::fs::CD_OpenExisting, + llvm::sys::fs::OF_None))); + auto old_time = std::chrono::system_clock::now() - std::chrono::hours(1); + ASSERT_FALSE(bool(llvm::sys::fs::setLastAccessAndModificationTime(fd, old_time, old_time))); + llvm::sys::Process::SafelyCloseFileDescriptor(fd); + + auto store = open_store(tmp); + register_paired(store); + ASSERT_TRUE(store.lookup("pch", "k1").has_value()); + ASSERT_FALSE(store.lookup_aux("pch", "k1").has_value()); + ASSERT_FALSE(fs::read(aux_path).has_value()); +} + +TEST_CASE(OrphanAuxSwept) { + TempDir tmp; + { + auto store = open_store(tmp); + register_paired(store); + store.shutdown(); + } + + // Crash residue: an aux blob whose primary is gone. Registration must + // remove it — nothing can ever reference it again. + tmp.touch("root/cache/v1/pch/ghost.pch.idx", "orphan"); + + auto store = open_store(tmp); + register_paired(store); + ASSERT_FALSE(store.lookup_aux("pch", "ghost").has_value()); + ASSERT_FALSE(fs::read(tmp.path("root/cache/v1/pch/ghost.pch.idx")).has_value()); +} + +TEST_CASE(InvalidateRemovesPair) { + TempDir tmp; + auto store = open_store(tmp); + register_paired(store); + + put(store, "pch", "k1", "primary"); + auto aux_path = put_aux(store, "pch", "k1", "aux"); + + store.invalidate("pch", "k1"); + ASSERT_FALSE(store.lookup("pch", "k1").has_value()); + ASSERT_FALSE(store.lookup_aux("pch", "k1").has_value()); + ASSERT_FALSE(fs::read(aux_path).has_value()); +} + }; // TEST_SUITE(CacheStore) } // namespace