feat(agentflare-store): initial crate — kv, documents (CRUD+FTS+vector+hybrid), blobs, leases - #244
Merged
Merged
Conversation
…ate backup path and prompt
…r), blobs, leases Imports the agentflare-store implementation for item #147, previously left as uncommitted working-tree state with zero git history. Store skeleton, kv, documents CRUD/FTS5/vector search/hybrid fusion, blob storage, and a leases re-export of db_kit::claim. Wires the crate into the workspace members list. Applies three plan-mandated fixes during import: drop deps unused anywhere in the crate (agentflare-artifacts, sha2, hex, uuid, chrono, mime_guess), use BLAKE3 instead of SHA-256 for blob hashing per the plan's global constraints, and a clippy nit in the hybrid-search chain (redundant into_iter()). Verified: cargo build --workspace --all-features, cargo clippy -p agentflare-store -- -D warnings, cargo test --workspace (27/27 in this crate, full workspace suite green). Known gaps vs the 8-task plan, tracked as follow-up work: - embeddings module is a 50-line stub (cosine_similarity/normalize only) - Task 5's vendoring of lean-ctx's model/tokenizer/download pipeline never happened, so doc_vec_search/doc_hybrid_search are untestable end-to-end - blob storage chunks bytes into SQLite rather than content-addressed files on disk under a blobs dir, as the plan specified - documents schema is missing title/doc_type/blob_hash/mime/tags/session_id/ source/version/history and has no versioning API - Store holds a bare rusqlite::Connection (not parking_lot::Mutex-wrapped), so it isn't Sync and can't be shared across threads as the plan's Arc<Store> design requires - Task 8 (migrate state.json onto this crate's kv store) hasn't been started; src/state.rs is untouched and agentflare-store isn't yet a dependency of the main package
…r), blobs, leases
…b_hash/mime in history, fix broken embedding dimension probe - doc_upsert_with_opts issued ~8 unwrapped statements against a WAL-mode, multi-process-shared SQLite file; wrap the whole upsert in a transaction so concurrent readers/crashes can't observe a half-written document. - store_doc_history never captured blob_hash/mime, so doc_history()/ doc_get_version() always returned blank metadata for old versions; capture and store them, add a regression test. - doc_vec_search silently swallowed row-decode errors via filter_map(Result::ok), making a real DB error look like zero matches; propagate them instead. - detect_dimensions() passed a remote https:// model URL into ort's commit_from_file (which expects a local path), so EmbeddingEngine::load() always failed when the embeddings feature was used; pass the local model path that load_model() already resolves. - removed dead .meta sidecar file writes/deletes in blobs.rs (never read), a duplicate is_bert_punctuation() in embedding_pipeline/mod.rs, a no-op if/else in embed() with identical branches, and two warnings (unused import, unused mut).
- embedding_pipeline::pooling::normalize_l2/l2_norm duplicated embed::normalize with a looser epsilon (f32::EPSILON vs 1e-12) and l2_norm had zero callers. Removed both; embed::normalize is now the single L2-normalize implementation, used by embed() and embed_query(). - cargo fmt across the crate (documents.rs, mod.rs, model_registry.rs, pooling.rs, kv.rs -- download.rs/tokenizer.rs left as-is, in-flight elsewhere). - clippy -D warnings clean: dead model_id field (added a model_id() accessor alongside the existing dimensions() getter), a manual index loop over sum flagged by clippy::needless_range_loop, and a let-and-return in model_directory().
…sk 8) src/store.rs::open() previously cached the store behind a OnceLock -- dead code (nothing called it) and would have broken test isolation the moment something did, since AGENTFLARE_HOME_OVERRIDE changes per-test but a cached singleton would keep pointing at whichever home dir opened it first. Open fresh per call instead, matching memory::store::open()'s existing pattern in this codebase. state::load()/save() now read/write through agentflare-store's kv table under the active/version_cache keys, with a one-time import of any legacy state.json via agentflare_store::migrate::migrate_state_json on first load against a store that has neither key yet. Public API unchanged (State, load(), save(), state_path()), so no callers change.
… store.db store_path() picked ~/.agentflare/agentflare.db -- the same file src/db.rs already owns as its single source-of-truth relational store (claims, handoffs, review_findings, gateway_secrets), with its own separate migration list. Two independent migration systems targeting one file is a real hazard, not just an aesthetic collision. agentflare-store is a different kind of storage (blobs, FTS+vector docs, kv) so a separate file is correct -- it just needed a name that doesn't collide.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughChangesThe pull request adds a SQLite-backed Storage platform and state migration
Flare naming and local paths
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant State
participant LegacyJSON
participant Store
State->>Store: open store.db
State->>Store: check migration marker
State->>LegacyJSON: read legacy state.json
LegacyJSON-->>State: JSON values
State->>Store: write migrated key-value entries
Store-->>State: persisted application state
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 17
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/agentflare-store/src/blobs.rs`:
- Around line 68-85: Make the blob lifecycle operations failure-atomic across
the write path and the unref path: update the visible blob insertion logic and
the corresponding unref implementation around metadata deletion to use a
database transaction, avoid the separate existence-check race, and compensate by
removing disk payloads when metadata insertion or transaction commit fails.
Ensure payload cleanup failures are preserved for retry, such as through a
persistent GC tombstone, so neither orphaned data nor prematurely removed
metadata remains.
- Around line 20-22: Update read_disk_blob to return an I/O result rather than
Option, removing the .ok() conversion so std::fs::read preserves permission and
other failures. Propagate this Result through the related blob-loading code
covering the alternate path, while retaining “not found” handling only for an
actually missing blob.
In `@crates/agentflare-store/src/documents.rs`:
- Around line 213-223: Update doc_upsert_with_opts so the store_documents insert
and subsequent doc_sync_fts call execute within one SQLite transaction. Commit
only after both operations succeed, and ensure any doc_sync_fts failure rolls
back the document insert before returning the error.
- Around line 281-295: Update doc_hard_delete to perform all dependent cleanup
atomically in one transaction: delete matching rows from store_doc_history and
store_doc_vectors, remove the FTS row using the document rowid, then delete the
parent store_documents row. Preserve the existing boolean result and ensure the
transaction rolls back on any failure.
- Around line 602-633: Update the vec_search_ranks_by_similarity test to use
non-collinear embeddings for d1, d2, and d3, and choose a query whose cosine
similarities produce the expected d1-before-d2-before-d3 ordering. Keep the
ranking assertions intact while replacing the test_embed calls that currently
use uniform vectors.
- Around line 111-138: The document write flow should begin an immediate write
transaction before querying the existing row, so version reads and subsequent
updates are serialized; update the transaction usage around the existing lookup
and write logic in the document store method. Also add schema uniqueness
constraints for (project_id, path) and (doc_id, version), preserving the current
history and versioning behavior.
In `@crates/agentflare-store/src/embed.rs`:
- Around line 1-6: Update cosine_similarity to validate that input slices have
equal lengths before calculating dot products or norms, returning an error or
None for mismatches instead of silently truncating via zip. Propagate the
changed return type through callers and add test coverage confirming unequal
dimensions are rejected.
In `@crates/agentflare-store/src/embedding_pipeline/download.rs`:
- Around line 174-200: Update read_lockfile to return a Result and propagate
missing, unreadable, or malformed lockfile errors instead of defaulting to an
empty map. In file_passes_verification, require every required file to have a
matching lock pin and reject unpinned files so ensure_model triggers a fresh
download; add tests covering missing and malformed lockfiles.
- Around line 31-38: Handle the result of std::fs::remove_file in the
verification flow instead of ignoring it: propagate the deletion error so
download setup aborts when a corrupt artifact cannot be removed. Ensure the
existing any_corrupt path only continues when the failed file was successfully
deleted, preventing later readiness logic from accepting the mismatched
artifact.
In `@crates/agentflare-store/src/embedding_pipeline/mod.rs`:
- Around line 58-92: Update load_model and detect_dimensions to resolve ONNX
input and output bindings by tensor names and expected shapes rather than
positional indices. Explicitly locate input_ids, attention_mask, and optional
token_type_ids, validate required bindings, and select the embedding output by
its expected shape instead of session.outputs().first(). Preserve the existing
error behavior by returning clear failures when required bindings or a valid
embedding output are absent.
In `@crates/agentflare-store/src/embedding_pipeline/model_registry.rs`:
- Around line 43-62: Update ModelRegistry::storage_slug to append a stable
digest derived from the complete repository and revision values, rather than
relying only on sanitized text and the truncated revision. Keep the slug
filesystem-safe and ensure distinct repository/revision specifications produce
distinct cache directory names, including revisions longer than 16 characters.
- Around line 211-214: Update resolve_model so the default model is used only
when AGENTFLARE_EMBEDDING_MODEL is absent; when the variable is present,
validate its value with EmbeddingModel::from_str_name and propagate an error
containing the invalid model name instead of falling back to
EmbeddingModel::DEFAULT.
In `@crates/agentflare-store/src/embedding_pipeline/tokenizer.rs`:
- Around line 143-173: Update wordpiece_encode so any failed WordPiece
segmentation returns exactly one unk_id for the entire word, discarding
previously matched subword tokens. Keep successful complete segmentation
unchanged, but when matched is false return immediately with a single unknown
token instead of advancing start and continuing.
- Around line 83-106: The HfTokenizerWrapper implementation must not silently
emulate only a narrow tokenizer subset. Update the wrapper’s tokenizer
construction and encoding flow to use the Hugging Face tokenizer pipeline,
including normalizer, pre_tokenizer, added_tokens, and post_processor, so
tokenizer.json behavior and token IDs round-trip faithfully; alternatively,
explicitly reject unsupported configurations and add parity tests covering the
supported subset.
In `@crates/agentflare-store/src/migrate.rs`:
- Around line 27-34: Update migrate_state_json to persist all legacy entries and
MIGRATION_MARKER through one atomic transaction or batch write, so partial
imports cannot become visible when any write fails. Preserve the existing marker
semantics and add a test that injects a mid-migration failure, verifies no
migrated keys or marker remain, and confirms a later load can retry the import.
In `@crates/flare-output/src/compress.rs`:
- Line 147: Preserve backward compatibility in the backup namespace path around
the "flare-output" join by retaining a fallback lookup for the legacy "caveman"
namespace. Ensure BackupExists checks both namespaces so existing out-of-tree
backups still prevent creating a new backup; add migration or tests and
documentation only if intentionally removing the fallback.
In `@crates/flare-output/src/llm.rs`:
- Around line 29-31: Update the model environment-variable resolution around
FLARE_OUTPUT_MODEL and CAVEMAN_MODEL to treat blank values as unset before
applying precedence. Preserve the order of FLARE_OUTPUT_MODEL, then
CAVEMAN_MODEL, then the "claude-sonnet-4-5" default, while ensuring empty values
cannot be selected.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9339df4e-f592-4b6a-b88f-f7fff0350a27
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (28)
.gitignoreCargo.tomlcrates/agentflare-store/Cargo.tomlcrates/agentflare-store/src/blobs.rscrates/agentflare-store/src/documents.rscrates/agentflare-store/src/embed.rscrates/agentflare-store/src/embedding_pipeline/download.rscrates/agentflare-store/src/embedding_pipeline/mod.rscrates/agentflare-store/src/embedding_pipeline/model_registry.rscrates/agentflare-store/src/embedding_pipeline/pooling.rscrates/agentflare-store/src/embedding_pipeline/tokenizer.rscrates/agentflare-store/src/kv.rscrates/agentflare-store/src/leases.rscrates/agentflare-store/src/lib.rscrates/agentflare-store/src/migrate.rscrates/agentflare-store/src/migrations.rscrates/flare-code/src/config.rscrates/flare-code/src/instructions.rscrates/flare-code/src/skill-no-hallucination.mdcrates/flare-code/src/skill-playbook.mdcrates/flare-code/src/skill-review.mdcrates/flare-code/src/skill.mdcrates/flare-output/src/compress.rscrates/flare-output/src/llm.rscrates/flare-output/src/prompt.rssrc/main.rssrc/state.rssrc/store.rs
Comment on lines
+83
to
+106
| fn pre_tokenize(&self, text: &str) -> Vec<String> { | ||
| let mut words = Vec::new(); | ||
| let mut current = String::new(); | ||
|
|
||
| for ch in text.chars() { | ||
| if ch.is_whitespace() { | ||
| if !current.is_empty() { | ||
| words.extend(self.split_identifier(¤t)); | ||
| current.clear(); | ||
| } | ||
| } else if is_bert_punctuation(ch) { | ||
| if !current.is_empty() { | ||
| words.extend(self.split_identifier(¤t)); | ||
| current.clear(); | ||
| } | ||
| words.push(ch.to_string()); | ||
| } else { | ||
| current.push(ch); | ||
| } | ||
| } | ||
| if !current.is_empty() { | ||
| words.extend(self.split_identifier(¤t)); | ||
| } | ||
| words.iter().map(|w| w.to_lowercase()).collect() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🌐 Web query:
Using official Hugging Face Tokenizers documentation, which tokenizer.json sections define normalization, pre-tokenization, added special tokens, and post-processing, and can model.vocab/model.merges alone reproduce BPE tokenization?
💡 Result:
In the Hugging Face Tokenizers library, the tokenizer.json file acts as a serialized representation of the entire tokenization pipeline [1][2]. The following top-level sections of the JSON structure define the corresponding components: 1. Normalization: The normalizer section [1][3]. 2. Pre-tokenization: The pre_tokenizer section [1][3]. 3. Added special tokens: The added_tokens section [4][5]. 4. Post-processing: The post_processor section [1][3][5]. Regarding the BPE algorithm, model.vocab and model.merges alone cannot reproduce full BPE tokenization [1][6]. While the BPE model itself relies on these two components to map sequences to subword IDs [7][8], the complete tokenization pipeline requires additional steps defined in the tokenizer.json file [1][6]. Specifically, a functional tokenizer requires a pre-tokenizer to split text into word-level units before BPE can be applied, as well as a normalizer to handle input text transformations (e.g., Unicode normalization or lowercasing) [1][6]. Without these components, the BPE model would receive raw, unprocessed input, leading to inconsistent or incorrect tokenization results compared to the original pipeline [6]. Sources: [1] https://huggingface.co/docs/tokenizers/main/en/api/tokenizer [4] huggingface/tokenizers#1410 [3] https://huggingface.co/docs/tokenizers/en/api/tokenizer [5] https://github.com/huggingface/transformers/blob/v4.52.3/src/transformers/tokenization_utils_fast.py [6] https://huggingface.co/docs/tokenizers/python/latest/components.html [7] https://huggingface.co/docs/tokenizers/main/en/api/models [2] huggingface/tokenizers#521 [8] https://github.com/huggingface/transformers/blob/main/docs/source/en/custom_tokenizers.md
Citations:
- 1: https://huggingface.co/docs/tokenizers/main/en/api/tokenizer
- 2: How to get both the vocabulary.json and the merges.txt file when saving a BPE tokenizer huggingface/tokenizers#521
- 3: https://huggingface.co/docs/tokenizers/en/api/tokenizer
- 4: How to create Tokenizer.json? huggingface/tokenizers#1410
- 5: https://github.com/huggingface/transformers/blob/v4.52.3/src/transformers/tokenization_utils_fast.py
- 6: https://huggingface.co/docs/tokenizers/python/latest/components.html
- 7: https://huggingface.co/docs/tokenizers/main/en/api/models
- 8: https://github.com/huggingface/transformers/blob/main/docs/source/en/custom_tokenizers.md
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the tokenizer implementation and related pipeline code.
git ls-files | grep -E '^crates/agentflare-store/src/embedding_pipeline/.*tokenizer\.rs$|^crates/agentflare-store/src/embedding_pipeline/'
# Show the relevant section with line numbers.
sed -n '1,240p' crates/agentflare-store/src/embedding_pipeline/tokenizer.rsRepository: getappz/agentflare
Length of output: 7936
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect any references to tokenizer.json or HF tokenizer behavior in the crate.
grep -RIn --exclude-dir=.git --exclude='*.lock' --exclude='*.png' \
-E 'tokenizer\.json|pre_tokenize|post_processor|added_tokens|normalizer|BPE|huggingface|tokenizers' \
crates/agentflare-store/src || trueRepository: getappz/agentflare
Length of output: 2529
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check for the tokenizer pipeline construction and what config fields are consumed.
sed -n '1,220p' crates/agentflare-store/src/embedding_pipeline/mod.rs
sed -n '1,240p' crates/agentflare-store/src/embedding_pipeline/*.rs 2>/dev/null | sed -n '1,240p'Repository: getappz/agentflare
Length of output: 15245
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Narrow read of the tokenizer file around the cited lines and related helpers.
nl -ba crates/agentflare-store/src/embedding_pipeline/tokenizer.rs | sed -n '1,420p'Repository: getappz/agentflare
Length of output: 196
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the tokenizer implementation with line numbers.
if [ -f crates/agentflare-store/src/embedding_pipeline/tokenizer.rs ]; then
nl -ba crates/agentflare-store/src/embedding_pipeline/tokenizer.rs | sed -n '1,420p'
fiRepository: getappz/agentflare
Length of output: 196
🌐 Web query:
Official Hugging Face tokenizers docs: does tokenizer.json include normalizer, pre_tokenizer, added_tokens, and post_processor sections, and is a BPE model described by vocab and merges only?
💡 Result:
Yes, the official Hugging Face tokenizer.json file is a self-contained serialization of the entire tokenizer pipeline [1]. It includes sections for the normalizer, pre_tokenizer, model, post_processor, and added_tokens [2][1]. Specifically, the structure typically reflects the Tokenizer pipeline, which consists of the following components [3][4]: 1. Normalizer: Processes and normalizes raw input text [3][5]. 2. PreTokenizer: Splits text into word-level tokens [3][5]. 3. Model: Maps tokens to IDs using the specified algorithm (e.g., BPE) [3][5]. 4. PostProcessor: Performs final transformations, such as adding special tokens like [CLS] or [SEP] [3][5][1]. Regarding the BPE (Byte-Pair Encoding) model, it is described within the model section of the tokenizer.json file [1]. While the model's core logic is defined by its vocabulary (mapping tokens to IDs) and merge rules (how to combine tokens) [6][7], the model entry in tokenizer.json may contain additional parameters—such as unk_token, dropout, byte_fallback, and other configuration settings—depending on how the model was initialized and saved [8][9]. Therefore, while vocab and merges are the fundamental components of a BPE model, the serialized model section often includes these extra configuration parameters as well [8].
Citations:
- 1: https://mbrenndoerfer.com/writing/tokenizer-training-guide-huggingface-custom-nlp
- 2: Tokenizers: read the tokenizer.json 'decoder' section in CreateFromTokenizerJson dotnet/machinelearning#7660
- 3: https://huggingface.co/docs/tokenizers/main/en/api/tokenizer
- 4: https://huggingface.co/docs/tokenizers/en/api/tokenizer
- 5: https://huggingface.co/docs/tokenizers/main/components
- 6: https://huggingface.co/docs/transformers/en/custom_tokenizers
- 7: https://huggingface.co/datasets/John6666/forum1/blob/main/bpe_distill.md
- 8: https://github.com/huggingface/tokenizers/blob/b6b1688b/bindings/python/py_src/tokenizers/models.pyi
- 9: https://huggingface.co/docs/tokenizers/main/api/models
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the tokenizer loader and wrapper implementation around the relevant lines.
sed -n '300,520p' crates/agentflare-store/src/embedding_pipeline/tokenizer.rsRepository: getappz/agentflare
Length of output: 5891
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the tokenizer tests and any tokenizer.json fixtures for supported/unsupported fields.
sed -n '520,760p' crates/agentflare-store/src/embedding_pipeline/tokenizer.rsRepository: getappz/agentflare
Length of output: 156
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the loader and tests around the tokenizer wrapper implementation.
sed -n '300,760p' crates/agentflare-store/src/embedding_pipeline/tokenizer.rsRepository: getappz/agentflare
Length of output: 5891
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('crates/agentflare-store/src/embedding_pipeline/tokenizer.rs')
lines = p.read_text().splitlines()
for start, end in [(300, 520), (520, 760)]:
print(f"\n--- {start}-{end} ---")
for i in range(start-1, min(end, len(lines))):
print(f"{i+1}: {lines[i]}")
PYRepository: getappz/agentflare
Length of output: 6915
HfTokenizerWrapper only rebuilds a narrow WordPiece/BPE subset. It ignores normalizer, pre_tokenizer, added_tokens, and post_processor, so saved tokenizer.json pipelines won’t round-trip faithfully and can yield different token IDs. Use the Hugging Face tokenizer directly, or reject configs outside a documented subset and add parity tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/agentflare-store/src/embedding_pipeline/tokenizer.rs` around lines 83
- 106, The HfTokenizerWrapper implementation must not silently emulate only a
narrow tokenizer subset. Update the wrapper’s tokenizer construction and
encoding flow to use the Hugging Face tokenizer pipeline, including normalizer,
pre_tokenizer, added_tokens, and post_processor, so tokenizer.json behavior and
token IDs round-trip faithfully; alternatively, explicitly reject unsupported
configurations and add parity tests covering the supported subset.
- blob_store/blob_unref: wrap the exists-check + insert/decrement + cascade in an Immediate transaction so concurrent connections can't race on the same hash; compensate for orphaned disk writes on failure. - doc_upsert_with_opts: open the transaction before reading the current version so concurrent upserts can't both read version N and both write N+1; also makes new-document insert + FTS sync atomic. - doc_hard_delete: delete history/vector/FTS rows before the parent row, all in one transaction. - migrate_state_json: one transaction for every key plus the completion marker, so a partial failure can't desync the marker from what migrated.
…them - cosine_similarity: return None on mismatched vector lengths instead of letting zip() silently drop the excess coordinates. - blob_get: propagate disk read I/O errors (permissions, disk failure) instead of folding them into 'blob not found'. - read_lockfile: distinguish a missing lockfile (fine, first run) from an unreadable or malformed one (error) instead of treating both as 'no pins', which made every present file look unverified and skip the tamper check. - ensure_model: abort if a corrupt model file can't be deleted, instead of proceeding while the corrupt file is still on disk. - resolve_model: error on an invalid AGENTFLARE_EMBEDDING_MODEL instead of silently falling back to the default model.
…X I/O, correct WordPiece fallback - CustomModelSpec::storage_slug: hash repo+revision into the cache dir name instead of truncating the revision to 16 chars, which let two different (repo, revision) pairs collide on the same cache directory. - EmbeddingEngine::load_model: resolve ONNX input/output tensors by declared name (with a positional fallback for the two inputs every model has) instead of blind index 0/1/2, which breaks if a model export ever reorders its inputs/outputs. - wordpiece_encode: an unsegmentable word now collapses to a single [UNK], matching standard WordPiece, instead of one UNK per unmatched character interleaved with whatever subwords did match. - vec_search_ranks_by_similarity test: use directionally distinct embeddings; the old ones were collinear and scored identically regardless of magnitude, so the test wasn't exercising rank order.
…me backups discoverable - call_via_api: a blank FLARE_OUTPUT_MODEL/CAVEMAN_MODEL value no longer wins over the next fallback in the chain. - backup_path_for (OutOfTree): the backup namespace directory was renamed from 'caveman' to 'flare-output' with a hard rename and no fallback, so a user's pre-rename backups became invisible to both the 'backup already exists' guard and Report::original_path. Now resolves to an existing legacy path if the new one doesn't have one yet.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/flare-output/src/compress.rs`:
- Around line 360-364: Update the legacy path construction around the `legacy`
variable to avoid global string replacement. Derive the base path with
`out_of_tree_backup_dir("caveman")`, preserve the relative suffix from
`resolved`, and join them so only the namespace component changes while
unrelated parent-directory names remain intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7a9484bb-1800-49b0-b6ac-6ee6c2ea3b1e
📒 Files selected for processing (10)
crates/agentflare-store/src/blobs.rscrates/agentflare-store/src/documents.rscrates/agentflare-store/src/embed.rscrates/agentflare-store/src/embedding_pipeline/download.rscrates/agentflare-store/src/embedding_pipeline/mod.rscrates/agentflare-store/src/embedding_pipeline/model_registry.rscrates/agentflare-store/src/embedding_pipeline/tokenizer.rscrates/agentflare-store/src/migrate.rscrates/flare-output/src/compress.rscrates/flare-output/src/llm.rs
🚧 Files skipped from review as they are similar to previous changes (8)
- crates/flare-output/src/llm.rs
- crates/agentflare-store/src/migrate.rs
- crates/agentflare-store/src/embedding_pipeline/model_registry.rs
- crates/agentflare-store/src/embedding_pipeline/tokenizer.rs
- crates/agentflare-store/src/embedding_pipeline/download.rs
- crates/agentflare-store/src/blobs.rs
- crates/agentflare-store/src/embedding_pipeline/mod.rs
- crates/agentflare-store/src/documents.rs
getappz
enabled auto-merge (squash)
July 18, 2026 08:08
2 tasks
This was referenced Jul 19, 2026
feat: on-demand third-party docs — flare-docs crate + flare_docs MCP tool + agentflare docs CLI
#316
Merged
5 tasks
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the
agentflare-storecrate (work tracked as item #147/#148 in the item tracker), previously sitting unmerged and PR-less onfeat/agentflare-store-v1, 45 commits behind master. Rebased onto current master, verified clean.Modules:
model.type), model download with SHA-256 verification (re-verifies already-present files, self-heals on corruption), pooling, model registrydb_kit::ClaimLedgerThis closes the gap I flagged (and then corrected myself on, after re-verifying with lean-ctx's code-intelligence tools) in issues #148/#149 — those issues asked whether embeddings/hybrid-search existed; the engine now lands here. Not in scope for this PR: wiring this into
src/memory's actualremember/recallMCP tools — that integration doesn't exist yet and is separate follow-up work (noted on #148/#149).Also resolves two known bugs already fixed on this branch (items #152, #153 in the tracker):
model.type(WordPiece vs BPE) instead of assuming WordPiece for all custom modelsItem #155 ("crate is not cargo fmt-clean") was flagged as an open blocker but turned out to already be resolved on this branch's tip (
cargo fmt --checkpasses clean) — no action needed there.Test plan
cargo build --workspacecleancargo test -p agentflare-store— 31/31 passcargo test --workspace— 609+ tests, 0 failed, 0 regressions from the rebasecargo clippy -p agentflare-store --all-targets --all-features -- -D warningscleancargo fmt --checkcleancargo checkrather than hand-merging the lockfileSummary by CodeRabbit