Repository navigation
fix(knowledge): harden document indexing state - #875
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds DB-backed indexing state ( Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant Orchestrator as Orchestrator
participant DB as Database
participant Redis as Redis Lock
participant Celery as Celery Worker
participant Embeddings as Embedding Model
Client->>Orchestrator: request reindex / update
Orchestrator->>DB: prepare_document_index_enqueue(doc_id)
alt should_enqueue = false
DB-->>Orchestrator: should_enqueue=False, reason
Orchestrator-->>Client: scheduled: False (reason)
else should_enqueue = true
DB->>DB: increment index_generation, set index_status=QUEUED
DB-->>Orchestrator: should_enqueue=True, generation=N
Orchestrator->>Celery: index_document_task.delay(doc_id, index_generation=N)
Orchestrator-->>Client: scheduled: True, generation=N
Celery->>Redis: acquire_watchdog_context(lock_name)
alt lock acquired
Redis-->>Celery: lock acquired
Celery->>DB: mark_document_index_started(doc_id, generation=N)
alt should_execute = true
Celery->>Embeddings: run_document_indexing()
Embeddings-->>Celery: result (success/failure/skipped)
Celery->>DB: mark_document_index_succeeded/failed(doc_id, generation=N)
DB-->>Celery: finalized
Celery-->>Client: completed (status)
else skipped
DB-->>Celery: should_execute=False (stale/other)
Celery-->>Client: skipped (reason)
end
Redis->>Redis: watchdog extends lock periodically
Celery->>Redis: release lock
else lock held
Redis-->>Celery: lock not acquired
Celery->>Celery: retry or return skipped after retries
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (5)
backend/tests/tasks/test_knowledge_tasks.py (1)
90-136: Consider reducing nested context managers for maintainability.The test has 7 levels of nested
withblocks, making it hard to read and modify. Consider usingunittest.mock.patch.multiple, pytest'smonkeypatchwith fixtures, or combining patches into a helper.Additionally, the
SessionLocalmock at lines 125-128 assumes exactly 2 calls in sequence. This is fragile if the implementation changes the number of database sessions created.♻️ Example using patch decorators or a setup helper
def test_index_document_task_marks_skip_result_as_failed( monkeypatch: pytest.MonkeyPatch, ): """Test that skipped indexing results are marked as failed.""" start_decision = MagicMock(should_execute=True, reason="started") success_finalize_mock = MagicMock() failed_finalize_mock = MagicMock(return_value=True) # Group all patches together patches = { "app.tasks.knowledge_tasks.distributed_lock.acquire_watchdog_context": _lock_context(True), "app.services.knowledge.index_state_machine.mark_document_index_started": start_decision, "app.services.knowledge.index_state_machine.mark_document_index_succeeded": success_finalize_mock, "app.services.knowledge.index_state_machine.mark_document_index_failed": failed_finalize_mock, "app.services.knowledge.indexing.run_document_indexing": { "status": "skipped", "reason": "unsupported_document", "document_id": 4, "knowledge_base_id": "1", }, } # Apply patches using monkeypatch or patch.multiple # ... rest of test🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/tests/tasks/test_knowledge_tasks.py` around lines 90 - 136, The test test_index_document_task_marks_skip_result_as_failed is overly nested and brittle; refactor the multiple with-patch blocks into a grouped patch application (e.g., use unittest.mock.patch.multiple or pytest.monkeypatch to set the same targets) for acquire_watchdog_context, mark_document_index_started, mark_document_index_succeeded, mark_document_index_failed, and run_document_indexing, then call index_document_task.run(**_task_kwargs()) once; also replace the brittle SessionLocal side_effect list with a stable factory that returns a new nullcontext(MagicMock()) per call (e.g., patch SessionLocal with a lambda or helper function that yields fresh context managers) so the test no longer depends on exact session call counts.backend/app/services/knowledge/orchestrator.py (1)
1361-1362: Consider replacingassertwith explicit error handling.The
assertstatement can be disabled with Python's-Oflag. While the logic guaranteesgenerationis notNonewhenshould_enqueueisTrue, using an explicit check would be safer in production.♻️ Replace assert with explicit check
- generation = enqueue_decision.generation - assert generation is not None + generation = enqueue_decision.generation + if generation is None: + logger.error( + f"[Orchestrator] Bug: should_enqueue=True but generation is None for document {document.id}" + ) + return { + "scheduled": False, + "reason": "internal_error", + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/orchestrator.py` around lines 1361 - 1362, Replace the bare Python assert for enqueue_decision.generation with an explicit runtime check: after assigning generation = enqueue_decision.generation, test if generation is None and raise a clear exception (e.g., ValueError or RuntimeError) with a descriptive message (or log and return) so the condition cannot be skipped with -O; reference the variable/enqueue_decision.generation in the error message and perform this change inside the same function in orchestrator.py where generation is used.frontend/src/features/knowledge/document/components/DocumentItem.tsx (1)
180-196: Minor: Deeply nested ternary could be simplified.The nested ternary for
unavailableHintis readable but has 4 levels of nesting. Consider extracting to a small helper function or using early returns for clarity in future refactors.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@frontend/src/features/knowledge/document/components/DocumentItem.tsx` around lines 180 - 196, The nested ternary assigned to unavailableHint (using isExcelExceedingSizeLimit, ragConfigured, isNotIndexed and document fields) is hard to read; replace it with a small helper function or an early-return if/else block (e.g., create a getUnavailableHint(document, isExcelExceedingSizeLimit, ragConfigured, isNotIndexed) or inline if/else) that returns the correct translation string for the four cases (excel size exceeded, rag not configured, not indexed, default unavailable) to improve readability and maintainability; keep unavailableLabel and unavailableDotColor as-is but reference the same predicate flags in the new helper.backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py (1)
97-107: Downgrade does not clean up the enum type.On PostgreSQL, dropping columns that use a custom enum type does not automatically drop the enum type itself. This leaves orphaned types in the database after downgrade.
♻️ Consider adding enum type cleanup for PostgreSQL
def downgrade() -> None: """Remove knowledge document indexing state columns.""" conn = op.get_bind() inspector = sa.inspect(conn) existing_columns = { column["name"] for column in inspector.get_columns("knowledge_documents") } for column_name in ["index_generation", "index_status"]: if column_name in existing_columns: op.drop_column("knowledge_documents", column_name) + + # Clean up enum type on PostgreSQL + if conn.dialect.name == "postgresql": + op.execute(sa.text("DROP TYPE IF EXISTS documentindexstatus"))🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py` around lines 97 - 107, The downgrade currently drops the "index_generation" and "index_status" columns but leaves their PostgreSQL enum type(s) behind; update the downgrade function to, after determining and/or dropping those columns, check if conn.dialect.name == "postgresql" and then drop the associated enum type(s) (e.g., the enum used by "index_generation" and "index_status") using op.execute("DROP TYPE IF EXISTS <enum_name>") for each unique enum; to find the enum name reliably, inspect the column type via inspector.get_columns("knowledge_documents") or capture the sa.Enum name used when the enum was created, and only drop those types if present to avoid errors.backend/app/services/knowledge/index_state_machine.py (1)
84-180: Instrument these state transitions.These helpers are now the source of truth for enqueue/start/finalize decisions, but they do not emit spans or transition attributes. At minimum, attach
document_id,generation, and the decision reason so stale-takeover and duplicate-task incidents are reconstructable in production.As per coding guidelines, "Use
@trace_sync(span_name, tracer_name, extract_attributes)decorator to trace entire sync functions in OpenTelemetry", "Useadd_span_event(name, attributes)to add event to current OpenTelemetry span", and "Useset_span_attribute(key, value)to set attribute on current OpenTelemetry span".Also applies to: 183-299
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/index_state_machine.py` around lines 84 - 180, Wrap the sync helpers with OpenTelemetry tracing and emit events/attributes: add the `@trace_sync`(...) decorator to prepare_document_index_enqueue and mark_document_index_enqueue_failed (and the other functions in the same range) using an appropriate span name and tracer, and inside each function call set_span_attribute for "document_id", "generation" (when known) and "decision_reason" (or "reason") before returning/committing; also call add_span_event at decision points (e.g., before returning not-enqueued/queued/failure) with attributes { "document_id": document_id, "generation": next_generation or generation, "reason": "<reason_string>" } so all enqueue/start/finalize decisions are recorded. Ensure you import trace helpers and choose extract_attributes per guidelines.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/app/services/knowledge/index_state_machine.py`:
- Around line 103-108: The early return when document is None (after the
with_for_update() query) leaves the DB transaction open; before returning the
IndexEnqueueDecision, roll back/close the session/transaction used in this
function (call session.rollback() or the appropriate session object’s
rollback/close) so the transaction is not left open, or refactor the
with_for_update() usage into a context manager that always closes the
transaction; target the branch that checks document is None and the surrounding
code that performs the with_for_update() query and returns IndexEnqueueDecision.
- Around line 123-148: Update the happy-path tests to assert that
document.updated_at advances after the DB commit in both
prepare_document_index_enqueue() and mark_document_index_started(): capture the
pre-commit updated_at, call the function under test (e.g.,
prepare_document_index_enqueue() in
test_prepare_document_index_enqueue_schedules_new_generation and
test_prepare_document_index_enqueue_can_replace_active_generation, and
mark_document_index_started() where applicable), commit/refresh the session,
then assert the post-commit/refresh updated_at is greater than the captured
timestamp to verify the onupdate=func.now() behavior. Ensure you refresh the
document from the session or database before comparing so the tests observe the
updated value.
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1480-1495: The return message uses schedule_result["reason"] to
choose between only two messages, which mislabels cases like
"document_not_found"; update the logic around schedule_result and reason in
orchestrator.py (where schedule_result is checked and logger.info is called) to
map each possible reason to a specific user-facing message (e.g.,
"already_in_progress" -> "Reindex already in progress", "document_not_found" ->
"Document not found", "already_indexed" -> "Document is already indexed", etc.),
by replacing the current inline conditional with a small dict or if/elif that
sets message before returning and include that message in both logger.info and
the returned dict.
In `@backend/app/tasks/knowledge_tasks.py`:
- Around line 225-245: The code is blocking the DB session by calling an async
summary service via a synchronous event loop; replace that pattern by enqueuing
a background Celery task so the SessionLocal context is released immediately.
Specifically, remove any loop.run_until_complete or direct async calls inside
the finalize_db block (related to trigger_document_summary_if_enabled /
summary_service.trigger_document_summary) and instead create or call an existing
Celery task (e.g., trigger_document_summary_task) that accepts document_id,
user_id, user_name, and serialized kb_info (or its id) and invoke it with
delay/apply_async before exiting the with SessionLocal() block; ensure you pass
only simple serializable arguments (not open DB objects or sessions) so the
background worker can open its own DB session and run
summary_service.trigger_document_summary asynchronously.
In `@backend/tests/services/knowledge/test_index_state_machine.py`:
- Around line 192-209: The test
test_prepare_document_index_enqueue_can_replace_active_generation currently sets
the document to DocumentIndexStatus.INDEXING so the allow_if_success=True branch
in prepare_document_index_enqueue is never exercised; either remove the
allow_if_success=True flag from this test (since it’s irrelevant for INDEXING)
or modify the fixture to create the document with DocumentIndexStatus.SUCCESS
(e.g., set index_status=DocumentIndexStatus.SUCCESS) so the success-override
branch in prepare_document_index_enqueue is actually exercised.
---
Nitpick comments:
In
`@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py`:
- Around line 97-107: The downgrade currently drops the "index_generation" and
"index_status" columns but leaves their PostgreSQL enum type(s) behind; update
the downgrade function to, after determining and/or dropping those columns,
check if conn.dialect.name == "postgresql" and then drop the associated enum
type(s) (e.g., the enum used by "index_generation" and "index_status") using
op.execute("DROP TYPE IF EXISTS <enum_name>") for each unique enum; to find the
enum name reliably, inspect the column type via
inspector.get_columns("knowledge_documents") or capture the sa.Enum name used
when the enum was created, and only drop those types if present to avoid errors.
In `@backend/app/services/knowledge/index_state_machine.py`:
- Around line 84-180: Wrap the sync helpers with OpenTelemetry tracing and emit
events/attributes: add the `@trace_sync`(...) decorator to
prepare_document_index_enqueue and mark_document_index_enqueue_failed (and the
other functions in the same range) using an appropriate span name and tracer,
and inside each function call set_span_attribute for "document_id", "generation"
(when known) and "decision_reason" (or "reason") before returning/committing;
also call add_span_event at decision points (e.g., before returning
not-enqueued/queued/failure) with attributes { "document_id": document_id,
"generation": next_generation or generation, "reason": "<reason_string>" } so
all enqueue/start/finalize decisions are recorded. Ensure you import trace
helpers and choose extract_attributes per guidelines.
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1361-1362: Replace the bare Python assert for
enqueue_decision.generation with an explicit runtime check: after assigning
generation = enqueue_decision.generation, test if generation is None and raise a
clear exception (e.g., ValueError or RuntimeError) with a descriptive message
(or log and return) so the condition cannot be skipped with -O; reference the
variable/enqueue_decision.generation in the error message and perform this
change inside the same function in orchestrator.py where generation is used.
In `@backend/tests/tasks/test_knowledge_tasks.py`:
- Around line 90-136: The test
test_index_document_task_marks_skip_result_as_failed is overly nested and
brittle; refactor the multiple with-patch blocks into a grouped patch
application (e.g., use unittest.mock.patch.multiple or pytest.monkeypatch to set
the same targets) for acquire_watchdog_context, mark_document_index_started,
mark_document_index_succeeded, mark_document_index_failed, and
run_document_indexing, then call index_document_task.run(**_task_kwargs()) once;
also replace the brittle SessionLocal side_effect list with a stable factory
that returns a new nullcontext(MagicMock()) per call (e.g., patch SessionLocal
with a lambda or helper function that yields fresh context managers) so the test
no longer depends on exact session call counts.
In `@frontend/src/features/knowledge/document/components/DocumentItem.tsx`:
- Around line 180-196: The nested ternary assigned to unavailableHint (using
isExcelExceedingSizeLimit, ragConfigured, isNotIndexed and document fields) is
hard to read; replace it with a small helper function or an early-return if/else
block (e.g., create a getUnavailableHint(document, isExcelExceedingSizeLimit,
ragConfigured, isNotIndexed) or inline if/else) that returns the correct
translation string for the four cases (excel size exceeded, rag not configured,
not indexed, default unavailable) to improve readability and maintainability;
keep unavailableLabel and unavailableDotColor as-is but reference the same
predicate flags in the new helper.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ed9c7fd5-b510-427d-9101-6b576dde4e60
📒 Files selected for processing (18)
backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.pybackend/app/core/config.pybackend/app/core/distributed_lock.pybackend/app/models/knowledge.pybackend/app/schemas/knowledge.pybackend/app/services/knowledge/index_state_machine.pybackend/app/services/knowledge/indexing.pybackend/app/services/knowledge/orchestrator.pybackend/app/tasks/knowledge_tasks.pybackend/tests/services/knowledge/test_index_state_machine.pybackend/tests/services/knowledge/test_orchestrator.pybackend/tests/tasks/test_knowledge_tasks.pydocs/en/developer-guide/knowledge-indexing-protection.mddocs/zh/developer-guide/knowledge-indexing-protection.mdfrontend/src/features/knowledge/document/components/DocumentItem.tsxfrontend/src/i18n/locales/en/knowledge.jsonfrontend/src/i18n/locales/zh-CN/knowledge.jsonfrontend/src/types/knowledge.ts
Comment on lines
+123
to
+148
| next_generation = (document.index_generation or 0) + 1 | ||
| document.index_generation = next_generation | ||
| document.index_status = DocumentIndexStatus.QUEUED | ||
| db.commit() | ||
|
|
||
| return IndexEnqueueDecision( | ||
| should_enqueue=True, | ||
| generation=next_generation, | ||
| reason="scheduled_after_stale_recovery", | ||
| previous_status=current_status, | ||
| ) | ||
|
|
||
| if current_status == DocumentIndexStatus.SUCCESS and not allow_if_success: | ||
| db.rollback() | ||
| return IndexEnqueueDecision( | ||
| should_enqueue=False, | ||
| generation=document.index_generation, | ||
| reason="already_indexed", | ||
| previous_status=current_status, | ||
| ) | ||
|
|
||
| next_generation = (document.index_generation or 0) + 1 | ||
| document.index_generation = next_generation | ||
| document.index_status = DocumentIndexStatus.QUEUED | ||
|
|
||
| db.commit() |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== KnowledgeDocument/Base timestamp configuration =="
rg -n -C2 'updated_at|created_at|onupdate|server_onupdate|func\.now' backend/app/models
echo
echo "== KnowledgeDocument definition =="
sed -n '1,220p' backend/app/models/knowledge.py
echo
echo "== Tests touching updated_at or these transitions =="
rg -n -C2 'updated_at|prepare_document_index_enqueue|mark_document_index_started' backend/testsRepository: wecode-ai/Wegent
Length of output: 34760
🏁 Script executed:
# Check the implementation of the two functions mentioned
sed -n '1,250p' backend/app/services/knowledge/index_state_machine.py | cat -nRepository: wecode-ai/Wegent
Length of output: 9091
🏁 Script executed:
# Look specifically at the implementation of these two functions
rg -n -A15 'def prepare_document_index_enqueue|def mark_document_index_started' backend/app/services/knowledge/index_state_machine.pyRepository: wecode-ai/Wegent
Length of output: 1142
🏁 Script executed:
# Check lines 231-232 referenced in "Also applies to"
sed -n '220,240p' backend/app/services/knowledge/index_state_machine.py | cat -nRepository: wecode-ai/Wegent
Length of output: 690
Add test assertions to verify updated_at advances during normal index enqueue and start transitions.
The model is correctly configured with onupdate=func.now(), but the current test suite only validates stale recovery by backdating updated_at—it does not verify that happy-path transitions in prepare_document_index_enqueue() (lines 123–126, 144–148) and mark_document_index_started() (lines 231–232) actually advance the timestamp. Add explicit assertions in the happy-path tests (test_prepare_document_index_enqueue_schedules_new_generation, test_prepare_document_index_enqueue_can_replace_active_generation, etc.) to confirm updated_at moves forward after db.commit().
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@backend/app/services/knowledge/index_state_machine.py` around lines 123 -
148, Update the happy-path tests to assert that document.updated_at advances
after the DB commit in both prepare_document_index_enqueue() and
mark_document_index_started(): capture the pre-commit updated_at, call the
function under test (e.g., prepare_document_index_enqueue() in
test_prepare_document_index_enqueue_schedules_new_generation and
test_prepare_document_index_enqueue_can_replace_active_generation, and
mark_document_index_started() where applicable), commit/refresh the session,
then assert the post-commit/refresh updated_at is greater than the captured
timestamp to verify the onupdate=func.now() behavior. Ensure you refresh the
document from the session or database before comparing so the tests observe the
updated value.
kissghosts
force-pushed
the
fix/knowledge-indexing-state
branch
from
March 27, 2026 14:04
d4ea3b6 to
e0bb557
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
backend/app/tasks/knowledge_tasks.py (1)
125-143:⚠️ Potential issue | 🟠 MajorStrip
chunks_dataout of the Celery task result.The worker only needs raw chunks for
mark_document_index_succeeded(). Returning them here means the Celery result backend and theseresult={result}logs can now carry full chunk payloads, which is both large and content-sensitive.🧹 Suggested change
- logger.info( - f"[Celery RAG Indexing] Completed without document guard: task_id={task_id}, " - f"attachment_id={attachment_id}, result={result}" - ) - return result + task_result = {k: v for k, v in result.items() if k != "chunks_data"} + logger.info( + "[Celery RAG Indexing] Completed without document guard: " + "task_id=%s, attachment_id=%s, status=%s, indexed_count=%s, index_name=%s", + task_id, + attachment_id, + task_result.get("status"), + task_result.get("indexed_count"), + task_result.get("index_name"), + ) + return task_result ... - logger.info( - f"[Celery RAG Indexing] Completed: task_id={task_id}, " - f"document_id={document_id}, attachment_id={attachment_id}, " - f"index_generation={index_generation}, result={result}" - ) - result["index_generation"] = index_generation - return result + task_result = {k: v for k, v in result.items() if k != "chunks_data"} + task_result["index_generation"] = index_generation + logger.info( + "[Celery RAG Indexing] Completed: task_id=%s, document_id=%s, " + "attachment_id=%s, index_generation=%s, status=%s, indexed_count=%s, " + "index_name=%s", + task_id, + document_id, + attachment_id, + index_generation, + task_result.get("status"), + task_result.get("indexed_count"), + task_result.get("index_name"), + ) + return task_resultAlso applies to: 293-299
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/tasks/knowledge_tasks.py` around lines 125 - 143, The Celery task is returning full chunks payloads from run_document_indexing which causes large/sensitive data to be stored/logged; after calling run_document_indexing (in the branch where document_id is None and the analogous branch later), remove or replace the chunks payload before returning or logging: inspect the returned value from run_document_indexing in knowledge_tasks.py, drop the "chunks_data" key (or replace it with a small summary/length) and then use that sanitized result for logger.info and as the function return; ensure mark_document_index_succeeded continues to receive raw chunks only via an internal call (not via the Celery task return).backend/app/services/knowledge/indexing.py (1)
372-387:⚠️ Potential issue | 🔴 CriticalUpdate docstring or refactor to clarify the actual contract.
The function's docstring claims support for "FastAPI BackgroundTasks" and "directly from other services," but it is only called from Celery tasks in production. Additionally, the
trigger_summaryparameter is unused within the function—it is handled by the Celery caller afterrun_document_indexing()returns. If this function were called from any context other than a Celery task that provides finalization and summary logic, callers would receive incomplete behavior (data only, no state mutations or summaries).Either update the docstring to accurately reflect Celery-only usage and remove the unused
trigger_summaryparameter, or refactor to restore the function's original side effects to match its advertised contract.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/indexing.py` around lines 372 - 387, The run_document_indexing function advertises support for BackgroundTasks/direct callers but is only used from Celery and still accepts an unused trigger_summary param; fix by either (A) removing the unused trigger_summary parameter and updating the run_document_indexing docstring to state the function is intended to be invoked by Celery tasks only (also update any callers/signatures in Celery tasks to drop the param), or (B) restore the original side-effect behavior so run_document_indexing performs finalization and summary triggering itself (implement the summary/state-mutation logic currently in the Celery task), and then update the docstring to describe that it performs those side effects—refer to run_document_indexing and the trigger_summary parameter when making the change.
🧹 Nitpick comments (1)
backend/app/services/knowledge/orchestrator.py (1)
1257-1408: Refactor_schedule_indexing_celeryinto smaller helpers.This function now handles config validation, owner resolution, enqueue state transitions, Celery dispatch, error rollback, and response shaping in one block. Splitting it will improve readability and testability.
As per coding guidelines, "Function length: Max 50 lines per function (preferred)".
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/orchestrator.py` around lines 1257 - 1408, Split _schedule_indexing_celery into small helpers: extract retrieval validation into validate_retrieval_config(spec) which checks retrievalConfig, retriever_name, embedding_config and returns (retriever_name, retriever_namespace, embedding_model_name, embedding_model_namespace) or the appropriate scheduled=False reason; extract owner resolution into resolve_index_owner_user_id(db, knowledge_base, user) which implements the namespace logic; move the enqueue/decision logic into prepare_enqueue_and_get_generation(db, document_id, allow_if_success, replace_active) that calls prepare_document_index_enqueue and handles the should_enqueue/generation check and error raising; encapsulate the Celery call into dispatch_index_document_task(...) that calls index_document_task.delay and on Exception calls mark_document_index_enqueue_failed; finally have a small coordinator that calls these helpers and returns the shaped response (scheduled, reason, index_generation, task_id) while keeping logging where needed. Ensure helper names reference prepare_document_index_enqueue, mark_document_index_enqueue_failed, index_document_task and reuse existing parameter names (document, knowledge_base, user, splitter_config, trigger_summary).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1478-1485: reindex_document is calling
_schedule_indexing_celery(...) with default behavior which allows a silent no-op
(reason="already_indexed"); update the reindex_document call to pass an explicit
flag that forces scheduling (for example force=True / force_reindex=True or
allow_skip=False depending on _schedule_indexing_celery’s signature) so the
scheduler will not short-circuit on "already_indexed", and if needed adjust
_schedule_indexing_celery to honor that flag and bypass the already-indexed
early return.
In `@backend/tests/services/knowledge/test_orchestrator.py`:
- Around line 419-462: The test currently only checks the returned scheduled
flag/reason but doesn't verify Celery isn't called; update
test_schedule_indexing_celery_skips_duplicate_enqueue to also assert that
index_document_task.delay was not invoked: patch or spy on
app.services.knowledge.tasks.index_document_task (or index_document_task.delay)
around the orchestrator._schedule_indexing_celery call and add an assertion that
the mocked delay method was not called (e.g., assert_not_called on the mock).
This ensures if orchestrator._schedule_indexing_celery mistakenly enqueues a
task despite should_enqueue=False, the test will fail.
In `@backend/tests/tasks/test_knowledge_tasks.py`:
- Around line 155-218: The test depends on the global settings.SUMMARY_ENABLED
and can flake; patch or monkeypatch the SUMMARY_ENABLED flag to True within the
test so generate_document_summary_task.delay is exercised deterministically.
Wrap a patch (or use monkeypatch) targeting the settings object used by
index_document_task (e.g., app.core.settings.SUMMARY_ENABLED) around the
existing ExitStack alongside the other patches before calling
index_document_task.run, ensuring SUMMARY_ENABLED is True for the duration of
the test so generate_document_summary_task.delay is always invoked.
---
Outside diff comments:
In `@backend/app/services/knowledge/indexing.py`:
- Around line 372-387: The run_document_indexing function advertises support for
BackgroundTasks/direct callers but is only used from Celery and still accepts an
unused trigger_summary param; fix by either (A) removing the unused
trigger_summary parameter and updating the run_document_indexing docstring to
state the function is intended to be invoked by Celery tasks only (also update
any callers/signatures in Celery tasks to drop the param), or (B) restore the
original side-effect behavior so run_document_indexing performs finalization and
summary triggering itself (implement the summary/state-mutation logic currently
in the Celery task), and then update the docstring to describe that it performs
those side effects—refer to run_document_indexing and the trigger_summary
parameter when making the change.
In `@backend/app/tasks/knowledge_tasks.py`:
- Around line 125-143: The Celery task is returning full chunks payloads from
run_document_indexing which causes large/sensitive data to be stored/logged;
after calling run_document_indexing (in the branch where document_id is None and
the analogous branch later), remove or replace the chunks payload before
returning or logging: inspect the returned value from run_document_indexing in
knowledge_tasks.py, drop the "chunks_data" key (or replace it with a small
summary/length) and then use that sanitized result for logger.info and as the
function return; ensure mark_document_index_succeeded continues to receive raw
chunks only via an internal call (not via the Celery task return).
---
Nitpick comments:
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1257-1408: Split _schedule_indexing_celery into small helpers:
extract retrieval validation into validate_retrieval_config(spec) which checks
retrievalConfig, retriever_name, embedding_config and returns (retriever_name,
retriever_namespace, embedding_model_name, embedding_model_namespace) or the
appropriate scheduled=False reason; extract owner resolution into
resolve_index_owner_user_id(db, knowledge_base, user) which implements the
namespace logic; move the enqueue/decision logic into
prepare_enqueue_and_get_generation(db, document_id, allow_if_success,
replace_active) that calls prepare_document_index_enqueue and handles the
should_enqueue/generation check and error raising; encapsulate the Celery call
into dispatch_index_document_task(...) that calls index_document_task.delay and
on Exception calls mark_document_index_enqueue_failed; finally have a small
coordinator that calls these helpers and returns the shaped response (scheduled,
reason, index_generation, task_id) while keeping logging where needed. Ensure
helper names reference prepare_document_index_enqueue,
mark_document_index_enqueue_failed, index_document_task and reuse existing
parameter names (document, knowledge_base, user, splitter_config,
trigger_summary).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: dc4f14eb-bf8f-47d1-98e6-2b2717cb85c1
📒 Files selected for processing (19)
backend/alembic/versions/20260325_b1c2d3e4f5g6_add_work_queue_tables.pybackend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.pybackend/app/core/config.pybackend/app/core/distributed_lock.pybackend/app/models/knowledge.pybackend/app/schemas/knowledge.pybackend/app/services/knowledge/index_state_machine.pybackend/app/services/knowledge/indexing.pybackend/app/services/knowledge/orchestrator.pybackend/app/tasks/knowledge_tasks.pybackend/tests/services/knowledge/test_index_state_machine.pybackend/tests/services/knowledge/test_orchestrator.pybackend/tests/tasks/test_knowledge_tasks.pydocs/en/developer-guide/knowledge-indexing-protection.mddocs/zh/developer-guide/knowledge-indexing-protection.mdfrontend/src/features/knowledge/document/components/DocumentItem.tsxfrontend/src/i18n/locales/en/knowledge.jsonfrontend/src/i18n/locales/zh-CN/knowledge.jsonfrontend/src/types/knowledge.ts
💤 Files with no reviewable changes (1)
- backend/alembic/versions/20260325_b1c2d3e4f5g6_add_work_queue_tables.py
✅ Files skipped from review due to trivial changes (4)
- frontend/src/i18n/locales/zh-CN/knowledge.json
- backend/app/core/config.py
- docs/en/developer-guide/knowledge-indexing-protection.md
- backend/tests/services/knowledge/test_index_state_machine.py
🚧 Files skipped from review as they are similar to previous changes (5)
- backend/app/schemas/knowledge.py
- backend/app/models/knowledge.py
- backend/app/core/distributed_lock.py
- backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py
- frontend/src/features/knowledge/document/components/DocumentItem.tsx
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/app/services/knowledge/orchestrator.py (1)
1834-1842:⚠️ Potential issue | 🟠 MajorDon't return a successful refresh when reindex was skipped.
This call still ignores
_schedule_indexing_celery(...)returningscheduled=False. Because Line 1819 already setdocument.is_active = False, a skipped enqueue leaves the refreshed document inactive while the API still returnssuccess=True.Based on learnings the frontend intentionally treats any inactive non-table document as "indexing".
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/orchestrator.py` around lines 1834 - 1842, Capture the boolean result returned by _schedule_indexing_celery(...) (e.g., scheduled = self._schedule_indexing_celery(...)) and use it to determine the API response: if scheduled is False, set the refresh response's success to False (and avoid returning success=True) so a skipped enqueue does not report a successful refresh; keep referencing _schedule_indexing_celery, document.is_active and the response object that is returned from the refresh path.
🧹 Nitpick comments (1)
backend/app/services/knowledge/orchestrator.py (1)
1257-1408: Extract and instrument_schedule_indexing_celery.This method now bundles config validation, owner resolution, state-machine mutation, broker dispatch, and recovery in one place. Pulling this into a smaller module and tracing the boundary/span events would make generation transitions much easier to audit and keep the orchestrator from growing further.
As per coding guidelines "File size limit: If a file exceeds 1000 lines, split it into multiple sub-modules", "Function length: Max 50 lines per function (preferred)", and "Use
@trace_sync(span_name, tracer_name, extract_attributes)decorator to trace entire sync functions in OpenTelemetry".🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/app/services/knowledge/orchestrator.py` around lines 1257 - 1408, The _schedule_indexing_celery function is too large and mixes validation, owner resolution, state-machine calls, Celery dispatch and recovery; refactor by extracting its sub-parts into a new submodule (e.g., services/knowledge/index_orchestration.py) and reduce _schedule_indexing_celery to a thin orchestrator that calls small helpers: validate_retrieval_config(retrieval_config), resolve_index_owner(knowledge_base, user, is_organization_namespace), enqueue_index_task(...) (which wraps index_document_task.delay), and handle_enqueue_failure(...) (which calls mark_document_index_enqueue_failed); then add tracing by annotating the top-level sync entrypoint with `@trace_sync`(span_name="orchestrator.schedule_indexing_celery", tracer_name="knowledge", extract_attributes=...) so the function is under 50 lines and the file is split out of the oversized module while keeping references to prepare_document_index_enqueue, mark_document_index_enqueue_failed, index_document_task, and is_organization_namespace intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1385-1395: The current except block may let
mark_document_index_enqueue_failed(...) raise and overwrite the original
broker/enqueue exception; modify the except that catches the enqueue error so
that you call mark_document_index_enqueue_failed(...) inside its own try/except,
logging any exception from that cleanup (e.g., logger.exception or logger.error
with context) but do NOT raise that cleanup exception — instead re-raise the
original exception variable (exc) so the root cause is preserved; reference the
existing mark_document_index_enqueue_failed and logger.error calls and ensure
the final action is "raise exc" (or simply "raise" after restoring) to propagate
the original error.
---
Outside diff comments:
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1834-1842: Capture the boolean result returned by
_schedule_indexing_celery(...) (e.g., scheduled =
self._schedule_indexing_celery(...)) and use it to determine the API response:
if scheduled is False, set the refresh response's success to False (and avoid
returning success=True) so a skipped enqueue does not report a successful
refresh; keep referencing _schedule_indexing_celery, document.is_active and the
response object that is returned from the refresh path.
---
Nitpick comments:
In `@backend/app/services/knowledge/orchestrator.py`:
- Around line 1257-1408: The _schedule_indexing_celery function is too large and
mixes validation, owner resolution, state-machine calls, Celery dispatch and
recovery; refactor by extracting its sub-parts into a new submodule (e.g.,
services/knowledge/index_orchestration.py) and reduce _schedule_indexing_celery
to a thin orchestrator that calls small helpers:
validate_retrieval_config(retrieval_config), resolve_index_owner(knowledge_base,
user, is_organization_namespace), enqueue_index_task(...) (which wraps
index_document_task.delay), and handle_enqueue_failure(...) (which calls
mark_document_index_enqueue_failed); then add tracing by annotating the
top-level sync entrypoint with
`@trace_sync`(span_name="orchestrator.schedule_indexing_celery",
tracer_name="knowledge", extract_attributes=...) so the function is under 50
lines and the file is split out of the oversized module while keeping references
to prepare_document_index_enqueue, mark_document_index_enqueue_failed,
index_document_task, and is_organization_namespace intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4961af96-7886-4839-84e0-f1dc40314314
📒 Files selected for processing (3)
backend/app/services/knowledge/orchestrator.pybackend/tests/services/knowledge/test_orchestrator.pybackend/tests/tasks/test_knowledge_tasks.py
✅ Files skipped from review due to trivial changes (1)
- backend/tests/services/knowledge/test_orchestrator.py
🚧 Files skipped from review as they are similar to previous changes (1)
- backend/tests/tasks/test_knowledge_tasks.py
Comment on lines
+1385
to
+1395
| except Exception as exc: | ||
| mark_document_index_enqueue_failed( | ||
| db=db, | ||
| document_id=document.id, | ||
| generation=generation, | ||
| ) | ||
| logger.error( | ||
| f"[Orchestrator] Failed to enqueue RAG indexing task for document {document.id}: {exc}", | ||
| exc_info=True, | ||
| ) | ||
| raise |
Contributor
There was a problem hiding this comment.
Don't let enqueue-failure cleanup hide the real exception.
If Line 1386 throws, it replaces the original broker/enqueue error and you lose the actual root cause. Wrap mark_document_index_enqueue_failed(...) in its own try/except, log that failure, then re-raise the original exc.
Suggested fix
except Exception as exc:
- mark_document_index_enqueue_failed(
- db=db,
- document_id=document.id,
- generation=generation,
- )
+ try:
+ mark_document_index_enqueue_failed(
+ db=db,
+ document_id=document.id,
+ generation=generation,
+ )
+ except Exception:
+ logger.exception(
+ "[Orchestrator] Failed to mark enqueue failure: "
+ f"document_id={document.id}, generation={generation}"
+ )
logger.error(
f"[Orchestrator] Failed to enqueue RAG indexing task for document {document.id}: {exc}",
exc_info=True,
)
raise🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@backend/app/services/knowledge/orchestrator.py` around lines 1385 - 1395, The
current except block may let mark_document_index_enqueue_failed(...) raise and
overwrite the original broker/enqueue exception; modify the except that catches
the enqueue error so that you call mark_document_index_enqueue_failed(...)
inside its own try/except, logging any exception from that cleanup (e.g.,
logger.exception or logger.error with context) but do NOT raise that cleanup
exception — instead re-raise the original exception variable (exc) so the root
cause is preserved; reference the existing mark_document_index_enqueue_failed
and logger.error calls and ensure the final action is "raise exc" (or simply
"raise" after restoring) to propagate the original error.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py (2)
27-82: Splitupgrade()into smaller helpers.
upgrade()is doing schema detection, column DDL, and data backfill in one 56-line function. Splitting improves readability and testability.As per coding guidelines: "Function length: Max 50 lines per function (preferred)".
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py` around lines 27 - 82, The upgrade() function is doing schema detection, DDL and backfill in one block; split it into small helpers: extract schema detection into a function (e.g., get_existing_columns(conn) or inspect_existing_columns using sa.inspect), move the column-add/alter logic into a helper (e.g., ensure_index_columns which calls op.add_column and op.alter_column for "index_status" and "index_generation"), and move the UPDATE backfill SQL into a separate backfill_indexing_state function that runs the op.execute(sa.text(...)) statement; have upgrade() simply get the bind, call these helpers in order so each helper is short and focused and names from the diff (upgrade, existing_columns, INDEX_STATUS_TYPE, op.add_column, op.alter_column, op.execute) can be located easily.
71-71: Consider dialect-aware BOOLEAN comparison for portability.Lines 71 and 76 use
is_active = 1for a BOOLEAN column. While this works in MySQL (where BOOLEAN is stored as TINYINT), it may fail in Postgres. The related migration20260313_a0b1c2d3e4f5_add_is_group_chat_to_tasks.pydemonstrates dialect-aware handling; consider a similar approach here or useIS TRUEfor better portability across databases.Suggested approach
UPDATE knowledge_documents SET index_status = CASE - WHEN is_active = 1 THEN 'success' + WHEN is_active IS TRUE THEN 'success' WHEN COALESCE(index_generation, 0) = 0 THEN 'not_indexed' ELSE index_status END, index_generation = CASE - WHEN is_active = 1 AND COALESCE(index_generation, 0) = 0 THEN 1 + WHEN is_active IS TRUE AND COALESCE(index_generation, 0) = 0 THEN 1 ELSE COALESCE(index_generation, 0) END🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py` at line 71, The CASE expression in the migration uses "is_active = 1" which is MySQL-specific and can break on Postgres; update the CASE/conditional to be dialect-aware by using a boolean-safe comparison (e.g., "is_active IS TRUE" or the same helper used in the other migration 20260313_a0b1c2d3e4f5_add_is_group_chat_to_tasks.py) so the CASE WHEN for the is_active column returns 'success'/'danger' portably; modify the CASE statement(s) referencing is_active in this migration (the CASE WHEN ... is_active ...) to use IS TRUE or the existing cross-dialect helper approach.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py`:
- Line 8: The migration has inconsistent ancestry: the module-level docstring
"Revises: a7b8c9d0e1f3" does not match the declared down_revision variable
(down_revision = "b1c2d3e4f5g6"); update the docstring to state the correct
parent revision "Revises: b1c2d3e4f5g6" so the declared down_revision and the
human-readable header match (look for the top-of-file docstring and the
down_revision assignment in this migration file and make them identical).
---
Nitpick comments:
In
`@backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.py`:
- Around line 27-82: The upgrade() function is doing schema detection, DDL and
backfill in one block; split it into small helpers: extract schema detection
into a function (e.g., get_existing_columns(conn) or inspect_existing_columns
using sa.inspect), move the column-add/alter logic into a helper (e.g.,
ensure_index_columns which calls op.add_column and op.alter_column for
"index_status" and "index_generation"), and move the UPDATE backfill SQL into a
separate backfill_indexing_state function that runs the op.execute(sa.text(...))
statement; have upgrade() simply get the bind, call these helpers in order so
each helper is short and focused and names from the diff (upgrade,
existing_columns, INDEX_STATUS_TYPE, op.add_column, op.alter_column, op.execute)
can be located easily.
- Line 71: The CASE expression in the migration uses "is_active = 1" which is
MySQL-specific and can break on Postgres; update the CASE/conditional to be
dialect-aware by using a boolean-safe comparison (e.g., "is_active IS TRUE" or
the same helper used in the other migration
20260313_a0b1c2d3e4f5_add_is_group_chat_to_tasks.py) so the CASE WHEN for the
is_active column returns 'success'/'danger' portably; modify the CASE
statement(s) referencing is_active in this migration (the CASE WHEN ...
is_active ...) to use IS TRUE or the existing cross-dialect helper approach.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f88f3108-cf22-4850-bdfb-a0e4740daeb0
📒 Files selected for processing (2)
backend/alembic/versions/20260326_b1c2d3e4f607_add_knowledge_document_index_state.pybackend/app/models/knowledge.py
🚧 Files skipped from review as they are similar to previous changes (1)
- backend/app/models/knowledge.py
FicoHu
pushed a commit
to FicoHu/Wegent
that referenced
this pull request
Apr 1, 2026
* fix(knowledge): harden document indexing state * fix(knowledge): harden indexing state transitions * fix(knowledge): tighten reindex review followups * fix(rag): KNOWLEDGE_INDEX_LOCK_MAX_RETRIES=1 * fix(knowledge): store index status without db enum * fix(alembic): correct migration revision metadata
FicoHu
pushed a commit
to FicoHu/Wegent
that referenced
this pull request
Apr 1, 2026
* fix(knowledge): harden document indexing state * fix(knowledge): harden indexing state transitions * fix(knowledge): tighten reindex review followups * fix(rag): KNOWLEDGE_INDEX_LOCK_MAX_RETRIES=1 * fix(knowledge): store index status without db enum * fix(alembic): correct migration revision metadata
qdaxb
added a commit
that referenced
this pull request
Apr 1, 2026
* feat: add workspace archive and recovery for Pod deletion Implement Pod recovery feature that allows users to continue conversations after executor Pods are deleted: - Add ArchiveInfo schema to store archive metadata in Task.status.archive - Add workspace_archive service for MinIO presigned URL generation - Add archive/restore endpoints in executor and executor_manager - Add ExecutorRecoveryService for Pod recreation with workspace restoration - Add skip_git_clone support to prevent git clone conflicts during restore - Add Celery Beat scheduled task for expired archive cleanup (30-day retention) - Integrate archive workflow in executor_job before Pod deletion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(backend): add missing minio dependency Add minio package to backend dependencies for workspace archive storage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(admin): add created_at field to device monitor list (#846) - Add created_at field to AdminDeviceInfo schema in backend - Add created_at to AdminDeviceInfo TypeScript interface - Display creation time in device list UI - Add i18n translations for created_at column (zh-CN and en) Co-authored-by: yunpeng7 <yunpeng7@staff.weibo.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * perf: use async heartbeat (#849) * fix(chat_shell): prevent orphaned tool call IDs in group chat history (#852) * fix(chat_shell): prevent tool call group splitting in group chat history truncation _truncate_history blindly sliced the flat message list, which could split assistant+tool_calls and their tool response messages across the truncation boundary. This produced orphaned function_call_output items without matching function_call entries in the OpenAI Responses API format, causing 'No tool call found for function call output' errors. Add _group_messages() to treat assistant+tool_calls+tool responses as atomic units, and modify _truncate_history() to truncate by groups instead of individual messages. * fix(chat_shell): ensure tool call group integrity across all compression paths Extract group_tool_call_messages() into shared messages/utils.py and apply atomic group handling to: - HistoryTruncationStrategy.compress(): adjust first/last boundaries to avoid splitting tool call groups, add post-removal orphan cleanup - _force_compression_to_target(): Step 2 removes middle groups instead of individual messages, Step 5 keeps first/last groups - Refactor loader.py to import shared utility instead of inline helper All 95 related tests pass. Comprehensive audit confirms no other code paths in chat_shell break tool call group integrity. * test(chat_shell): add tool-call group truncation regressions Add regression coverage for preserving assistant tool calls with their matching tool results across group-chat history truncation, history compression boundaries, and forced compression fallback paths. * fix: 在里面智能体的创建人和机器人的创建人不是一个人时会报错 (#853) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 在里面智能体的创建人和机器人的创建人不是一个人时会报错 * fix: 迁移其他的sql到kindReader --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fail fast on invalid tool call IDs across chat protocols (#854) * fix: 订阅任务执行间隔调整 & trace http接口的匹配规则调整 (#855) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: telemetry的匹配规则调整 * fix: 订阅任务执行间隔调整 * fix: 调整逻辑 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: sanitize PyInstaller LD_LIBRARY_PATH for child processes (#859) PyInstaller's bootloader prepends its extraction directory (/tmp/_MEIxxxxxx) to LD_LIBRARY_PATH, causing child processes (Claude Code, shell commands) to load wrong shared libraries (e.g., libtinfo.so.6). Extract `sanitize_ld_library_path()` into platform_compat and apply it in both local_mode_strategy (Claude SDK subprocess) and extension_handler (extension script subprocess) to restore the original library path. See: https://pyinstaller.org/en/stable/common-issues-and-pitfalls.html Co-authored-by: sifang <sifang@staff.weibo.com> * refactor(rag): move internal retrieve routing into backend (#847) * refactor(rag): move internal retrieve routing into backend * refactor(rag): move retrieve persistence into backend * refactor(rag): move kb head persistence into backend * fix(rag): tighten backend retrieve routing * refactor(rag): simplify token estimation fallback * refactor(rag): batch route token estimation * fix(rag): token estimator decimal * fix(rag): harden kb read persistence * frontend: show 'indexing' status for documents pending indexing (#858) * fix(frontend): show 'indexing' status for documents pending indexing When a document is uploaded, is_active is false while indexing is in progress. Previously, this showed 'index unavailable' which was misleading. Now documents with status='enabled' but is_active=false will show 'indexing' instead of 'index unavailable', matching the existing translation key. Changes: - Updated compact mode status indicator (line 247) - Updated normal mode status badge (line 461) Fixes: knowledge base document shows 'index unavailable' during indexing * fix(i18n): update Developer role description in Chinese locale * fix: 修复订阅器中脏数据导致的无法展示的问题 (#860) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复订阅器中脏数据导致的无法展示的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * feat(backend): add knowledge base RAG support to /v1/responses API (#844) * feat(backend): add knowledge base RAG support to /v1/responses API Implement knowledge base Q&A functionality for the OpenAI-compatible /v1/responses endpoint with the following changes: Schema Changes: - Add knowledge_base_names field to WegentTool schema - Support namespace#name format for KB specification New Services: - KnowledgeBaseNameResolver: resolves KB display names to IDs with permission checking via TaskKnowledgeBaseService - KnowledgeBaseContextCreator: creates SubtaskContext records for API-specified KBs to integrate with existing RAG pipeline Integration: - Extend parse_wegent_tools() to handle knowledge_base tool type - Add parse_knowledge_base_name() for namespace#name parsing - Modify build_execution_request() to accept knowledge_base_names - Add _create_kb_contexts_from_api_request() for KB context injection - Update openapi_responses.py endpoints to pass KB names parameter Testing: - Add comprehensive unit tests for KB name resolution - Add tests for KB context creation - Add tests for helper functions with KB support - All 32 new tests passing The implementation reuses the existing RAG flow through prepare_contexts_for_chat() and KnowledgeBaseTool, ensuring consistency with the web UI behavior. * refactor(kb_resolver): optimize database query complexity from O(n) to O(m) - Add batch_get_knowledge_base_by_names() to TaskKnowledgeBaseService for efficient batch queries - Extract _check_kb_access_permission() for unified permission checking logic - Update kb_resolver to use service methods instead of direct queries - Reduce DB queries from O(n) to O(m) where m is unique namespace count - Update tests to mock service methods instead of internal functions All 15 tests pass. * refactor(backend): optimize KB permission check in kb_resolver Simplify permission checking by using get_all_knowledge_bases_grouped() which already returns all KBs accessible to the user. Changes: - Replace _batch_query_kbs() and _check_kb_access_permission() with _get_accessible_kb_lookup() that builds a lookup dict from grouped results - Remove unused resolve_single() method and batch_get_knowledge_base_by_names() - Update tests to mock KnowledgeService.get_all_knowledge_bases_grouped() This reduces database queries and simplifies the permission check logic. * feat(backend): integrate knowledge base context creation into unified AI trigger - Add knowledge_base_names parameter to build_execution_request for OpenAPI v1/responses - Create _create_kb_contexts_from_api_request helper to process KB names from API requests - Integrate KB context processing into unified trigger flow - Add KnowledgeBaseContextCreator class for creating SubtaskContext records - Add comprehensive unit tests for KnowledgeBaseContextCreator This enables RAG functionality for OpenAPI v1/responses endpoint by creating SubtaskContext records that are processed by the existing context pipeline. * fix(chat): propagate HTTPException from KnowledgeBaseNameResolver in unified trigger - Add HTTPException import from fastapi - Catch and re-raise HTTPException explicitly in _create_kb_contexts_from_api_request - Keep generic exception handling for non-HTTP exceptions to preserve best-effort behavior - This ensures 403/404 errors from KB resolution are properly propagated to callers instead of being swallowed and logged as warnings * fix(chat_shell): use model_type for provider detection instead of model_id (#861) - Extract shared detect_provider() into models/providers.py with PROVIDER_ALIASES as single source of truth - model_type (env.model) is authoritative; raise ValueError on unknown - Remove _PROVIDER_ID_PATTERNS model_id prefix fallback (same model_id can serve multiple protocols) - Unify truncation notice logic for all providers (no system-role notice in middle of messages, preventing Anthropic non-consecutive system message errors) - Pass model_type from model_config through MessageCompressor to TokenCounter for accurate provider detection - Remove duplicate provider constants from factory.py and token_counter.py Co-authored-by: wenbo17 <wenbo17@staff.weibo.com> * ix: claude的子代理模式不可用修复 (#862) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: claude的子代理模式不可用修复 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * feat(inbox): implement work queue and message forwarding system (#851) * feat(inbox): implement work queue and message forwarding system Add a comprehensive message forwarding and work queue system that enables users to forward conversation messages to other users. Recipients can manage received messages through work queues and optionally configure AI agents for automatic processing. Backend changes: - Add QueueMessage and RecentContact database models - Add WorkQueue CRD schema stored in kinds table - Implement work queue services for CRUD operations - Implement message forwarding service with queue selection - Add WebSocket events for real-time notifications - Add API endpoints for work queue management - Add ForbiddenException to core exceptions Frontend changes: - Add inbox page with queue sidebar and message list - Implement ForwardMessageDialog for recipient selection - Add queue edit/create dialog with auto-processing config - Add message detail dialog with content preview - Add i18n translations (zh-CN and en) - Register inbox route and navigation * fix: resolve lint errors in inbox feature - Remove unused imports (RefreshCw, Settings, Menu, Button, Input, Clock) - Format Python files with Black * fix: resolve TypeScript errors in inbox feature - Add 'flow' and 'inbox' to TopNavigation activePage type - Replace Badge variant 'outline' with 'info' (outline not supported) * fix: address CodeRabbit review comments for work queue feature - Fix status parameter shadowing fastapi status module in work_queue.py - Fix NamespaceMember.namespace to use group_name column reference - Replace func.json_set with Python-level update for PostgreSQL compatibility - Fix refreshQueues dependency loop using functional state update - Remove Chinese text from inbox page comment - Add NaN validation for schedule interval input - Replace clickable Badge with accessible button for recent contacts --------- Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> * Wegent/work queue and message forwarding (#863) * feat(inbox): implement work queue and message forwarding system Add a comprehensive message forwarding and work queue system that enables users to forward conversation messages to other users. Recipients can manage received messages through work queues and optionally configure AI agents for automatic processing. Backend changes: - Add QueueMessage and RecentContact database models - Add WorkQueue CRD schema stored in kinds table - Implement work queue services for CRUD operations - Implement message forwarding service with queue selection - Add WebSocket events for real-time notifications - Add API endpoints for work queue management - Add ForbiddenException to core exceptions Frontend changes: - Add inbox page with queue sidebar and message list - Implement ForwardMessageDialog for recipient selection - Add queue edit/create dialog with auto-processing config - Add message detail dialog with content preview - Add i18n translations (zh-CN and en) - Register inbox route and navigation * fix: resolve lint errors in inbox feature - Remove unused imports (RefreshCw, Settings, Menu, Button, Input, Clock) - Format Python files with Black * fix: resolve TypeScript errors in inbox feature - Add 'flow' and 'inbox' to TopNavigation activePage type - Replace Badge variant 'outline' with 'info' (outline not supported) * fix: address CodeRabbit review comments for work queue feature - Fix status parameter shadowing fastapi status module in work_queue.py - Fix NamespaceMember.namespace to use group_name column reference - Replace func.json_set with Python-level update for PostgreSQL compatibility - Fix refreshQueues dependency loop using functional state update - Remove Chinese text from inbox page comment - Add NaN validation for schedule interval input - Replace clickable Badge with accessible button for recent contacts * feat: update work queue --------- Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> * fix: fix column default value * fix: fix forward message click * feat: remove unused area * feat: add inbox default message * fix: 修复脏数据导致的订阅任务无法执行的问题 (#865) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复脏数据导致的订阅任务无法执行的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix(browser): harden browser-tool linux recovery (#867) Co-authored-by: sifang <sifang@staff.weibo.com> * fix: 修复无法使用自定义镜像以及自定义镜像无法校验的问题 (#866) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复无法使用自定义镜像 * fix: 修复无法使用自定义镜像以及自定义镜像无法校验的问题 * fix: 修复可能的bug --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fix welcome timezone (#868) * docs: update inbox docs (#869) * fix: 修复单测 (#870) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复单测 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fix page error (#871) * fix: data-roleid="api-key-create-button" not in html attr (#873) * fix: 修复停止按钮偶发变成发送的问题 & 修复失败的消息会丢失的问题 (#876) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复停止按钮偶发变成发送的问题 * fix: 修复失败的消息会丢失的问题 * fix: 修复类型报错 * fix: 修复前端复制按钮内容错误的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: 修复“skill not found” (#877) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 技能错乱 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * chore(executor): upgrade claude-agent-sdk from 0.1.31 to 0.1.51 (#880) Co-authored-by: sifang <sifang@staff.weibo.com> * feat(frontend): add HTML attachment preview and share link generation (#872) - Add HtmlPreview component for previewing HTML files with iframe and source view modes - Update FilePreview to support HTML file type detection and rendering - Add createAttachmentShareLink API for generating public share links - Add share button to FilePreviewDialog and FilePreviewPage (visible for owners only) - Add Web Page type label and icon for HTML attachments in AttachmentCard - Fix preview page height to fill viewport using h-screen and h-full - Update AttachmentPreview to support HTML file preview button Fixes share link browsing and download for table-type attachments * fix(knowledge): harden document indexing state (#875) * fix(knowledge): harden document indexing state * fix(knowledge): harden indexing state transitions * fix(knowledge): tighten reindex review followups * fix(rag): KNOWLEDGE_INDEX_LOCK_MAX_RETRIES=1 * fix(knowledge): store index status without db enum * fix(alembic): correct migration revision metadata * docs: update contributors list (zh) (#879) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * docs: update contributors list (#878) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat: 支持skill认证 (#882) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * docs: add skill http identity design and plan * feat(skill): add http identity token flow * feat: 修改样式 * feat: 修改样式 * feat: 移除计划 * feat: 调整为带有认证的接口 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> # Conflicts: # executor_manager/services/sandbox/manager.py * Feature/knowledge document read (#888) * docs(backend): add knowledge document read spec and plan * feat(backend): add orchestrated knowledge document reads * fix(backend): tighten knowledge document read handling * refactor(web-scraper): add configuration for specific sites that require special handling (#884) * refactor(web-scraper): extract site-specific config to scraper_config.py - Add scraper_config.py for site-specific crawling configurations * fix(web_scraper): load WEB_SCRAPER_SITE_CONFIG from Pydantic settings instead of os.environ - Add WEB_SCRAPER_SITE_CONFIG to Settings class in config.py - Refactor scraper_config.py to use settings.WEB_SCRAPER_SITE_CONFIG with proper logging - Replace module-level config loading with cached get_site_config() function - Update scraper_service.py to use new get_site_config() function - Add detailed logging for site config matching and application - Fix remove_overlay_elements duplicate key in CrawlerRunConfig This ensures the configuration is properly loaded from .env file via Pydantic settings." * refactor(backend): improve scraper_config security and type safety - Remove sensitive config value from INFO level logging - Add return type annotations to _load_site_config and get_site_config - Replace broad Exception catch with specific exception types (AttributeError, KeyError, TypeError, ValueError) - Import Dict and Any from typing module * refactor(backend): replace auto-cleanup with archive expiration check Remove automatic MinIO archive cleanup (Celery Beat task) and instead block task execution when workspace archive has expired (>30 days). Users will receive a clear error message and must create a new task. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: 添加恢复行为测试和脚本路径修正 * fix: 使用 timezone.utc 替换 UTC 以确保时区正确 * test(backend): fix workspace recovery test patches --------- Co-authored-by: yunpeng7 <yunpeng7@staff.weibo.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: axb <qindi@staff.weibo.com> Co-authored-by: moqimoqidea <39821951+moqimoqidea@users.noreply.github.com> Co-authored-by: MicroLee <yudexiaozhou@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> Co-authored-by: icycrystal4 <icycrystal4@gmail.com> Co-authored-by: sifang <sifang@staff.weibo.com> Co-authored-by: Yanhe <yanhe1@staff.weibo.com> Co-authored-by: parabala <chenyong1@staff.weibo.com> Co-authored-by: wenbo17 <wenbo17@staff.weibo.com> Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> Co-authored-by: 李枨煊 <chengxuan@staff.weibo.com> Co-authored-by: johnny0120 <jiangyang7@staff.weibo.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
qdaxb
added a commit
that referenced
this pull request
Apr 3, 2026
* feat: add workspace archive and recovery for Pod deletion Implement Pod recovery feature that allows users to continue conversations after executor Pods are deleted: - Add ArchiveInfo schema to store archive metadata in Task.status.archive - Add workspace_archive service for MinIO presigned URL generation - Add archive/restore endpoints in executor and executor_manager - Add ExecutorRecoveryService for Pod recreation with workspace restoration - Add skip_git_clone support to prevent git clone conflicts during restore - Add Celery Beat scheduled task for expired archive cleanup (30-day retention) - Integrate archive workflow in executor_job before Pod deletion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(backend): add missing minio dependency Add minio package to backend dependencies for workspace archive storage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(admin): add created_at field to device monitor list (#846) - Add created_at field to AdminDeviceInfo schema in backend - Add created_at to AdminDeviceInfo TypeScript interface - Display creation time in device list UI - Add i18n translations for created_at column (zh-CN and en) Co-authored-by: yunpeng7 <yunpeng7@staff.weibo.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * perf: use async heartbeat (#849) * fix(chat_shell): prevent orphaned tool call IDs in group chat history (#852) * fix(chat_shell): prevent tool call group splitting in group chat history truncation _truncate_history blindly sliced the flat message list, which could split assistant+tool_calls and their tool response messages across the truncation boundary. This produced orphaned function_call_output items without matching function_call entries in the OpenAI Responses API format, causing 'No tool call found for function call output' errors. Add _group_messages() to treat assistant+tool_calls+tool responses as atomic units, and modify _truncate_history() to truncate by groups instead of individual messages. * fix(chat_shell): ensure tool call group integrity across all compression paths Extract group_tool_call_messages() into shared messages/utils.py and apply atomic group handling to: - HistoryTruncationStrategy.compress(): adjust first/last boundaries to avoid splitting tool call groups, add post-removal orphan cleanup - _force_compression_to_target(): Step 2 removes middle groups instead of individual messages, Step 5 keeps first/last groups - Refactor loader.py to import shared utility instead of inline helper All 95 related tests pass. Comprehensive audit confirms no other code paths in chat_shell break tool call group integrity. * test(chat_shell): add tool-call group truncation regressions Add regression coverage for preserving assistant tool calls with their matching tool results across group-chat history truncation, history compression boundaries, and forced compression fallback paths. * fix: 在里面智能体的创建人和机器人的创建人不是一个人时会报错 (#853) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 在里面智能体的创建人和机器人的创建人不是一个人时会报错 * fix: 迁移其他的sql到kindReader --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fail fast on invalid tool call IDs across chat protocols (#854) * fix: 订阅任务执行间隔调整 & trace http接口的匹配规则调整 (#855) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: telemetry的匹配规则调整 * fix: 订阅任务执行间隔调整 * fix: 调整逻辑 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: sanitize PyInstaller LD_LIBRARY_PATH for child processes (#859) PyInstaller's bootloader prepends its extraction directory (/tmp/_MEIxxxxxx) to LD_LIBRARY_PATH, causing child processes (Claude Code, shell commands) to load wrong shared libraries (e.g., libtinfo.so.6). Extract `sanitize_ld_library_path()` into platform_compat and apply it in both local_mode_strategy (Claude SDK subprocess) and extension_handler (extension script subprocess) to restore the original library path. See: https://pyinstaller.org/en/stable/common-issues-and-pitfalls.html Co-authored-by: sifang <sifang@staff.weibo.com> * refactor(rag): move internal retrieve routing into backend (#847) * refactor(rag): move internal retrieve routing into backend * refactor(rag): move retrieve persistence into backend * refactor(rag): move kb head persistence into backend * fix(rag): tighten backend retrieve routing * refactor(rag): simplify token estimation fallback * refactor(rag): batch route token estimation * fix(rag): token estimator decimal * fix(rag): harden kb read persistence * frontend: show 'indexing' status for documents pending indexing (#858) * fix(frontend): show 'indexing' status for documents pending indexing When a document is uploaded, is_active is false while indexing is in progress. Previously, this showed 'index unavailable' which was misleading. Now documents with status='enabled' but is_active=false will show 'indexing' instead of 'index unavailable', matching the existing translation key. Changes: - Updated compact mode status indicator (line 247) - Updated normal mode status badge (line 461) Fixes: knowledge base document shows 'index unavailable' during indexing * fix(i18n): update Developer role description in Chinese locale * fix: 修复订阅器中脏数据导致的无法展示的问题 (#860) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复订阅器中脏数据导致的无法展示的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * feat(backend): add knowledge base RAG support to /v1/responses API (#844) * feat(backend): add knowledge base RAG support to /v1/responses API Implement knowledge base Q&A functionality for the OpenAI-compatible /v1/responses endpoint with the following changes: Schema Changes: - Add knowledge_base_names field to WegentTool schema - Support namespace#name format for KB specification New Services: - KnowledgeBaseNameResolver: resolves KB display names to IDs with permission checking via TaskKnowledgeBaseService - KnowledgeBaseContextCreator: creates SubtaskContext records for API-specified KBs to integrate with existing RAG pipeline Integration: - Extend parse_wegent_tools() to handle knowledge_base tool type - Add parse_knowledge_base_name() for namespace#name parsing - Modify build_execution_request() to accept knowledge_base_names - Add _create_kb_contexts_from_api_request() for KB context injection - Update openapi_responses.py endpoints to pass KB names parameter Testing: - Add comprehensive unit tests for KB name resolution - Add tests for KB context creation - Add tests for helper functions with KB support - All 32 new tests passing The implementation reuses the existing RAG flow through prepare_contexts_for_chat() and KnowledgeBaseTool, ensuring consistency with the web UI behavior. * refactor(kb_resolver): optimize database query complexity from O(n) to O(m) - Add batch_get_knowledge_base_by_names() to TaskKnowledgeBaseService for efficient batch queries - Extract _check_kb_access_permission() for unified permission checking logic - Update kb_resolver to use service methods instead of direct queries - Reduce DB queries from O(n) to O(m) where m is unique namespace count - Update tests to mock service methods instead of internal functions All 15 tests pass. * refactor(backend): optimize KB permission check in kb_resolver Simplify permission checking by using get_all_knowledge_bases_grouped() which already returns all KBs accessible to the user. Changes: - Replace _batch_query_kbs() and _check_kb_access_permission() with _get_accessible_kb_lookup() that builds a lookup dict from grouped results - Remove unused resolve_single() method and batch_get_knowledge_base_by_names() - Update tests to mock KnowledgeService.get_all_knowledge_bases_grouped() This reduces database queries and simplifies the permission check logic. * feat(backend): integrate knowledge base context creation into unified AI trigger - Add knowledge_base_names parameter to build_execution_request for OpenAPI v1/responses - Create _create_kb_contexts_from_api_request helper to process KB names from API requests - Integrate KB context processing into unified trigger flow - Add KnowledgeBaseContextCreator class for creating SubtaskContext records - Add comprehensive unit tests for KnowledgeBaseContextCreator This enables RAG functionality for OpenAPI v1/responses endpoint by creating SubtaskContext records that are processed by the existing context pipeline. * fix(chat): propagate HTTPException from KnowledgeBaseNameResolver in unified trigger - Add HTTPException import from fastapi - Catch and re-raise HTTPException explicitly in _create_kb_contexts_from_api_request - Keep generic exception handling for non-HTTP exceptions to preserve best-effort behavior - This ensures 403/404 errors from KB resolution are properly propagated to callers instead of being swallowed and logged as warnings * fix(chat_shell): use model_type for provider detection instead of model_id (#861) - Extract shared detect_provider() into models/providers.py with PROVIDER_ALIASES as single source of truth - model_type (env.model) is authoritative; raise ValueError on unknown - Remove _PROVIDER_ID_PATTERNS model_id prefix fallback (same model_id can serve multiple protocols) - Unify truncation notice logic for all providers (no system-role notice in middle of messages, preventing Anthropic non-consecutive system message errors) - Pass model_type from model_config through MessageCompressor to TokenCounter for accurate provider detection - Remove duplicate provider constants from factory.py and token_counter.py Co-authored-by: wenbo17 <wenbo17@staff.weibo.com> * ix: claude的子代理模式不可用修复 (#862) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: claude的子代理模式不可用修复 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * feat(inbox): implement work queue and message forwarding system (#851) * feat(inbox): implement work queue and message forwarding system Add a comprehensive message forwarding and work queue system that enables users to forward conversation messages to other users. Recipients can manage received messages through work queues and optionally configure AI agents for automatic processing. Backend changes: - Add QueueMessage and RecentContact database models - Add WorkQueue CRD schema stored in kinds table - Implement work queue services for CRUD operations - Implement message forwarding service with queue selection - Add WebSocket events for real-time notifications - Add API endpoints for work queue management - Add ForbiddenException to core exceptions Frontend changes: - Add inbox page with queue sidebar and message list - Implement ForwardMessageDialog for recipient selection - Add queue edit/create dialog with auto-processing config - Add message detail dialog with content preview - Add i18n translations (zh-CN and en) - Register inbox route and navigation * fix: resolve lint errors in inbox feature - Remove unused imports (RefreshCw, Settings, Menu, Button, Input, Clock) - Format Python files with Black * fix: resolve TypeScript errors in inbox feature - Add 'flow' and 'inbox' to TopNavigation activePage type - Replace Badge variant 'outline' with 'info' (outline not supported) * fix: address CodeRabbit review comments for work queue feature - Fix status parameter shadowing fastapi status module in work_queue.py - Fix NamespaceMember.namespace to use group_name column reference - Replace func.json_set with Python-level update for PostgreSQL compatibility - Fix refreshQueues dependency loop using functional state update - Remove Chinese text from inbox page comment - Add NaN validation for schedule interval input - Replace clickable Badge with accessible button for recent contacts --------- Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> * Wegent/work queue and message forwarding (#863) * feat(inbox): implement work queue and message forwarding system Add a comprehensive message forwarding and work queue system that enables users to forward conversation messages to other users. Recipients can manage received messages through work queues and optionally configure AI agents for automatic processing. Backend changes: - Add QueueMessage and RecentContact database models - Add WorkQueue CRD schema stored in kinds table - Implement work queue services for CRUD operations - Implement message forwarding service with queue selection - Add WebSocket events for real-time notifications - Add API endpoints for work queue management - Add ForbiddenException to core exceptions Frontend changes: - Add inbox page with queue sidebar and message list - Implement ForwardMessageDialog for recipient selection - Add queue edit/create dialog with auto-processing config - Add message detail dialog with content preview - Add i18n translations (zh-CN and en) - Register inbox route and navigation * fix: resolve lint errors in inbox feature - Remove unused imports (RefreshCw, Settings, Menu, Button, Input, Clock) - Format Python files with Black * fix: resolve TypeScript errors in inbox feature - Add 'flow' and 'inbox' to TopNavigation activePage type - Replace Badge variant 'outline' with 'info' (outline not supported) * fix: address CodeRabbit review comments for work queue feature - Fix status parameter shadowing fastapi status module in work_queue.py - Fix NamespaceMember.namespace to use group_name column reference - Replace func.json_set with Python-level update for PostgreSQL compatibility - Fix refreshQueues dependency loop using functional state update - Remove Chinese text from inbox page comment - Add NaN validation for schedule interval input - Replace clickable Badge with accessible button for recent contacts * feat: update work queue --------- Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> * fix: fix column default value * fix: fix forward message click * feat: remove unused area * feat: add inbox default message * fix: 修复脏数据导致的订阅任务无法执行的问题 (#865) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复脏数据导致的订阅任务无法执行的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix(browser): harden browser-tool linux recovery (#867) Co-authored-by: sifang <sifang@staff.weibo.com> * fix: 修复无法使用自定义镜像以及自定义镜像无法校验的问题 (#866) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复无法使用自定义镜像 * fix: 修复无法使用自定义镜像以及自定义镜像无法校验的问题 * fix: 修复可能的bug --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fix welcome timezone (#868) * docs: update inbox docs (#869) * fix: 修复单测 (#870) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复单测 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: fix page error (#871) * fix: data-roleid="api-key-create-button" not in html attr (#873) * fix: 修复停止按钮偶发变成发送的问题 & 修复失败的消息会丢失的问题 (#876) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 修复停止按钮偶发变成发送的问题 * fix: 修复失败的消息会丢失的问题 * fix: 修复类型报错 * fix: 修复前端复制按钮内容错误的问题 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * fix: 修复“skill not found” (#877) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * fix: 技能错乱 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> * chore(executor): upgrade claude-agent-sdk from 0.1.31 to 0.1.51 (#880) Co-authored-by: sifang <sifang@staff.weibo.com> * feat(frontend): add HTML attachment preview and share link generation (#872) - Add HtmlPreview component for previewing HTML files with iframe and source view modes - Update FilePreview to support HTML file type detection and rendering - Add createAttachmentShareLink API for generating public share links - Add share button to FilePreviewDialog and FilePreviewPage (visible for owners only) - Add Web Page type label and icon for HTML attachments in AttachmentCard - Fix preview page height to fill viewport using h-screen and h-full - Update AttachmentPreview to support HTML file preview button Fixes share link browsing and download for table-type attachments * fix(knowledge): harden document indexing state (#875) * fix(knowledge): harden document indexing state * fix(knowledge): harden indexing state transitions * fix(knowledge): tighten reindex review followups * fix(rag): KNOWLEDGE_INDEX_LOCK_MAX_RETRIES=1 * fix(knowledge): store index status without db enum * fix(alembic): correct migration revision metadata * docs: update contributors list (zh) (#879) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * docs: update contributors list (#878) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat: 支持skill认证 (#882) * docs: update contributors list * docs: update contributors list (zh) * docs: update contributors list * docs: update contributors list (zh) * docs: add skill http identity design and plan * feat(skill): add http identity token flow * feat: 修改样式 * feat: 修改样式 * feat: 移除计划 * feat: 调整为带有认证的接口 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> # Conflicts: # executor_manager/services/sandbox/manager.py * Feature/knowledge document read (#888) * docs(backend): add knowledge document read spec and plan * feat(backend): add orchestrated knowledge document reads * fix(backend): tighten knowledge document read handling * refactor(web-scraper): add configuration for specific sites that require special handling (#884) * refactor(web-scraper): extract site-specific config to scraper_config.py - Add scraper_config.py for site-specific crawling configurations * fix(web_scraper): load WEB_SCRAPER_SITE_CONFIG from Pydantic settings instead of os.environ - Add WEB_SCRAPER_SITE_CONFIG to Settings class in config.py - Refactor scraper_config.py to use settings.WEB_SCRAPER_SITE_CONFIG with proper logging - Replace module-level config loading with cached get_site_config() function - Update scraper_service.py to use new get_site_config() function - Add detailed logging for site config matching and application - Fix remove_overlay_elements duplicate key in CrawlerRunConfig This ensures the configuration is properly loaded from .env file via Pydantic settings." * refactor(backend): improve scraper_config security and type safety - Remove sensitive config value from INFO level logging - Add return type annotations to _load_site_config and get_site_config - Replace broad Exception catch with specific exception types (AttributeError, KeyError, TypeError, ValueError) - Import Dict and Any from typing module * refactor(backend): replace auto-cleanup with archive expiration check Remove automatic MinIO archive cleanup (Celery Beat task) and instead block task execution when workspace archive has expired (>30 days). Users will receive a clear error message and must create a new task. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: 添加恢复行为测试和脚本路径修正 * fix: 使用 timezone.utc 替换 UTC 以确保时区正确 * test(backend): fix workspace recovery test patches * feat: 添加工作区归档配置测试与默认值 * feat: 添加工作区归档时区配置及测试 --------- Co-authored-by: yunpeng7 <yunpeng7@staff.weibo.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: axb <qindi@staff.weibo.com> Co-authored-by: moqimoqidea <39821951+moqimoqidea@users.noreply.github.com> Co-authored-by: MicroLee <yudexiaozhou@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: hongyu9 <hongyu9@staff.weibo.com> Co-authored-by: icycrystal4 <icycrystal4@gmail.com> Co-authored-by: sifang <sifang@staff.weibo.com> Co-authored-by: Yanhe <yanhe1@staff.weibo.com> Co-authored-by: parabala <chenyong1@staff.weibo.com> Co-authored-by: wenbo17 <wenbo17@staff.weibo.com> Co-authored-by: qdaxb <4157870+qdaxb@users.noreply.github.com> Co-authored-by: 李枨煊 <chengxuan@staff.weibo.com> Co-authored-by: johnny0120 <jiangyang7@staff.weibo.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
cocowh
pushed a commit
to cocowh/Wegent
that referenced
this pull request
May 11, 2026
fix(knowledge): harden document indexing state (wecode-ai#875) See merge request weibo_rd/common/wecode/wegent!1858
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Documentation
Localization