[Feature] Add 'datus package' command: export a self-contained project zip - #1262
Merged
Merged
Conversation
…t zip
Interactive wizard (datus package, -y for defaults) that packs the current
project into a zip the receiver can unzip + pip install + run, without
touching their ~/.datus. conf/agent.yml is regenerated (home: ., pinned
project_name, every secret field rewritten to ${VAR} placeholders) and a
final content scan fails the build on any real credential material.
Ships metric/semantic YAML sources with a generated rebuild_kb.sh, plugin
install commands from the activation list, README with the env-var list,
and a sha256 package manifest.
Also removes BLOB_REQUIRED from ask_report so it falls back to disk
artifact binding like ask_dashboard: datus-agent is not SaaS-aware, local
reports/<slug>/ directories are the source of truth, and no artifact_blob
injector exists in this repo — this also fixes ask_report being unusable
in standalone datus-api deployments. Blob injection still wins when a
host provides one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR adds a ChangesProject packaging
Artifact binding
Estimated code review effort: 5 (Critical) | ~120 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Test Audit ReportMode: diff-only (P0 blocks, P1 warns)
P1 Issues (warn-only)
Breakdown by rule
Static test-quality audit — rules from xUnit Test Patterns / Google Hermetic Tests / F.I.R.S.T. / Khorikov. Source: ci/audit_tests.py. To reproduce locally: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
datus/agent/node/base_artifact_ask_agentic_node.py (1)
554-592: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winModel
artifact_blobwith Pydantic.
artifact_blobcrosses the host-to-agent boundary and is currently parsed with manualAnychecks in_is_usable_blob()and_bind_artifact_from_blob(). DefineArtifactBlobFile/ArtifactBlobPydantic models, validate once when loading the agentic_nodes entry, and fall back to disk binding on validation errors. This follows the Python guideline to use type hints and Pydantic for data structures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/agent/node/base_artifact_ask_agentic_node.py` around lines 554 - 592, Define Pydantic models ArtifactBlobFile and ArtifactBlob for the injected artifact_blob shape, including manifest and non-empty files validation as needed. Validate artifact_blob once while loading the agentic_nodes entry, use the validated model in _bind_artifact_from_blob, and fall back to _bind_artifact_from_disk when validation fails; remove the manual Any-based validation in _is_usable_blob.Source: Coding guidelines
🧹 Nitpick comments (5)
datus/cli/package_builder.py (3)
1041-1050: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winShell-quote every interpolated value in the generated scripts. Both generators build bash command lines that the receiver executes, and neither escapes the interpolated values. A datasource directory, a YAML filename, or a plugin distribution name that contains
",$, a backtick, or whitespace breaks the script or injects a command. Useshlex.quoteon each interpolated value.
datus/cli/package_builder.py#L1041-L1050: wrapdsandrelwithshlex.quotein thebootstrap-kblines instead of relying on literal double quotes.datus/cli/package_builder.py#L1090-L1096: wrap thespecvalue withshlex.quotein thedatus plugin installline.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/cli/package_builder.py` around lines 1041 - 1050, Shell-quote every interpolated command value using shlex.quote in the generated script commands: update the bootstrap-kb lines at datus/cli/package_builder.py:1041-1050 to quote both ds and rel, and update the datus plugin install line at datus/cli/package_builder.py:1090-1096 to quote spec; replace reliance on literal double quotes with the escaped values while preserving the existing command structure.
1194-1216: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winHash the staged files in chunks.
entry.read_bytes()loads each staged file fully into memory to compute its SHA-256. The builder already warns at 100 MB per file, so a single read can allocate that much, andwrite_zipreads the same bytes again afterwards. A chunked hash keeps memory bounded.♻️ Proposed refactor
+_HASH_CHUNK_BYTES = 1024 * 1024 + + +def _sha256_of(entry: StagedEntry) -> str: + digest = hashlib.sha256() + if entry.content is not None: + digest.update(entry.content) + return digest.hexdigest() + assert entry.source is not None + with open(entry.source, "rb") as fh: + for chunk in iter(lambda: fh.read(_HASH_CHUNK_BYTES), b""): + digest.update(chunk) + return digest.hexdigest() + + def build_package_manifest( @@ files = [ - {"path": entry.arcname, "sha256": hashlib.sha256(entry.read_bytes()).hexdigest(), "size": entry.size()} + {"path": entry.arcname, "sha256": _sha256_of(entry), "size": entry.size()} for entry in sorted(entries, key=lambda e: e.arcname) ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/cli/package_builder.py` around lines 1194 - 1216, Update build_package_manifest to compute each file’s SHA-256 incrementally in chunks instead of calling StagedEntry.read_bytes(), while preserving the existing path, digest, and size fields. Reuse the staged-file access mechanism exposed by StagedEntry and ensure the hash loop bounds memory independently of file size.
749-756: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winApply the recursive secret sweep to provider entries too.
Datasources, BI platforms, schedulers, and channels all get a
_rewrite_secret_named_keyssweep after their explicit fields. Providers only getapi_key. A provider entry that carries another credential key, for examplesecret_key,token, oraccess_key_secret, passes through into the generatedconf/agent.ymluntouched. The final scan then fails the build withplaintext_secret_key, so nothing leaks, but the user has no way to package that project.♻️ Proposed change
providers = data.get("providers") if isinstance(providers, dict): for name, entry in providers.items(): if isinstance(entry, dict): # Conventional names (OPENAI_API_KEY, …) — matches agent.yml.example. _rewrite_secret_value( entry, "api_key", f"{_sanitize_var_component(name)}_API_KEY", f"providers.{name}.api_key", alloc ) + _rewrite_secret_named_keys( + entry, f"DATUS_PROVIDER_{_sanitize_var_component(name)}", f"providers.{name}", alloc + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/cli/package_builder.py` around lines 749 - 756, Apply the same recursive secret sweep used for datasources, BI platforms, schedulers, and channels to each provider entry in package_builder.py after the explicit api_key rewrite. Update the providers loop in the package-building flow so provider dicts call _rewrite_secret_named_keys with the correct provider path, while preserving the existing _rewrite_secret_value handling for api_key. Ensure any additional credential fields on providers such as secret_key, token, or access_key_secret are rewritten before the final plaintext_secret_key scan.tests/unit_tests/cli/test_package_builder.py (1)
394-456: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for
generate_mcp_json.The fixture never creates
{home}/conf/.mcp.json, sogenerate_mcp_jsonreturnsNonein every test. The header andenvplaceholder rewriting for MCP servers, plus theconf/.mcp.jsonmember and its entry in_scan_generated_config, are untested. Add one test that writes an.mcp.jsonwith a plaintextAuthorizationheader and asserts the staged member holds a${VAR}placeholder.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit_tests/cli/test_package_builder.py` around lines 394 - 456, Add a test in TestGeneratedFiles that creates the fixture’s home conf/.mcp.json with an MCP server containing a plaintext Authorization header, builds the package, and verifies the generated conf/.mcp.json member exists with that header rewritten to the expected ${VAR} placeholder. Also assert the member is included by _scan_generated_config.DatusPackage-review.md (1)
58-58: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value为代码块补充语言标识。
markdownlint 报告 MD040。此处的目录树代码块没有语言标识。使用
text可以消除告警。♻️ 建议修改
-``` +```text sales-project.zip🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@DatusPackage-review.md` at line 58, Update the directory-tree markdown code block in DatusPackage-review.md to include a language identifier by changing the fenced block around the sales-project.zip tree to use text. Keep the existing block content unchanged and apply the fix to the code fence itself so markdownlint MD040 is satisfied.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@datus/cli/package_builder.py`:
- Around line 1250-1267: Update scan_for_secrets to detect when entry.read_bytes
reaches _SCAN_READ_CAP_BYTES and record a truncation warning instead of silently
scanning only the prefix. Propagate these warnings through _build_package
alongside the existing warnings, while preserving current secret findings and
generated-config scanning behavior.
- Around line 25-28: Update the module docstring’s configuration guidance to
reference the actual extension points, _sanitize_agent_tree and
_SECRET_KEY_NAMES, instead of the nonexistent _SECRET_PATH_TABLE symbol.
- Around line 689-703: Update the URI handling around _URI_CREDENTIAL_RE and
_rewrite_secret_value so passwordless userinfo URIs such as user@host are
preserved rather than replaced wholesale. Only invoke whole-value replacement
when the URI contains an actual password component; otherwise leave the original
URI unchanged and continue allocating or preserving credentials only when
_URI_CREDENTIAL_RE identifies a password.
- Around line 588-605: Update _stage_index_html to validate that report_dist /
_DIST_CSS_NAME and report_dist / _DIST_JS_NAME both exist before rewriting
index.html or returning asset StagedEntry objects. If either file is missing,
raise PackageError with a clear message identifying the missing dist asset,
while preserving the existing staged-as-is behavior for unreadable HTML or
absent CDN URLs.
- Around line 1270-1278: Update write_zip to use one fixed timestamp for every
archive member instead of calling datetime.now(timezone.utc) inside the entry
loop. Reuse the build timestamp already carried by package_manifest.json,
passing it into or otherwise making it available to write_zip, and apply that
same value when constructing each ZipInfo.
- Around line 899-911: Guard the plugin_config_schema lookup and secret-field
extraction inside the plugin-processing loop so any registry or malformed-spec
exception is handled without escaping _build_package. On failure, treat the
schema as unavailable by leaving secret_fields unset, preserve the
all-string-leaves fallback, and emit the existing warning through the
established logging path.
In `@datus/cli/package_cli.py`:
- Line 8: Update the module docstring in package_cli.py to use English only by
replacing the Chinese character in the section reference with the numeric
equivalent, keeping the rest of the docstring unchanged. Anchor the edit on the
top-level docstring near the ``DatusPackage-review.md`` reference and ensure the
text reads with 3 instead of 三.
- Line 90: The helper interfaces around _run_wizard and the downstream helpers
at the referenced locations lack complete typing. Add Python 3.12+ annotations
for raw’s key/value types and replace untyped pb, options, and result parameters
or returns with the appropriate builder and result contract types, covering each
helper signature without changing runtime behavior.
In `@DatusPackage-review.md`:
- Line 76: 将 README.md 的目录注释“env 清单 + 四步快速启动”更新为“五步快速启动”，使其与第 14 行和第 141
行描述的验收流程保持一致。
---
Outside diff comments:
In `@datus/agent/node/base_artifact_ask_agentic_node.py`:
- Around line 554-592: Define Pydantic models ArtifactBlobFile and ArtifactBlob
for the injected artifact_blob shape, including manifest and non-empty files
validation as needed. Validate artifact_blob once while loading the
agentic_nodes entry, use the validated model in _bind_artifact_from_blob, and
fall back to _bind_artifact_from_disk when validation fails; remove the manual
Any-based validation in _is_usable_blob.
---
Nitpick comments:
In `@datus/cli/package_builder.py`:
- Around line 1041-1050: Shell-quote every interpolated command value using
shlex.quote in the generated script commands: update the bootstrap-kb lines at
datus/cli/package_builder.py:1041-1050 to quote both ds and rel, and update the
datus plugin install line at datus/cli/package_builder.py:1090-1096 to quote
spec; replace reliance on literal double quotes with the escaped values while
preserving the existing command structure.
- Around line 1194-1216: Update build_package_manifest to compute each file’s
SHA-256 incrementally in chunks instead of calling StagedEntry.read_bytes(),
while preserving the existing path, digest, and size fields. Reuse the
staged-file access mechanism exposed by StagedEntry and ensure the hash loop
bounds memory independently of file size.
- Around line 749-756: Apply the same recursive secret sweep used for
datasources, BI platforms, schedulers, and channels to each provider entry in
package_builder.py after the explicit api_key rewrite. Update the providers loop
in the package-building flow so provider dicts call _rewrite_secret_named_keys
with the correct provider path, while preserving the existing
_rewrite_secret_value handling for api_key. Ensure any additional credential
fields on providers such as secret_key, token, or access_key_secret are
rewritten before the final plaintext_secret_key scan.
In `@DatusPackage-review.md`:
- Line 58: Update the directory-tree markdown code block in
DatusPackage-review.md to include a language identifier by changing the fenced
block around the sales-project.zip tree to use text. Keep the existing block
content unchanged and apply the fix to the code fence itself so markdownlint
MD040 is satisfied.
In `@tests/unit_tests/cli/test_package_builder.py`:
- Around line 394-456: Add a test in TestGeneratedFiles that creates the
fixture’s home conf/.mcp.json with an MCP server containing a plaintext
Authorization header, builds the package, and verifies the generated
conf/.mcp.json member exists with that header rewritten to the expected ${VAR}
placeholder. Also assert the member is included by _scan_generated_config.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1d3b6a64-b25c-450c-94e1-ce0cd8e87449
📒 Files selected for processing (12)
DatusPackage-review.mddatus/agent/node/ask_report_agentic_node.pydatus/agent/node/base_artifact_ask_agentic_node.pydatus/cli/main.pydatus/cli/package_builder.pydatus/cli/package_cli.pydatus/plugins/store.pytests/unit_tests/agent/node/test_ask_artifact_agentic_node.pytests/unit_tests/cli/test_main.pytests/unit_tests/cli/test_package_builder.pytests/unit_tests/cli/test_package_cli.pytests/unit_tests/plugins/test_store.py
💤 Files with no reviewable changes (1)
- datus/agent/node/ask_report_agentic_node.py
… list
Found while packaging a real project (baisheng): datasource host/port/
database placeholders never pass through the secret-path rewriters, so the
generated README's required-env table missed them and the receiver could
not connect. After sanitization, walk the generated agent.yml (and
.mcp.json) and record every remaining ${VAR} / ${VAR:-default} occurrence
— whole-value or embedded — as a preexisting binding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
📊 PR Test Coverage Report
Coverage DetailsDiff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
datus/agent/agent.pyLines 672-681 672 sql_summary_dir, "SQL summary directory", force=force
673 ):
674 return {"status": "cancelled", "message": "User cancelled deletion of SQL summary directory"}
675 self.global_config.save_storage_config("reference_sql")
! 676 elif kb_update_strategy == "overwrite":
! 677 self.global_config.save_storage_config("reference_sql")
678 else:
679 self.global_config.check_init_storage_config("reference_sql")
680
681 # Initialize reference SQL storageLines 682-694 682 from datus.storage.reference_sql import ReferenceSqlRAG
683 from datus.storage.reference_sql.reference_sql_init import init_reference_sql
684
685 self.reference_sql_store = ReferenceSqlRAG(self.global_config)
! 686 if from_summaries:
687 # Re-index the committed summary YAML verbatim — no LLM.
! 688 from datus.storage.reference_sql.reference_sql_init import init_reference_sql_from_summaries
689
! 690 results[component] = init_reference_sql_from_summaries(
691 self.reference_sql_store,
692 self.global_config,
693 summaries_dir=getattr(self.args, "summaries_dir", "") or "",
694 build_mode=kb_update_strategy,Lines 692-700 692 self.global_config,
693 summaries_dir=getattr(self.args, "summaries_dir", "") or "",
694 build_mode=kb_update_strategy,
695 )
! 696 continue
697
698 if kb_update_strategy == "overwrite":
699 self.reference_sql_store.truncate()
700 self._reset_reference_sql_stream_state()datus/cli/_cli_utils.pyLines 37-45 37
38
39 def _raise_if_cancelled(result):
40 if result is _CANCELLED:
! 41 raise KeyboardInterrupt
42 return result
43
44
45 def prompt_with_back(label: str, default: str = "", password: bool = False) -> str:Lines 332-340 332
333 except (KeyboardInterrupt, EOFError):
334 print_warning(console, "\nInput cancelled")
335 if cancellable:
! 336 raise KeyboardInterrupt from None
337 return default
338 except Exception as e:
339 logger.error(f"Interactive select error: {e}")
340 print_error(console, f"Selection error: {str(e)}")Lines 408-420 408
409 def _depth(self, key: str) -> int:
410 depth, seen = 0, {key}
411 while key in self._parent_of:
! 412 key = self._parent_of[key]
! 413 if key in seen: # malformed hierarchy — stop instead of looping
! 414 break
! 415 seen.add(key)
! 416 depth += 1
417 return depth
418
419 def _sync_parents(self) -> None:
420 # Deepest first, so a grandparent sees its children's fresh state.Lines 616-624 616
617 except (KeyboardInterrupt, EOFError):
618 print_warning(console, "\nInput cancelled")
619 if cancellable:
! 620 raise KeyboardInterrupt from None
621 return []
622 except Exception as e:
623 logger.error(f"Interactive multi-select error: {e}")
624 print_error(console, f"Multi-select error: {str(e)}")Lines 916-922 916 """
917 console.print(f"[bold]{message}[/bold]")
918 choices = {"y": "Yes", "n": "No"}
919 default_key = "y" if default else "n"
! 920 result = select_choice(console, choices, default=default_key, cancellable=cancellable)
921 return result == "y"datus/cli/package_builder.pyLines 282-297 282 leak into the generated file. Returns ``None`` when no config exists.
283 """
284 try:
285 path = parse_config_path("")
! 286 except DatusException:
! 287 return None
288 try:
289 with open(path, "r", encoding="utf-8") as fh:
290 data = yaml.safe_load(fh) or {}
! 291 except (OSError, yaml.YAMLError) as exc:
! 292 logger.warning("package: cannot read agent config %s: %s", path, exc)
! 293 return None
294 agent = data.get("agent")
295 return agent if isinstance(agent, dict) else None
296 Lines 299-315 299 """Project name to pin into the package: override > agent.yml > CWD-derived."""
300 override = load_project_override(cwd=str(root))
301 if override is not None and override.project_name:
302 return override.project_name
! 303 raw_name = raw.get("project_name")
! 304 if isinstance(raw_name, str) and raw_name.strip():
! 305 try:
306 # Private-by-convention validator; packaging must apply the same
307 # shape rule the loader does or the receiver diverges.
! 308 return _validate_project_name(raw_name.strip())
! 309 except DatusException:
! 310 logger.warning("package: agent.yml project_name %r invalid; deriving from CWD", raw_name)
! 311 return _normalize_project_name(str(root))
312
313
314 def resolve_source_home(raw: Dict[str, Any], root: Path) -> Path:
315 """The *source* project's ``home`` (where templates live)."""Lines 316-326 316 home = raw.get("home")
317 if isinstance(home, str) and home.strip():
318 expanded = Path(home).expanduser()
319 if not expanded.is_absolute():
! 320 expanded = (root / expanded).resolve()
321 return expanded
! 322 return DatusPathManager.resolve_home(None)
323
324
325 def list_subagents(raw: Dict[str, Any]) -> Dict[str, str]:
326 """``{name: description}`` for every ``agentic_nodes`` entry."""Lines 340-348 340 path that does NOT follow ``home: .`` on the receiver."""
341 out: Dict[str, Path] = {}
342 for base in (Path.home() / ".datus" / "skills", root / ".datus" / "skills"):
343 if not base.is_dir():
! 344 continue
345 for entry in sorted(base.iterdir()):
346 if entry.is_dir() and (entry / "SKILL.md").is_file():
347 out[entry.name] = entry
348 return outLines 396-404 396 def _summary_subject_values(root: Path) -> List[str]:
397 """``subject_tree`` of every committed reference-SQL summary."""
398 base = root / _SQL_SUMMARIES_REL
399 if not base.is_dir():
! 400 return []
401 return [str(_read_yaml_mapping(path).get("subject_tree") or "") for path in sorted(base.rglob("*.y*ml"))]
402
403
404 def _metric_subject_values(root: Path) -> List[str]:Lines 411-419 411
412 def _metric_yaml_files(root: Path) -> List[Path]:
413 base = root / "subject" / "semantic_models"
414 if not base.is_dir():
! 415 return []
416 return sorted(p for p in base.rglob("*.y*ml") if p.is_file() and "metrics" in p.relative_to(base).parts[:-1])
417
418
419 def _metric_subject_path(doc: Any) -> str:Lines 424-432 424 shorter form.
425 """
426 metric = doc.get("metric") if isinstance(doc, dict) else None
427 if not isinstance(metric, dict):
! 428 return ""
429 locked = metric.get("locked_metadata")
430 tag_lists = [metric.get("tags"), locked.get("tags") if isinstance(locked, dict) else None]
431 for tags in tag_lists:
432 for tag in tags or []:Lines 460-470 460 roots: List[str] = []
461 try:
462 with open(path, "r", encoding="utf-8") as fh:
463 docs = list(yaml.safe_load_all(fh))
! 464 except (OSError, yaml.YAMLError) as exc:
! 465 logger.warning("package: unreadable metric yaml %s: %s", path, exc)
! 466 return roots
467 for doc in docs:
468 subject_path = _metric_subject_path(doc)
469 if subject_path:
470 roots.append(subject_path)Lines 481-491 481 document is kept, so the caller can ship the file untouched.
482 """
483 try:
484 text = path.read_text(encoding="utf-8")
! 485 except OSError as exc:
! 486 logger.warning("package: unreadable metric yaml %s: %s", path, exc)
! 487 return None
488
489 wanted = set(selected_subjects)
490 chunks = _split_yaml_documents(text)
491 kept: List[str] = []Lines 491-501 491 kept: List[str] = []
492 for chunk in chunks:
493 try:
494 doc = yaml.safe_load(chunk)
! 495 except yaml.YAMLError:
! 496 kept.append(chunk) # unparseable: keep rather than silently drop
! 497 continue
498 subject_path = _metric_subject_path(doc)
499 # Untagged metrics belong to no subject and would match no selection;
500 # keep them rather than dropping them from every filtered package.
501 if not subject_path or _subject_matches(subject_path, wanted):Lines 500-508 500 # keep them rather than dropping them from every filtered package.
501 if not subject_path or _subject_matches(subject_path, wanted):
502 kept.append(chunk)
503 if len(kept) == len(chunks):
! 504 return None
505 return ("\n---\n".join(kept) + "\n").encode("utf-8")
506
507
508 def _read_yaml_mapping(path: Path) -> Dict[str, Any]:Lines 508-518 508 def _read_yaml_mapping(path: Path) -> Dict[str, Any]:
509 try:
510 with open(path, "r", encoding="utf-8") as fh:
511 doc = yaml.safe_load(fh)
! 512 except (OSError, yaml.YAMLError) as exc:
! 513 logger.warning("package: unreadable yaml %s: %s", path, exc)
! 514 return {}
515 return doc if isinstance(doc, dict) else {}
516
517
518 def _vector_db_subject_roots(root: Path, raw: Dict[str, Any], project_name: str) -> Dict[str, str]:Lines 528-536 528 from datus.utils.path_manager import get_path_manager, set_current_path_manager
529
530 datasources = list(((raw.get("services") or {}).get("datasources")) or {})
531 if not datasources:
! 532 return {}
533 try:
534 previous = get_path_manager()
535 except Exception: # pragma: no cover - no context installed yet
536 previous = NoneLines 545-566 545 for datasource in datasources:
546 try:
547 store = SubjectTreeStore(project=project_name, datasource_id=str(datasource))
548 for node in store.get_children(None) or []:
! 549 name = str(node.get("name") or "").strip()
! 550 if not name:
! 551 continue
! 552 roots.setdefault(name, str(node.get("description") or ""))
553 # Second level: a root alone is usually too coarse to be a
554 # useful selection (baisheng keeps all 22 metrics under one).
! 555 for child in store.get_children(node.get("node_id")) or []:
! 556 child_name = str(child.get("name") or "").strip()
! 557 if child_name:
! 558 roots.setdefault(f"{name}/{child_name}", str(child.get("description") or ""))
! 559 except Exception as exc:
! 560 logger.debug("package: subject tree unavailable for %s: %s", datasource, exc)
! 561 except Exception as exc:
! 562 logger.warning("package: cannot read the subject tree from the vector store: %s", exc)
563 finally:
564 if previous is not None:
565 set_current_path_manager(previous)
566 return rootsLines 597-605 597 parts.append(f"{metric_counts[path]} metrics")
598 if sql_counts.get(path):
599 parts.append(f"{sql_counts[path]} reference SQL")
600 if not parts:
! 601 parts.append("no packaged entries")
602 description = tree_nodes.get(path) or ""
603 suffix = f" — {description}" if description else ""
604 indent = " └ " if depth else ""
605 labels[path] = f"{indent}{leaf} ({', '.join(parts)}){suffix}"Lines 651-662 651 if override is not None and override.default_datasource:
652 return override.default_datasource
653 datasources = ((raw.get("services") or {}).get("datasources")) or {}
654 if not isinstance(datasources, dict) or not datasources:
! 655 return None
656 for name, entry in datasources.items():
657 if isinstance(entry, dict) and entry.get("default"):
! 658 return str(name)
659 return str(next(iter(datasources))) if len(datasources) == 1 else None
660
661
662 def list_metric_datasources(root: Path) -> List[str]:Lines 661-669 661
662 def list_metric_datasources(root: Path) -> List[str]:
663 base = root / "subject" / "semantic_models"
664 if not base.is_dir():
! 665 return []
666 return sorted(p.name for p in base.iterdir() if p.is_dir())
667
668
669 # kind → (top-level dir name, per-prefix walker allowlist). The single mapLines 678-686 678 """Slugs with a ``manifest.json`` under ``reports/`` / ``dashboards/``."""
679 kind_dir, _ = _ARTIFACT_KIND_DIRS[kind]
680 base = root / kind_dir
681 if not base.is_dir():
! 682 return []
683 slugs = []
684 for entry in sorted(base.iterdir()):
685 if entry.is_dir() and ARTIFACT_SLUG_RE.fullmatch(entry.name) and (entry / "manifest.json").is_file():
686 slugs.append(entry.name)Lines 739-748 739 # Selector-owned: metric / reference-SQL selection stages
740 # these subtrees, filtered by the chosen subject roots.
741 pruned.append(d)
742 elif (Path(dirpath) / d).is_symlink():
! 743 warnings.append(f"skipped symlinked directory: {(rel_dir / d).as_posix()}")
! 744 pruned.append(d)
745 for d in pruned:
746 dirnames.remove(d)
747 dirnames.sort()Lines 757-766 757 continue
758 try:
759 if fpath.resolve() == output_resolved:
760 continue
! 761 except OSError:
! 762 continue
763 if include_res and not any(p.search(rel) for p in include_res):
764 continue
765 if any(p.search(rel) for p in exclude_res):
766 continueLines 766-783 766 continue
767 if fpath.is_symlink():
768 try:
769 resolved = fpath.resolve(strict=True)
! 770 except OSError:
! 771 warnings.append(f"skipped broken symlink: {rel}")
! 772 continue
773 if not resolved.is_relative_to(root):
774 warnings.append(f"skipped symlink escaping project root: {rel}")
775 continue
! 776 entries.append(StagedEntry(arcname=rel, source=resolved))
! 777 continue
778 if not fpath.is_file():
! 779 continue
780 entries.append(StagedEntry(arcname=rel, source=fpath))
781 return entries, warnings
782 Lines 817-825 817 entries.append(StagedEntry(arcname=f"template/{template.name}", source=template))
818 elif entry.get("system_prompt"):
819 # Built-in node names fall back to packaged templates; a custom
820 # system_prompt with no template file will fail on the receiver.
! 821 warnings.append(f"subagent {name!r}: template {template.name} not found under {template_dir}")
822 return kept, entries, warnings
823
824
825 def select_skills(Lines 915-923 915 disappearing from every package.
916 """
917 base = root / _SQL_SUMMARIES_REL
918 if not base.is_dir():
! 919 return [], 0, []
920 wanted = set(selected_subjects)
921 entries: List[StagedEntry] = []
922 warnings: List[str] = []
923 for path in sorted(base.rglob("*.y*ml")):Lines 921-929 921 entries: List[StagedEntry] = []
922 warnings: List[str] = []
923 for path in sorted(base.rglob("*.y*ml")):
924 if not path.is_file() or _is_junk_path(path.relative_to(base)):
! 925 continue
926 subject_path = str(_read_yaml_mapping(path).get("subject_tree") or "")
927 if not _subject_segments(subject_path):
928 warnings.append(f"{path.name}: no subject_tree — packaged regardless of the subject selection")
929 elif not _subject_matches(subject_path, wanted):Lines 938-946 938 resolved_root = artifact_dir.resolve()
939 for sub, (suffixes, recursive) in dirs_spec.items():
940 base = artifact_dir / sub
941 if not base.is_dir():
! 942 continue
943 iterator = base.rglob("*") if recursive else base.iterdir()
944 for path in iterator:
945 if not path.is_file() or _is_junk_path(path.relative_to(base)):
946 continueLines 948-957 948 if not any(name_lower.endswith(sfx) for sfx in suffixes):
949 continue
950 try:
951 path.resolve().relative_to(resolved_root)
! 952 except ValueError:
! 953 continue # symlink escaping the artifact — drop
954 out.append(path)
955 return sorted(out)
956 Lines 986-995 986 staged_assets = False
987 for asset_name in (_DIST_CSS_NAME, _DIST_JS_NAME):
988 asset = assets_dir / asset_name
989 if asset.is_file():
! 990 entries.append(StagedEntry(arcname=f"{kind_dir}/{slug}/_assets/{asset_name}", source=asset))
! 991 staged_assets = True
992 index_html = artifact_dir / "index.html"
993 if index_html.is_file():
994 entries.extend(_stage_index_html(index_html, kind, kind_dir, slug, report_dist, staged_assets, warnings))
995 return kept, entries, warningsLines 1011-1024 1011 from datus.agent.node.visual_artifact._artifact_html_renderer import CDN_BUNDLE_CSS, CDN_BUNDLE_JS
1012
1013 try:
1014 html = index_html.read_text(encoding="utf-8")
! 1015 except OSError as exc:
! 1016 warnings.append(f"{arc}: unreadable ({exc}); staged as-is")
! 1017 return [StagedEntry(arcname=arc, source=index_html)]
1018 if CDN_BUNDLE_CSS not in html and CDN_BUNDLE_JS not in html:
! 1019 warnings.append(f"{arc}: no CDN bundle URLs found; staged as-is")
! 1020 return [StagedEntry(arcname=arc, source=index_html)]
1021 missing = [name for name in (_DIST_CSS_NAME, _DIST_JS_NAME) if not (report_dist / name).is_file()]
1022 if missing:
1023 # The CLI validates the dist dir up front, but the builder API can be
1024 # handed any path — never rewrite to assets we cannot actually ship.Lines 1098-1107 1098 for key, value in tree.items():
1099 child = f"{config_path}.{key}" if config_path else str(key)
1100 self.harvest(value, child)
1101 elif isinstance(tree, list):
! 1102 for idx, item in enumerate(tree):
! 1103 self.harvest(item, f"{config_path}[{idx}]")
1104 elif isinstance(tree, str):
1105 for var in _PLACEHOLDER_ANY_RE.findall(tree):
1106 if var not in self._used:
1107 self._record(var, config_path, preexisting=True)Lines 1136-1145 1136 match = _URI_CREDENTIAL_RE.match(value)
1137 if match:
1138 pwd = match.group("pwd")
1139 if _PLACEHOLDER_RE.match(pwd):
! 1140 alloc.keep_preexisting(pwd, config_path)
! 1141 return
1142 placeholder = alloc.allocate(pwd, preferred_var, config_path)
1143 container[key] = f"{match.group('prefix')}{placeholder}{match.group('suffix')}"
1144 elif re.search(r"://[^/@]+:[^@]+@", value):
1145 # A password component IS present but the URI doesn't parse —Lines 1172-1182 1172 elif isinstance(value, str) and (all_string_leaves or _is_secret_key(key)):
1173 _rewrite_secret_value(
1174 tree, key, f"{scope_var_prefix}_{_sanitize_var_component(key)}", child_path, alloc
1175 )
! 1176 elif isinstance(tree, list):
! 1177 for idx, item in enumerate(tree):
! 1178 _rewrite_secret_named_keys(
1179 item,
1180 f"{scope_var_prefix}_{idx}",
1181 f"{config_path}[{idx}]",
1182 alloc,Lines 1204-1212 1204 models = data.get("models")
1205 if isinstance(models, dict):
1206 for name, entry in models.items():
1207 if not isinstance(entry, dict):
! 1208 continue
1209 prefix = f"DATUS_MODEL_{_sanitize_var_component(name)}"
1210 _rewrite_secret_value(entry, "api_key", f"{prefix}_API_KEY", f"models.{name}.api_key", alloc)
1211 headers = entry.get("default_headers")
1212 if isinstance(headers, dict):Lines 1224-1232 1224 datasources = services.get("datasources")
1225 if isinstance(datasources, dict):
1226 for name, entry in datasources.items():
1227 if not isinstance(entry, dict):
! 1228 continue
1229 prefix = f"DATUS_DS_{_sanitize_var_component(name)}"
1230 path = f"services.datasources.{name}"
1231 for fld in ("password", "username", "account", "private_key_file_pwd"):
1232 _rewrite_secret_value(Lines 1237-1248 1237 _rewrite_secret_named_keys(entry, prefix, path, alloc)
1238 for section, scope in (("bi_platforms", "DATUS_BI"), ("schedulers", "DATUS_SCHEDULER")):
1239 block = services.get(section)
1240 if not isinstance(block, dict):
! 1241 continue
1242 for name, entry in block.items():
1243 if not isinstance(entry, dict):
! 1244 continue
1245 prefix = f"{scope}_{_sanitize_var_component(name)}"
1246 path = f"services.{section}.{name}"
1247 for fld in ("password", "username", "api_key", "token"):
1248 _rewrite_secret_value(Lines 1250-1263 1250 )
1251 _rewrite_secret_named_keys(entry, prefix, path, alloc)
1252 semantic = services.get("semantic_layer")
1253 if isinstance(semantic, dict):
! 1254 _rewrite_secret_named_keys(semantic, "DATUS_SEMANTIC", "services.semantic_layer", alloc)
1255 mcp = services.get("mcp_servers")
1256 if isinstance(mcp, dict):
1257 for name, entry in mcp.items():
1258 if not isinstance(entry, dict):
! 1259 continue
1260 prefix = f"DATUS_MCP_{_sanitize_var_component(name)}"
1261 headers = entry.get("headers")
1262 if isinstance(headers, dict):
1263 for header in list(headers):Lines 1273-1293 1273 _rewrite_secret_named_keys(env, f"{prefix}_ENV", f"services.mcp_servers.{name}.env", alloc)
1274
1275 plugins = data.get("plugins")
1276 if isinstance(plugins, dict):
! 1277 _sanitize_plugin_profiles(plugins, alloc, warnings)
1278
1279 channels = data.get("channels")
1280 if isinstance(channels, dict):
! 1281 for name, entry in channels.items():
! 1282 if isinstance(entry, dict):
! 1283 _rewrite_secret_named_keys(
1284 entry, f"DATUS_CHANNEL_{_sanitize_var_component(name)}", f"channels.{name}", alloc
1285 )
! 1286 extra = entry.get("extra")
! 1287 if isinstance(extra, dict):
! 1288 for fld in ("bot_token", "app_token", "app_secret", "app_id"):
! 1289 _rewrite_secret_value(
1290 extra,
1291 fld,
1292 f"DATUS_CHANNEL_{_sanitize_var_component(name)}_{_sanitize_var_component(fld)}",
1293 f"channels.{name}.extra.{fld}",Lines 1295-1312 1295 )
1296
1297 observability = data.get("observability")
1298 if isinstance(observability, dict):
! 1299 _rewrite_secret_named_keys(observability, "DATUS_TRACING", "observability", alloc)
! 1300 tracing = observability.get("tracing")
! 1301 if isinstance(tracing, dict):
! 1302 adapters = tracing.get("adapters")
! 1303 if isinstance(adapters, list):
! 1304 for idx, adapter in enumerate(adapters):
! 1305 if isinstance(adapter, dict) and isinstance(adapter.get("headers"), dict):
! 1306 headers = adapter["headers"]
! 1307 for header in list(headers):
! 1308 _rewrite_secret_value(
1309 headers,
1310 header,
1311 f"DATUS_TRACING_{idx}_HEADER_{_sanitize_var_component(header)}",
1312 f"observability.tracing.adapters[{idx}].headers.{header}",Lines 1317-1325 1317 if isinstance(document, dict):
1318 _rewrite_secret_value(document, "tavily_api_key", "TAVILY_API_KEY", "document.tavily_api_key", alloc)
1319 for name, entry in document.items():
1320 if isinstance(entry, dict):
! 1321 _rewrite_secret_value(
1322 entry,
1323 "github_token",
1324 "GITHUB_TOKEN",
1325 f"document.{name}.github_token",Lines 1327-1337 1327 )
1328
1329 api = data.get("api")
1330 if isinstance(api, dict):
! 1331 auth = api.get("auth_provider")
! 1332 if isinstance(auth, dict) and isinstance(auth.get("kwargs"), dict):
! 1333 _rewrite_secret_named_keys(
1334 auth["kwargs"], "DATUS_API_AUTH", "api.auth_provider.kwargs", alloc, all_string_leaves=True
1335 )
1336 Lines 1348-1356 1348 plugin_config_schema = None # type: ignore[assignment]
1349
1350 for plugin_name, profiles in plugins.items():
1351 if not isinstance(profiles, dict):
! 1352 continue
1353 secret_fields: Optional[Set[str]] = None
1354 if plugin_config_schema is not None:
1355 # A broken plugin manifest must degrade to "no schema" (all string
1356 # leaves become placeholders), not crash the build.Lines 1355-1364 1355 # A broken plugin manifest must degrade to "no schema" (all string
1356 # leaves become placeholders), not crash the build.
1357 try:
1358 specs = plugin_config_schema(str(plugin_name))
! 1359 if specs:
! 1360 secret_fields = {spec["name"] for spec in specs if isinstance(spec, dict) and spec.get("secret")}
1361 except Exception as exc:
1362 logger.warning("package: plugin_config_schema(%r) failed: %s", plugin_name, exc)
1363 secret_fields = None
1364 for profile_name, profile in profiles.items():Lines 1362-1370 1362 logger.warning("package: plugin_config_schema(%r) failed: %s", plugin_name, exc)
1363 secret_fields = None
1364 for profile_name, profile in profiles.items():
1365 if not isinstance(profile, dict):
! 1366 continue
1367 prefix = f"DATUS_PLUGIN_{_sanitize_var_component(plugin_name)}_{_sanitize_var_component(profile_name)}"
1368 path = f"plugins.{plugin_name}.{profile_name}"
1369 if secret_fields is None:
1370 warnings.append(Lines 1374-1388 1374 _rewrite_secret_named_keys(profile, prefix, path, alloc, all_string_leaves=True)
1375 else:
1376 # Sorted: allocation order decides collision suffixes (_2, _3),
1377 # so set order would rename variables between builds.
! 1378 for dotted in sorted(secret_fields):
! 1379 container: Any = profile
! 1380 parts = dotted.split(".")
! 1381 for part in parts[:-1]:
! 1382 container = container.get(part) if isinstance(container, dict) else None
! 1383 if isinstance(container, dict):
! 1384 _rewrite_secret_value(
1385 container,
1386 parts[-1],
1387 f"{prefix}_{_sanitize_var_component(dotted)}",
1388 f"{path}.{dotted}",Lines 1388-1396 1388 f"{path}.{dotted}",
1389 alloc,
1390 )
1391 # Belt & braces: secret-named keys outside the schema.
! 1392 _rewrite_secret_named_keys(profile, prefix, path, alloc)
1393
1394
1395 def generate_agent_yml(
1396 raw: Dict[str, Any],Lines 1416-1424 1416 filtered = {name: nodes[name] for name in kept_subagents if name in nodes}
1417 if filtered:
1418 data["agentic_nodes"] = filtered
1419 else:
! 1420 data.pop("agentic_nodes", None)
1421
1422 _sanitize_agent_tree(data, alloc, warnings)
1423 alloc.harvest(data, "")
1424 text = yaml.safe_dump({"agent": data}, allow_unicode=True, sort_keys=False)Lines 1429-1453 1429 """Sanitized copy of ``{home}/conf/.mcp.json`` (headers → placeholders)."""
1430 source = source_home / "conf" / ".mcp.json"
1431 if not source.is_file():
1432 return None
! 1433 try:
! 1434 payload = json.loads(source.read_text(encoding="utf-8"))
! 1435 except (OSError, json.JSONDecodeError) as exc:
! 1436 logger.warning("package: unreadable %s (%s); skipping", source, exc)
! 1437 return None
! 1438 if not isinstance(payload, dict):
! 1439 return None
! 1440 servers = payload.get("mcpServers")
! 1441 if isinstance(servers, dict):
! 1442 for name, entry in servers.items():
! 1443 if not isinstance(entry, dict):
! 1444 continue
! 1445 prefix = f"DATUS_MCP_{_sanitize_var_component(name)}"
! 1446 headers = entry.get("headers")
! 1447 if isinstance(headers, dict):
! 1448 for header in list(headers):
! 1449 _rewrite_secret_value(
1450 headers,
1451 header,
1452 f"{prefix}_HEADER_{_sanitize_var_component(header)}",
1453 f".mcp.json:{name}.headers.{header}",Lines 1452-1464 1452 f"{prefix}_HEADER_{_sanitize_var_component(header)}",
1453 f".mcp.json:{name}.headers.{header}",
1454 alloc,
1455 )
! 1456 env = entry.get("env")
! 1457 if isinstance(env, dict):
! 1458 _rewrite_secret_named_keys(env, f"{prefix}_ENV", f".mcp.json:{name}.env", alloc)
! 1459 alloc.harvest(payload, ".mcp.json")
! 1460 return (json.dumps(payload, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
1461
1462
1463 def generate_project_config(root: Path, project_name: str) -> bytes:
1464 """Regenerate ``.datus/config.yml`` with the pinned project name."""Lines 1468-1477 1468 try:
1469 loaded = yaml.safe_load(source.read_text(encoding="utf-8"))
1470 if isinstance(loaded, dict):
1471 payload = loaded
! 1472 except (OSError, yaml.YAMLError) as exc:
! 1473 logger.warning("package: unreadable %s (%s); regenerating minimal config", source, exc)
1474 payload["project_name"] = project_name
1475 text = yaml.safe_dump(payload, allow_unicode=True, sort_keys=False, default_flow_style=False)
1476 return text.encode("utf-8")Lines 1586-1594 1586 for name in sorted(kept):
1587 distribution, version = versions.get(name, (name, ""))
1588 spec = f"{distribution}=={version}" if version else distribution
1589 if not version:
! 1590 warnings.append(f"plugin {name!r}: installed version unknown; install line left unpinned")
1591 # --force replaces an already-installed plugin, so re-running the
1592 # script (or init.sh) is idempotent instead of erroring out.
1593 commands.append(f"datus plugin install {spec} --force")
1594 script = "\n".join(Lines 1719-1727 1719 grouped.setdefault(binding.var, []).append(binding.config_path)
1720 for var in sorted(grouped):
1721 lines.append(f"| `{var}` | {', '.join(sorted(set(grouped[var])))} |")
1722 else:
! 1723 lines.append("None — this package carries no credential-bearing config.")
1724 lines += [
1725 "",
1726 "`.env` files are NOT auto-loaded when datus-agent is installed via pip.",
1727 "Export the variables in your shell, or run `set -a; source .env; set +a`.",Lines 1807-1822 1807 _walk(value, child)
1808 elif isinstance(value, str) and value.strip() and _is_secret_key(str(key)):
1809 if not _PLACEHOLDER_RE.match(value.strip()):
1810 findings.append(SecretFinding(arcname=arcname, locator=child, kind="plaintext_secret_key"))
! 1811 elif isinstance(node, list):
! 1812 for idx, item in enumerate(node):
! 1813 _walk(item, f"{path}[{idx}]")
1814
1815 try:
1816 parsed = json.loads(text) if arcname.endswith(".json") else yaml.safe_load(text)
! 1817 except (ValueError, yaml.YAMLError):
! 1818 return findings
1819 _walk(parsed, "")
1820 return findings
1821 Lines 1832-1842 1832 generated_conf = {"conf/agent.yml", "conf/.mcp.json", PROJECT_CONFIG_REL}
1833 for entry in entries:
1834 try:
1835 head = entry.read_bytes(cap=_SCAN_READ_CAP_BYTES)
! 1836 except OSError as exc:
! 1837 findings.append(SecretFinding(arcname=entry.arcname, locator=str(exc), kind="unreadable"))
! 1838 continue
1839 if entry.size() > _SCAN_READ_CAP_BYTES:
1840 warnings.append(
1841 f"secret scan truncated for {entry.arcname}: only the first "
1842 f"{_SCAN_READ_CAP_BYTES // (1024 * 1024)} MB of {entry.size()} bytes were scanned"Lines 1880-1890 1880 try:
1881 return _build_package(options)
1882 except PackageError as exc:
1883 return PackageResult(ok=False, error=str(exc))
! 1884 except (OSError, zipfile.BadZipFile) as exc:
! 1885 logger.error("package build failed: %s", exc)
! 1886 return PackageResult(ok=False, error=str(exc))
1887
1888
1889 def _build_package(options: PackageOptions) -> PackageResult:
1890 root = options.root.resolve()Lines 1889-1897 1889 def _build_package(options: PackageOptions) -> PackageResult:
1890 root = options.root.resolve()
1891 raw = load_raw_agent_config()
1892 if raw is None:
! 1893 raise PackageError("no agent configuration found (conf/agent.yml or ~/.datus/conf/agent.yml)")
1894 project_name = resolve_effective_project_name(root, raw)
1895 source_home = resolve_source_home(raw, root)
1896 output_path = (options.output or default_output_path(root, project_name)).resolve()Lines 1900-1908 1900
1901 def _add(new_entries: Sequence[StagedEntry]) -> None:
1902 for entry in new_entries:
1903 if entry.arcname in entries_by_arc:
! 1904 warnings.append(f"duplicate staging of {entry.arcname}; keeping the later entry")
1905 entries_by_arc[entry.arcname] = entry
1906
1907 walk_entries, walk_warnings = collect_project_files(root, options.include, options.exclude, output_path)
1908 warnings.extend(walk_warnings)Lines 1943-1951 1943 _add([StagedEntry(arcname="conf/agent.yml", content=agent_yml)])
1944
1945 mcp_json = generate_mcp_json(source_home, alloc)
1946 if mcp_json is not None:
! 1947 _add([StagedEntry(arcname="conf/.mcp.json", content=mcp_json)])
1948
1949 _add([StagedEntry(arcname=PROJECT_CONFIG_REL, content=generate_project_config(root, project_name))])
1950
1951 packages = enumerate_datus_packages()Lines 2000-2008 2000 staged.append(entry)
2001
2002 for entry in staged:
2003 if entry.source is not None and entry.size() > _LARGE_FILE_WARN_BYTES:
! 2004 warnings.append(f"large file packaged: {entry.arcname} ({entry.size() // (1024 * 1024)} MB)")
2005
2006 selections = {
2007 "subagents": kept_subagents,
2008 "skills": kept_skills,datus/cli/package_cli.pyLines 54-65 54
55
56 def _is_interactive() -> bool:
57 """Both streams must be TTYs — mirrors ``service_bootstrap._is_interactive``."""
! 58 try:
! 59 return bool(sys.stdin.isatty() and sys.stdout.isatty())
! 60 except (AttributeError, OSError):
! 61 return False
62
63
64 def run_package_command(argv: List[str]) -> int:
65 parser = _build_parser()Lines 106-115 106 path = Path(target)
107 if path.is_file():
108 path.unlink()
109 print_warning(console, f"Removed the partially written {path}")
! 110 except OSError as exc:
! 111 print_warning(console, f"Could not remove the partially written {target}: {exc}")
112
113
114 # --------------------------------------------------------------------------- #
115 # Wizard #Lines 122-130 122 print_info(console, f"Packaging project {project_name!r} from {root}")
123
124 output = _step_output_path(console, pb, root, project_name)
125 if output is None:
! 126 return None
127
128 include, exclude = _step_file_scope(console)
129
130 subagents = _step_multi(Lines 181-189 181
182 def _is_under(path: Path, root: Path) -> bool:
183 try:
184 path.resolve().relative_to(root.resolve())
! 185 return True
186 except ValueError:
187 return False
188 Lines 199-207 199 continue
200 if candidate.exists() and not confirm_prompt(
201 console, f"{candidate} exists — overwrite?", default=False, cancellable=True
202 ):
! 203 continue
204 return candidate
205
206
207 def _step_file_scope(console: Console) -> Tuple[List[str], List[str]]:Lines 215-223 215 def _prompt_patterns(console: Console, message: str) -> List[str]:
216 while True:
217 answer = prompt_input(console, message, default="", allow_interrupt=True).strip()
218 if not answer:
! 219 return []
220 patterns = [part.strip() for part in answer.split(",") if part.strip()]
221 for pattern in patterns:
222 try:
223 re.compile(pattern)Lines 278-286 278 nodes = raw.get("agentic_nodes")
279 if isinstance(nodes, dict):
280 gen_report = nodes.get("gen_visual_report")
281 if isinstance(gen_report, dict) and isinstance(gen_report.get("report_dist"), str):
! 282 configured = gen_report["report_dist"]
283 while True:
284 answer = prompt_input(
285 console,
286 "Path to the web-artifact-render dist directory (empty = skip)",Lines 287-296 287 default=configured,
288 allow_interrupt=True,
289 ).strip()
290 if not answer:
! 291 print_info(console, "Skipping the local dist; reports will use the CDN.")
! 292 return None
293 resolved = _resolve_dist(Path(answer))
294 if resolved is None:
295 print_warning(console, f"{answer} is not a valid dist (needs index.css + index.umd.js)")
296 continueLines 301-309 301 from datus.cli._render_utils import build_row_table
302
303 def _fmt(values: Optional[Tuple[str, ...]]) -> str:
304 if values is None:
! 305 return "(all)"
306 return ", ".join(values) if values else "(none)"
307
308 rows = [
309 {"item": "Project", "value": project_name},Lines 333-355 333 def _selection_summary_lines(selections: Dict) -> List[str]:
334 """Human-readable "what went in" lines for the build result."""
335 if not selections:
336 return []
! 337 lines: List[str] = []
! 338 subjects = selections.get("subjects")
! 339 if subjects is not None:
! 340 shown = ", ".join(subjects) if subjects else "(none)"
! 341 lines.append(f"Subject areas: {shown} → {selections.get('reference_sql_entries', 0)} reference-SQL summaries")
! 342 for key, label in (("subagents", "Subagents"), ("skills", "Skills"), ("metrics", "Metric datasources")):
! 343 values = selections.get(key)
! 344 if values is not None:
! 345 lines.append(f"{label}: {', '.join(values) if values else '(none)'}")
! 346 return lines
347
348
349 def _report_result(console: Console, result: "PackageResult") -> int:
350 for warning in result.warnings:
! 351 print_warning(console, warning)
352 if not result.ok:
353 print_status(console, "Package build failed.", ok=False)
354 for finding in result.secret_findings:
355 print_error(Lines 369-381 369 # Spell out what the selection actually produced — counting zip entries by
370 # hand is easy to get wrong (``unzip -l`` wraps long/CJK names onto several
371 # lines), so a filtered package can look unfiltered.
372 for line in _selection_summary_lines(result.selections):
! 373 print_info(console, line)
374 env_vars = sorted({binding.var for binding in result.env_vars})
375 if env_vars:
! 376 print_info(console, "Receiver must export: " + ", ".join(env_vars))
! 377 print_info(console, "The generated README.md lists where each variable is used.")
378 return 0
379
380
381 __all__ = ["run_package_command"]datus/storage/reference_sql/reference_sql_init.pyLines 477-487 477
478 Returns:
479 Dict with the same shape as :func:`init_reference_sql`
480 """
! 481 directory = Path(summaries_dir) if summaries_dir else Path(global_config.path_manager.sql_summary_path())
! 482 if not directory.is_dir():
! 483 return {
484 "status": "success",
485 "message": f"reference_sql summaries directory not found: {directory}",
486 "valid_entries": 0,
487 "processed_entries": 0,Lines 487-511 487 "processed_entries": 0,
488 "total_stored_entries": storage.get_reference_sql_size(),
489 }
490
! 491 items: List[Dict[str, Any]] = []
! 492 invalid: List[str] = []
! 493 for path in sorted(directory.rglob("*.y*ml")):
! 494 if not path.is_file():
! 495 continue
! 496 try:
! 497 with open(path, "r", encoding="utf-8") as fh:
! 498 doc = yaml.safe_load(fh)
! 499 except (OSError, yaml.YAMLError) as exc:
! 500 invalid.append(f"{path.name}: unreadable ({exc})")
! 501 continue
! 502 if not isinstance(doc, dict):
! 503 invalid.append(f"{path.name}: not a YAML mapping")
! 504 continue
505
! 506 subject_path = [part.strip() for part in str(doc.get("subject_tree") or "").split("/") if part.strip()]
! 507 item = {
508 "id": doc.get("id"),
509 "name": doc.get("name"),
510 "sql": doc.get("sql"),
511 "comment": doc.get("comment", ""),Lines 514-536 514 "subject_path": subject_path,
515 "tags": doc.get("tags", ""),
516 "filepath": doc.get("filepath") or str(path),
517 }
! 518 missing = [key for key in ("name", "sql", "summary", "search_text") if not item.get(key)]
! 519 if missing or not subject_path:
! 520 invalid.append(f"{path.name}: missing {', '.join(missing + ([] if subject_path else ['subject_tree']))}")
! 521 continue
! 522 items.append(item)
523
! 524 if build_mode == "overwrite":
! 525 storage.truncate()
! 526 storage.store_batch(items)
527 else:
! 528 storage.upsert_batch(items)
! 529 storage.after_init()
530
! 531 logger.info("Reindexed %d reference SQL entries from %s (no LLM)", len(items), directory)
! 532 return {
533 "status": "success",
534 "message": f"reference_sql reindexed from summaries ({build_mode} mode)",
535 "valid_entries": len(items),
536 "processed_entries": len(items),datus/storage/subject_tree/store.pyLines 583-591 583 return _node_to_dict(rows[0])
584
585 target = normalize_subject_node_name(name)
586 if not target:
! 587 return None
588 siblings = self._table.query(
589 SubjectNodeRecord,
590 where={"parent_id": db_parent_id, "datasource_id": self.datasource_id},
591 )Test Failure Details17867/19853 tests passed, 1986 skipped Generated by pytest-cov + diff-cover |
Found during the baisheng end-to-end run: bootstrap-kb defaults to the 'check' strategy, which only reports counts — the generated script ingested nothing on the receiver. The script now uses 'overwrite' for the first semantic_model call only (it truncates the whole project-scoped store, safe exactly once on a fresh unzip) and 'incremental' for every subsequent call so multi-file/multi-datasource selections don't wipe each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Louis-Law
force-pushed
the
feat/package-to-server
branch
from
August 9, 2026 08:11
594ffee to
4d9b584
Compare
Standards axis: PackageError now derives from DatusException with a dedicated PACKAGE_BUILD_ERROR code (mirrors plugins.store.StoreError); non-ASCII section markers in comments replaced; missing type hints filled in (wizard signatures, allocator/plugin sets). Duplicated selection resolution extracted into _resolve_selection, the report/dashboard kind switch collapsed into one _ARTIFACT_KIND_DIRS map, and placeholder harvesting moved onto _PlaceholderAllocator so nothing reaches into its private state. Spec axis: the final secret scan now covers Fernet tokens (gAAAAA...) as the design review requires; editable installs are surfaced as a warning even under --yes instead of silently vanishing when the confirmation is skipped; the module docstring states the scan's binary-sniff/read-cap limits honestly; and package_manifest.json now carries per-file provenance (generated vs project). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
datus/cli/package_cli.py (1)
182-194: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
_prompt_patternsreports only the first failing pattern name reliably.The
trywraps the whole loop.patternin theexcepthandler holds the last bound value, which is the failing one. The behavior is correct. Consider moving thetryinside the loop for clarity.♻️ Proposed refactor
- patterns = [part.strip() for part in answer.split(",") if part.strip()] - try: - for pattern in patterns: - re.compile(pattern) - except re.error as exc: - print_warning(console, f"Invalid regex {pattern!r}: {exc}") - continue - return patterns + patterns = [part.strip() for part in answer.split(",") if part.strip()] + invalid = False + for pattern in patterns: + try: + re.compile(pattern) + except re.error as exc: + print_warning(console, f"Invalid regex {pattern!r}: {exc}") + invalid = True + break + if invalid: + continue + return patterns🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/cli/package_cli.py` around lines 182 - 194, Refactor _prompt_patterns so each pattern is compiled inside its own try/except block, with the corresponding pattern directly available when reporting Invalid regex warnings. Preserve the current retry behavior and return values for valid and empty input.tests/unit_tests/cli/test_package_builder.py (2)
562-567: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value
extractallon an untrusted archive is unsafe; here the archive is self-produced.Static analysis flags
zf.extractall(extract)as Zip Slip. The archive comes frombuild_packagein the same test, and Line 541 already asserts that no member path contains.., a leading/, or a backslash. The current call is safe. Add a short comment so the pattern is not copied into code that reads external archives.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit_tests/cli/test_package_builder.py` around lines 562 - 567, Add a concise comment immediately before the zf.extractall(extract) call explaining that the archive is produced by build_package and its member paths were already validated against traversal, absolute, and backslash paths. Leave the extraction logic unchanged.Source: Linters/SAST tools
238-245: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value
symlink_toneeds privileges on Windows.
Path.symlink_toraisesOSErroron Windows without Developer Mode or the create-symlink privilege. If CI runs Windows, guard this test.🛡️ Proposed fix
+ import os + def test_symlink_escaping_root_dropped(self, project, tmp_path): + if os.name == "nt": + pytest.skip("symlink creation requires elevated privileges on Windows")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit_tests/cli/test_package_builder.py` around lines 238 - 245, Guard test_symlink_escaping_root_dropped so it is skipped on Windows when symlink creation is unavailable, while preserving the existing assertions on supported platforms. Use the test’s existing platform-detection conventions or pytest’s skip mechanism around the symlink setup.datus/cli/package_builder.py (1)
941-953: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winIterate
secret_fieldsin sorted order.
secret_fieldsis aset. The iteration order varies between runs._PlaceholderAllocator.allocateassigns collision suffixes (_2,_3) in allocation order, so two schema fields that sanitize to the same variable name can receive swapped names across builds. Sorting makes the generatedconf/agent.ymland the README table stable.♻️ Proposed fix
- for dotted in secret_fields: + for dotted in sorted(secret_fields):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@datus/cli/package_builder.py` around lines 941 - 953, Update the loop in the secret-field rewriting flow to iterate over secret_fields in sorted order before calling _rewrite_secret_value, ensuring _PlaceholderAllocator allocation and generated outputs remain deterministic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unit_tests/cli/test_package_builder.py`:
- Around line 83-95: Update the project fixture to set USERPROFILE to fake_home
alongside HOME before changing directories, ensuring Path.home() and ~ expansion
resolve within the isolated test profile on Windows.
---
Nitpick comments:
In `@datus/cli/package_builder.py`:
- Around line 941-953: Update the loop in the secret-field rewriting flow to
iterate over secret_fields in sorted order before calling _rewrite_secret_value,
ensuring _PlaceholderAllocator allocation and generated outputs remain
deterministic.
In `@datus/cli/package_cli.py`:
- Around line 182-194: Refactor _prompt_patterns so each pattern is compiled
inside its own try/except block, with the corresponding pattern directly
available when reporting Invalid regex warnings. Preserve the current retry
behavior and return values for valid and empty input.
In `@tests/unit_tests/cli/test_package_builder.py`:
- Around line 562-567: Add a concise comment immediately before the
zf.extractall(extract) call explaining that the archive is produced by
build_package and its member paths were already validated against traversal,
absolute, and backslash paths. Leave the extraction logic unchanged.
- Around line 238-245: Guard test_symlink_escaping_root_dropped so it is skipped
on Windows when symlink creation is unavailable, while preserving the existing
assertions on supported platforms. Use the test’s existing platform-detection
conventions or pytest’s skip mechanism around the symlink setup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 58e4716b-141d-4b63-890c-701618d2e264
📒 Files selected for processing (4)
datus/cli/package_builder.pydatus/cli/package_cli.pydatus/utils/exceptions.pytests/unit_tests/cli/test_package_builder.py
- Guard the plugin_config_schema call itself, not just its import: a broken plugin manifest now degrades to the all-string-leaves fallback with a warning instead of escaping build_package's never-raise contract. - Surface secret-scan truncation: scan_for_secrets returns warnings and flags any file larger than the read cap, so a partial scan can no longer read as a full one. - Leave passwordless URIs (scheme://user@host/db) untouched — only URIs that actually carry a password component are rewritten, so receivers keep host/port/database instead of exporting a whole URI under a *_URI_PASSWORD variable. - Validate dist assets exist before rewriting report index.html to relative _assets/ URLs; missing files keep the CDN html and warn. - Use one timestamp for all zip members; fix the stale _SECRET_PATH_TABLE docstring reference; set USERPROFILE alongside HOME in the test fixture for Windows isolation. Already addressed in the previous commit: English-only docstrings and wizard type annotations. The Pydantic artifact_blob suggestion is pre-existing tolerant-by-design wire handling owned by the injecting host and is left as a follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tests/unit_tests/cli/test_package_builder.py (1)
577-584: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd concrete type hints for the local test values.
Annotate
boomwith its parameter type andNeverreturn type. Changewarnings: listtowarnings: list[str].As per coding guidelines, “Use Python 3.12+ type hints throughout.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit_tests/cli/test_package_builder.py` around lines 577 - 584, Update the local test values in the _sanitize_plugin_profiles setup: annotate boom’s _name parameter with its concrete string type and its return type with Never, and change warnings from an unparameterized list to list[str].Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unit_tests/cli/test_package_builder.py`:
- Around line 304-309: Update
test_unparseable_uri_with_password_component_fully_replaced to assert that
container["uri"] exactly equals "${DATUS_DS_X_URI_PASSWORD}" after
_rewrite_uri_password, replacing the weaker substring-only assertion while
preserving the existing malformed URI setup.
- Around line 425-441: Update test_report_dist_missing_asset_keeps_cdn_html to
save the original index.html content before calling _build, then assert the
packaged HTML exactly equals that original content when a required dist asset is
missing. Retain the existing archive and warning assertions, and ensure the test
verifies the complete CDN HTML rather than only checking that some CDN reference
remains.
---
Nitpick comments:
In `@tests/unit_tests/cli/test_package_builder.py`:
- Around line 577-584: Update the local test values in the
_sanitize_plugin_profiles setup: annotate boom’s _name parameter with its
concrete string type and its return type with Never, and change warnings from an
unparameterized list to list[str].
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1d1c4997-43b6-430e-9911-66e90f7320be
📒 Files selected for processing (2)
datus/cli/package_builder.pytests/unit_tests/cli/test_package_builder.py
🚧 Files skipped from review as they are similar to previous changes (1)
- datus/cli/package_builder.py
Self-test feedback on the package command: - Editable/source installs no longer gate the build behind a confirmation — they are pinned as-is with a warning in both interactive and --yes runs. The now-unused confirm_cb hook is removed from PackageOptions. - A vim swap file open during packaging (conf/.agent.yml.swp) was collected by the walk and vanished before zip write, failing the whole build with FileNotFoundError. Editor swap/backup junk (*.swp/*.swo/*.swx, *~, .DS_Store) is now excluded up front, and any disk-backed entry that disappears between collection and finalize is dropped with a warning instead of aborting the build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AppleDouble xattr sidecars (._*) — which macOS writes next to real files on SMB/FAT volumes — were not excluded anywhere, and the component selectors rglob their own subtrees without the generic walk's pruning: ._orders.yml would be staged as a semantic-model file and even written into rebuild_kb.sh for the receiver to bootstrap against. A shared _is_junk_path predicate now guards the walk and all three selectors (skills, metrics, artifacts), and the any-depth dir exclusions gain __MACOSX plus external-volume metadata dirs (.Spotlight-V100, .Trashes, .fseventsd, .TemporaryItems, .DocumentRevisions-V100). Deliberately not handled: xattrs/resource forks are dropped by design (quarantine flags and Finder metadata must not travel), and receiver- side quarantine does not affect `bash scripts/*.sh` invocations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under home: . the REPL command history lands at ./history in the project root — user activity, potentially sensitive queries — and the walk shipped it (confirmed leaked in a real self-test package). A new top-level-only file exclusion drops it; a nested project file named "history" (e.g. docs/history) still ships. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-test found both components unselectable. Reference SQL: candidate corpora are discovered by content (top-level dirs holding *.sql) and get a wizard step plus a reference_sql bootstrap-kb line in rebuild_kb.sh, running against the project's default datasource (project pin -> `default: true` -> single-datasource shortcut; an ambiguous setup emits a commented manual command instead). Selection drives the KB rebuild only — every .sql directory still ships via the generic walk, so deselecting can never drop a user's files — and the default is the conventionally-named subset: a real baisheng run showed an init/ directory of DDL being discovered, and bootstrapping that as reference SQL would poison the store. The generated script also warns that the rebuild re-runs one LLM summary per statement rather than reusing the shipped subject/sql_summaries/. Plugins: candidates are the union of the project's activation list and the managed store, so a project that never wrote a `plugins:` key can now ship install lines at all. The wizard picks which ones; generate_install_plugins_script takes the selection and reports what it kept for the manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reference SQL now behaves exactly like metrics rather than being a special case: - Canonical path. Only reference_sql/ (or reference_sqls/ ref_sql/) is treated as reference SQL, the same property that lets the metric selector trust subject/semantic_models/. Content-sniffing arbitrary *.sql directories is gone — a migrations/ or init/ tree of DDL ships as ordinary project content and is never bootstrapped. - Per-datasource units. reference_sql/<datasource>/ mirrors subject/semantic_models/<datasource>/: one selectable unit per datasource, and the rebuild line takes --datasource from the directory name. A flat reference_sql/*.sql layout stays supported as a single unit bound to the project's default datasource; when that cannot be resolved the corpus still ships and the rebuild is emitted as a commented manual command. - Selection gates packaging, like every other selector: selected corpora ship whole, unselected ones do not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…om summaries The KB indexes metrics and reference SQL under one subject tree per datasource, held in the vector store — so that tree, not a directory layout, is the natural unit for choosing what a package carries. Agent side: new `bootstrap-kb --components reference_sql --from_summaries [--summaries_dir DIR]` re-indexes the committed subject/sql_summaries/*.yaml verbatim. Those files already carry sql, summary, search_text, subject_tree and tags, so re-deriving them through the LLM costs time and credits and drifts from the reviewed originals; this path is the restore route for a fresh checkout, a machine migration, or an unzipped package (81 statements: ~40 min of LLM before, seconds now). The overwrite branch no longer wipes the summaries directory under --from_summaries — that directory is the input. Package side: one "Subject areas" wizard step, sourced from the vector store's subject tree (falling back to the artifacts when the KB was never built on this machine) and labelled with what each root actually costs. The selection gates metric documents (matched on their `subject_tree:` tag) and reference-SQL summaries (matched on their subject_tree field); semantic-model documents are table definitions and always travel with their datasource, and the raw .sql corpus ships as ordinary project content. rebuild_kb.sh now ends with the --from_summaries call instead of an LLM re-derivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Subject paths are written by an LLM one entry at a time and node lookup matched names exactly, so a single area accumulated siblings that only differed in case or separators — each splitting the same content into its own subtree and its own line in the package selector. Node lookup now falls back to a normalized comparison (case-folded, separators and surrounding punctuation ignored) and reuses the existing node; the stored name is unchanged. Genuinely different wording still stays distinct — collapsing 营销 into 营销分析 needs a curated tree via bootstrap-kb --subject_tree. Also: a reference-SQL summary with no subject_tree matched no selection and silently disappeared from every package. It now ships regardless, with a warning naming the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A filtered package can look unfiltered: counting zip entries by hand is error-prone because `unzip -l` wraps long or CJK filenames onto several lines, so a run that packaged one summary out of 77 reads as if nothing was filtered. The build result now carries the resolved selections and the CLI prints them, e.g. Subject areas: activity → 1 reference-SQL summaries Subagents: baisheng_metadata, baisheng_metrics, baisheng_ref_sql The counts come from the same numbers written into package_manifest.json, so what is reported is what shipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Metric documents record their subject as a `subject_tree:` tag, but the tag is optional — hand-written or older metric YAML has none. Those files matched no subject root, so any package with a subject selection dropped them silently, and with them the metrics rebuild line: a real run selecting two subjects produced a rebuild_kb.sh with no `--components metrics` step at all. Untagged metric files now travel regardless of the selection (same rule already applied to untagged reference-SQL summaries) and the build reports a warning naming the file, so the missing tag is visible rather than silently costing the receiver its metrics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ctrl+C could not cancel the wizard, and on the final prompt it did the opposite of what the user asked: the shared prompts swallow the interrupt and return a plausible answer — select_multi_choice returns [] (identical to "deselect everything") and select_choice returns its default, so Ctrl+C on "Build the package now?" (default yes) started the build. The shared helpers gain an opt-in `cancellable=True` that turns Ctrl+C / EOF into a KeyboardInterrupt instead; the default stays exactly as it was, so existing callers and their tests are untouched. The wizard opts in everywhere and exits 130 (the shell convention for SIGINT) without writing anything. An interrupt during zip assembly also removes the truncated archive — a half-written package that looks complete is worse than none. Verified against a real terminal: Ctrl+C on the Subagents screen exits 130 with "Aborted — nothing was written." and leaves an existing baisheng.zip byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Setting up a received package meant reading the README and running three things in the right order. `scripts/init.sh` is that order, generated with only the steps a given package actually needs: dependencies, then plugins, then the knowledge base. The per-step scripts stay because they are worth running on their own later (re-index after editing subject YAML, reinstall a plugin). Everything in it is re-runnable: plugin install lines now pass --force, so a second run replaces the installed plugin instead of failing, and the KB steps already overwrite. Two things the first real run surfaced. Dependency install targets an explicit interpreter — uv first (uv-created virtualenvs have no pip module at all), then `python -m pip`, with a clear error when neither exists; a bare `pip` had resolved to a different Python than the venv running datus. And the closing echo is single-quoted: inside a double-quoted echo the backticks around `datus` were command substitution, which would have launched the REPL at the end of setup. Verified end to end: unzip, `bash scripts/init.sh` → deps installed, airflow plugin installed, KB rebuilt (10 semantic objects, 3 metrics, 77 reference SQL), exit 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The subject menu listed only tree roots, so the smallest selectable unit was far larger than users wanted: baisheng has 61 distinct subject paths under 22 roots, and picking one root pulled in everything beneath it. The menu is now the subject tree rendered two levels deep — roots plus their direct children, with deeper paths folded into their depth-2 parent. Counts roll up, so a root still shows the full cost of taking everything under it. Selecting either level matches by path prefix, so "marketing" keeps its old meaning while "marketing/campaigns" narrows. Verified on baisheng: root selection ships 24 reference-SQL summaries, the depth-2 child ships 15.
The two-level subject menu rendered as a tree but still behaved like a flat list: checking a parent left its children unchecked, and checking every child left the parent unchecked. Both read as bugs against the indentation the menu shows. MultiSelectState now holds the checkbox state for select_multi_choice and, when the caller passes a child -> parent map, enforces both directions: toggling a parent applies the same state to its children, and a parent is checked exactly when all of its children are. Pre- selection is reconciled the same way, so the wizard's "everything selected" default still comes up fully checked. A parent whose children are only partly selected renders as [-] instead of looking empty. Only the subject screen passes a hierarchy; every other multi-select keeps flat behaviour.
Two test assertions were weak enough to pass on a wrong implementation: the unparseable-URI case only checked that the password substring was gone (a partial redaction would have passed), and the missing-dist case only checked that some CDN reference survived (a half-rewritten page that swapped the CSS but not the JS would have passed). Both now assert the exact expected value. Plugin secret fields are now rewritten in sorted order — allocation order decides collision suffixes (_2, _3), so set iteration order could rename variables between builds of the same project. Also: compile each include/exclude regex in its own try block instead of reading the loop variable after the handler; note why the test's blanket extractall is safe on a self-produced archive; skip the symlink test on Windows, where creating one needs Developer Mode; and type the local helpers in the broken-plugin-schema test.
Adds cli/package_command.md (EN + zh) and wires it into the CLI nav in
both locales. Covers the wizard flow, the two-level subject tree and its
cascade, the produced package layout, what never ships, how credentials
become ${VAR} placeholders, and the receiver's one-command setup.
Content is checked against package_cli.py / package_builder.py; examples
come from a real package built from a project (env-var table, generated
agent.yml shape, rebuild_kb.sh output).
validate_coverage_map.py --strict failed the coverage job: every page in the mkdocs nav must be claimed by a flow or excluded with a rationale, and docs/cli/package_command.md was neither. Adds the flow extension.project_package rather than an exclusion — `datus package` is a user-facing surface with its own contracts (self- contained home: ., zero-secret placeholders, selection-driven contents, sources instead of binary indexes). PR and merge-queue layers point at the existing unit suites; nightly and weekly are not applicable because packaging touches no LLM, remote service, or network.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Sharing a finished Datus project today means hand-copying config, subject YAML, skills, templates, and reports — and hoping no credential travels along. There is no supported way to hand a working project to another person or machine.
Additionally,
ask_reportdeclaredBLOB_REQUIRED = True, a constraint serving only an external SaaS host: this repo has zeroartifact_blobinjectors (report_routes.pystates publish + ask_report live in a separate SaaS wrapper), whileagent_service.pyvalidates ask-agent creation against disk directories. The result:ask_reportcould be created but never started in plain CLI mode and in standalonedatus-apideployments. datus-agent should not be SaaS-aware — localreports/<slug>/directories are the source of truth.Solution
datus package(interactive) — newdatus/cli/package_cli.py(wizard) +datus/cli/package_builder.py(pure logic, mirrors theplugin_cli.py/plugin_pack.pysplit). The only flag is-y/--yes(all-defaults, for scripts/non-TTY); everything else is collected via a linear wizard reusing the existing_cli_utilsselectors (select_multi_choice's first production caller). Key mechanisms:conf/agent.ymlis regenerated, never copied —home: .plus a pinnedproject_namemake the unzipped directory the entire runtime; the receiver's~/.datusis untouched.AgentConfigis a lossy projection, so the raw dict is the only lossless base — and overwrites every known secret path with${VAR}placeholders (existing placeholders are preserved and harvested into the README env table). A final content scan (PEM headers, token prefixes, secret-named keys in generated configs) fails the build with no bypass.scripts/rebuild_kb.sh(onebootstrap-kbcall per file;semantic_modelbeforemetricsper datasource). Binary LanceDB indexes never ship.scripts/install_plugins.shgenerated from the.datus/config.ymlactivation list with versions pinned from the plugin store.index.htmlto relative_assets/URLs in memory (source tree never mutated).main.pyepilog +_RESERVED_SUBCOMMANDS+ an interception block cloned fromplugin, andstore.RESERVED_PLUGIN_NAMESkept in sync.BLOB_REQUIREDremoval —ask_reportnow binds exactly likeask_dashboard: an injectedartifact_blobstill wins when present and usable (the degenerate-blob gate is untouched), otherwise both kinds fall back to the on-disk artifact directory. This is also what makes packaged reports answerable viaask_reporton the receiver.Test Cases
tests/unit_tests/cli/test_package_builder.py(new, 40 cases): built-in exclusions (.datus/memory,.datus/plans, sessions/data/logs/…,.env,*.duckdb.wal), include/exclude regex semantics, escaping-symlink drop, output-zip self-exclusion; agent.yml generation (secrets replaced across providers/models/datasources/BI/schedulers/MCP/document, URI password-component rewrite, placeholder preservation, unknown-section passthrough, var collision suffixing); selectors (unknown-name errors, template staging, global-skill materialization, rebuild-script ordering, artifact allowlist, dist rewrite); requirements + editable confirmation flow; secret scan (PEM, ghp_/AKIA/xoxb/sk- prefixes, binary sniff, generated-config self-check catching a simulated future secret section); end-to-end zip round-trip with sha256 manifest verification.tests/unit_tests/cli/test_package_cli.py(new, 14 cases):--yeszero-prompt defaults, non-TTY guard, wizard happy path with capturedPackageOptions, empty-selection double-confirm, summary decline, per-step validation reprompts, dist directory validation, exit codes 0/1/2/3.tests/unit_tests/agent/node/test_ask_artifact_agentic_node.py: report disk-fallback contract tests (degenerate blob → disk; no blob → disk; degenerate blob + missing dir → still raises), missing-dir and symlink-rejection parametrized over both kinds, and the blob↔disk parity test upgraded to a same-kind byte-identical comparison (previously impossible for reports).tests/unit_tests/cli/test_main.py:packageinterception test;test_plugin_dispatch.pyreserved-token loop andtest_store.pyreserved-name rejection cover the new token.🤖 Generated with Claude Code
Backport
Summary by CodeRabbit
New Features
datus packagecommand to export projects as self-contained ZIP archives.--yes.Security
Bug Fixes
Summary by CodeRabbit
New Features
datus packagecommand with an interactive wizard for exporting selected project content as a self-contained ZIP.Bug Fixes