feat: persist DiskSage file lineage ontology

[seonghobae]

Product slice

Persist validated DiskSage file-lineage envelopes in Naruon so tenants can inspect metadata-first production context, provider evidence, ontology relations, synchronization state, and copy-readiness provenance without granting Naruon authority to copy, evict, or mutate the source file.

Current exact state

• Protected base: develop@dd8d15191338b841f9e6f3a06507c6a5643b95d0.
• Current branch head: 32b00ce304a39af35f7c20980e965ad369461062.
• Ancestry: 0 commits behind and 26 commits ahead of the live protected base.
• Semantic diff: 17 files, limited to the DiskSage lineage projection, handoff contract, migrations, tests, ERD documentation, and one-line DAV OpenAPI compatibility guard required by the Python 3.14 full-suite gate.
• The previously identified browser-origin and text-safety scope pollution has been removed from this branch.

Migration proof

The DiskSage chain is explicit and single-headed:

0017_merge_newsdom_carddav_heads
→ 0018_disksage_file_lineage
→ 0019_disksage_sync_state
→ 0020_disksage_columns
→ 0021_disksage_scope_idx

Fresh local evidence at this head:

• uv run pytest -q: 1810 passed, 34 skipped.
• uv run pytest -q tests/test_alembic_migrations.py: 20 passed.
• Focused lineage, API, handoff, and migration set: 74 passed.

DiskSage implementation

• versioned lineage-envelope validation and canonical JSON hashing;
• metadata-first production evidence precedence and context/lineage retention;
• encrypted, idempotent, workspace-scoped persistence;
• provider evidence including pending-upload without source-eviction authority;
• ontology class/relation projection and pg-erd-cloud DBML handoff;
• copy-readiness handoff with all-or-none approval evidence.

Compatibility guard

The existing multi-method DAV protocol route is excluded from OpenAPI schema generation. FastAPI otherwise emits duplicate operation-ID warnings under Python 3.14 when the lineage resource test requests the application schema. DAV runtime behavior is unchanged; the guard keeps the full suite warning-clean.

Ownership boundary

DiskSage remains authoritative for copy execution, hashes, provider evidence, review decisions, dynamic ADR/Goal projections, and reversible source trash. Naruon stores and exposes a validated tenant-scoped provenance projection. The semantic-data-portal preview remains a separate consumer boundary.

This PR remains Draft until current-head required checks, independent review, and governance gates are terminal-success. No predecessor-head approval or queued check is treated as merge evidence.

[coderabbitai]

Warning

Review limit reached

Next included review available in 58 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4c59ec06-5fc9-4a6c-b98a-ecf0ac0e9fb6

📥 Commits

Reviewing files that changed from the base of the PR and between b6167d8 and 7c1f936.

📒 Files selected for processing (16)

• .github/workflows/app-ci.yml
• backend/alembic/versions/0022_disksage_organization_lineage.py
• backend/api/dav.py
• backend/api/disksage.py
• backend/db/models.py
• backend/main.py
• backend/scripts/disksage_copy_readiness_handoff.py
• backend/services/disksage_file_lineage.py
• backend/services/disksage_organization_lineage.py
• backend/tests/test_disksage_copy_readiness_handoff.py
• backend/tests/test_disksage_file_lineage.py
• backend/tests/test_disksage_file_lineage_api.py
• backend/tests/test_disksage_file_lineage_postgres.py
• backend/tests/test_disksage_organization_lineage.py
• docs/architecture/disksage-file-lineage-handoff.md
• docs/architecture/erd/disksage-file-lineage.dbml

📝 Walkthrough

Walkthrough

Adds strict DiskSage file-lineage validation, encrypted workspace-scoped persistence, authenticated ingestion and listing endpoints, migrations, tests, application wiring, architecture documentation, separate DAV operation identifiers, and HTML sanitizer cleanup.

Changes

DiskSage file lineage

Layer / File(s)	Summary
Lineage contracts and validation backend/services/disksage_file_lineage.py, backend/tests/test_disksage_file_lineage.py	Defines strict envelope and evidence models. Validates paths, review evidence, cloud-copy state, provider synchronization evidence, and remote locations. Adds canonical hashing and deterministic ontology projection tests.
Lineage persistence schema backend/db/models.py, backend/alembic/versions/0018_disksage_file_lineage.py, backend/alembic/versions/0019_disksage_provider_sync_state.py, backend/alembic/versions/0020_disksage_lineage_column_names.py, backend/alembic/versions/0021_disksage_lineage_scope_index.py, docs/architecture/erd/disksage-file-lineage.dbml	Adds the scoped ORM record and migrations for encrypted envelopes, lineage metadata, synchronization state, constraints, indexes, column renames, and scope index changes.
Lineage API integration backend/api/disksage.py, backend/core/runtime_secrets.py, backend/main.py	Adds authenticated POST and GET endpoints. Ingestion hashes, deduplicates, persists, and redacts lineage records. Listing applies scope, pagination, and newest-first ordering. Missing encryption keys use a dedicated exception. Registers the router with private API dependencies.
Handoff architecture record docs/architecture/disksage-file-lineage-handoff.md	Documents the API boundary, validation rules, encrypted storage, scope restrictions, provider-proof semantics, provenance relationships, deferred integrations, and operational consequences.
DAV route operation identifiers backend/api/dav.py	Registers separate DAV routes for supported HTTP methods and assigns each route a distinct OpenAPI operation ID.
HTML sanitizer cleanup backend/services/text_safety.py	Removes stray --> markers from sanitized output.

Estimated code review effort: 4 (Complex) | ~60 minutes

Mergeability Score: 🟡 Moderate · up to b6167

The current head can corrupt legitimate text during sanitization and may publish invalid DAV OpenAPI schemas that break schema consumers; these bounded correctness and integration issues should be fixed before merging.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ingest_file_lineage
  participant FileLineageEnvelope
  participant Database
  Client->>ingest_file_lineage: POST /api/disksage/file-lineage
  ingest_file_lineage->>FileLineageEnvelope: Validate envelope
  FileLineageEnvelope-->>ingest_file_lineage: Validated lineage data
  ingest_file_lineage->>Database: Hash, deduplicate, and store encrypted envelope
  Database-->>ingest_file_lineage: Stored lineage record
  ingest_file_lineage-->>Client: FileLineageSummary

Loading

Possibly related PRs

• ContextualWisdomLab/naruon#1296: Both changes modify DAV route registration and supported-method advertising.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: persisting DiskSage file-lineage ontology data.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/disksage-file-lineage-ontology

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@cwl-noema-review please review the current head.

[github-actions]

PR governance metadata gate is not ready for 7c1f936221523544300f7e528e11bd57168934cd:

• Required check strix is FAILURE on the current head.

[coderabbitai]

Rate Limit Exceeded

@seonghobae have exceeded the limit for the number of chat messages per hour. Please wait 18 minutes and 21 seconds before sending another message.

[seonghobae]

@opencode-agent review current exact head 208f746728d7737833a2168f525dc4243d5f7354. Review the complete 21-file current-head DiskSage lineage integration diff and publish a formal GitHub review verdict anchored to this SHA. All six repository-local CI/security/dependency/container workflows are terminal-success and every actionable inline thread is resolved. Verify tenant/workspace isolation, concurrent idempotency, typed encryption failure handling, strict versioned provider/copy evidence, migration/index consistency, DAV OpenAPI exclusion, normalized relative paths, and source-eviction authority boundaries.

[seonghobae]

@coderabbitai review current exact head 208f746728d7737833a2168f525dc4243d5f7354; all previously actionable findings are resolved. Publish a current-head verdict and do not reuse the predecessor-head rate-limit state.

[coderabbitai]

Rate Limit Exceeded

@seonghobae have exceeded the limit for the number of chat messages per hour. Please wait 22 minutes and 37 seconds before sending another message.

[seonghobae]

Fixed in dd947ed: organization-lineage IntegrityError replay now uses the user/workspace uniqueness identity and returns HTTP 409 for a conflicting organization, with a regression test. The two informational comments remain safe by design (201 is the endpoint contract for idempotent POST and session teardown rolls back non-key StatementError paths).

[devin-ai-integration]

Devin Review found 3 new potential issues.

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

Reviewed the three new informational notes from the current-head analysis: the scope index intentionally covers user/workspace/order and applies organization as a residual tenant guard; cross-organization fingerprint reuse intentionally returns 409 under the fingerprint-binding contract; EncryptionKeyMissingError remains RuntimeError-compatible for existing callers. No code change is warranted. Resolving these non-blocking threads so the metadata gate reflects the current review state.