feat(ingestion): Outline wiki connector with Confluence-parity silver staging - #1312
Merged
mitasovr merged 3 commits intoJun 15, 2026
Merged
Conversation
…lver staging
Second contributor to the wiki silver classes (class_wiki_pages,
class_wiki_activity, class_wiki_engagement) alongside Confluence.
Connector (wiki/outline, nocode, fully-inlined manifest — passes both
validate and validate-strict):
- wiki_spaces <- collections.list (full refresh)
- wiki_pages <- documents.list, client-side incremental on
updated_at, statusFilter [published, archived]
- wiki_page_versions <- revisions.list (substream of wiki_pages)
- wiki_comments <- comments.list workspace-wide (single threaded
stream: reply = parentCommentId, inline = anchorText)
- wiki_users <- users.list (email directory for identity resolution)
The Outline API is RPC-style (POST {instance}/api/{method}, Bearer key,
offset/limit pagination in the JSON body); 429 handled via Retry-After
capped at 600s, same policy as Confluence.
Live-tested against a self-hosted instance (47 collections / 11314 docs /
1758 comments / 484 users; all 5 streams green, incremental resume
verified). Findings baked in:
- statusFilter ["draft"] returns HTTP 500 on self-hosted Outline and
drafts are owner-scoped anyway -> drafts excluded by design
- embedded createdBy/updatedBy objects carry no email field on
self-hosted instances -> staging models LEFT JOIN the connector's own
wiki_users stream (coalesce embedded-first for cloud deployments)
- 11k+ documents -> 11k+ substream partitions exceeds the 10k CDK
futures limit -> default_concurrency: 4 (same deadlock class fixed
for jira/confluence in constructorfabric#1308)
dbt: outline__bronze_promoted (RMT promotion for 5 bronze tables) +
3 staging models tagged silver:class_wiki_*; column order kept
positionally identical to the confluence__* models (union_by_tag emits
positional UNION ALL). silver/wiki/schema.yml accepted_values extended
with outline / insight_outline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
📝 WalkthroughWalkthroughAdds the Outline wiki connector with Bearer token authentication, five RPC-based ingestion streams (spaces, pages, versions, comments, users), Bronze deduplication and RMT promotion, identity pipeline (snapshot, fields history, identity inputs), and three Silver staging models that compute page metadata, per-author activity sessions, and per-page engagement metrics. ChangesOutline Wiki Connector
Sequence DiagramsequenceDiagram
participant OutlineAPI as Outline API
participant Connector as Connector CDK
participant Bronze as Bronze Layer
participant RMT as RMT
participant Silver as Silver Staging
OutlineAPI->>Connector: /collections.list (Bearer auth)
OutlineAPI->>Connector: /documents.list (incremental via updated_at)
OutlineAPI->>Connector: /revisions.list (partitioned by page_id)
OutlineAPI->>Connector: /comments.list
OutlineAPI->>Connector: /users.list
Connector->>Connector: stamp tenant/source, derive unique_key, normalize timestamps
Connector->>Bronze: write deduplicated records
Bronze->>RMT: promote via outline__bronze_promoted
RMT->>Silver: outline__wiki_pages (join spaces + users for email)
RMT->>Silver: outline__wiki_activity (collapse sessions, aggregate)
RMT->>Silver: outline__wiki_engagement (aggregate comments per page)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/ingestion/connectors/wiki/outline/dbt/outline__wiki_pages.sql (1)
48-48: 💤 Low valueConsider the semantics of
version_count = 0.The
coalesce(version_number, 0)at line 48 ensuresversion_countis never NULL (satisfying the not_null test at schema.yml:36-38), but it allows pages to haveversion_count = 0. Semantically, a page should have at least one version (the creation event). Ifversion_numbercan legitimately be NULL in the bronze table, consider whether downstream consumers expectversion_count ≥ 1or if zero is a valid sentinel for "uncounted."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/connectors/wiki/outline/dbt/outline__wiki_pages.sql` at line 48, The select currently uses toUInt32(coalesce(version_number, 0)) AS version_count which allows pages to have version_count = 0; change the coalesce default to 1 (or otherwise enforce a minimum of 1) so a page always reports at least one version: e.g. toUInt32(coalesce(version_number, 1)) AS version_count or toUInt32(greatest(coalesce(version_number, 0), 1)) AS version_count, and adjust any downstream expectations/tests if you intentionally want to keep 0 as a sentinel.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/ingestion/connectors/wiki/outline/descriptor.yaml`:
- Around line 15-18: The descriptor.yaml's required_fields list erroneously
includes outline_start_date causing a contract mismatch; remove
outline_start_date from the required_fields array in
src/ingestion/connectors/wiki/outline/descriptor.yaml so the descriptor matches
the connector spec (connector.yaml) which provides a default for
outline_start_date and treats it as optional.
---
Nitpick comments:
In `@src/ingestion/connectors/wiki/outline/dbt/outline__wiki_pages.sql`:
- Line 48: The select currently uses toUInt32(coalesce(version_number, 0)) AS
version_count which allows pages to have version_count = 0; change the coalesce
default to 1 (or otherwise enforce a minimum of 1) so a page always reports at
least one version: e.g. toUInt32(coalesce(version_number, 1)) AS version_count
or toUInt32(greatest(coalesce(version_number, 0), 1)) AS version_count, and
adjust any downstream expectations/tests if you intentionally want to keep 0 as
a sentinel.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1809479b-8ecd-4d47-b004-4b4193bf783f
📒 Files selected for processing (10)
src/ingestion/connectors/wiki/outline/README.mdsrc/ingestion/connectors/wiki/outline/connector.yamlsrc/ingestion/connectors/wiki/outline/dbt/outline__bronze_promoted.sqlsrc/ingestion/connectors/wiki/outline/dbt/outline__wiki_activity.sqlsrc/ingestion/connectors/wiki/outline/dbt/outline__wiki_engagement.sqlsrc/ingestion/connectors/wiki/outline/dbt/outline__wiki_pages.sqlsrc/ingestion/connectors/wiki/outline/dbt/schema.ymlsrc/ingestion/connectors/wiki/outline/descriptor.yamlsrc/ingestion/secrets/connectors/outline.yaml.examplesrc/ingestion/silver/wiki/schema.yml
Comment on lines
+15
to
+18
| required_fields: | ||
| - outline_instance_url | ||
| - outline_api_token | ||
| - outline_start_date |
There was a problem hiding this comment.
Configuration contract inconsistency: outline_start_date required status.
outline_start_date is listed as required here, but the connector spec (connector.yaml lines 764-768) does NOT include it in the required array, and it has a default value (lines 798-806: default: "2020-01-01"). The README (line 41) also documents it as optional.
Either remove outline_start_date from this required_fields list, or add it to the connector spec's required array and remove the default. The typical pattern is to make it optional with a default (as the connector spec currently does), so the fix would be to remove it from this list.
🔧 Proposed fix
secret:
required_fields:
- outline_instance_url
- outline_api_token
- - outline_start_date📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| required_fields: | |
| - outline_instance_url | |
| - outline_api_token | |
| - outline_start_date | |
| required_fields: | |
| - outline_instance_url | |
| - outline_api_token |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/ingestion/connectors/wiki/outline/descriptor.yaml` around lines 15 - 18,
The descriptor.yaml's required_fields list erroneously includes
outline_start_date causing a contract mismatch; remove outline_start_date from
the required_fields array in
src/ingestion/connectors/wiki/outline/descriptor.yaml so the descriptor matches
the connector spec (connector.yaml) which provides a default for
outline_start_date and treats it as optional.
Add the standard identity-inputs macro chain for the outline connector — wiki_users exposes emails, so the connector must contribute observations to identity.identity_inputs like zoom/zulip-proxy/bamboohr/ms-entra do: - outline__users_snapshot: SCD2 snapshot of wiki_users (name/email/role/is_suspended) - outline__users_fields_history: field-level change log, entity_id = user_id (Outline user UUID) - outline__identity_inputs: email + display_name observations plus the canonical id binding row (emitted by the macro per ADR-0002); suspension (is_suspended=true) emits DELETE rows - silver/_shared/identity_inputs.sql: depends_on line for first-run build ordering Skill docs updated to close the gap that allowed this omission: - connector-create.md gains §3.6b — when the chain is REQUIRED, model templates for all three macros, and the rules (canonical id row is automatic, bool fields stringify to 'true'/'false', snapshot source must be RMT-promoted, shared-union depends_on) - connector-validate.md gains an "Identity Resolution inputs" checklist section Verified: dbt compile of the new chain plus the shared identity_inputs union is clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/ingestion/connectors/wiki/outline/dbt/schema.yml (1)
14-34: 💤 Low valueConsider adding column-level tests for
outline__users_fields_historyandoutline__identity_inputs.The
outline__users_snapshotmodel includes anot_nulltest onunique_key(line 21-22), butoutline__users_fields_historyandoutline__identity_inputshave no column-level tests. Consider adding tests for critical columns such asentity_id,field_name, or identity key columns to catch data quality issues early.📋 Example tests to add
- name: outline__users_fields_history description: > Field-level change log derived from the users snapshot. One row per changed field per version transition. columns: - name: entity_id tests: - not_null - name: field_name tests: - not_null - name: outline__identity_inputs description: > Identity Manager input rows emitted from the user fields-history. Contributes `email` and `display_name` (from `name`) value types plus the canonical `id` binding row. Deactivation triggered when `is_suspended=true`. columns: - name: source_type tests: - not_null - name: value_type tests: - not_null🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/connectors/wiki/outline/dbt/schema.yml` around lines 14 - 34, Add column-level tests to the two models missing them: update the outline__users_fields_history and outline__identity_inputs entries in schema.yml to declare critical columns and attach tests (e.g., for outline__users_fields_history add columns entity_id and field_name with a not_null test; for outline__identity_inputs add columns like source_type and value_type and/or identity key columns with not_null tests). Keep the existing outline__users_snapshot tests intact and mirror its style when adding the new columns/tests so dbt picks them up.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/ingestion/connectors/wiki/outline/dbt/schema.yml`:
- Around line 14-34: Add column-level tests to the two models missing them:
update the outline__users_fields_history and outline__identity_inputs entries in
schema.yml to declare critical columns and attach tests (e.g., for
outline__users_fields_history add columns entity_id and field_name with a
not_null test; for outline__identity_inputs add columns like source_type and
value_type and/or identity key columns with not_null tests). Keep the existing
outline__users_snapshot tests intact and mirror its style when adding the new
columns/tests so dbt picks them up.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1a4df81c-f812-43bb-b566-a6c24622d2f6
📒 Files selected for processing (8)
cypilot/.core/skills/connector/workflows/create.mdcypilot/.core/skills/connector/workflows/validate.mdsrc/ingestion/connectors/wiki/outline/README.mdsrc/ingestion/connectors/wiki/outline/dbt/outline__identity_inputs.sqlsrc/ingestion/connectors/wiki/outline/dbt/outline__users_fields_history.sqlsrc/ingestion/connectors/wiki/outline/dbt/outline__users_snapshot.sqlsrc/ingestion/connectors/wiki/outline/dbt/schema.ymlsrc/ingestion/silver/_shared/identity_inputs.sql
✅ Files skipped from review due to trivial changes (2)
- src/ingestion/silver/_shared/identity_inputs.sql
- cypilot/.core/skills/connector/workflows/validate.md
…hen-764588 # Conflicts: # src/ingestion/silver/_shared/identity_inputs.sql
cyberantonz
approved these changes
Jun 15, 2026
This was referenced Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an Outline (getoutline.com) connector as the second contributor to the wiki silver classes (
class_wiki_pages,class_wiki_activity,class_wiki_engagement), collecting values analogous to the existingwiki/confluenceconnector.Streams (
src/ingestion/connectors/wiki/outline/)wiki_spacescollections.listwiki_pagesdocuments.list(statusFilter: [published, archived])updated_atwiki_page_versionsrevisions.list(substream ofwiki_pages)wiki_commentscomments.listworkspace-widewiki_usersusers.list(email directory)The Outline API is RPC-style: every endpoint is
POST {instance}/api/{method}with a JSON body, Bearer API key,offset/limitpagination injected intobody_json. 429 is handled viaRetry-Aftercapped at 600s (same policy as Confluence). The manifest is fully inlined (no$ref) and passes bothvalidateandvalidate-strict.Mapping vs Confluence
parentCommentIdset, inline-vs-footer =anchorTextnon-empty (includeAnchorText: true)row_number()overcreatedAt;pages_createdcounts ordinal 1confluence__*(union_by_tagemits positionalUNION ALL);silver/wiki/schema.ymlaccepted_values extended withoutline/insight_outlineLive-tested (self-hosted instance, 2026-06-12)
47 collections / 11,314 documents / 1,758 comments / 484 users — all 5 streams emit with 0 errors, every record carries
tenant_id/source_id/unique_key, incremental resume returns a strict subset (8 of 11,314), substream verified end-to-end on a small collection.Findings baked into the design:
statusFilter: ["draft"]→ HTTP 500 on self-hosted Outline, and drafts are owner-scoped anyway → drafts excluded by design (documented in README)createdBy/updatedByobjects carry no email field on self-hosted instances (0 of 11,314 pages) → staging models LEFT JOIN the connector's ownwiki_usersstream,coalesce(embedded, wiki_users.email)keeps cloud deployments workingdefault_concurrency: 4, same deadlock class fixed for jira/confluence in fix(ingestion): bump jira/confluence concurrency to 4 to break CDK partition deadlock #1308Validation
source.sh validate+validate-strict: both passdbt parse+dbt compileof alltag:outlinemodels: cleanaudit_rmt_read_dedup.py: no findings for outline models (every bronze read dedups viaQUALIFY row_number())outline__bronze_promoted) covers all 5 streamsOperational note
A full
wiki_page_versionssync makes onerevisions.listcall per document (~1h for an 11k-doc instance at concurrency 4); the other four streams finish in minutes. A possible follow-up is deriving activity fromevents.listto avoid the per-document sweep.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation