From ba38b540c12c18faacf68a9a5e37b6e64a6995f0 Mon Sep 17 00:00:00 2001 From: Roman Mitasov Date: Thu, 11 Jun 2026 23:34:35 +0300 Subject: [PATCH 1/3] fix(tooling): correct INGESTION_DIR in generate-catalog.sh and generate-schema.sh Both scripts live in tools/declarative-connector/ but resolved INGESTION_DIR as SCRIPT_DIR/.., which points at tools/ and breaks every invocation (./tools/tools/declarative-connector/source.sh not found). Resolve two levels up instead. Co-Authored-By: Claude Fable 5 Signed-off-by: Roman Mitasov --- src/ingestion/tools/declarative-connector/generate-catalog.sh | 2 +- src/ingestion/tools/declarative-connector/generate-schema.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/ingestion/tools/declarative-connector/generate-catalog.sh b/src/ingestion/tools/declarative-connector/generate-catalog.sh index d1ec810c4a..4b4453d61a 100755 --- a/src/ingestion/tools/declarative-connector/generate-catalog.sh +++ b/src/ingestion/tools/declarative-connector/generate-catalog.sh @@ -13,7 +13,7 @@ set -euo pipefail # --------------------------------------------------------------------------- SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -INGESTION_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +INGESTION_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" CONNECTORS_DIR="${INGESTION_DIR}/connectors" TOOLS_DIR="${INGESTION_DIR}/tools/declarative-connector" diff --git a/src/ingestion/tools/declarative-connector/generate-schema.sh b/src/ingestion/tools/declarative-connector/generate-schema.sh index f853a99074..8f893c17aa 100755 --- a/src/ingestion/tools/declarative-connector/generate-schema.sh +++ b/src/ingestion/tools/declarative-connector/generate-schema.sh @@ -14,7 +14,7 @@ set -euo pipefail # --------------------------------------------------------------------------- SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -INGESTION_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +INGESTION_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" CONNECTORS_DIR="${INGESTION_DIR}/connectors" TOOLS_DIR="${INGESTION_DIR}/tools/declarative-connector" From caf548a6f34bb84d5fa722fd5df499e8c70dd4d6 Mon Sep 17 00:00:00 2001 From: Roman Mitasov Date: Thu, 11 Jun 2026 23:34:50 +0300 Subject: [PATCH 2/3] feat(ingestion): add Figma connector (ui-design, bronze-only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nocode declarative connector for the Figma REST API v1. Five raw streams into bronze_figma: design_projects, design_files (client-side incremental on last_modified), design_file_meta, design_file_versions (start-date bounded), design_file_comments. dbt step is RMT promotion only — Silver (class_design_activity) is a follow-up. API facts verified against figma/rest-api-spec and encoded in the manifest: - no team or member enumeration endpoints (team IDs are required config; User objects carry id/handle only, never email — identity resolution deferred to dbt via handle matching or Enterprise SCIM) - per-endpoint rate-limit tiers bound to the token owner's seat type; the Tier 1 full-document endpoint is deliberately not used - 429 honours Retry-After (600s cap); file-level 403/404 are IGNOREd (invite-only projects, deleted files), team-level ones FAIL the run Manifest is fully inlined (no $refs except schemas) and passes both validate-strict and validate. Live-tested against a real workspace: check/discover/per-stream read green, mandatory fields present, incremental cursor advances on resume; empty versions/comments confirmed as genuine API responses (30-day version retention on Starter plan). Docs: ui-design domain spec and figma.md corrected to v1.1 — the March draft assumed a nonexistent GET /v1/teams/{id}/members endpoint and user emails in version/comment payloads. Co-Authored-By: Claude Fable 5 Signed-off-by: Roman Mitasov --- .../components/connectors/ui-design/README.md | 56 +- .../connectors/ui-design/figma/figma.md | 307 ++--- .../connectors/ui-design/figma/README.md | 100 ++ .../connectors/ui-design/figma/connector.yaml | 1071 +++++++++++++++++ .../figma/dbt/figma__bronze_promoted.sql | 22 + .../connectors/ui-design/figma/dbt/schema.yml | 19 + .../ui-design/figma/descriptor.yaml | 20 + .../secrets/connectors/figma.yaml.example | 15 + 8 files changed, 1389 insertions(+), 221 deletions(-) create mode 100644 src/ingestion/connectors/ui-design/figma/README.md create mode 100644 src/ingestion/connectors/ui-design/figma/connector.yaml create mode 100644 src/ingestion/connectors/ui-design/figma/dbt/figma__bronze_promoted.sql create mode 100644 src/ingestion/connectors/ui-design/figma/dbt/schema.yml create mode 100644 src/ingestion/connectors/ui-design/figma/descriptor.yaml create mode 100644 src/ingestion/secrets/connectors/figma.yaml.example diff --git a/docs/components/connectors/ui-design/README.md b/docs/components/connectors/ui-design/README.md index d90a578ecf..b6bf6515b8 100644 --- a/docs/components/connectors/ui-design/README.md +++ b/docs/components/connectors/ui-design/README.md @@ -1,7 +1,7 @@ # Design Tools Connector Specification (Multi-Source) -> Version 1.0 — March 2026 -> Based on: Figma (Source TBD — to be assigned in CONNECTORS_REFERENCE.md) +> Version 1.1 — June 2026 (Figma API facts verified against figma/rest-api-spec; see `figma/figma.md`) +> Based on: Figma (implemented: `src/ingestion/connectors/ui-design/figma/`, bronze-only) Data-source agnostic specification for design tool connectors. Defines unified Bronze schemas that work across Figma (and future sources: Sketch, Adobe XD) using a `data_source` discriminator column. @@ -45,9 +45,19 @@ Data-source agnostic specification for design tool connectors. Defines unified B - **Version creation** — who created a version of a file, and when (proxy for active editing) - **Comments** — who posted a comment on a file, and when (proxy for async design review/collaboration) -Figma's Analytics API (Enterprise plan only) provides user-level activity summaries via an admin dashboard, but this is not available through a public REST API for non-Enterprise plans. `design_file_activity` is therefore a **derived table** populated at collection time by aggregating version and comment records per user per file per day. +Figma's Analytics API (Enterprise plan only) provides user-level activity summaries via an admin dashboard, but this is not available through a public REST API for non-Enterprise plans. `design_file_activity` is therefore a **derived table** — but per the thin-extractor architecture (ADR `cpt-insightspec-adr-connector-responsibility-scope`) it is derived **in dbt (Silver)**, not by the connector. The connector ships raw streams. -**Why four tables**: Design tool data naturally separates into: (1) per-user activity aggregates (the primary metric table), (2) a file/project directory (the dimension table for file metadata), (3) a user directory (identity anchor), and (4) the standard collection run log. Keeping them separate avoids wide denormalized rows and allows each table to evolve independently. +**Implemented bronze streams (Figma, June 2026)** — raw extraction in `bronze_figma`, each record carrying `tenant_id`/`source_id`/`unique_key`/`data_source`: + +| Stream | Contents | Sync mode | +|--------|----------|-----------| +| `design_projects` | Projects per configured team | Full refresh | +| `design_files` | File directory: `file_key`, `file_name`, `last_modified`, project/team denormalized | Incremental (client-side on `last_modified`) | +| `design_file_meta` | Creator, last_touched_by, `editor_type`, `folder_name`, `link_access` | Full refresh (substream) | +| `design_file_versions` | Raw version history: author id/handle, timestamps, labels | Full refresh (substream, bounded by start date) | +| `design_file_comments` | Raw comments incl. replies, resolution, reactions | Full refresh (substream) | + +The `design_file_activity` / `design_users` tables below remain the **target derived model** for the dbt layer and are kept for reference — with one major correction: `design_users` cannot be populated from the Figma REST API (see below). **Terminology mapping**: @@ -58,7 +68,7 @@ Figma's Analytics API (Enterprise plan only) provides user-level activity summar | File viewed | Figma Analytics (Enterprise only) | — | `design_file_activity.files_viewed` | | File | `GET /v1/projects/{id}/files` | — | `design_files` | | Project | `GET /v1/teams/{id}/projects` | — | `design_files.project_name` (denormalized) | -| User | `GET /v1/teams/{id}/members` | — | `design_users` | +| User | **No REST endpoint** — `GET /v1/teams/{id}/members` does not exist (open feature request). Enterprise SCIM only. | — | `design_users` (Enterprise SCIM, planned) | --- @@ -120,7 +130,9 @@ Directory of all design files and the projects they belong to. Used as a dimensi ### `design_users` — User directory -Identity anchor for design tool analytics. Populated from the team members endpoint. Used to resolve source-native `user_id` to `email` and thence to canonical `person_id` in Silver. +> **Correction (June 2026): not implementable from the Figma REST API.** There is no team/org member enumeration endpoint at any plan tier, and `User` objects in versions/comments/meta responses carry only `id`/`handle`/`img_url` — never email. The only programmatic directory sources are Enterprise-only: the SCIM API (requires SSO + a dedicated SCIM token; `userName` = email) or the Activity Logs API (org admin). This table is therefore **planned for Enterprise tenants only**, populated from SCIM. For everyone else, identity resolution falls back to handle-based name matching in Silver (see Identity Resolution). + +Identity anchor for design tool analytics. Used to resolve source-native `user_id` to `email` and thence to canonical `person_id` in Silver. | Field | Type | Description | |-------|------|-------------| @@ -171,32 +183,30 @@ Monitoring table — not an analytics source. | Unified table | Figma endpoint | Key mapping notes | |---------------|----------------|-------------------| -| `design_files` | `GET /v1/teams/{team_id}/projects` → `GET /v1/projects/{project_id}/files` | `key` → `file_key`; `name` → `file_name`; `last_modified` → `last_modified` | -| `design_users` | `GET /v1/teams/{team_id}/members` | `id` → `user_id`; `email` → `email`; `handle` → `display_name`; `role` → `role` | -| `design_file_activity` (versions) | `GET /v1/files/{file_key}/versions` | Aggregate by `(created_by.id, date(created_at))` → `versions_created` | -| `design_file_activity` (comments) | `GET /v1/files/{file_key}/comments` | Aggregate by `(user.id, date(created_at))` → `comments_posted` | -| `design_file_activity` (views) | Figma Analytics API (Enterprise only) | `files_viewed` — NULL for non-Enterprise plans | +| `design_projects` (bronze) | `GET /v1/teams/{team_id}/projects` | `id` → `project_id`; `name` → `project_name`; team id from config partition | +| `design_files` (bronze) | `GET /v1/projects/{project_id}/files` | `key` → `file_key`; `name` → `file_name`; `last_modified` passthrough | +| `design_file_meta` (bronze) | `GET /v1/files/{file_key}/meta` | `creator`/`last_touched_by` → id + handle (no email); `editorType` → `editor_type` | +| `design_file_versions` (bronze) | `GET /v1/files/{file_key}/versions` | `user` → `author_id`/`author_handle`; raw rows, no aggregation | +| `design_file_comments` (bronze) | `GET /v1/files/{file_key}/comments` | `user` → `author_id`/`author_handle`; `parent_id` → `parent_comment_id`; raw rows | +| `design_users` | Enterprise SCIM `/scim/v2/Users` only (planned) | No REST member endpoint exists; `userName` = email | +| `design_file_activity` (derived, dbt) | — | Aggregated in Silver from `design_file_versions` + `design_file_comments` | +| `design_file_activity` (views) | Figma Analytics (Enterprise dashboard only) | `files_viewed` — no public API; NULL | -**`design_file_activity` construction**: The connector iterates over all files in all projects for all configured teams. For each file: -1. Fetch version history (`/v1/files/{key}/versions`) — extract all versions created since last sync, group by `(created_by.id, date)`, count → `versions_created` -2. Fetch comments (`/v1/files/{key}/comments`) — extract all comments posted since last sync, group by `(user.id, date)`, count → `comments_posted` -3. Merge version and comment aggregates into `design_file_activity` rows; upsert on `(user_id, file_key, date)` +**`design_file_activity` construction (dbt, planned)**: aggregate `design_file_versions` by `(author_id, file_key, date(created_at))` → `versions_created` (with autosave-checkpoint session collapse per the `confluence__wiki_activity` precedent) and `design_file_comments` by `(author_id, file_key, date(created_at))` → `comments_posted`; merge on `(author_id, file_key, date)`. -**Rate limiting**: Figma REST API is rate-limited per token. Large teams with many files will require cursor-based pagination and rate-limit backoff. +**Rate limiting**: per-endpoint tiers tied to the token owner's seat type (verified June 2026). All endpoints used are Tier 2/3: 50–150 req/min on a Dev/Full seat, 5–10 req/min on a viewer seat. The full-document endpoint (`GET /v1/files/{key}`) is Tier 1 (6 req/month on viewer seats) and is not used. `429` carries `Retry-After`. See `figma/figma.md` for the tier table. --- ## Identity Resolution -**Identity anchor**: `design_users` table (`email` field from team members endpoint). +**Identity anchor (corrected June 2026)**: the Figma REST API exposes **no emails** for record authors — `User` objects carry only `id`/`handle`/`img_url`. Resolution paths, in order of preference: -**Resolution process**: -1. Collect `email` from `design_users` (populated from `GET /v1/teams/{team_id}/members`) -2. In `design_file_activity`, `user_id` is the Figma numeric ID — resolve to `email` via `design_users` join at Silver step 1 -3. Normalize email (lowercase, trim) -4. Map to canonical `person_id` via Identity Manager in Silver step 2 +1. **Enterprise SCIM directory** (tenants with Figma Enterprise + SSO): `design_users` populated from `/scim/v2/Users` (`userName` = email) → email join → `person_id` via Identity Manager. +2. **Handle-based matching** (all other tenants): `author_handle` (display name) matched against HR/git display names via the Identity Manager name-matching mechanism (`inbox/IDENTITY_RESOLUTION.md`). +3. The stable `author_id` preserves per-person continuity within Figma even when unresolved. -**`email` availability**: Figma team members endpoint exposes email only for users within the organisation's team. Guest users and external collaborators may have email withheld. +This mirrors confluence (no email in the v2 API, resolved via the `jira_user` join) — except Figma has no companion-product user table, so SCIM or name-matching is the path. --- diff --git a/docs/components/connectors/ui-design/figma/figma.md b/docs/components/connectors/ui-design/figma/figma.md index eb0de2ac3b..bd7a62c130 100644 --- a/docs/components/connectors/ui-design/figma/figma.md +++ b/docs/components/connectors/ui-design/figma/figma.md @@ -1,33 +1,29 @@ # Figma Connector Specification -> Version 1.0 — March 2026 -> Based on: `docs/connectors/design/README.md` (Design Tools domain) +> Version 1.1 — June 2026 (verified against the official OpenAPI spec, figma/rest-api-spec) +> Based on: `docs/components/connectors/ui-design/README.md` (Design Tools domain) +> Implementation: `src/ingestion/connectors/ui-design/figma/` (nocode declarative manifest, bronze-only) -Standalone specification for the Figma (Design Tools) connector. Expands the Design Tools domain schema with Figma-specific API details, authentication options, endpoint mapping, and known limitations. +Standalone specification for the Figma (Design Tools) connector. Expands the Design Tools domain schema with Figma-specific API details, authentication, endpoint mapping, and known limitations. + +**v1.1 corrections over v1.0** (March 2026 draft was written before API verification): + +- `GET /v1/teams/{team_id}/members` **does not exist** in the public REST API — there is no team/org member enumeration endpoint (it is an open feature request on the Figma forum). `design_users` cannot be populated from the REST API. +- The `User` object returned by versions/comments/meta endpoints carries **only `id`, `handle`, `img_url` — never `email`**. Email exists solely on `GET /v1/me` (the token owner). +- `design_file_activity` is **not** built by the connector — connectors are thin extractors (ADR `cpt-insightspec-adr-connector-responsibility-scope`); aggregation is a dbt concern. The connector ships raw streams instead. +- Rate limits are now publicly documented as per-endpoint tiers tied to the token owner's seat type (see below) — the old "~100–120 req/min" estimate is obsolete. - [Overview](#overview) - [Authentication](#authentication) - - [Option 1: Personal Access Token](#option-1-personal-access-token) - - [Option 2: OAuth 2.0 (Figma OAuth App)](#option-2-oauth-20-figma-oauth-app) -- [API Endpoints](#api-endpoints) - - [`GET /v1/me` — Current user info](#get-v1me-current-user-info) - - [`GET /v1/teams/{team_id}/projects` — List projects](#get-v1teamsteamidprojects-list-projects) - - [`GET /v1/projects/{project_id}/files` — List files in project](#get-v1projectsprojectidfiles-list-files-in-project) - - [`GET /v1/teams/{team_id}/members` — Team member directory](#get-v1teamsteamidmembers-team-member-directory) - - [`GET /v1/files/{file_key}/versions` — Version history](#get-v1filesfilekeyversions-version-history) - - [`GET /v1/files/{file_key}/comments` — Comments on file](#get-v1filesfilekeycomments-comments-on-file) -- [Bronze Table Mapping](#bronze-table-mapping) -- [Activity Inference Strategy](#activity-inference-strategy) +- [API Endpoints Used](#api-endpoints-used) +- [Bronze Streams (implemented)](#bronze-streams-implemented) - [Rate Limiting and Pagination](#rate-limiting-and-pagination) - [Known Limitations](#known-limitations) - [Identity Resolution](#identity-resolution) - [Silver / Gold Notes](#silver-gold-notes) - [Open Questions](#open-questions) - - [OQ-FIGMA-1: Figma Analytics API (Enterprise)](#oq-figma-1-figma-analytics-api-enterprise) - - [OQ-FIGMA-2: Team ID configuration vs. organisation-level access](#oq-figma-2-team-id-configuration-vs-organisation-level-access) - - [OQ-FIGMA-3: Version vs. autosave — activity undercount](#oq-figma-3-version-vs-autosave-activity-undercount) @@ -37,197 +33,106 @@ Standalone specification for the Figma (Design Tools) connector. Expands the Des **API**: Figma REST API v1 — `https://api.figma.com/v1/` -**Category**: Design Tools +**Category**: Design Tools (`ui-design`) **`data_source`**: `insight_figma` -**Authentication**: Personal Access Token or OAuth 2.0 +**Authentication**: Personal Access Token (`X-Figma-Token` header) -**Identity**: `email` from `GET /v1/teams/{team_id}/members` — resolved to canonical `person_id` via Identity Manager in Silver step 2. +**Identity**: `author_id` / `author_handle` (Figma user ID + display name) on versions, comments, and file metadata. **No email is available anywhere in the REST API** except `/v1/me`. Resolution to canonical `person_id` is deferred to Silver (handle-based matching via Identity Manager, or an Enterprise SCIM-sourced directory later). -**Critical limitation**: The standard Figma REST API does **not** expose per-user activity aggregates or a user activity feed. There is no `/v1/activity` endpoint equivalent to GitHub's event stream. Activity must be inferred from version history (who created versions) and comments (who posted comments). File view counts are available only on the Figma Enterprise plan via an Analytics feature that has no public REST API as of March 2026. +**Critical limitations**: -**Figma plan tiers relevant to this connector**: -- **Starter / Professional**: version history + comments accessible via REST API — sufficient for `versions_created` and `comments_posted` -- **Organization**: same as Professional for API access -- **Enterprise**: adds Analytics dashboard with user-level activity data — no public API available for programmatic access +- No per-user activity aggregates or activity feed (no `/v1/activity` equivalent). Activity is inferred from version history and comments. +- No team enumeration — `figma_team_ids` is required config, copied from team URLs. +- No member enumeration — user directory requires Enterprise SCIM API (separate token, SSO required) or Enterprise Activity Logs API (org admin, `org:activity_log_read`). +- File view counts: Enterprise-only Library Analytics / admin Analytics; no usable public API for per-file views. +- `GET /v1/files/{key}` (full document tree) is rate-limit **Tier 1** — 6 requests/month on viewer seats — and is deliberately not used. Design content never leaves Figma. --- ## Authentication -### Option 1: Personal Access Token +**Personal Access Token** (implemented): -Simpler to configure; scoped to a single Figma user account. - -- Generate at: **Figma → Account Settings → Personal access tokens** +- Generate at: **Figma → Settings → Security tab → Personal access tokens** — any user can create one, no admin required - Header: `X-Figma-Token: {token}` -- Access scope: read access to all files and teams the generating user can see -- Limitation: tied to a specific user account; if the user leaves, the token is revoked - -### Option 2: OAuth 2.0 (Figma OAuth App) +- Scopes: select read scopes covering projects, file metadata, file versions, file comments +- Access scope: the token sees exactly what its owner sees — the owner must be a member of every configured team +- Rate limits are tied to the owner's **seat type** — use a Dev/Full seat account +- The token is displayed once at creation -Preferred for production deployments; supports org-level access. +**OAuth 2.0** (not implemented): Figma supports OAuth apps (`https://www.figma.com/oauth`), but PAT is sufficient for server-side batch collection and avoids refresh-token management. Revisit if a customer requires it. -- Authorization endpoint: `https://www.figma.com/oauth` -- Token endpoint: `https://www.figma.com/api/oauth/token` -- Required scopes: `file_read` (read files, versions, comments), `team_projects:read` (list projects) -- Flow: Authorization Code with PKCE -- Token lifetime: access token expires; refresh token must be stored securely - -**Connector configuration fields**: +**Connector configuration fields** (see `src/ingestion/connectors/ui-design/figma/README.md` for the K8s Secret): | Field | Required | Description | |-------|----------|-------------| -| `team_ids` | REQUIRED | One or more Figma team IDs to collect from | -| `auth_type` | REQUIRED | `pat` (Personal Access Token) or `oauth2` | -| `access_token` | REQUIRED | PAT value or OAuth access token | -| `refresh_token` | OAuth only | OAuth refresh token for token renewal | -| `client_id` | OAuth only | OAuth app client ID | -| `client_secret` | OAuth only | OAuth app client secret (stored in secret manager) | - ---- - -## API Endpoints - -### `GET /v1/me` — Current user info - -Used at connector startup to validate authentication and retrieve the authenticated user's ID. - -**Response fields used**: - -| Field | Type | Description | -|-------|------|-------------| -| `id` | String | Figma user ID of the authenticated user | -| `email` | String | Email address | -| `handle` | String | Display name | +| `figma_token` | Yes | Personal access token | +| `figma_team_ids` | Yes | Comma-separated team IDs (no default — fail-fast) | +| `figma_start_date` | No | Earliest date of interest, default `2020-01-01` | +| `figma_page_size` | No | Versions page size, default 50 (API max) | --- -### `GET /v1/teams/{team_id}/projects` — List projects - -Enumerate all projects under a configured team. Projects are the top-level organisational units under a team. +## API Endpoints Used -**Path parameter**: `team_id` — from connector configuration. +| Endpoint | Tier | Pagination | Used for | +|----------|------|------------|----------| +| `GET /v1/teams/{team_id}/projects` | 2 | none | `design_projects` | +| `GET /v1/projects/{project_id}/files` | 2 | none | `design_files` (`key`, `name`, `last_modified`) | +| `GET /v1/files/{file_key}/meta` | 3 | none | `design_file_meta` (`creator`, `last_touched_by`, `editorType`, `folder_name`, `link_access`) | +| `GET /v1/files/{file_key}/versions` | 2 | `pagination.next_page` URL, `page_size` ≤ 50 | `design_file_versions` | +| `GET /v1/files/{file_key}/comments` | 2 | none | `design_file_comments` (top-level + replies via `parent_id`) | -**Response fields used**: +Endpoints **not** used and why: -| Field | Type | Description | -|-------|------|-------------| -| `projects[].id` | String | Project ID → `design_files.project_id` | -| `projects[].name` | String | Project name → `design_files.project_name` (denormalized) | +| Endpoint | Why not | +|----------|---------| +| `GET /v1/files/{key}` (full node tree) | Tier 1 rate limit; design content not needed for analytics | +| `GET /v1/teams/{id}/components` / `styles` | Design-system metrics — out of scope for v1 | +| SCIM `/Users` | Enterprise + SSO only; candidate for an optional `design_users` stream later | +| `GET /v1/activity_logs` | Enterprise only, org admin token | +| Webhooks v2 | Batch polling model; no push infrastructure needed | --- -### `GET /v1/projects/{project_id}/files` — List files in project - -List all files within a project. Populates `design_files`. - -**Response fields used**: - -| Field | Type | Description | -|-------|------|-------------| -| `files[].key` | String | File key → `design_files.file_key` and `design_file_activity.file_key` | -| `files[].name` | String | File name → `design_files.file_name` | -| `files[].last_modified` | String (ISO 8601) | Last modification timestamp → `design_files.last_modified` | -| `files[].thumbnail_url` | String | Not collected — not relevant for analytics | - ---- - -### `GET /v1/teams/{team_id}/members` — Team member directory - -Retrieve team members for identity resolution. Populates `design_users`. - -**Response fields used**: - -| Field | Type | Description | -|-------|------|-------------| -| `members[].id` | String | Figma user ID → `design_users.user_id` | -| `members[].email` | String | User email → `design_users.email` — primary identity anchor | -| `members[].handle` | String | Display name → `design_users.display_name` | -| `members[].role` | String | Team role: `owner` / `admin` / `editor` / `viewer` → `design_users.role` | - -**Note**: `email` is only returned for users within the organisation's team. Guest/external collaborators may not have their email exposed. See OQ-DESIGN-3 in domain README. - ---- - -### `GET /v1/files/{file_key}/versions` — Version history - -Retrieve the full version history for a file. The primary source for `versions_created` in `design_file_activity`. +## Bronze Streams (implemented) -**No date filter parameter** — the API returns the complete version history. Incremental sync must be implemented client-side by filtering on `created_at > last_sync_cursor`. +Raw extraction, namespace `bronze_figma`. Every record carries `tenant_id`, `source_id`, `unique_key`, `data_source='insight_figma'`, `collected_at`. -**Response fields used**: +| Stream | Natural key | Sync mode | Notes | +|--------|-------------|-----------|-------| +| `design_projects` | `project_id` | full refresh | Partitioned by configured `figma_team_ids`; `team_id` denormalized | +| `design_files` | `file_key` | incremental (client-side cursor on `last_modified`) | `project_id`, `project_name`, `team_id` denormalized from parent | +| `design_file_meta` | `file_key` | full refresh (substream) | `creator_id/handle`, `last_touched_by_id/handle`, `last_touched_at`, `editor_type` (figma/figjam/slides/…), `link_access` | +| `design_file_versions` | `file_key` + `version_id` | full refresh (substream) | `author_id/handle`, `created_at`, `label`, `description`; bounded by `figma_start_date` (record filter + pagination stop) | +| `design_file_comments` | `file_key` + `comment_id` | full refresh (substream) | `parent_comment_id` for replies, `resolved_at`, `message`, `order_id`, `reaction_count` | -| Field | Type | Description | -|-------|------|-------------| -| `versions[].id` | String | Version ID — used for deduplication and cursor tracking | -| `versions[].created_at` | String (ISO 8601) | Version creation timestamp — extract date for `design_file_activity.date` | -| `versions[].user.id` | String | ID of the user who created the version → maps to `design_users.user_id` | -| `versions[].user.email` | String | Email of the version creator (when available) | -| `versions[].label` | String | Optional human-readable version label — not collected | +Differences from the v1.0 draft's `design_*` table plan: -**Aggregation**: Group by `(user.id, date(created_at))` → count → `design_file_activity.versions_created`. - ---- - -### `GET /v1/files/{file_key}/comments` — Comments on file - -Retrieve all comments on a file. The primary source for `comments_posted` in `design_file_activity`. - -**No date filter parameter** — returns all comments. Incremental sync uses `created_at > last_sync_cursor` client-side. - -**Response fields used**: - -| Field | Type | Description | -|-------|------|-------------| -| `comments[].id` | String | Comment ID — deduplication | -| `comments[].created_at` | String (ISO 8601) | Comment creation timestamp — extract date for `design_file_activity.date` | -| `comments[].user.id` | String | ID of the user who posted the comment → maps to `design_users.user_id` | -| `comments[].user.email` | String | Email of the commenter (when available) | -| `comments[].resolved_at` | String (ISO 8601) | Null if unresolved — not collected for activity metrics | - -**Aggregation**: Group by `(user.id, date(created_at))` → count → `design_file_activity.comments_posted`. - ---- - -## Bronze Table Mapping - -| Bronze table | Source endpoint(s) | Collection strategy | -|-------------|-------------------|---------------------| -| `design_files` | `GET /v1/teams/{id}/projects` + `GET /v1/projects/{id}/files` | Full refresh on each run — file directory changes rarely; overwrite on `file_key` | -| `design_users` | `GET /v1/teams/{id}/members` | Full refresh on each run — team membership changes infrequently | -| `design_file_activity` | `GET /v1/files/{key}/versions` + `GET /v1/files/{key}/comments` | Incremental — filter `created_at > last_sync_cursor`; upsert on `(user_id, file_key, date)` | -| `design_collection_runs` | Internal connector state | One row inserted per run | - ---- - -## Activity Inference Strategy - -Because Figma provides no native activity feed, `design_file_activity` is constructed as follows for each connector run: - -1. **Collect file list**: iterate `team_ids` → projects → files. Upsert all files into `design_files`. -2. **Collect user list**: iterate `team_ids` → members. Upsert all members into `design_users`. -3. **For each file in `design_files`**: - a. Fetch version history since `last_sync_cursor`. For each version where `created_at > last_sync_cursor`: record `(created_by.id, date(created_at))`. - b. Fetch comments since `last_sync_cursor`. For each comment where `created_at > last_sync_cursor`: record `(user.id, date(created_at))`. -4. **Aggregate**: Group version records by `(user_id, file_key, date)` → `versions_created`. Group comment records by `(user_id, file_key, date)` → `comments_posted`. -5. **Merge**: For each `(user_id, file_key, date)` combination, produce one `design_file_activity` row. Upsert into Bronze. -6. **Resolve email**: For each `user_id` in the aggregated rows, look up `email` from `design_users`. If not found, emit a warning and skip (see OQ-DESIGN-3). -7. **Update cursor**: set `last_sync_cursor = max(created_at)` across all collected version and comment records. +- `design_users` — **dropped**: no REST endpoint exists. See Identity Resolution. +- `design_file_activity` — **moved to dbt** (planned staging model), not a connector output. +- `design_collection_runs` — superseded by platform-level run tracking (Argo/Airbyte job logs); modern connectors do not emit a runs stream. --- ## Rate Limiting and Pagination -**Rate limit**: Figma API enforces per-token rate limits. Exact limits are not publicly documented but observed to be in the range of 100–120 requests per minute per token. +Figma publishes per-endpoint **tiers**; budgets depend on the token owner's seat type and the workspace plan ([developers.figma.com/docs/rest-api/rate-limits](https://developers.figma.com/docs/rest-api/rate-limits/)): -**Pagination**: Most list endpoints return results in a single response (no standard pagination). Version history for files with many versions may be large — filter client-side. +| Tier | Example endpoints | Viewer/Collab seat | Dev/Full seat (Pro / Org) | +|------|-------------------|--------------------|---------------------------| +| 1 | full file, images | **6/month** | 15–20/min | +| 2 | projects, files, versions, comments | 5/min | 50–100/min | +| 3 | file meta, components, users | 10/min | 100–150/min | -**Recommended strategy**: -- Add `time.sleep(0.6)` between file-level requests (100 req/min ceiling) -- Implement exponential backoff on HTTP 429 responses -- For large teams (>500 files), spread collection across multiple runs using a round-robin file queue +- All five endpoints used by the connector are Tier 2/3. +- On `429` the API returns `Retry-After`; the manifest honours it with a 600 s cap (confluence pattern), retries 5xx with backoff. +- Per-file fan-out is 3 requests (meta + versions + comments). At 50 req/min (Pro, Dev seat) a 500-file workspace takes ~30 min per full pass; client-side incremental on `last_modified` keeps subsequent syncs short. +- File-level `403`/`404` (invite-only projects, deleted files) are IGNOREd; team-level `401`/`403`/`404` FAIL the run (bad token or team ID). + +**Pagination**: only `/versions` paginates — `pagination.next_page` is a full URL injected via `RequestPath`, `page_size` max 50, newest-first ordering (which makes the `figma_start_date` pagination stop-condition correct). --- @@ -235,34 +140,48 @@ Because Figma provides no native activity feed, `design_file_activity` is constr | Limitation | Impact | Mitigation | |-----------|--------|------------| -| No per-user activity feed endpoint | `design_file_activity` is inferred, not directly observed — activity is undercounted if a user edits without saving a named version | Document as proxy metric; recommend "Auto-save creates versions" Figma setting in team guidelines | -| `files_viewed` unavailable (non-Enterprise) | `design_file_activity.files_viewed` is NULL for all non-Enterprise tenants | Field present in schema; populate when/if Figma adds API access for Enterprise Analytics | -| Version history — no server-side date filter | Full version history fetched per file on first sync; large files with long history are slow | Store `last_version_id` as cursor; skip already-seen versions by ID rather than re-fetching | -| Guest users — email may be absent | Activity from external collaborators cannot be attributed to a `person_id` | Omit unresolvable rows; track count in `design_collection_runs.errors` as a quality signal | -| Comments — no edit/delete tracking | Deleted or edited comments are not tracked | Acceptable for activity counting — comment deletion is rare and does not affect aggregate counts significantly | -| OAuth token expiry | If the refresh token is not renewed, collection stops silently | Alert on HTTP 401 response; surface in `design_collection_runs.status = "failed"` | +| No member enumeration in REST API | No `design_users` stream; no emails for authors | Identity via handle-matching in Silver; optional SCIM-based directory stream for Enterprise tenants later | +| `User` objects have no email | Author attribution is `id` + display name only | Same as above | +| No activity feed | Activity inferred from versions + comments; editing without version checkpoints is undercounted | Document as proxy metric; recommend enabling autosave-version setting in Figma org settings | +| No server-side date filter on versions/comments | First sync walks history | `figma_start_date` record filter + pagination stop on versions; comments are a single response per file | +| Seat-type-dependent rate limits | Viewer-seat token makes the connector unusably slow | README requires a Dev/Full seat token owner | +| No team enumeration | New Figma teams require a config update | Documented in README; `figma_team_ids` has no default (fail-fast) | +| View counts unavailable | No `files_viewed` metric | Revisit if Figma ships a public Analytics API | --- ## Identity Resolution -**Source field**: `email` from `GET /v1/teams/{team_id}/members` → `design_users.email` +**What the API gives us**: `author_id` (stable Figma user ID) and `author_handle` (display name) on versions, comments, file meta. No email. + +**Resolution chain (Silver, future)**: + +``` +design_file_versions.author_id / author_handle + → Identity Manager name-matching (handle ≈ display name in HR/git sources) + → person_id +``` -**In `design_file_activity`**: `user_id` (Figma numeric ID) is the native key. At Silver step 1, join `design_file_activity.user_id` → `design_users.user_id` → `design_users.email` to resolve to email. At Silver step 2, the Identity Manager maps `email` → `person_id`. +Fallbacks, in order of preference: -**Email format**: Figma emails are standard corporate email addresses. In most organisations they match the email used in HR, git, and other tools — making them a reliable cross-system identity key. +1. **Enterprise SCIM directory** (if tenant has Figma Enterprise + SSO): `userName` = email, full name → straightforward email join. Candidate optional stream. +2. **Handle-based matching** via Identity Manager (display-name aliases, same mechanism as first-name alias matching in `inbox/IDENTITY_RESOLUTION.md`). +3. Stable `author_id` keeps per-person continuity inside Figma even when unresolved to `person_id`. + +This mirrors the confluence situation (no email in the v2 API; resolved via `jira_user` join) — except Figma has no companion product table to join, so name-matching or SCIM is the path. --- ## Silver / Gold Notes -`design_file_activity` feeds `class_design_activity` (Silver, planned). Aggregation from Bronze to Silver collapses the per-file dimension: +Bronze-only delivery for now; `dbt_select: tag:figma+` runs only `figma__bronze_promoted` (RMT promotion). + +Planned Silver step 1 (staging) models, by analogy with confluence: -- `class_design_activity.files_edited` = count of distinct `file_key` values where `versions_created > 0` on a given `(person_id, date)` -- `class_design_activity.versions_created` = sum of `design_file_activity.versions_created` across all files on `(person_id, date)` -- `class_design_activity.comments_posted` = sum of `design_file_activity.comments_posted` across all files on `(person_id, date)` +- `figma__design_activity` → `class_design_activity`: per-author per-day rollup from `design_file_versions` (session-collapse like `confluence__wiki_activity` — Figma autosave checkpoints need the same 30-min gap treatment) plus comment counts from `design_file_comments`. +- `design_files` + `design_file_meta` serve as dimension tables. -**Designer↔engineer correlation (Gold)**: `class_design_activity` can be joined with `class_commits` on `(person_id, date ± N days)` and with `design_files.project_name` correlated against repository names or YouTrack/Jira project keys. High overlap between design activity and commit activity on the same project — within the same sprint window — is a signal of tight designer↔engineer collaboration. +**Designer↔engineer correlation (Gold)**: `class_design_activity` joined with `class_commits` on `(person_id, date ± N days)`; `design_files.project_name` correlated with repo / Jira project names. --- @@ -270,24 +189,16 @@ Because Figma provides no native activity feed, `design_file_activity` is constr ### OQ-FIGMA-1: Figma Analytics API (Enterprise) -Figma Enterprise plan exposes activity analytics via its admin dashboard (files viewed, time in editor, active users). There is no documented public REST API for this data as of March 2026. - -**Question**: Should the connector attempt to access Figma's internal analytics endpoints (via browser session or undocumented API) for Enterprise customers, or wait for a public API release? - -**Current decision**: No undocumented endpoints. `files_viewed` remains NULL until a public API is available. Revisit when Figma announces an Analytics API. - -### OQ-FIGMA-2: Team ID configuration vs. organisation-level access - -Figma's API is team-scoped — the connector must be configured with explicit `team_ids`. Large organisations may have many teams. +Unchanged from v1.0: no public API for view/edit analytics. Decision: no undocumented endpoints; revisit when Figma announces an Analytics API. -**Question**: Is there an organisation-level endpoint to enumerate all teams? Or must each team be added to `connector.settings.team_ids` manually? +### OQ-FIGMA-2: Team enumeration -**Current approach**: Manual `team_ids` configuration per connector instance. An admin-level OAuth app with `org:read` scope may provide team enumeration — verify with Figma API documentation. +**Resolved (June 2026)**: there is no org-level team enumeration endpoint in the REST API at any plan tier. `figma_team_ids` manual configuration is the only option. The Enterprise Activity Logs API exposes team-related events but is not a directory. ### OQ-FIGMA-3: Version vs. autosave — activity undercount -Figma distinguishes between autosave events (frequent, not exposed via API) and named or auto-generated versions (exposed via `/versions`). A designer who edits a file without triggering a version save will not appear in `versions_created`. +Unchanged: versions are checkpoint-based; editing without checkpoints is invisible. Mitigation: recommend the org setting that auto-creates versions; treat `versions_created` as a proxy metric. The Silver session-collapse must also avoid over-counting autosave checkpoint bursts (see `confluence__wiki_activity` precedent). -**Question**: What is the typical version creation frequency relative to editing sessions? Is `versions_created` a reliable enough proxy for editing activity? +### OQ-FIGMA-4 (new): SCIM-based `design_users` stream -**Current approach**: Document as a known proxy metric. Encourage teams to enable "Autosave creates a version" in Figma Organisation settings to improve signal quality. +For Enterprise + SSO tenants, the SCIM API (`GET /scim/v2/Users`, max 3000/page) provides the full directory with emails — exactly what identity resolution wants. Needs a separate SCIM token and applies to a minority of tenants. Decide whether to add it as an optional conditional stream when the first Enterprise tenant lands. diff --git a/src/ingestion/connectors/ui-design/figma/README.md b/src/ingestion/connectors/ui-design/figma/README.md new file mode 100644 index 0000000000..718b610747 --- /dev/null +++ b/src/ingestion/connectors/ui-design/figma/README.md @@ -0,0 +1,100 @@ +# Figma Connector + +Extracts the design workspace inventory and collaboration signals from the +Figma REST API v1 (projects, files, file metadata, version history, comments) +using a personal access token. Bronze-only: raw streams land in +`bronze_figma`, no Silver transformations yet. + +## Prerequisites + +1. **Personal access token** — any Figma user can create one: Figma → + Settings → **Security** tab → Personal access tokens → Generate new token. + Select read-only scopes covering projects, file metadata, file versions + and file comments. The token is shown once — copy immediately. +2. **Token owner matters twice**: + - The token sees exactly what the owner sees — the account must be a + member of every team you want to collect. + - API rate limits are tied to the owner's seat type. A Dev/Full seat gets + 50–150 req/min on the endpoints this connector uses; a viewer/collab + seat gets 5–10 req/min. Use a Dev/Full seat account. +3. **Team IDs** — the API cannot enumerate teams. Open each team in the + browser and copy the ID from the URL: `figma.com/files/team//...` + +## K8s Secret + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-figma-main + labels: + app.kubernetes.io/part-of: insight + annotations: + insight.cyberfabric.com/connector: figma + insight.cyberfabric.com/source-id: figma-main +type: Opaque +stringData: + figma_token: "CHANGE_ME" + figma_team_ids: "CHANGE_ME" # comma-separated team IDs + figma_start_date: "2020-01-01" +``` + +### Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `figma_token` | Yes | Personal access token (`X-Figma-Token` header). | +| `figma_team_ids` | Yes | Comma-separated Figma team IDs. No default — must be set explicitly. | +| `figma_start_date` | No | Earliest date of interest (YYYY-MM-DD). Files not modified since, and versions created before, this date are skipped. Defaults to `2020-01-01`. | +| `figma_page_size` | No | Page size for the versions endpoint (default 50 = API max). | + +### Automatically injected + +| Field | Source | +|-------|--------| +| `insight_tenant_id` | `tenant_id` from tenant YAML | +| `insight_source_id` | `insight.cyberfabric.com/source-id` annotation | + +### Local development + +Create `src/ingestion/secrets/connectors/figma.yaml` (gitignored) from the example: + +```bash +cp src/ingestion/secrets/connectors/figma.yaml.example src/ingestion/secrets/connectors/figma.yaml +# Fill in real values, then apply: +kubectl apply -f src/ingestion/secrets/connectors/figma.yaml +``` + +## Streams + +| Stream | Description | Sync Mode | +|--------|-------------|-----------| +| `design_projects` | Projects per configured team (`GET /v1/teams/{id}/projects`). | Full Refresh | +| `design_files` | Files per project: key, name, `last_modified`, denormalized project/team (`GET /v1/projects/{id}/files`). | Incremental (client-side cursor on `last_modified`) | +| `design_file_meta` | Per-file metadata: creator, last_touched_by, editor type (figma/figjam/slides/…), link access (`GET /v1/files/{key}/meta`). | Full Refresh (substream) | +| `design_file_versions` | Version history: author, timestamp, label/description (`GET /v1/files/{key}/versions`). Bounded by `figma_start_date` via record filter + pagination stop. | Full Refresh (substream) | +| `design_file_comments` | Comments incl. replies (`parent_comment_id`), resolution timestamps, reaction counts (`GET /v1/files/{key}/comments`). | Full Refresh (substream) | + +API behaviors encoded in the manifest: + +- **Rate limits** are tiered per endpoint and per token-owner seat. All five + endpoints used are Tier 2/3. The full-document endpoint + (`GET /v1/files/{key}`, Tier 1 — 6 req/month on viewer seats) is + deliberately not used; design content never leaves Figma. +- **429** is retried honouring `Retry-After` (capped at 600 s, like + confluence); 5xx retried with backoff. +- **403/404 on per-file endpoints are IGNOREd** — invite-only projects and + deleted files are routine and must not kill the sync. On the team-level + `design_projects` stream they FAIL, because there they mean a bad token or + team ID. +- **No user directory**: the REST API has no team/org member enumeration + (Enterprise-only SCIM does), and `User` objects carry only `id`/`handle` — + never email. Identity resolution is deferred to dbt (handle-based matching + via the Identity Manager, or an Enterprise SCIM directory later). + +## Silver Targets + +None yet — bronze-only delivery. `dbt_select: tag:figma+` runs +`figma__bronze_promoted` (ReplacingMergeTree promotion of the five bronze +tables). The planned Silver stream is `class_design_activity` per +`docs/components/connectors/ui-design/README.md`. diff --git a/src/ingestion/connectors/ui-design/figma/connector.yaml b/src/ingestion/connectors/ui-design/figma/connector.yaml new file mode 100644 index 0000000000..92ee979004 --- /dev/null +++ b/src/ingestion/connectors/ui-design/figma/connector.yaml @@ -0,0 +1,1071 @@ +version: 7.0.4 + +type: DeclarativeSource + +# Figma REST API v1 connector (ui-design domain, data_source=insight_figma). +# +# Raw extraction only — no aggregation in the connector (thin-extractor per +# ADR cpt-insightspec-adr-connector-responsibility-scope). Activity rollups +# (design_file_activity per the ui-design domain spec) are a downstream dbt +# concern. +# +# API facts that shape this manifest (verified against figma/rest-api-spec, +# June 2026): +# - No endpoint enumerates teams — figma_team_ids is required config, taken +# from the team URL (figma.com/files/team//...). +# - No endpoint enumerates team/org members; the User object in responses +# carries only id/handle/img_url (email exists solely on /v1/me). Identity +# resolution happens later in dbt from author_id/author_handle. +# - GET /v1/files/{key} (full document) is rate-limit Tier 1 (6 req/month on +# viewer seats) and is deliberately NOT used. All endpoints below are +# Tier 2/3. +# - /versions is the only paginated endpoint used (page_size max 50, +# pagination.next_page full URL); projects/files/comments return a single +# response. +# - 429 carries Retry-After; honoured with the same 600s cap as confluence. +# - File-level 403/404 are routine (invite-only projects, deleted files) and +# must not kill the sync — IGNOREd on per-file substreams, FAILed on the +# team-level stream where they mean a bad token or team id. +# +# Builder-UI compatibility notes: substream parents are INLINED full stream +# definitions (whole-object $ref like "#/streams/N" fails validate-strict), +# and url_base is an inline literal (string-typed slots reject granular +# $ref). Inline parent copies are minimal — no transformations except where +# a partition value must be denormalized (team_id on design_projects), with +# parent_key pointing at raw API fields (id, key). Repetition is the price +# of Builder compatibility. + +check: + type: CheckStream + stream_names: + - design_projects + +definitions: + design_projects_schema: + type: object + $schema: http://json-schema.org/schema# + required: + - unique_key + - tenant_id + - source_id + properties: + unique_key: {type: ["string", "null"]} + tenant_id: {type: ["string", "null"]} + source_id: {type: ["string", "null"]} + data_source: {type: ["string", "null"]} + project_id: {type: ["string", "null"]} + project_name: {type: ["string", "null"]} + team_id: {type: ["string", "null"]} + collected_at: {type: ["string", "null"]} + additionalProperties: true + + design_files_schema: + type: object + $schema: http://json-schema.org/schema# + required: + - unique_key + - tenant_id + - source_id + properties: + unique_key: {type: ["string", "null"]} + tenant_id: {type: ["string", "null"]} + source_id: {type: ["string", "null"]} + data_source: {type: ["string", "null"]} + file_key: {type: ["string", "null"]} + file_name: {type: ["string", "null"]} + project_id: {type: ["string", "null"]} + project_name: {type: ["string", "null"]} + team_id: {type: ["string", "null"]} + last_modified: {type: ["string", "null"]} + collected_at: {type: ["string", "null"]} + additionalProperties: true + + design_file_meta_schema: + type: object + $schema: http://json-schema.org/schema# + required: + - unique_key + - tenant_id + - source_id + properties: + unique_key: {type: ["string", "null"]} + tenant_id: {type: ["string", "null"]} + source_id: {type: ["string", "null"]} + data_source: {type: ["string", "null"]} + file_key: {type: ["string", "null"]} + folder_name: {type: ["string", "null"]} + creator_id: {type: ["string", "null"]} + creator_handle: {type: ["string", "null"]} + last_touched_by_id: {type: ["string", "null"]} + last_touched_by_handle: {type: ["string", "null"]} + last_touched_at: {type: ["string", "null"]} + editor_type: {type: ["string", "null"]} + link_access: {type: ["string", "null"]} + collected_at: {type: ["string", "null"]} + additionalProperties: true + + design_file_versions_schema: + type: object + $schema: http://json-schema.org/schema# + required: + - unique_key + - tenant_id + - source_id + properties: + unique_key: {type: ["string", "null"]} + tenant_id: {type: ["string", "null"]} + source_id: {type: ["string", "null"]} + data_source: {type: ["string", "null"]} + version_id: {type: ["string", "null"]} + file_key: {type: ["string", "null"]} + created_at: {type: ["string", "null"]} + label: {type: ["string", "null"]} + description: {type: ["string", "null"]} + author_id: {type: ["string", "null"]} + author_handle: {type: ["string", "null"]} + collected_at: {type: ["string", "null"]} + additionalProperties: true + + design_file_comments_schema: + type: object + $schema: http://json-schema.org/schema# + required: + - unique_key + - tenant_id + - source_id + properties: + unique_key: {type: ["string", "null"]} + tenant_id: {type: ["string", "null"]} + source_id: {type: ["string", "null"]} + data_source: {type: ["string", "null"]} + comment_id: {type: ["string", "null"]} + file_key: {type: ["string", "null"]} + parent_comment_id: {type: ["string", "null"]} + author_id: {type: ["string", "null"]} + author_handle: {type: ["string", "null"]} + created_at: {type: ["string", "null"]} + resolved_at: {type: ["string", "null"]} + message: {type: ["string", "null"]} + order_id: {type: ["string", "null"]} + reaction_count: {type: ["integer", "null"]} + collected_at: {type: ["string", "null"]} + additionalProperties: true + +streams: + # ── Stream 0: design_projects ────────────────────────────────────────── + # One partition per configured team id. GET /v1/teams/{id}/projects + # returns all projects in a single response (no pagination). + - type: DeclarativeStream + name: design_projects + primary_key: + - unique_key + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/definitions/design_projects_schema" + retriever: + type: SimpleRetriever + partition_router: + type: ListPartitionRouter + values: "{{ config['figma_team_ids'].replace(' ', '').split(',') }}" + cursor_field: team_id + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/teams/{{ stream_partition.team_id }}/projects + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401, 403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - projects + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [tenant_id] + value: "{{ config['insight_tenant_id'] }}" + - type: AddedFieldDefinition + path: [source_id] + value: "{{ config['insight_source_id'] }}" + - type: AddedFieldDefinition + path: [data_source] + value: insight_figma + - type: AddedFieldDefinition + path: [project_id] + value: "{{ record['id'] }}" + - type: AddedFieldDefinition + path: [project_name] + value: "{{ record.get('name', '') }}" + - type: AddedFieldDefinition + path: [team_id] + value: "{{ stream_partition.team_id }}" + - type: AddedFieldDefinition + path: [collected_at] + value: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.000Z') }}" + - type: AddedFieldDefinition + path: [unique_key] + value: >- + {{ config['insight_tenant_id'] }}-{{ config['insight_source_id'] + }}-{{ record['id'] }} + + # ── Stream 1: design_files ───────────────────────────────────────────── + # GET /v1/projects/{id}/files per parent project, single response. + # Client-side incremental on last_modified (no server-side filter exists), + # same pattern as confluence wiki_pages. The inline parent copy of + # design_projects carries one AddFields (team_id from the partition) so + # team_id can be denormalized onto file records via extra_fields. + - type: DeclarativeStream + name: design_files + primary_key: + - unique_key + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/definitions/design_files_schema" + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: id + partition_field: project_id + extra_fields: + - [name] + - [team_id] + stream: + type: DeclarativeStream + name: design_projects + primary_key: + - id + retriever: + type: SimpleRetriever + partition_router: + type: ListPartitionRouter + values: "{{ config['figma_team_ids'].replace(' ', '').split(',') }}" + cursor_field: team_id + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/teams/{{ stream_partition.team_id }}/projects + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401, 403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - projects + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [team_id] + value: "{{ stream_partition.team_id }}" + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/projects/{{ stream_partition.project_id }}/files + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - files + incremental_sync: + type: DatetimeBasedCursor + cursor_field: last_modified + is_client_side_incremental: true + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%SZ" + - "%Y-%m-%dT%H:%M:%S.%fZ" + - "%Y-%m-%d" + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('figma_start_date', '2020-01-01') }}" + datetime_format: "%Y-%m-%d" + end_datetime: + type: MinMaxDatetime + datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}" + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [tenant_id] + value: "{{ config['insight_tenant_id'] }}" + - type: AddedFieldDefinition + path: [source_id] + value: "{{ config['insight_source_id'] }}" + - type: AddedFieldDefinition + path: [data_source] + value: insight_figma + - type: AddedFieldDefinition + path: [file_key] + value: "{{ record['key'] }}" + - type: AddedFieldDefinition + path: [file_name] + value: "{{ record.get('name', '') }}" + - type: AddedFieldDefinition + path: [project_id] + value: "{{ stream_partition.project_id }}" + - type: AddedFieldDefinition + path: [project_name] + value: "{{ (stream_slice.extra_fields or {}).get('name', '') }}" + - type: AddedFieldDefinition + path: [team_id] + value: "{{ (stream_slice.extra_fields or {}).get('team_id', '') }}" + - type: AddedFieldDefinition + path: [collected_at] + value: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.000Z') }}" + - type: AddedFieldDefinition + path: [unique_key] + value: >- + {{ config['insight_tenant_id'] }}-{{ config['insight_source_id'] + }}-{{ record['key'] }} + + # ── Stream 2: design_file_meta ───────────────────────────────────────── + # GET /v1/files/{key}/meta — the only place the API exposes creator and + # last_touched_by (User objects: id/handle only, no email). Tier 3. + - type: DeclarativeStream + name: design_file_meta + primary_key: + - unique_key + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/definitions/design_file_meta_schema" + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: key + partition_field: file_key + stream: + type: DeclarativeStream + name: design_files + primary_key: + - key + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: id + partition_field: project_id + stream: + type: DeclarativeStream + name: design_projects + primary_key: + - id + retriever: + type: SimpleRetriever + partition_router: + type: ListPartitionRouter + values: "{{ config['figma_team_ids'].replace(' ', '').split(',') }}" + cursor_field: team_id + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/teams/{{ stream_partition.team_id }}/projects + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401, 403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - projects + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/projects/{{ stream_partition.project_id }}/files + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - files + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/files/{{ stream_partition.file_key }}/meta + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - file + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [tenant_id] + value: "{{ config['insight_tenant_id'] }}" + - type: AddedFieldDefinition + path: [source_id] + value: "{{ config['insight_source_id'] }}" + - type: AddedFieldDefinition + path: [data_source] + value: insight_figma + - type: AddedFieldDefinition + path: [file_key] + value: "{{ stream_partition.file_key }}" + - type: AddedFieldDefinition + path: [folder_name] + value: "{{ record.get('folder_name', '') }}" + - type: AddedFieldDefinition + path: [creator_id] + value: "{{ (record.get('creator') or {}).get('id', '') }}" + - type: AddedFieldDefinition + path: [creator_handle] + value: "{{ (record.get('creator') or {}).get('handle', '') }}" + - type: AddedFieldDefinition + path: [last_touched_by_id] + value: "{{ (record.get('last_touched_by') or {}).get('id', '') }}" + - type: AddedFieldDefinition + path: [last_touched_by_handle] + value: "{{ (record.get('last_touched_by') or {}).get('handle', '') }}" + - type: AddedFieldDefinition + path: [last_touched_at] + value: "{{ record.get('last_touched_at', '') }}" + - type: AddedFieldDefinition + path: [editor_type] + value: "{{ record.get('editorType', '') }}" + - type: AddedFieldDefinition + path: [link_access] + value: "{{ record.get('link_access', '') }}" + - type: AddedFieldDefinition + path: [collected_at] + value: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.000Z') }}" + - type: AddedFieldDefinition + path: [unique_key] + value: >- + {{ config['insight_tenant_id'] }}-{{ config['insight_source_id'] + }}-{{ stream_partition.file_key }} + + # ── Stream 3: design_file_versions ───────────────────────────────────── + # GET /v1/files/{key}/versions — newest first, paginated via + # pagination.next_page (full URL), page_size max 50. History is unbounded + # (autosave checkpoints), so two guards against walking it all: + # - record_filter drops versions older than figma_start_date + # - stop_condition halts pagination once the page tail crosses the date + - type: DeclarativeStream + name: design_file_versions + primary_key: + - unique_key + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/definitions/design_file_versions_schema" + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: key + partition_field: file_key + stream: + type: DeclarativeStream + name: design_files + primary_key: + - key + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: id + partition_field: project_id + stream: + type: DeclarativeStream + name: design_projects + primary_key: + - id + retriever: + type: SimpleRetriever + partition_router: + type: ListPartitionRouter + values: "{{ config['figma_team_ids'].replace(' ', '').split(',') }}" + cursor_field: team_id + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/teams/{{ stream_partition.team_id }}/projects + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401, 403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - projects + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/projects/{{ stream_partition.project_id }}/files + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - files + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/files/{{ stream_partition.file_key }}/versions + request_parameters: + page_size: "{{ config.get('figma_page_size', 50) }}" + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ (response.get('pagination', {}) or {}).get('next_page', '') }}" + stop_condition: >- + {{ not (response.get('pagination', {}) or {}).get('next_page') + or ((last_record or {}).get('created_at') or '9999') + < (config.get('figma_start_date') or '2020-01-01') }} + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - versions + record_filter: + type: RecordFilter + condition: >- + {{ record.get('created_at', '') >= (config.get('figma_start_date') + or '2020-01-01') }} + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [tenant_id] + value: "{{ config['insight_tenant_id'] }}" + - type: AddedFieldDefinition + path: [source_id] + value: "{{ config['insight_source_id'] }}" + - type: AddedFieldDefinition + path: [data_source] + value: insight_figma + - type: AddedFieldDefinition + path: [version_id] + value: "{{ record['id'] }}" + - type: AddedFieldDefinition + path: [file_key] + value: "{{ stream_partition.file_key }}" + - type: AddedFieldDefinition + path: [label] + value: "{{ record.get('label') or '' }}" + - type: AddedFieldDefinition + path: [description] + value: "{{ record.get('description') or '' }}" + - type: AddedFieldDefinition + path: [author_id] + value: "{{ (record.get('user') or {}).get('id', '') }}" + - type: AddedFieldDefinition + path: [author_handle] + value: "{{ (record.get('user') or {}).get('handle', '') }}" + - type: AddedFieldDefinition + path: [collected_at] + value: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.000Z') }}" + - type: AddedFieldDefinition + path: [unique_key] + value: >- + {{ config['insight_tenant_id'] }}-{{ config['insight_source_id'] + }}-{{ stream_partition.file_key }}-{{ record['id'] }} + + # ── Stream 4: design_file_comments ───────────────────────────────────── + # GET /v1/files/{key}/comments — single response, top-level comments and + # replies together (parent_id discriminates), unlike confluence's 4-stream + # matrix. resolved_at is the resolution marker. + - type: DeclarativeStream + name: design_file_comments + primary_key: + - unique_key + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/definitions/design_file_comments_schema" + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: key + partition_field: file_key + stream: + type: DeclarativeStream + name: design_files + primary_key: + - key + retriever: + type: SimpleRetriever + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: id + partition_field: project_id + stream: + type: DeclarativeStream + name: design_projects + primary_key: + - id + retriever: + type: SimpleRetriever + partition_router: + type: ListPartitionRouter + values: "{{ config['figma_team_ids'].replace(' ', '').split(',') }}" + cursor_field: team_id + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/teams/{{ stream_partition.team_id }}/projects + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401, 403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - projects + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/projects/{{ stream_partition.project_id }}/files + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - files + requester: + type: HttpRequester + url_base: https://api.figma.com + authenticator: + type: ApiKeyAuthenticator + api_token: "{{ config['figma_token'] }}" + inject_into: + type: RequestOption + inject_into: header + field_name: X-Figma-Token + request_headers: + Accept: application/json + http_method: GET + path: /v1/files/{{ stream_partition.file_key }}/comments + error_handler: + type: DefaultErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + max_waiting_time_in_seconds: 600 + response_filters: + - type: HttpResponseFilter + action: RETRY + http_codes: [429, 500, 502, 503] + - type: HttpResponseFilter + action: FAIL + http_codes: [401] + - type: HttpResponseFilter + action: IGNORE + http_codes: [403, 404] + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - comments + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: [tenant_id] + value: "{{ config['insight_tenant_id'] }}" + - type: AddedFieldDefinition + path: [source_id] + value: "{{ config['insight_source_id'] }}" + - type: AddedFieldDefinition + path: [data_source] + value: insight_figma + - type: AddedFieldDefinition + path: [comment_id] + value: "{{ record['id'] }}" + - type: AddedFieldDefinition + path: [file_key] + value: "{{ stream_partition.file_key }}" + - type: AddedFieldDefinition + path: [parent_comment_id] + value: "{{ record.get('parent_id', '') }}" + - type: AddedFieldDefinition + path: [author_id] + value: "{{ (record.get('user') or {}).get('id', '') }}" + - type: AddedFieldDefinition + path: [author_handle] + value: "{{ (record.get('user') or {}).get('handle', '') }}" + - type: AddedFieldDefinition + path: [resolved_at] + value: "{{ record.get('resolved_at') or '' }}" + - type: AddedFieldDefinition + path: [order_id] + value: "{{ record.get('order_id') or '' }}" + - type: AddedFieldDefinition + path: [reaction_count] + value: "{{ (record.get('reactions') or []) | length }}" + - type: AddedFieldDefinition + path: [collected_at] + value: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.000Z') }}" + - type: AddedFieldDefinition + path: [unique_key] + value: >- + {{ config['insight_tenant_id'] }}-{{ config['insight_source_id'] + }}-{{ stream_partition.file_key }}-{{ record['id'] }} + +concurrency_level: + type: ConcurrencyLevel + default_concurrency: 1 + +spec: + type: Spec + connection_specification: + type: object + $schema: http://json-schema.org/draft-07/schema# + required: + - insight_tenant_id + - insight_source_id + - figma_token + - figma_team_ids + properties: + insight_tenant_id: + type: string + description: Insight tenant identifier stamped onto every emitted record. + title: Insight Tenant ID + order: 0 + insight_source_id: + type: string + description: >- + Instance discriminator for multi-instance support (e.g. figma-acme). + title: Insight Source ID + order: 1 + figma_token: + type: string + description: >- + Figma personal access token (Figma → Settings → Security → Personal + access tokens). Needs read scopes for projects, file metadata, file + versions and file comments. The token sees exactly what its owner + sees, and rate limits depend on the owner's seat type — use an + account with a Dev/Full seat that is a member of every configured + team. + title: Figma Personal Access Token + airbyte_secret: true + order: 2 + figma_team_ids: + type: string + description: >- + Comma-separated Figma team IDs to collect. The API cannot enumerate + teams — copy each ID from the team URL + (figma.com/files/team//...). + title: Figma Team IDs + order: 3 + figma_start_date: + type: string + description: >- + Earliest date of interest (YYYY-MM-DD). Files not modified since, + and file versions created before, this date are skipped. Defaults + to 2020-01-01. + title: Start Date + format: date + default: "2020-01-01" + order: 4 + figma_page_size: + type: integer + description: >- + Page size for the file versions endpoint (default 50, which is also + the API maximum). + title: Page Size + default: 50 + minimum: 1 + maximum: 50 + order: 5 + additionalProperties: true + +metadata: + autoImportSchema: + design_projects: false + design_files: false + design_file_meta: false + design_file_versions: false + design_file_comments: false diff --git a/src/ingestion/connectors/ui-design/figma/dbt/figma__bronze_promoted.sql b/src/ingestion/connectors/ui-design/figma/dbt/figma__bronze_promoted.sql new file mode 100644 index 0000000000..b9e6473bd2 --- /dev/null +++ b/src/ingestion/connectors/ui-design/figma/dbt/figma__bronze_promoted.sql @@ -0,0 +1,22 @@ +{# ------------------------------------------------------------------------- + Bootstrap model for Figma bronze → RMT promotion. + + Counterpart of `confluence__bronze_promoted` for Figma. See ADR-0002. The + `promote_bronze_to_rmt` macro is idempotent — already-RMT tables are + detected and skipped on subsequent runs. + ------------------------------------------------------------------------- #} + +-- @cpt-principle:cpt-dataflow-principle-promote-bronze:p1 +{{ config( + materialized='view', + schema='staging', + tags=['figma'] +) }} + +{% do promote_bronze_to_rmt(table='bronze_figma.design_projects', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_figma.design_files', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_figma.design_file_meta', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_figma.design_file_versions', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_figma.design_file_comments', order_by='unique_key') %} + +SELECT 1 AS promoted diff --git a/src/ingestion/connectors/ui-design/figma/dbt/schema.yml b/src/ingestion/connectors/ui-design/figma/dbt/schema.yml new file mode 100644 index 0000000000..403c566269 --- /dev/null +++ b/src/ingestion/connectors/ui-design/figma/dbt/schema.yml @@ -0,0 +1,19 @@ +version: 2 + +sources: + - name: bronze_figma + schema: bronze_figma + tables: + - name: design_projects + - name: design_files + - name: design_file_meta + - name: design_file_versions + - name: design_file_comments + +# Bronze-only delivery: no staging/Silver models yet. Future staging +# projections (e.g. figma__design_activity → class_design_activity per the +# ui-design domain spec) will reference these sources. Identity note for +# that work: Figma User objects carry only id/handle (no email anywhere in +# the REST API except /v1/me), so author resolution must go through the +# Identity Manager name-matching path or an Enterprise SCIM-sourced +# directory — there is no jira_user-style email join available. diff --git a/src/ingestion/connectors/ui-design/figma/descriptor.yaml b/src/ingestion/connectors/ui-design/figma/descriptor.yaml new file mode 100644 index 0000000000..c58b5d89dc --- /dev/null +++ b/src/ingestion/connectors/ui-design/figma/descriptor.yaml @@ -0,0 +1,20 @@ +name: figma +type: nocode +version: "1.0.0" + +schedule: "0 5 * * *" +workflow: sync + +# Bronze-only delivery: dbt step promotes the five bronze tables to +# ReplacingMergeTree (figma__bronze_promoted) and nothing else. Silver +# routing (class_design_activity per the ui-design domain spec) is a +# follow-up — when staging models land, they pick up the same tag. +dbt_select: "tag:figma+" + +connection: + namespace: bronze_figma +secret: + required_fields: + - figma_token + - figma_team_ids + - figma_start_date diff --git a/src/ingestion/secrets/connectors/figma.yaml.example b/src/ingestion/secrets/connectors/figma.yaml.example new file mode 100644 index 0000000000..5aa87d90de --- /dev/null +++ b/src/ingestion/secrets/connectors/figma.yaml.example @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Secret +metadata: + name: insight-figma-main + labels: + app.kubernetes.io/part-of: insight + annotations: + insight.cyberfabric.com/connector: figma + insight.cyberfabric.com/source-id: figma-main +type: Opaque +stringData: + figma_token: "CHANGE_ME" + # Comma-separated team IDs from the team URL: figma.com/files/team//... + figma_team_ids: "CHANGE_ME" + figma_start_date: "2020-01-01" From e8b3f0cd9f9fcea1f410e5b2a99e02a1624b9aa5 Mon Sep 17 00:00:00 2001 From: Roman Mitasov Date: Fri, 12 Jun 2026 07:18:52 +0300 Subject: [PATCH 3/3] fix(figma): address CodeRabbit review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - validate figma_team_ids format in the spec (minLength + pattern) so trailing/double commas fail at check time instead of producing /v1/teams//projects requests - align design_file_activity derivation wording in the ui-design domain spec with the dbt/Silver contract (was: "at collection time") - add language hint to a fenced block in figma.md (MD040) figma_start_date stays in descriptor secret.required_fields by design — confluence and zoom require their start_date the same way (explicit deployment config over silent defaults). Co-Authored-By: Claude Fable 5 Signed-off-by: Roman Mitasov --- docs/components/connectors/ui-design/README.md | 4 ++-- docs/components/connectors/ui-design/figma/figma.md | 2 +- src/ingestion/connectors/ui-design/figma/connector.yaml | 2 ++ 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/components/connectors/ui-design/README.md b/docs/components/connectors/ui-design/README.md index b6bf6515b8..80959542c6 100644 --- a/docs/components/connectors/ui-design/README.md +++ b/docs/components/connectors/ui-design/README.md @@ -76,7 +76,7 @@ The `design_file_activity` / `design_users` tables below remain the **target der ### `design_file_activity` — Per-user per-file per-day activity -Derived daily aggregates of design activity per user per file. Populated at collection time by aggregating version history and comment records. This is the primary analytics table for design team productivity. +Derived daily aggregates of design activity per user per file. Populated in dbt (Silver) by aggregating the raw `design_file_versions` and `design_file_comments` bronze streams. This is the primary analytics table for design team productivity. | Field | Type | Description | |-------|------|-------------| @@ -96,7 +96,7 @@ Derived daily aggregates of design activity per user per file. Populated at coll - `idx_design_activity_user`: `(insight_source_id, email, date)` - `idx_design_activity_file`: `(file_key, date, data_source)` -**Derivation note**: One `design_file_activity` row per `(user_id, file_key, date)`. At collection time: +**Derivation note**: One `design_file_activity` row per `(user_id, file_key, date)`. In the dbt derivation: - `versions_created` = count of version records where `created_by.id = user_id` on `date` - `comments_posted` = count of comment records where `user.id = user_id` on `date` - `files_viewed` = populated only when Figma Analytics API is available (Enterprise plan); NULL otherwise diff --git a/docs/components/connectors/ui-design/figma/figma.md b/docs/components/connectors/ui-design/figma/figma.md index bd7a62c130..a4529c85d0 100644 --- a/docs/components/connectors/ui-design/figma/figma.md +++ b/docs/components/connectors/ui-design/figma/figma.md @@ -156,7 +156,7 @@ Figma publishes per-endpoint **tiers**; budgets depend on the token owner's seat **Resolution chain (Silver, future)**: -``` +```text design_file_versions.author_id / author_handle → Identity Manager name-matching (handle ≈ display name in HR/git sources) → person_id diff --git a/src/ingestion/connectors/ui-design/figma/connector.yaml b/src/ingestion/connectors/ui-design/figma/connector.yaml index 92ee979004..79fd6a51b6 100644 --- a/src/ingestion/connectors/ui-design/figma/connector.yaml +++ b/src/ingestion/connectors/ui-design/figma/connector.yaml @@ -1034,6 +1034,8 @@ spec: order: 2 figma_team_ids: type: string + minLength: 1 + pattern: "^\\s*[^,\\s]+(\\s*,\\s*[^,\\s]+)*\\s*$" description: >- Comma-separated Figma team IDs to collect. The API cannot enumerate teams — copy each ID from the team URL