Refactor: migrate connection-manager to credentials and intelligence to discovery

TECHNICAL_DEBT.md

@@ -6,7 +6,26 @@ This document tracks known technical debt items that should be addressed in futu
 
 ## High Priority
 
-### 1. Hardened L3 & L4 Pipeline Outbox Refactoring
+### 1. AI Orchestrator & MCP Architecture Refactoring
+
+**Location**: `engine/application/ai`, `engine/platform/ai` (or similar MCP directories), and `engine/application/pieces`
+**Added**: 2026-04-08
+**Impact**: Code Architecture, Domain-Driven Design, Platform Scalability
+**Effort**: High (1 sprint)
+
+**Current State**:
+- The AI Copilot, MCP (Model Context Protocol), and the Sync Engine are currently conflated under the `engine` directory.
+- This mixes two fundamentally opposed execution contexts: Sync Engine (async, batch, high-throughput ETL) and AI Orchestrator (sync, low-latency, real-time LLM streaming).
+- MCP logic is incorrectly positioned within `engine/platform/`, despite having no relevance to database replication or webhook syncing.
+- `pieces` (Integrations) are trapped under `engine/application/pieces`, making them appear bound strictly to the Sync Engine when they should be universally accessible.
+
+**Recommended Solution**:
+- **Extract Integrations**: Move `engine/application/pieces` into a shared top-level library (e.g., `packages/integrations` or `/integrations`). Both Sync and AI domains will import from this single source of truth.
+- **Promote AI Domain**: Extract AI components out of `engine` into a dedicated top-level `ai/` or `engines/ai/` directory. This ensures AI logic doesn't inherit unnecessary ETL pipeline dependencies.
+- **Relocate MCP**: Move the Model Context Protocol abstractions out of `engine/platform/` into the new dedicated AI domain architecture.
+
+
+### 2. Hardened L3 & L4 Pipeline Outbox Refactoring
 
 **Location**: `apps/worker/src/modules/pipeline/normalization.service.ts`, `apps/worker/src/modules/pipeline/fanout.service.ts`  
 **Added**: 2026-03-29  
@@ -25,7 +44,7 @@ This document tracks known technical debt items that should be addressed in futu
 
 ---
 
-### 1. DatabaseManager Duplication
+### 3. DatabaseManager Duplication
 
 **Location**: `apps/api/src/db/database-manager.ts`, `apps/worker/src/db/database-manager.ts`  
 **Added**: 2026-03-27  
@@ -44,7 +63,7 @@ This document tracks known technical debt items that should be addressed in futu
 - Export a global `DbManagerModule` from that package.
 - Delete the redundant files in both `apps/api` and `apps/worker` and refactor them to import the unified library service.
 
-### 2. PII Cleanup Job for Sessions
+### 4. PII Cleanup Job for Sessions
 
 **Location**: `packages/database/src/schema/identity.ts` & `apps/api/src/modules/background`  
 **Added**: 2026-03-06  
@@ -64,7 +83,7 @@ This document tracks known technical debt items that should be addressed in futu
 - This job will find sessions older than 30 days and anonymize their PII (nullify or hash), emitting audit logs.
 - Add a Drizzle migration to backfill and anonymize existing old sessions.
 
-### 2. Permission Caching Architecture
+### 5. Permission Caching Architecture
 
 **Location**: `apps/web/src/app/providers/auth-provider.ts`  
 **Added**: 2026-02-12  
@@ -104,7 +123,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 - [Permission Architecture Pattern](file:///.gemini/antigravity/brain/d18e0ee0-ce96-4db3-9e9a-041242a6c761/permission_architecture_pattern.md)
 
-### 2. CI Integration for E2E Tests
+### 6. CI Integration for E2E Tests
 
 **Location**: `apps/web/e2e`  
 **Added**: 2026-02-19  
@@ -135,7 +154,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 - **dotenv auto-resolution**: `drizzle.config.ts` in `@nexiom/database` loads `DATABASE_URL` from `apps/api/.env` automatically so all root commands work without manual env sourcing.
 - **Enterprise Piece Loader**: `PiecesModule` is now a DynamicModule with `forRoot({ anchorUrl: import.meta.url })`. All 6 host modules (ConnectionsModule, StitchesModule, TriggerModule, SchedulerModule, WebhooksModule, PipelineModule) pass their own `import.meta.url` as the resolution anchor, bypassing pnpm strict package containment in any working directory or container.
 
-### 4. Shadow Mode Direct Trigger Imports
+### 7. Shadow Mode Direct Trigger Imports
 
 **Location**: `packages/pieces/salesforce/src/lib/trigger/universal-trigger.ts` & Quickbooks  
 **Added**: 2026-03-04  
@@ -152,7 +171,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 - **Short term**: Complete Phase 3 testing (i.e., validate that the universal polling engine achieves 100% data parity and stability over a 2-week dual-run window; refer to the [QA Test Plan](/docs/qa/shadow_mode_test_plan.md) for exit criteria) and delete the legacy stubs immediately, removing the imports.
 - **Long term (if kept)**: Implement a Dependency Injection registry where legacy triggers self-register for shadow testing, keeping `universal-trigger.ts` completely unaware and decoupled.
 
-### 5. Drizzle Schema Consolidation (Modular Monolith)
+### 8. Drizzle Schema Consolidation (Modular Monolith)
 
 **Location**: `packages/database`, `packages/identity`  
 **Added**: 2026-03-05  
@@ -194,7 +213,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### 2. True Enterprise DX Setup (DevContainers & Secret Manager)
+### 1. True Enterprise DX Setup (DevContainers & Secret Manager)
 
 **Location**: Workspace Root  
 **Added**: 2026-04-02  
@@ -213,7 +232,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### 3. Vector Sink Alerting / Dropped Logs
+### 2. Vector Sink Alerting / Dropped Logs
 
 **Location**: `vector/vector.toml`
 **Added**: 2026-03-25
@@ -252,7 +271,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### 2. Enterprise-Grade Global Frontend UI Polish
+### 1. Enterprise-Grade Global Frontend UI Polish
 
 **Location**: `apps/web/src/*`
 **Added**: 2026-02-24
@@ -275,7 +294,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### ~~3. Drizzle-Kit ESM Module Resolution~~ ✅ RESOLVED (2026-03-31)
+### ~~2. Drizzle-Kit ESM Module Resolution~~ ✅ RESOLVED (2026-03-31)
 
 **Location**: `packages/database/drizzle.config.ts`, `packages/database/package.json`
 **Added**: 2026-03-30
@@ -288,7 +307,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### 4. Centralized Mock Gateway / Mock Service Worker (MSW)
+### 3. Centralized Mock Gateway / Mock Service Worker (MSW)
 
 **Location**: `docker-compose.yml`, `apps/api`, `apps/worker`
 **Added**: 2026-03-31
@@ -311,7 +330,7 @@ Adopt industry-standard data-fetching library (React Query or SWR):
 
 ---
 
-### Low Priority
+## Low Priority
 
 ### 1. Replace custom Logger with Pino
 
@@ -437,4 +456,4 @@ Items proposed or resolved will be tracked here.
 **Proposed Changes**:
 
 1. **API Modernization**: Proposed: delete `ProviderRegistryService` and update `connectors.controller.ts` and `connectors.service.ts` to use `PieceAuth` definitions from `PieceRegistryService` for mapping `tokenUrl`, `authUrl`, and `clientId` dynamically.
-2. **Frontend Simplification**: Proposed: remove hardcoded `env` in `DynamicAuthForm.tsx` and `oauth-state.service.ts`; render environment via `uiSchema`/`vendorParams` so any custom field flows through generically.
+2. **Frontend Simplification**: Proposed: remove hardcoded `env` in `DynamicAuthForm.tsx` and `oauth-state.service.ts`; render environment via `uiSchema`/`vendorParams` so any custom field flows through generically.

apps/api/package.json

@@ -39,7 +39,7 @@
     "@nestjs/schedule": "^6.1.1",
     "@nexiom/auth": "workspace:*",
     "@nexiom/cache": "workspace:*",
-    "@nexiom/connectors": "workspace:*",
+    "@nexiom/credentials": "workspace:*",
     "@nexiom/database": "workspace:*",
     "@nexiom/dbmanager": "workspace:*",
     "@nexiom/engine": "workspace:*",

apps/api/src/modules/ai/_services/orchestrator.service.spec.ts

@@ -4,7 +4,7 @@ import { Test, TestingModule } from '@nestjs/testing';
 import { OrchestratorService } from './orchestrator.service.js';
 import { PinoLogger } from 'nestjs-pino';
 import { DATABASE_CONNECTION } from '@nexiom/database';
-import { TokenManagerService } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
 import { PieceRegistryService } from '@nexiom/engine';
 import { MetadataDiscoveryService } from '../../metadata/metadata-discovery.service.js';
 

apps/api/src/modules/ai/_services/orchestrator.service.ts

@@ -17,8 +17,8 @@ import {
   AppConnectionStatus,
 } from '@nexiom/database';
 import type { DrizzleDb } from '@nexiom/database';
-import { TokenManagerService } from '@nexiom/connectors';
-import type { OAuthCredentialBlob } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
+import type { OAuthCredentialBlob } from '@nexiom/credentials';
 import { PieceRegistryService } from '@nexiom/engine';
 import { MetadataDiscoveryService } from '../../metadata/metadata-discovery.service.js';
 import type { Piece } from '@nexiom/piece-framework';

apps/api/src/modules/connections/connections.module.ts

@@ -4,7 +4,7 @@ import {
   TokenManagerService,
   AesEncryptionService,
   OAuthRefreshClient,
-} from '@nexiom/connectors';
+} from '@nexiom/credentials';
 import { DbModule } from '../../db/db.module.js';
 import { REDIS_CLIENT } from '@nexiom/cache';
 import type { Redis } from '@nexiom/cache';

apps/api/src/modules/connections/connections/connectors.controller.spec.ts

@@ -2,7 +2,7 @@ import type { Piece } from '@nexiom/piece-framework';
 /* eslint-disable @typescript-eslint/unbound-method */
 import { Test, TestingModule } from '@nestjs/testing';
 import { ConnectorsController } from './connectors.controller.js';
-import { EncryptionService } from '@nexiom/connectors';
+import { EncryptionService } from '@nexiom/credentials';
 import { ConnectorsService } from '../connectors.service.js';
 import { OauthStateService } from '../oauth-state.service.js';
 import { AppConnectionStatus, DATABASE_CONNECTION } from '@nexiom/database';

apps/api/src/modules/connections/connections/connectors.controller.ts

@@ -25,7 +25,7 @@ import type { Response } from 'express';
 
 import { AuthContext, type RequestAuthContext, AuthGuard } from '@nexiom/auth';
 import { getAdminRoleId, getOwnerRoleId } from '@nexiom/identity/constants';
-import { EncryptionService, AppCredentialError } from '@nexiom/connectors';
+import { EncryptionService, AppCredentialError } from '@nexiom/credentials';
 import type { AnyProperty } from '@nexiom/piece-framework';
 import { ConnectorsService } from '../connectors.service.js';
 import { OauthStateService } from '../oauth-state.service.js';

apps/api/src/modules/connections/connections/token-refresh.service.spec.ts

@@ -1,5 +1,5 @@
 import { DefaultOAuthRefreshClient } from './token-refresh.service.js';
-import { EncryptionService, OAuthRefreshError } from '@nexiom/connectors';
+import { EncryptionService, OAuthRefreshError } from '@nexiom/credentials';
 import { PieceRegistryService } from '@nexiom/engine';
 import type { Piece } from '@nexiom/piece-framework';
 import {

apps/api/src/modules/connections/connections/token-refresh.service.ts

@@ -3,7 +3,7 @@ import {
   OAuthRefreshClient,
   OAuthRefreshError,
   EncryptionService,
-} from '@nexiom/connectors';
+} from '@nexiom/credentials';
 import { PropertyType, resolveOAuth2Url } from '@nexiom/piece-framework';
 import {
   appConnections,

apps/api/src/modules/connections/connectors.service.spec.ts

@@ -2,7 +2,7 @@ import { Test, TestingModule } from '@nestjs/testing';
 import { DATABASE_CONNECTION } from '@nexiom/database';
 import { ConfigService } from '@nestjs/config';
 import { ConnectorsService } from './connectors.service.js';
-import { EncryptionService, AppCredentialError } from '@nexiom/connectors';
+import { EncryptionService, AppCredentialError } from '@nexiom/credentials';
 import { PieceRegistryService } from '@nexiom/engine';
 import type { Piece } from '@nexiom/piece-framework';
 import { DB_MANAGER } from '../dbmanager/dbmanager.module.js';

apps/api/src/modules/connections/connectors.service.ts

@@ -10,9 +10,9 @@ import {
   ConflictException,
 } from '@nestjs/common';
 import { ConfigService } from '@nestjs/config';
-import { AppCredentialError } from '@nexiom/connectors';
+import { AppCredentialError } from '@nexiom/credentials';
 import { resolveOAuth2Url, PropertyType } from '@nexiom/piece-framework';
-import type { OAuthCredentialBlob } from '@nexiom/connectors';
+import type { OAuthCredentialBlob } from '@nexiom/credentials';
 import type { OAuth2Auth } from '@nexiom/piece-framework';
 import {
   appConnections,
@@ -32,7 +32,7 @@ import { extractPgError, PG_UNIQUE_VIOLATION } from '../../shared/db.utils.js';
 
 /**
  * Encrypted value blob stored in app_connection.value.
- * Aliased from the connectors package so all callers share a single source of truth.
+ * Aliased from the credentials package so all callers share a single source of truth.
  */
 export type ConnectionValueBlob = OAuthCredentialBlob;
 

apps/api/src/modules/metadata/metadata-discovery.service.spec.ts

@@ -5,7 +5,7 @@ import {
 } from '@nestjs/common';
 import { Test } from '@nestjs/testing';
 import { ConfigService } from '@nestjs/config';
-import { TokenManagerService } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
 import { MetadataDiscoveryService } from './metadata-discovery.service.js';
 import { DATABASE_CONNECTION } from '@nexiom/database';
 import { REDIS_CLIENT } from '@nexiom/cache';

apps/api/src/modules/metadata/metadata-discovery.service.ts

@@ -16,9 +16,9 @@ import {
   appConnections,
 } from '@nexiom/database';
 import { REDIS_CLIENT, type Redis } from '@nexiom/cache';
-import { TokenManagerService } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
 import { PieceRegistryService } from '@nexiom/engine';
-import type { OAuthCredentialBlob } from '@nexiom/connectors';
+import type { OAuthCredentialBlob } from '@nexiom/credentials';
 import type {
   ObjectDescriptor,
   FieldDescriptor,

apps/api/src/modules/scheduler/poll-sync-runner.spec.ts

@@ -9,7 +9,7 @@ import {
   syncCursors,
 } from '@nexiom/database';
 import { REDIS_CLIENT } from '@nexiom/cache';
-import { TokenManagerService } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
 import { CursorManagerService, PieceRegistryService } from '@nexiom/engine';
 import type { StreamDescriptor, PollPage, PollRecord } from '@nexiom/engine';
 import { SyncRunner } from './sync-runner.js';

apps/api/src/modules/scheduler/poll-sync-runner.ts

@@ -16,8 +16,8 @@ import {
   appConnections,
   syncCursors,
 } from '@nexiom/database';
-import { TokenManagerService } from '@nexiom/connectors';
-import type { OAuthCredentialBlob } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
+import type { OAuthCredentialBlob } from '@nexiom/credentials';
 import type { Piece } from '@nexiom/piece-framework';
 import { REDIS_CLIENT, type Redis } from '@nexiom/cache';
 import {

apps/api/src/modules/scheduler/scheduler.module.ts

@@ -6,7 +6,7 @@ import {
   PiecesModule,
   PieceRegistryService,
 } from '@nexiom/engine';
-import { TokenManagerService } from '@nexiom/connectors';
+import { TokenManagerService } from '@nexiom/credentials';
 import { REDIS_CLIENT } from '@nexiom/cache';
 import { DATABASE_CONNECTION } from '@nexiom/database';
 import { DbModule } from '../../db/db.module.js';

apps/api/src/modules/stitches/stitches.module.ts

@@ -1,7 +1,7 @@
 import { Module } from '@nestjs/common';
 import { AuthModule } from '@nexiom/auth';
 import { CacheModule } from '@nexiom/cache';
-import { EncryptionService, AesEncryptionService } from '@nexiom/connectors';
+import { EncryptionService, AesEncryptionService } from '@nexiom/credentials';
 import { DbModule } from '../../db/db.module.js';
 import { PiecesModule } from '@nexiom/engine';
 import { ConnectionsModule } from '../connections/connections.module.js';

apps/web/src/modules/connections/api/connections.api.ts

@@ -98,4 +98,4 @@ export async function exchangeOAuthCode(payload: {
     connectionId?: string;
 }): Promise<void> {
     await apiClient.post('/connectors/oauth-exchange', payload);
-}
+}

apps/web/src/modules/connections/hooks/useConnections.ts

@@ -142,7 +142,7 @@ export function useConnections() {
                     vendorParams,
                 });
 
-                const popupUrl = `${apiUrl}/connectors/${providerName}?session=${sessionId}`;
+                const popupUrl = `${apiUrl}/connection-manager/${providerName}?session=${sessionId}`;
                 // Re-call openPopup with the actual URL to redirect the already-opened window
                 openPopup(popupUrl);
             } catch (err: unknown) {

apps/web/src/modules/stitches/api/metadata.api.ts

@@ -9,7 +9,7 @@ export interface ObjectDescriptor {
 /**
  * Mirrors the FieldDescriptor returned by piece.describeFields and the
  * /stitches/metadata/:connectionId/objects/:objectName/fields endpoint.
- * Matches packages/connectors/src/framework/piece.ts FieldDescriptor exactly.
+ * Matches packages/connection-manager/src/framework/piece.ts FieldDescriptor exactly.
  */
 export interface FieldDescriptor {
   name: string;

apps/worker/package.json

@@ -38,7 +38,7 @@
     "@nestjs/schedule": "^6.1.1",
     "@nexiom/auth": "workspace:*",
     "@nexiom/cache": "workspace:*",
-    "@nexiom/connectors": "workspace:*",
+    "@nexiom/credentials": "workspace:*",
     "@nexiom/database": "workspace:*",
     "@nexiom/dbmanager": "workspace:*",
     "@nexiom/engine": "workspace:*",

apps/worker/src/modules/pipeline/delivery.service.spec.ts

@@ -4,7 +4,7 @@ import { DeliveryService } from "./delivery.service.js";
 import { QueueService, QueueName } from "@nexiom/queue";
 import { DATABASE_CONNECTION } from "@nexiom/database";
 import { StorageResolverService, PieceRegistryService } from "@nexiom/engine";
-import { TokenManagerService } from "@nexiom/connectors";
+import { TokenManagerService } from "@nexiom/credentials";
 import { describe, it, expect, vi, beforeEach } from "vitest";
 
 describe("DeliveryService", () => {

apps/worker/src/modules/pipeline/delivery.service.ts

@@ -18,7 +18,7 @@ import {
 } from "@nexiom/database";
 import type { DrizzleDb } from "@nexiom/database";
 import { StorageResolverService, PieceRegistryService } from "@nexiom/engine";
-import { TokenManagerService } from "@nexiom/connectors";
+import { TokenManagerService } from "@nexiom/credentials";
 import { RetryableException } from "@nexiom/piece-framework";
 import {
   sanitizeError,