Move internal docs to docs/internal/

docs/internal/PLUGIN_REFACTOR.md

@@ -0,0 +1,145 @@
+# Technical Proposal: Data Source Plugin Architecture
+
+## 1. Overview
+The current `ContextManager` supports `LanguagePlugin` (for parsing code) and `ExporterPlugin` (for formatting output). To extend capabilities to ingest external knowledge (e.g., Slack conversations, Jira tickets, Linear issues), we propose adding a new plugin category: **Data Source Plugins**.
+
+This document outlines the architectural changes required to support these plugins, including a new base class, updates to the `PluginManager`, and the data ingestion lifecycle.
+
+## 2. Current Architecture
+*   **PluginManager**: Generic loader that scans directories and lazy-loads classes.
+*   **LanguagePlugin**: Base class for code analysis.
+*   **ExporterPlugin**: Base class for output formatting.
+*   **Discovery**: Scans `./lib/languages` and `./lib/exporters`.
+
+## 3. Proposed Architecture
+
+### 3.1 New `DataSourcePlugin` Base Class
+We will introduce a `DataSourcePlugin` class that defines the contract for external data providers.
+
+**Key Responsibilities:**
+*   **Authentication**: Handling API keys or OAuth tokens.
+*   **Fetching**: Retrieving data based on queries or filters.
+*   **Normalization**: Converting external API responses into a standard `ContextItem` format (uniform structure for the context manager).
+*   **Validation**: Verifying if a query is supported.
+
+**Interface Definition:**
+
+```javascript
+class DataSourcePlugin {
+  constructor() {
+    this.type = 'datasource';
+    this.name = 'base-source';
+  }
+
+  /**
+   * Configure the plugin with credentials/settings
+   * @param {object} config - { apiKey, baseUrl, ... }
+   */
+  configure(config) {
+    this.config = config;
+    this.validateConfig();
+  }
+
+  /**
+   * Fetch data from the source
+   * @param {object} query - { types: ['issue'], limit: 10, query: "search term" }
+   * @returns {Promise<Array<ContextItem>>}
+   */
+  async fetch(query) {
+    throw new Error('Not implemented');
+  }
+
+  /**
+   * Test the connection
+   * @returns {Promise<boolean>}
+   */
+  async testConnection() {
+    return false;
+  }
+
+  /**
+   * Normalize external data to standard format
+   * @param {object} rawData
+   * @returns {ContextItem}
+   */
+  normalize(rawData) {
+    return {
+      id: rawData.id,
+      content: rawData.text,
+      metadata: { source: this.name, ... }
+    };
+  }
+}
+```
+
+### 3.2 PluginManager Enhancements
+
+To support the new plugin type effectively, `PluginManager.js` needs the following updates:
+
+1.  **Expanded Discovery**:
+    *   Add `./lib/sources` (or `./lib/datasources`) to the default `pluginPaths`.
+    *   Allow plugins to self-identify their `type` (language, exporter, datasource) in `getMetadata()`.
+
+2.  **Configuration Injection**:
+    *   Currently, plugins are instantiated with no arguments.
+    *   Add a `configure(pluginName, config)` method to the manager that looks up a loaded plugin and calls its `configure()` method.
+
+3.  **Type-Based Retrieval**:
+    *   Add `getPluginsByType(type)` to easily retrieve all "datasource" plugins for UI lists or automated fetching.
+
+### 3.3 Data Ingestion Flow
+
+1.  **Registration**: `PluginManager` discovers `JiraPlugin` in `./lib/sources`.
+2.  **Configuration**: System (or user) calls `pluginManager.configure('jira', { apiKey: '...' })`.
+3.  **Request**: User asks: "Summarize recent bugs."
+4.  **Resolution**: The Context Agent identifies "bugs" as a Jira-related query.
+5.  **Execution**: Agent calls `pluginManager.get('jira').fetch({ type: 'bug', limit: 5 })`.
+6.  **Normalization**: The plugin converts Jira JSON into text/markdown `ContextItems`.
+7.  **Integration**: Items are added to the prompt context.
+
+## 4. Implementation Plan
+
+### Phase 1: Core Framework
+1.  Create `projects/context-manager/lib/plugins/DataSourcePlugin.js`.
+2.  Update `PluginManager.js` to include `./lib/datasources` in default paths.
+3.  Add `getPluginsByType()` and `configurePlugin()` methods to `PluginManager`.
+
+### Phase 2: Reference Implementation
+1.  Create a `MockSourcePlugin` for testing.
+2.  Implement a real `LinearPlugin` or `JiraPlugin` (if credentials available) or a generic `HttpSourcePlugin`.
+
+### Phase 3: Integration
+1.  Update the main Context Manager logic to allow querying data sources alongside file reading.
+
+## 5. Example: Linear Plugin
+
+```javascript
+class LinearPlugin extends DataSourcePlugin {
+  constructor() {
+    super();
+    this.name = 'linear';
+  }
+
+  async fetch({ query, limit = 5 }) {
+    const response = await axios.post('https://api.linear.app/graphql', {
+      query: `query { issueSearch(query: "${query}", first: ${limit}) { nodes { title description url } } }`
+    }, {
+      headers: { Authorization: this.config.apiKey }
+    });
+
+    return response.data.data.issueSearch.nodes.map(issue => this.normalize(issue));
+  }
+
+  normalize(issue) {
+    return {
+      source: 'linear',
+      title: issue.title,
+      content: `${issue.title}\n${issue.description}\nURL: ${issue.url}`
+    };
+  }
+}
+```
+
+## 6. Security Considerations
+*   **Credential Storage**: `PluginManager` should not store secrets persistently in plain text. Credentials should be passed in at runtime or managed via environment variables.
+*   **Sanitization**: Data sources might return sensitive info. Plugins should implement a `redact` or `filter` step if necessary.

docs/internal/RAG_DESIGN.md

@@ -0,0 +1,57 @@
+# Local RAG Design for Context Manager
+
+This document outlines the design for the local Retrieval-Augmented Generation (RAG) engine.
+
+## Architecture
+
+The RAG engine consists of two main components:
+1.  **Embedding Provider**: Converts text into vector embeddings.
+2.  **Vector Store**: Stores embeddings and metadata, and performs semantic search.
+
+### Libraries Used
+-   **Vector DB**: `@lancedb/lancedb` (embedded, zero-config, runs in-process).
+-   **Embeddings**: `@xenova/transformers` (local ONNX runtime, default model: `Xenova/all-MiniLM-L6-v2`).
+
+### Components
+
+#### 1. EmbeddingProvider (`lib/rag/EmbeddingProvider.js`)
+Abstracts the embedding generation.
+-   `TransformersEmbeddingProvider`: Uses local ONNX models.
+-   `MockEmbeddingProvider`: For testing.
+
+#### 2. VectorStore (`lib/rag/VectorStore.js`)
+Manages the LanceDB connection and document lifecycle.
+-   **Storage**: Filesystem based (default: `.context-manager/rag-store`).
+-   **Schema**:
+    -   `vector`: Float32 array (dimension depends on model, e.g., 384).
+    -   `text`: Original text content.
+    -   `id`: Unique identifier.
+    -   `timestamp`: Insertion time.
+    -   `metadata`: JSON object for arbitrary metadata (source, language, etc.).
+
+## Usage
+
+```javascript
+import { TransformersEmbeddingProvider } from '../lib/rag/EmbeddingProvider.js';
+import { VectorStore } from '../lib/rag/VectorStore.js';
+
+// Initialize
+const provider = new TransformersEmbeddingProvider();
+const store = new VectorStore(provider);
+
+// Indexing
+await store.addDocument('function hello() { console.log("world"); }', { 
+  source: 'src/main.js', 
+  language: 'javascript' 
+});
+
+// Search
+const results = await store.search('hello world function');
+console.log(results);
+// Output: [{ text: '...', score: 0.12, metadata: { ... } }]
+```
+
+## Integration Plan
+1.  When indexing code files, pass content to `VectorStore.addDocument`.
+2.  Store file path and other context in `metadata`.
+3.  During context generation, query `VectorStore` with the user prompt to retrieve relevant code snippets.

docs/internal/UNBLOCKED_ANALYSIS.md

@@ -0,0 +1,86 @@
+# Unblocked Analysis & Feature Roadmap
+
+## 🎯 What is Unblocked?
+Unblocked is an AI-powered development assistant that connects source code with external knowledge sources (Slack, Linear, Confluence, GitHub) to provide **deep context**. Unlike standard AI coding tools that only see the code in your editor, Unblocked understands *why* the code was written that way by referencing discussions, tickets, and docs.
+
+## 🌟 Feature Breakdown (Gap Analysis)
+
+We have analyzed the Unblocked documentation and identified the following key feature sets.
+
+### 1. Core Platform & Knowledge Graph
+| Feature | Unblocked | Context Manager (Us) | Status |
+| :--- | :--- | :--- | :--- |
+| **Code Indexing** | ✅ Git-based | ✅ Advanced (Exact tokens) | **Done** |
+| **External Data Sources** | ✅ Slack, Jira, Linear, Notion, Drive, Confluence | ❌ None | **Critical Priority** |
+| **Vector Search (RAG)** | ✅ Semantic Search | ⚠️ Basic Regex only | **Critical Priority** |
+| **Data Shield** | ✅ Permission-aware answers (ACLs) | ❌ None | **High Priority** |
+| **Identity Linking** | ✅ Map users across platforms (Slack <-> GitHub) | ❌ None | **Medium Priority** |
+
+### 2. Specialized Agents
+| Feature | Unblocked | Context Manager (Us) | Status |
+| :--- | :--- | :--- | :--- |
+| **PR Failure Agent** | ✅ Analyzes CI logs, suggests fixes | ❌ None | **High Priority** |
+| **Code Review Agent** | ✅ Automated PR reviews | ❌ None | **High Priority** |
+| **Expert Finder** | ✅ "Who knows about this?" | ❌ Basic Git Blame only | **Medium Priority** |
+
+### 3. Enterprise & Security
+| Feature | Unblocked | Context Manager (Us) | Status |
+| :--- | :--- | :--- | :--- |
+| **SSO / RBAC** | ✅ SAML, Okta, Google | ❌ None | **Low Priority (MVP)** |
+| **Incognito Mode** | ✅ Private questions | ❌ N/A (Local only) | **N/A** |
+| **Audit Logs** | ✅ Usage tracking | ⚠️ Basic Telemetry | **Low Priority** |
+
+### 4. Interfaces
+| Feature | Unblocked | Context Manager (Us) | Status |
+| :--- | :--- | :--- | :--- |
+| **Web Dashboard** | ✅ Management & Chat | ❌ CLI only | **Medium Priority** |
+| **IDE Extensions** | ✅ VSCode, JetBrains | ❌ None | **High Priority** |
+| **Mac App** | ✅ Native Desktop App | ⚠️ `desktop-app` (Incomplete) | **Medium Priority** |
+| **Slack Bot** | ✅ Conversational Bot | ❌ None | **Medium Priority** |
+
+---
+
+## 🗺️ Execution Roadmap
+
+### Phase 1: The Engine (Architecture Refactor)
+**Goal:** Enable ingestion of non-code data.
+- [ ] **Refactor `PluginManager`:** Support `DataSourcePlugin` interface.
+- [ ] **Implement Vector Store:** Integrate a local vector database (e.g., LanceDB) for semantic search.
+- [ ] **Universal Indexer:** Unified interface to index Code + Docs + Issues.
+
+### Phase 2: The Connectors (Data Sources)
+**Goal:** Connect the most popular tools.
+- [ ] **GitHub Issues/PRs Plugin:** Fetch context from PR descriptions and comments.
+- [ ] **Slack Plugin:** Ingest public channel history.
+- [ ] **Linear/Jira Plugin:** Index tickets and specs.
+
+### Phase 3: The Intelligence (Agents)
+**Goal:** Active assistance, not just passive chat.
+- [ ] **Build "PR Failure Agent":**
+    - Input: CI Log
+    - Process: Analyze error -> Search Context -> Suggest Fix
+    - Output: Comment on PR
+- [ ] **Build "Code Review Agent":**
+    - Input: Git Diff
+    - Process: Style check + Bug hunt + Context verification
+    - Output: Line-level comments
+
+### Phase 4: The Interface (Consumption)
+**Goal:** Meet the developer where they are.
+- [ ] **Enhance MCP Server:** Serve rich context to Cursor/Claude.
+- [ ] **VSCode Extension:** Lightweight wrapper around the MCP server.
+
+---
+
+## 🛠️ Implementation Strategy
+
+We are using a multi-agent approach to accelerate development:
+
+1.  **Agent A (Architecture):** Focus on `PluginManager` and `DataSource` interfaces.
+2.  **Agent B (RAG Engine):** Focus on `VectorStore` implementation and Embedding generation.
+3.  **Agent C (Features):** Focus on specific agents like `PR Failure Agent`.
+
+**Current Status:**
+- Agent A is currently analyzing `PluginManager`.
+- Agent B (RAG) is queuing for launch.
+- Agent C (PR Agent) is queuing for launch.