diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index 2668a3dff..66d038b90 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -183,7 +183,7 @@ Read subagents on-demand. Full index: `subagent-index.toon`. | Design | `tools/design/ui-ux-inspiration.md`, `tools/design/ui-ux-catalogue.toon`, `tools/design/brand-identity.md` | | SEO | `seo/dataforseo.md`, `seo/google-search-console.md` | | WordPress | `tools/wordpress/wp-dev.md`, `tools/wordpress/mainwp.md` | -| Communications | `services/communications/matterbridge.md`, `services/communications/discord.md`, `services/communications/simplex.md`, `services/communications/matrix-bot.md`, `services/communications/bitchat.md`, `services/communications/xmtp.md` | +| Communications | `services/communications/matterbridge.md`, `services/communications/simplex.md`, `services/communications/signal.md`, `services/communications/telegram.md`, `services/communications/whatsapp.md`, `services/communications/matrix-bot.md`, `services/communications/slack.md`, `services/communications/discord.md`, `services/communications/msteams.md`, `services/communications/google-chat.md`, `services/communications/nextcloud-talk.md`, `services/communications/nostr.md`, `services/communications/urbit.md`, `services/communications/imessage.md`, `services/communications/bitchat.md`, `services/communications/xmtp.md` | | Email | `tools/ui/react-email.md`, `services/email/email-testing.md`, `services/email/email-agent.md` | | Payments | `services/payments/revenuecat.md`, `services/payments/stripe.md`, `services/payments/procurement.md` | | Security/Encryption | `tools/security/tirith.md`, `tools/security/opsec.md`, `tools/security/prompt-injection-defender.md`, `tools/credentials/encryption-stack.md` | diff --git a/.agents/services/communications/google-chat.md b/.agents/services/communications/google-chat.md new file mode 100644 index 000000000..0b1d23ca4 --- /dev/null +++ b/.agents/services/communications/google-chat.md @@ -0,0 +1,547 @@ +--- +description: Google Chat Bot integration — HTTP webhook/REST API setup, card messages, security considerations (no E2E, Gemini AI training), Matterbridge (unsupported), and aidevops dispatch +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Google Chat Bot Integration + + + +## Quick Reference + +- **Type**: Google Workspace messaging — no E2E encryption, Google has full access +- **License**: Proprietary (Google). No official TypeScript SDK — HTTP API + webhook +- **Bot tool**: Google Chat API (HTTP webhook mode, REST API) +- **Protocol**: Google Chat API (HTTP/JSON) +- **Encryption**: TLS in transit, Google-managed at rest — NO end-to-end encryption +- **Script**: `google-chat-dispatch-helper.sh [setup|start|stop|status|map|unmap|mappings|test|logs]` +- **Config**: `~/.config/aidevops/google-chat-bot.json` (600 permissions) +- **Data**: `~/.aidevops/.agent-workspace/google-chat-bot/` +- **Docs**: https://developers.google.com/workspace/chat/api/reference/rest | https://developers.google.com/workspace/chat +- **Console**: https://console.cloud.google.com/apis/api/chat.googleapis.com + +**Quick start**: + +```bash +google-chat-dispatch-helper.sh setup # Interactive wizard +google-chat-dispatch-helper.sh map spaces/AAAA general-assistant +google-chat-dispatch-helper.sh start --daemon +``` + + + +## Architecture + +```text +┌──────────────────────────┐ +│ Google Chat Space │ +│ │ +│ User sends message │ +│ or @mentions bot │ +└────────────┬─────────────┘ + │ + │ HTTP POST (webhook push) OR Pub/Sub subscription + │ (Google pushes events to URL) (async event delivery) + │ + │ ⚠ Requires public URL: + │ Tailscale Funnel (recommended) + │ Caddy reverse proxy + │ Cloudflare Tunnel + │ ngrok + │ +┌────────────▼─────────────┐ ┌──────────────────────┐ +│ Webhook Handler (Bun) │ │ aidevops Dispatch │ +│ │ │ │ +│ ├─ Event router │────▶│ runner-helper.sh │ +│ ├─ Message handler │ │ → AI session │ +│ ├─ Card action handler │◀────│ → response │ +│ ├─ Access control │ │ │ +│ └─ Entity resolution │ │ │ +└────────────┬─────────────┘ └──────────────────────┘ + │ +┌────────────▼─────────────┐ +│ memory.db (shared) │ +│ ├── entities │ Entity profiles +│ ├── entity_channels │ Cross-channel identity +│ ├── interactions │ Layer 0: Immutable log +│ └── conversations │ Layer 1: Context summaries +└───────────────────────────┘ +``` + +**Message flow**: User sends message/mention → Google pushes HTTP POST to webhook URL → handler verifies authenticity and checks access control → space-to-runner mapping lookup → entity resolution via `entity-helper.sh` → Layer 0 logging → context loading (entity profile + conversation summary) → dispatch to runner via `runner-helper.sh` → response sent back via REST API → emoji reaction added (success/failure). + +## Installation + +### Prerequisites + +1. **Google Workspace account** — consumer Google accounts cannot create Chat bots +2. **Google Cloud project** with billing enabled +3. **Node.js >= 18** or **Bun** runtime +4. **Public URL** for webhook delivery (see architecture diagram above) + +### Step 1: Create a Google Cloud Project + +1. Go to https://console.cloud.google.com and create a new project (or select an existing one) +2. Note the **Project ID** — you will need it for API calls + +### Step 2: Enable the Google Chat API + +```bash +# Via gcloud CLI +gcloud services enable chat.googleapis.com --project=YOUR_PROJECT_ID + +# Or via Console: APIs & Services > Library > search "Google Chat API" > Enable +``` + +### Step 3: Create a Service Account + +```bash +# Create service account and download key +gcloud iam service-accounts create aidevops-chat-bot \ + --display-name="aidevops Chat Bot" --project=YOUR_PROJECT_ID + +gcloud iam service-accounts keys create /tmp/chat-bot-sa-key.json \ + --iam-account=aidevops-chat-bot@YOUR_PROJECT_ID.iam.gserviceaccount.com + +# Store securely via gopass (preferred) — then delete the plaintext key +gopass insert -m aidevops/google-chat/service-account-key < /tmp/chat-bot-sa-key.json +rm /tmp/chat-bot-sa-key.json +``` + +### Step 4: Configure the Chat App + +1. Go to https://console.cloud.google.com/apis/api/chat.googleapis.com/hangouts-chat > **Configuration** +2. Set app name ("aidevops Bot"), description, functionality (1:1 messages + spaces) +3. Connection settings: **HTTP endpoint URL** — enter your public webhook URL +4. Set visibility (who can discover and install the bot) and click **Save** + +### Step 5: Set Up a Public URL + +Google Chat requires a publicly accessible HTTPS endpoint — **no Socket Mode equivalent**. Recommended: Tailscale Funnel (no open ports, persistent URL). Alternatives: Caddy reverse proxy, Cloudflare Tunnel, ngrok. + +```bash +# Expose local port via Tailscale Funnel +tailscale funnel 8443 +# Endpoint: https://your-machine.your-tailnet.ts.net:8443 +``` + +### Step 6: Install Dependencies + +```bash +# Using Bun (preferred) +bun add googleapis google-auth-library + +# Using npm +npm install googleapis google-auth-library +``` + +## Bot API Integration + +### HTTP Webhook Handler + +Google Chat delivers events as HTTP POST requests with JSON payloads. This is the primary integration mode — there is no WebSocket or Socket Mode equivalent. + +```typescript +import { google } from "googleapis"; +import { GoogleAuth } from "google-auth-library"; + +const PORT = Number(process.env.PORT) || 8443; + +// Authenticate with service account +const auth = new GoogleAuth({ + keyFile: process.env.GOOGLE_CHAT_SA_KEY_PATH, + scopes: ["https://www.googleapis.com/auth/chat.bot"], +}); +const chat = google.chat({ version: "v1", auth }); + +// Webhook handler (Bun.serve) +Bun.serve({ + port: PORT, + async fetch(req) { + if (req.method !== "POST") { + return new Response("Method not allowed", { status: 405 }); + } + + const event = await req.json(); + + // Route by event type + switch (event.type) { + case "ADDED_TO_SPACE": + return handleAddedToSpace(event); + case "MESSAGE": + return handleMessage(event); + case "CARD_CLICKED": + return handleCardClicked(event); + case "REMOVED_FROM_SPACE": + return handleRemovedFromSpace(event); + default: + return Response.json({ text: "" }); + } + }, +}); + +async function handleAddedToSpace(event: any): Promise { + return Response.json({ text: "Hello! I'm the aidevops assistant. Mention me or send a DM." }); +} + +async function handleMessage(event: any): Promise { + const userMessage = event.message.argumentText?.trim() || event.message.text?.trim(); + if (!userMessage) return Response.json({ text: "Send me a prompt and I'll help!" }); + + try { + const response = await dispatchToRunner(userMessage, event.user, event.space.name); + return Response.json({ text: response }); + } catch (error) { + return Response.json({ text: `Error: ${error.message}` }); + } +} + +async function handleCardClicked(event: any): Promise { + return Response.json({ text: `Action "${event.action.actionMethodName}" received.` }); +} + +async function handleRemovedFromSpace(_event: any): Promise { + return Response.json({}); +} + +console.log(`Google Chat webhook handler running on port ${PORT}`); +``` + +### Sending Messages via REST API + +Synchronous webhook responses work for simple replies. For async responses (long-running tasks), use the REST API: + +```typescript +// Send a message to a space +async function sendMessage(spaceName: string, text: string, threadKey?: string) { + const requestBody: any = { text }; + + if (threadKey) { + requestBody.thread = { threadKey }; + // Reply in existing thread + } + + const res = await chat.spaces.messages.create({ + parent: spaceName, + requestBody, + // messageReplyOption: "REPLY_MESSAGE_FALLBACK_TO_NEW_THREAD", + }); + + return res.data; +} + +// Send to a specific space +await sendMessage("spaces/AAAAxxxxxx", "Deployment complete."); +``` + +### Card Messages + +Google Chat supports rich card messages with headers, sections, widgets, and buttons (Cards v2): + +```typescript +// Minimal card with header, decorated text, and action button +const requestBody = { + cardsV2: [{ + cardId: "status-card", + card: { + header: { title: "Deployment Status", subtitle: "v2.3.1" }, + sections: [{ + widgets: [ + { decoratedText: { topLabel: "Status", text: "Deployed", startIcon: { knownIcon: "STAR" } } }, + { buttonList: { buttons: [{ + text: "View Logs", + onClick: { action: { actionMethodName: "view_logs", parameters: [{ key: "id", value: "dep-123" }] } }, + }] } }, + ], + }], + }, + }], +}; + +await chat.spaces.messages.create({ parent: spaceName, requestBody }); +``` + +See: https://developers.google.com/workspace/chat/api/reference/rest/v1/cards + +### Reactions and Typing Indicators + +```typescript +// Add an emoji reaction to a message +await chat.spaces.messages.reactions.create({ + parent: event.message.name, // "spaces/AAAA/messages/BBBB" + requestBody: { + emoji: { unicode: "U+1F440" }, // eyes emoji + }, +}); + +// Note: Google Chat does not have a typing indicator API. +// The bot cannot show "typing..." status to users. +``` + +### Access Control + +```typescript +// Space allowlist +const ALLOWED_SPACES = new Set(["spaces/AAAAxxxx", "spaces/BBBByyyy"]); + +// User allowlist (Google Workspace email addresses) +const ALLOWED_USERS = new Set(["admin@company.com", "dev@company.com"]); + +function isAllowed(user: any, spaceName: string): boolean { + if (ALLOWED_SPACES.size > 0 && !ALLOWED_SPACES.has(spaceName)) { + return false; + } + if (ALLOWED_USERS.size > 0 && !ALLOWED_USERS.has(user.email)) { + return false; + } + return true; +} + +// Apply in webhook handler +async function handleMessage(event: any): Promise { + if (!isAllowed(event.user, event.space.name)) { + return Response.json({ text: "Access denied." }); + } + // ... dispatch +} +``` + +## Security Considerations + +**CRITICAL: Read this section carefully before deploying any bot that processes sensitive information via Google Chat.** + +### Encryption + +Google Chat provides **TLS 1.2+ in transit** and **AES-256 at rest** using Google-managed encryption keys. There is **NO end-to-end encryption**. Google has full technical access to ALL message content — space messages, DMs, file uploads, message edits, deleted messages (retained per retention policies), and card interactions. + +### Google's Data Access + +Google Chat is a Google Workspace product. All messages are stored on Google's servers. Google retains the technical ability to access all content. Workspace admins have full access via Admin Console, Google Vault (eDiscovery — legal holds, search, export of all Chat data including DMs and deleted messages), and audit logs. + +### Metadata Collection + +Google stores comprehensive metadata: full message history with timestamps, edit/deletion history, space membership, reactions, file sharing records, read receipts, login times, IP addresses, device info, user agents, search queries, bot interaction logs, and usage patterns. + +### AI Training and Data Processing + +**CRITICAL WARNING**: Google has integrated **Gemini AI directly into Google Chat**. This is arguably the most aggressive AI integration of any major messaging platform. + +- **Gemini in Chat**: Gemini can summarise conversations, answer questions about chat history, generate content, and suggest replies — all by processing message content on Google's servers. This is enabled by default in many Workspace configurations. +- **Gemini app in spaces**: Users can @mention Gemini directly in Chat spaces. Gemini reads and processes the conversation context to generate responses. Other users in the space may invoke Gemini on messages you sent. +- **Google's AI terms**: Google's Generative AI Additional Terms of Service and Privacy Notice govern how Chat data is processed by Gemini. Under default Workspace terms, Google states it does not use Workspace Core Services data to train its AI models — but this is a policy commitment, not a technical guarantee, and applies only to organisations with Workspace agreements. +- **Workspace admin control**: Admins can disable Gemini features in Chat via Admin Console (Apps > Google Workspace > Google Chat > Gemini settings). However, the **default configuration enables Gemini** in most Workspace editions. +- **Data processing agreements (DPAs)**: Enterprise customers can negotiate DPAs that restrict Google's data processing. But default Workspace terms are broad, and Google has been fined for GDPR violations. Without a specific DPA, assume Google processes Chat data for service improvement. +- **Smart features**: Beyond Gemini, Google Chat has "smart features" (Smart Reply, smart compose) that process message content to generate suggestions. These can be disabled at the organisation level. + +**Practical impact**: Assume that any message sent in Google Chat is processed by Gemini and Google's AI systems unless the Workspace admin has explicitly disabled all AI features and smart features, AND has a specific DPA in place. + +### Push Notifications + +Push notifications are delivered via **Firebase Cloud Messaging (FCM)** for Android and **Apple Push Notification Service (APNs)** for iOS. FCM is Google's own service — Google sees all notification metadata and content. On Android, notification content is unencrypted in transit to the device via FCM. Admins cannot configure notification content redaction. + +### Open Source and Auditability + +Google Chat is entirely **closed source** — no independent audit of data handling, encryption, or access controls is possible. Google client libraries (`googleapis`, `google-auth-library`) are open source under Apache 2.0, so bot-side code is auditable, but Google's server-side processing is opaque. Clients (web, mobile) are closed source with no reproducible builds. + +### Jurisdiction and Legal + +**Entity**: Google LLC, Mountain View, California, USA. Subject to US federal law including the CLOUD Act, FISA Section 702, and National Security Letters. EU data residency is available on Enterprise plans but controls storage location only — Google US personnel may still access EU-resident data. Google publishes a Transparency Report for government requests. + +### Bot-Specific Security + +- **Service account auth**: Bot authenticates as a service account with access only to spaces where it has been added +- **Webhook verification**: Google includes a bearer token in the `Authorization` header. Verification is recommended but **not enforced** — bots can function without checking, creating a security risk if the webhook URL is discovered +- **Public URL exposure**: If the URL leaks and payload verification is not implemented, anyone can send fake events to the bot + +### Comparison with Other Platforms + +| Aspect | Google Chat | Slack | Matrix (self-hosted) | SimpleX | +|--------|-------------|-------|---------------------|---------| +| E2E encryption | No | No | Yes (Megolm) | Yes (Double ratchet) | +| Server access to content | Full (Google) | Full (Salesforce) | None (if E2E on) | None (stateless) | +| Admin message export | Yes (Vault) | Yes (all plans) | Server admin only | N/A | +| AI integration default | Gemini (most aggressive) | Opt-out required | No | No | +| Open source server | No | No | Yes (Synapse) | Yes (SMP) | +| User identifiers | Google Workspace email | Workspace email | `@user:server` | None | +| Metadata retention | Comprehensive | Comprehensive | Moderate | Minimal | +| Self-hostable | No | No | Yes | Yes | +| Jurisdiction | USA (Google) | USA (Salesforce) | Self-determined | Self-determined | + +**Summary**: Google Chat is among the **least private** mainstream messaging platforms, comparable to Slack and Microsoft Teams. No E2E encryption, full admin and platform access to all content, and **Gemini AI integration that actively processes conversations by default**. Google's AI integration is arguably the most aggressive of the three major corporate platforms. **Treat all Google Chat messages as fully observable by the Workspace admin AND Google.** Use Google Chat for work communication where corporate oversight is expected and acceptable. Never use it for sensitive personal communication, confidential legal matters, or information that should not be accessible to the organisation owner or Google. + +## aidevops Integration + +### google-chat-dispatch-helper.sh + +The helper script follows the same pattern as `slack-dispatch-helper.sh` and `matrix-dispatch-helper.sh`: + +```bash +# Setup wizard — prompts for project ID, service account, webhook URL +google-chat-dispatch-helper.sh setup + +# Map Google Chat spaces to runners +google-chat-dispatch-helper.sh map spaces/AAAAxxxx code-reviewer +google-chat-dispatch-helper.sh map spaces/BBBByyyy seo-analyst + +# List mappings +google-chat-dispatch-helper.sh mappings + +# Remove a mapping +google-chat-dispatch-helper.sh unmap spaces/AAAAxxxx + +# Start/stop the webhook handler +google-chat-dispatch-helper.sh start --daemon +google-chat-dispatch-helper.sh stop +google-chat-dispatch-helper.sh status + +# Test dispatch +google-chat-dispatch-helper.sh test code-reviewer "Review src/auth.ts" + +# View logs +google-chat-dispatch-helper.sh logs +google-chat-dispatch-helper.sh logs --follow +``` + +### Public URL Setup + +The webhook handler requires a public URL. Tailscale Funnel is the recommended approach: + +```bash +# Start the webhook handler on a local port +google-chat-dispatch-helper.sh start --port 8443 + +# Expose via Tailscale Funnel +tailscale funnel 8443 + +# Configure the public URL in Google Cloud Console: +# https://your-machine.your-tailnet.ts.net:8443 +``` + +### Runner Dispatch + +The bot dispatches to runners via `runner-helper.sh`, which handles: + +- Runner AGENTS.md (personality/instructions) +- Headless session management +- Memory namespace isolation +- Entity-aware context loading +- Run logging + +### Entity Resolution + +When a Google Chat user sends a message, the bot resolves their Google user identity to an entity: + +- **Known user**: Match on `entity_channels` table (`channel=google-chat`, `channel_id=users/USER_ID`) +- **New user**: Creates entity via `entity-helper.sh create` with Google user ID linked +- **Cross-channel**: If the same person is linked on other channels (Slack, Matrix, email), their full profile is available +- **Profile enrichment**: Google Chat's user object provides display name and email (Workspace directory) — used to populate entity profile on first contact + +### Configuration + +`~/.config/aidevops/google-chat-bot.json` (600 permissions): + +```json +{ + "projectId": "your-gcp-project-id", + "serviceAccountKeyPath": "", + "webhookPort": 8443, + "allowedSpaces": ["spaces/AAAAxxxx", "spaces/BBBByyyy"], + "allowedUsers": [], + "defaultRunner": "", + "spaceMappings": { + "spaces/AAAAxxxx": "code-reviewer", + "spaces/BBBByyyy": "seo-analyst" + }, + "verifyWebhookToken": true, + "maxPromptLength": 3000, + "responseTimeout": 600, + "sessionIdleTimeout": 300 +} +``` + +**Notes**: + +- `serviceAccountKeyPath` should point to the key file or be empty if using `gopass` for credential retrieval +- `verifyWebhookToken` should always be `true` in production — set to `false` only during local development +- Google Chat bots are invoked via @mention or DM — there is no text prefix or slash command equivalent + +## Matterbridge Integration + +Google Chat is **NOT natively supported** by [Matterbridge](https://github.com/42wim/matterbridge). There is no built-in gateway for Google Chat. + +### Bridging Feasibility + +Bridging would require a custom Go gateway using the Matterbridge plugin architecture + Google Chat API. Challenges: webhook-only model (no Socket Mode) complicates bidirectional bridging, service account auth adds complexity, and Workspace-only nature limits adoption. The Matterbridge project has not prioritised Google Chat support. + +**Alternative**: Use Google Chat API directly in a custom bridge, or use a paid integration platform (Zapier, Make, Workato) as an intermediary. + +**Privacy warning**: Bridging Google Chat to E2E-encrypted platforms (Matrix, SimpleX) means messages from encrypted platforms will be stored unencrypted on Google's servers. Users on the encrypted side should be informed. See `services/communications/matterbridge.md` for full bridging considerations. + +## Limitations + +### Google Workspace Required + +Google Chat bots can only be created within Google Workspace organisations. Consumer Google accounts (`@gmail.com`) cannot create or configure Chat apps. This limits deployment to organisations with paid Workspace subscriptions. + +### Public URL Required + +Google Chat delivers webhook events via HTTP POST to a public URL. There is **no Socket Mode, WebSocket, or long-polling alternative**. You must expose a publicly accessible HTTPS endpoint. This is a fundamental architectural difference from Slack (which offers Socket Mode) and adds operational complexity for developers behind NAT or firewalls. + +### No End-to-End Encryption + +Google Chat does not support E2E encryption. All messages are readable by Google and Workspace administrators. This is a platform design choice — Google Vault eDiscovery and Gemini AI features depend on server-side access to message content. + +### Gemini AI Processes Chat Data by Default + +Gemini is integrated directly into Google Chat and processes conversation content by default in most Workspace editions. Users can @mention Gemini in spaces, and Gemini reads the full conversation context. Workspace admins can disable this, but it requires explicit action. + +### Limited Bot Interactivity + +Compared to Slack and Discord: no slash commands (bots respond to @mentions and DMs only), no modal dialogs (dialog cards are less flexible than Slack's `views.open`), no app home tab, no shortcut menus, and card messages are less flexible than Slack Block Kit. + +### No Socket Mode Equivalent + +Every Google Chat bot must expose a public HTTPS endpoint or use Pub/Sub. No outbound-only connections. Firewall/NAT traversal required, local development needs a tunnel (Tailscale Funnel, ngrok), and the public endpoint adds security surface. + +### Rate Limits + +| API | Rate Limit | Notes | +|-----|-----------|-------| +| Chat API (per project) | 60 requests per minute per space | Applies to messages.create, messages.update | +| Chat API (global) | 600 requests per minute per project | Aggregate across all spaces | +| Webhook responses | Synchronous (must reply within 30 seconds) | For async work, respond immediately and use REST API later | +| File attachments | Via Google Drive API | Separate Drive API quotas apply | + +### Service Account Complexity + +Setting up a Google Chat bot requires: Google Cloud project creation, Chat API enablement, service account creation and key management, Chat app configuration in Admin Console, OAuth consent screen configuration, and Workspace admin approval. This is significantly more complex than Slack or Discord (create app, paste token). + +### No Consumer Support + +Unlike Telegram, Discord, or Slack (free workspaces), Google Chat bots are exclusively for Google Workspace organisations. There is no way to create a public bot that any Google user can interact with. + +## Related + +- `services/communications/slack.md` — Slack bot integration (closest comparable — corporate, no E2E, but has Socket Mode) +- `services/communications/matrix-bot.md` — Matrix bot integration (E2E encrypted, self-hostable) +- `services/communications/simplex.md` — SimpleX Chat (no identifiers, maximum privacy) +- `services/communications/matterbridge.md` — Multi-platform chat bridging (no Google Chat support) +- `scripts/entity-helper.sh` — Entity memory system (identity resolution, Layer 0/1/2) +- `scripts/runner-helper.sh` — Runner management +- `tools/security/opsec.md` — Operational security guidance +- `tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- Google Chat API: https://developers.google.com/workspace/chat +- Google Chat REST reference: https://developers.google.com/workspace/chat/api/reference/rest +- Google Cloud Console: https://console.cloud.google.com +- Google Workspace Admin: https://admin.google.com +- Google Vault: https://vault.google.com +- Google Transparency Report: https://transparencyreport.google.com diff --git a/.agents/services/communications/imessage.md b/.agents/services/communications/imessage.md new file mode 100644 index 000000000..ece4c176a --- /dev/null +++ b/.agents/services/communications/imessage.md @@ -0,0 +1,418 @@ +--- +description: iMessage / BlueBubbles — Apple encrypted messaging, BlueBubbles REST API bot integration, imsg CLI send-only, macOS-only requirement, security model, SMS fallback risks +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# iMessage / BlueBubbles Bot Integration + + + +## Quick Reference + +- **Type**: Apple's encrypted messaging — E2E encrypted, Apple ecosystem only +- **License**: BlueBubbles (Apache-2.0), imsg CLI (MIT, [github.com/steipete/imsg](https://github.com/steipete/imsg)) +- **Bot tools**: BlueBubbles REST API (recommended, full-featured) OR imsg CLI (simple send-only) +- **Protocol**: Apple Push Notification service (APNs) + iMessage protocol +- **Encryption**: E2E (ECDSA P-256 for newer devices, RSA-2048 + AES-128-CTR for legacy) +- **BlueBubbles server**: [github.com/BlueBubblesApp/bluebubbles-server](https://github.com/BlueBubblesApp/bluebubbles-server) +- **BlueBubbles docs**: [docs.bluebubbles.app](https://docs.bluebubbles.app/) +- **Requirement**: macOS host with Messages.app (always-on Mac, Apple ID signed in) +- **Platforms**: iMessage users only (iPhone, iPad, Mac, Apple Watch, Apple Vision Pro) + +**Key differentiator**: iMessage is the default messaging platform for over 1 billion Apple users. Unlike open protocols (Matrix, SimpleX, XMTP), iMessage has no official bot API — BlueBubbles provides an unofficial bridge by wrapping Messages.app on a macOS host. This makes it the only viable path for programmatic iMessage interaction. + +**When to use iMessage vs other protocols**: + +| Criterion | iMessage (BlueBubbles) | SimpleX | Matrix | Signal | +|-----------|----------------------|---------|--------|--------| +| User base | 1B+ Apple users | Growing niche | Enterprise/tech | 40M+ | +| Identity | Apple ID / phone | None | `@user:server` | Phone number | +| E2E encryption | Yes (closed source) | Yes (audited) | Optional (Megolm) | Yes (audited) | +| Official bot API | No (unofficial only) | WebSocket JSON | First-class SDK | No | +| Open source | No (BlueBubbles is) | Yes (AGPL-3.0) | Yes | Yes | +| Platform | Apple-only | Cross-platform | Cross-platform | Cross-platform | +| Best for | Reaching Apple users | Privacy-first | Team collaboration | Privacy + mainstream | + + + +## Architecture + +```text +Path 1: BlueBubbles (bidirectional) Path 2: imsg CLI (send-only) + +┌───────────────────┐ ┌───────────────────┐ +│ iPhone / iPad / │ │ aidevops runner / │ +│ Mac Users │ │ cron job / script │ +└────────┬──────────┘ └────────┬──────────┘ + │ iMessage (E2E via APNs) │ shell exec +┌────────▼──────────┐ ┌────────▼──────────┐ +│ macOS Host │ │ imsg CLI (Swift) │ +│ ├─ Messages.app │ │ → Messages.app │ +│ ├─ BlueBubbles │ │ → iMessage / SMS │ +│ │ (Electron, │ └───────────────────┘ +│ │ private APIs) │ +│ └─ REST :1234 │ +│ + WebSocket │ +└────────┬──────────┘ + │ HTTP REST + WebSocket +┌────────▼──────────┐ +│ Bot Process │ +│ ├─ Webhook recv │ +│ ├─ Command router │ +│ └─ aidevops │ +└───────────────────┘ +``` + +**Message flow (BlueBubbles)**: User sends iMessage → APNs delivers to macOS host → Messages.app decrypts → BlueBubbles detects via private API → fires webhook/WebSocket to bot → bot replies via REST API → BlueBubbles instructs Messages.app → encrypted send via APNs. + +## Installation + +### Path 1: BlueBubbles Server (Recommended) + +**Requirements**: + +- macOS 11 (Big Sur) or later +- Apple ID signed into Messages.app +- Always-on Mac (Mac mini recommended for servers) +- Full Disk Access permission for BlueBubbles +- Accessibility permission for BlueBubbles + +**Setup**: + +1. Download BlueBubbles `.dmg` from [GitHub releases](https://github.com/BlueBubblesApp/bluebubbles-server/releases) +2. Install to `/Applications`, open, grant Full Disk Access + Accessibility permissions +3. Sign into Messages.app with your Apple ID (must be running) +4. Configure: set server password, port (default: 1234), enable Private API +5. Verify: `curl -s "http://localhost:1234/api/v1/server?password=YOUR_PASSWORD" | jq .` + +**Private API setup** (required for full features): + +BlueBubbles uses a "Private API" helper that hooks into macOS internals for features like typing indicators, read receipts, reactions, and message editing. This requires: + +1. Disable SIP (System Integrity Protection) — **only on the server Mac** +2. Install the Private API helper bundle +3. Restart BlueBubbles + +See: [docs.bluebubbles.app/private-api](https://docs.bluebubbles.app/server/private-api-setup) + +**Headless / VM setup**: Prevent display sleep (`sudo pmset -a displaysleep 0 sleep 0`), create a launchd plist (`sh.aidevops.messages-keepalive.plist`) with `KeepAlive: true` to keep Messages.app running. For macOS VMs, use screen sharing for initial setup, then run headless. + +### Path 2: imsg CLI (Send-Only) + +```bash +# Install via Homebrew +brew install steipete/tap/imsg + +# Or build from source +git clone https://github.com/steipete/imsg.git +cd imsg +swift build -c release +cp .build/release/imsg /usr/local/bin/ + +# Verify +imsg --version +``` + +**Usage**: + +```bash +# Send a message to a phone number +imsg send "+1234567890" "Hello from aidevops" + +# Send to an email (Apple ID) +imsg send "user@example.com" "Deployment complete" + +# Send to a group chat (by group name) +imsg send --group "DevOps Team" "Build passed" +``` + +**Limitations**: imsg can only send messages. It cannot receive, read, or react to messages. For bidirectional communication, use BlueBubbles. + +## Bot API (BlueBubbles) + +### REST API Endpoints + +All requests require the `password` query parameter or `Authorization` header. + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/v1/message` | GET | List messages (with pagination, filtering) | +| `/api/v1/message/:guid` | GET | Get specific message | +| `/api/v1/message/text` | POST | Send text message | +| `/api/v1/message/:chatGuid/tapback` | POST | Send reaction/tapback | +| `/api/v1/message/:guid/edit` | PUT | Edit a sent message | +| `/api/v1/message/:guid/unsend` | POST | Unsend a message | +| `/api/v1/chat` | GET | List chats (DMs and groups) | +| `/api/v1/chat/:guid` | GET | Get specific chat | +| `/api/v1/chat/new` | POST | Create new chat | +| `/api/v1/contact` | GET | List contacts | +| `/api/v1/attachment/:guid` | GET | Download attachment | +| `/api/v1/attachment/upload` | POST | Upload attachment | +| `/api/v1/server` | GET | Server info and status | +| `/api/v1/handle` | GET | List handles (phone numbers/emails) | + +### WebSocket Real-Time Events + +Connect to `ws://localhost:1234` for real-time events: + +| Event | Description | +|-------|-------------| +| `new-message` | New message received | +| `updated-message` | Message edited or tapback added | +| `typing-indicator` | Contact started/stopped typing | +| `group-name-change` | Group chat renamed | +| `participant-added` | Member added to group | +| `participant-removed` | Member removed from group | +| `chat-read-status-changed` | Chat marked as read | + +### Features + +| Feature | BlueBubbles API | imsg CLI | +|---------|----------------|----------| +| Send DM | Yes | Yes | +| Send group message | Yes | Yes (by name) | +| Receive messages | Yes (webhook/WebSocket) | No | +| Reactions/tapbacks | Yes (Private API) | No | +| Edit message | Yes (Private API, macOS 13+) | No | +| Unsend message | Yes (Private API, macOS 13+) | No | +| Reply threading | Yes (Private API) | No | +| Attachments (send) | Yes | No | +| Attachments (receive) | Yes | No | +| Typing indicators | Yes (Private API) | No | +| Read receipts | Yes (Private API) | No | +| Contact info | Yes | No | +| Group management | Limited | No | +| Message search | Yes | No | + +### Access Control Patterns + +```typescript +// allowlist of Apple IDs / phone numbers permitted to interact with bot +const ALLOWED_SENDERS = new Set([ + "+1234567890", + "admin@example.com", +]); + +function isAuthorized(handle: string): boolean { + return ALLOWED_SENDERS.has(handle); +} +``` + +### Webhook-Based Message Handling + +```typescript +import express from "express"; + +const app = express(); +app.use(express.json()); + +const BB_PASSWORD = process.env.BLUEBUBBLES_PASSWORD; +const BB_URL = process.env.BLUEBUBBLES_URL || "http://localhost:1234"; + +app.post("/webhook", async (req, res) => { + const { type, data } = req.body; + if (type !== "new-message" || data.isFromMe) { res.sendStatus(200); return; } + + const { text, handle, chats } = data; + if (!isAuthorized(handle?.address)) { res.sendStatus(200); return; } + + if (text.startsWith("/")) { + const chatGuid = chats?.[0]?.guid; + const response = await handleCommand(text.slice(1).trim()); + await fetch(`${BB_URL}/api/v1/message/text?password=${BB_PASSWORD}`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ chatGuid, message: response, method: "private-api" }), + }); + } + res.sendStatus(200); +}); + +app.listen(3000); +``` + +**Webhook setup**: In BlueBubbles, navigate to API & Webhooks → add URL `http://your-bot-host:3000/webhook` → select events (`new-message`, `updated-message`). + +## Security Considerations + +### Encryption + +- **Newer devices (2020+)**: ECDSA P-256 key agreement, AES-256-GCM message encryption +- **Legacy devices**: RSA-2048 key exchange + AES-128-CTR message encryption +- **Group chats**: Each message individually encrypted per recipient (no group key) +- **Key verification**: Contact Key Verification (iOS 17.2+) — manual verification like Signal's safety numbers +- Apple **cannot** read iMessage content in transit + +### Metadata + +Apple sees metadata despite E2E encryption: sender/recipient identifiers (phone/Apple ID), timestamps, IP addresses, device info. Apple retains metadata for up to 30 days. Message content, attachments, and reactions are NOT visible to Apple. + +### iCloud Backups (CRITICAL) + +Default iCloud backups are **NOT E2E encrypted** — Apple holds the keys, meaning Apple and law enforcement (with warrant) can access message content from backups. **Advanced Data Protection** (opt-in, iOS 16.3+) enables E2E encrypted backups. **Recommendation**: Enable Advanced Data Protection or disable iCloud backup for Messages. This is the single biggest practical privacy risk for iMessage. + +### Push Notifications + +Delivered via APNs. Apple sees notification metadata (device token, timestamp) but **not** message content. Push tokens linkable to Apple ID. No way to use iMessage without APNs. + +### AI Training + +Apple states they do **not** use iMessage content for AI training. Apple Intelligence features process data on-device where possible; server-side processing uses "Private Cloud Compute" with published transparency logs. No third-party AI model has access to iMessage content via Apple's systems. + +### Open Source Status + +- **iMessage protocol + Apple servers**: CLOSED source — no independent audit of E2E implementation (though security researchers including Johns Hopkins have analyzed and found it sound) +- **BlueBubbles**: OPEN source (Apache-2.0) — auditable bridge layer +- **imsg CLI**: OPEN source (MIT) — auditable send-only tool +- Apple publishes a [Platform Security Guide](https://support.apple.com/guide/security/welcome/web) with protocol details (self-reported, not independently verified) + +### Jurisdiction + +Apple Inc., USA (Cupertino, CA). Subject to US law enforcement (NSLs, FISA, subpoenas). Apple has historically fought for user privacy (FBI vs Apple, 2016) and publishes transparency reports. Non-US user data may transit US infrastructure. + +### Bot-Specific Security + +- BlueBubbles Mac has access to **all decrypted messages** for the signed-in Apple ID — secure the host: FileVault, strong password, auto-lock, firewall, minimal software +- REST API must be localhost-only or behind reverse proxy with TLS + auth — **never** expose port 1234 to the internet +- Use a **dedicated Apple ID** for the bot, not a personal one +- Store BlueBubbles password via gopass or env vars — never in code or logs + +### SMS Fallback (CRITICAL) + +When iMessage is unavailable, messages **fall back to SMS** which is **completely unencrypted**. There is **no way to prevent this programmatically**. Users can disable it per-device (Settings > Messages > "Send as SMS" off), but bots cannot control this. **Recommendation**: Check the BlueBubbles `service` field (`iMessage` vs `SMS`) before sending sensitive content. + +### Comparison Summary + +Strong privacy for a mainstream platform: + +| Aspect | iMessage | Signal | WhatsApp | SimpleX | +|--------|----------|--------|----------|---------| +| E2E encryption | Yes | Yes | Yes | Yes | +| Open source | No | Yes | No (client) | Yes | +| Metadata collection | Moderate (30d) | Minimal | Extensive | None | +| Independent audit | No | Yes | Yes (protocol) | Yes | +| iCloud backup risk | Yes (default) | N/A | Yes (default) | N/A | +| SMS fallback | Yes (unencrypted) | No | No | No | +| Closed protocol | Yes | No | Yes | No | + +**Bottom line**: Better than WhatsApp (less metadata harvesting by the service provider). Worse than Signal (Apple still sees metadata, protocol is closed source, iCloud backup is a practical risk). The iCloud backup issue and SMS fallback are the biggest real-world privacy risks. + +## Integration with aidevops + +### Components + +| Component | Purpose | +|-----------|---------| +| `imessage-dispatch-helper.sh` | Shell helper for sending notifications via imsg CLI | +| BlueBubbles webhook handler | Receives inbound messages, dispatches to runners | +| `entity-helper.sh` | Resolves user identifiers (phone/Apple ID) to aidevops entities | +| Config file | `~/.config/aidevops/imessage-bot.json` | + +### imessage-dispatch-helper.sh Pattern + +```bash +#!/usr/bin/env bash +# imessage-dispatch-helper.sh — send notifications via iMessage +set -euo pipefail + +send_imessage() { + local recipient="$1" + local message="$2" + if ! command -v imsg &>/dev/null; then + echo "ERROR: imsg not installed. brew install steipete/tap/imsg" >&2 + return 1 + fi + imsg send "$recipient" "$message" + return 0 +} +# Usage: imessage-dispatch-helper.sh "+1234567890" "Build #42 passed" +``` + +### BlueBubbles Webhook to Runner Dispatch + +Flow: iMessage user sends command → BlueBubbles webhook → verify sender via entity-helper.sh → parse command → dispatch via runner-helper.sh → return result → reply via BlueBubbles API. + +### Entity Resolution + +```bash +# Resolve phone/Apple ID to aidevops entity +entity-helper.sh resolve "+1234567890" +# → { "entity": "marcus", "role": "admin", "platforms": ["imessage", "matrix"] } + +# Check authorization +entity-helper.sh check-auth "+1234567890" "deploy" +# → exit 0 (authorized) or exit 1 (denied) +``` + +### Configuration + +```json +{ + "server_url": "http://localhost:1234", + "server_password_ref": "gopass:aidevops/bluebubbles/password", + "webhook_port": 3000, + "webhook_path": "/webhook", + "allowed_handles": [ + "+1234567890", + "admin@example.com" + ], + "command_prefix": "/", + "send_method": "private-api", + "notifications": { + "build_status": true, + "deploy_alerts": true, + "error_alerts": true + } +} +``` + +Store at `~/.config/aidevops/imessage-bot.json`. Server password must reference a gopass secret, never stored in plaintext in the config file. + +## Matterbridge Integration + +### No Native Support + +Matterbridge does **not** have native support for iMessage. There is no official or community gateway for iMessage in the Matterbridge ecosystem. + +### Custom Gateway Feasibility + +A custom gateway could bridge BlueBubbles REST API → custom Node.js adapter → Matterbridge API (:4242) → Matrix, Telegram, Discord, SimpleX, etc. + +**Considerations**: Poll BlueBubbles or listen on WebSocket for inbound; send via REST API for outbound. Main complexity is identity mapping and iMessage-specific features (tapbacks → reactions). Inherits all BlueBubbles limitations. Effort: medium. + +**Alternative**: For simple 1:1 bridging (e.g., iMessage ↔ Matrix), a direct bridge bot avoids the Matterbridge abstraction — BlueBubbles webhook receives iMessage, bot forwards to Matrix via matrix-bot-sdk, and vice versa. + +## Limitations + +- **macOS only**: Requires always-on Mac with Messages.app. No Linux/Windows. Mac mini (M-series, ~$500-600) recommended. macOS VMs possible on Apple hardware only. +- **Apple ID required**: Dedicated Apple ID needed, phone number for verification, mandatory 2FA. Apple may lock accounts used for automated messaging. +- **No official bot API**: BlueBubbles uses unofficial private APIs + AppleScript — can break with macOS updates. Apple has not endorsed third-party iMessage automation and could actively block it. +- **SMS fallback is unencrypted**: Non-iMessage recipients silently get SMS. No programmatic prevention. Check `service` field in API responses. +- **BlueBubbles requirements**: Electron app (~200-400MB RAM), Full Disk Access + Accessibility permissions, Private API requires SIP disabled, restart needed after macOS updates. +- **Group management limited**: Programmatic group creation supported but minimal admin controls compared to Matrix/Telegram. +- **Rate limiting**: Apple imposes undocumented limits. Bulk messaging violates Apple ToS. Implement bot-side rate limiting (max 10-20 messages/minute). +- **No cross-platform**: Apple-only. No web client. RCS (iOS 18+) improves SMS interop but is not iMessage. +- **Unofficial integration risk**: Apple may break private APIs with any macOS update. Treat iMessage as a "best-effort" integration, not mission-critical. + +## Related + +- `services/communications/simplex.md` — SimpleX Chat (zero-knowledge, no identifiers) +- `services/communications/matrix-bot.md` — Matrix bot integration (federated, official SDK) +- `services/communications/matterbridge.md` — Multi-platform chat bridge +- `services/communications/twilio.md` — SMS/voice via Twilio (for non-Apple messaging) +- `tools/security/opsec.md` — Operational security guidance +- `tools/credentials/gopass.md` — Secure credential storage +- BlueBubbles: https://bluebubbles.app/ +- BlueBubbles GitHub: https://github.com/BlueBubblesApp/bluebubbles-server +- BlueBubbles API Docs: https://docs.bluebubbles.app/ +- imsg CLI: https://github.com/steipete/imsg +- Apple Platform Security Guide: https://support.apple.com/guide/security/welcome/web diff --git a/.agents/services/communications/matterbridge.md b/.agents/services/communications/matterbridge.md index fd8332a68..724be5618 100644 --- a/.agents/services/communications/matterbridge.md +++ b/.agents/services/communications/matterbridge.md @@ -462,9 +462,28 @@ See `tools/security/opsec.md` for full platform trust matrix and threat modeling ## Related +### Platforms with native Matterbridge support + - `services/communications/matrix-bot.md` — Matrix bot for aidevops runner dispatch -- `services/communications/simplex.md` — SimpleX install, bot API, self-hosted servers +- `services/communications/simplex.md` — SimpleX (via custom adapter) +- `services/communications/telegram.md` — Telegram Bot API +- `services/communications/signal.md` — Signal (via signal-cli) +- `services/communications/whatsapp.md` — WhatsApp (via whatsmeow) +- `services/communications/slack.md` — Slack Bot API +- `services/communications/discord.md` — Discord Bot API +- `services/communications/msteams.md` — MS Teams (webhook/Bot Framework) +- `services/communications/nextcloud-talk.md` — Nextcloud Talk API + +### Platforms without native Matterbridge support + +- `services/communications/nostr.md` — Nostr (would require custom gateway) +- `services/communications/imessage.md` — iMessage (would require BlueBubbles gateway) +- `services/communications/google-chat.md` — Google Chat (would require custom gateway) +- `services/communications/urbit.md` — Urbit (would require Eyre HTTP gateway) + +### Other + - `services/communications/bitchat.md` — Bitchat (Bluetooth mesh, offline P2P) - `services/communications/xmtp.md` — XMTP (Web3 messaging, agent SDK, payments) -- `tools/security/opsec.md` — Platform trust matrix, E2E status, metadata warnings +- `tools/security/opsec.md` — Platform trust matrix, privacy comparison, AI training risks - `tools/ai-assistants/headless-dispatch.md` — Headless dispatch patterns diff --git a/.agents/services/communications/nextcloud-talk.md b/.agents/services/communications/nextcloud-talk.md new file mode 100644 index 000000000..6c0d1c5bf --- /dev/null +++ b/.agents/services/communications/nextcloud-talk.md @@ -0,0 +1,750 @@ +--- +description: Nextcloud Talk — self-hosted team communication with strongest corporate privacy, Talk Bot API (webhook-based, OCC CLI), server-side encryption, Matterbridge bridging, and aidevops dispatch +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Nextcloud Talk Bot Integration + + + +## Quick Reference + +- **Type**: Self-hosted team communication — you own everything, strongest privacy for corporate use +- **License**: AGPL-3.0 (Nextcloud server + Talk app) +- **Bot tool**: Talk Bot API (webhook-based, OCC CLI registration) +- **Protocol**: Nextcloud Talk API (HTTP REST + webhook) +- **Encryption**: TLS in transit, server-side at rest (you control the keys), E2E for 1:1 calls (WebRTC) +- **Script**: `nextcloud-talk-dispatch-helper.sh [setup|start|stop|status|map|unmap|mappings|test|logs]` +- **Config**: `~/.config/aidevops/nextcloud-talk-bot.json` (600 permissions) +- **Data**: `~/.aidevops/.agent-workspace/nextcloud-talk-bot/` +- **Docs**: https://nextcloud-talk.readthedocs.io/ | https://docs.nextcloud.com/server/latest/developer_manual/digging_deeper/bots.html +- **Server**: https://nextcloud.com/install/ | https://docs.nextcloud.com/server/latest/admin_manual/ + +**Key differentiator**: Nextcloud Talk is the strongest privacy option for corporate/team communication. You own the server, the database, the encryption keys, the backups — everything. No third party (including Nextcloud GmbH) has access to any of your data. Unlike Slack/Teams/Discord, there is ZERO external data access. Unlike SimpleX/Signal, you also get a full collaboration suite (files, calendar, office, contacts). + +**Quick start**: + +```bash +nextcloud-talk-dispatch-helper.sh setup # Interactive wizard +nextcloud-talk-dispatch-helper.sh map "general" code-reviewer +nextcloud-talk-dispatch-helper.sh start --daemon +``` + + + +## Architecture + +```text +┌──────────────────────┐ +│ Nextcloud Talk Room │ +│ │ +│ User sends message │ +│ @bot or in mapped │ +│ conversation │ +└──────────┬───────────┘ + │ + │ Talk Bot API (webhook POST) + │ HMAC-SHA256 signature verification + │ +┌──────────▼───────────┐ ┌──────────────────────┐ +│ Bot Webhook Endpoint │ │ aidevops Dispatch │ +│ (Bun/Node HTTP) │ │ │ +│ │ │ runner-helper.sh │ +│ ├─ Signature verify │────▶│ → AI session │ +│ ├─ Access control │ │ → response │ +│ ├─ Message parsing │◀────│ │ +│ ├─ Entity resolution │ │ │ +│ └─ Reply via OCS API │ │ │ +└──────────┬───────────┘ └──────────────────────┘ + │ +┌──────────▼───────────┐ +│ Nextcloud Server │ +│ (YOUR infrastructure) │ +│ │ +│ ├── PostgreSQL/MySQL │ Message storage (encrypted at rest) +│ ├── Talk app │ Conversations, participants, bots +│ ├── Files app │ File sharing, attachments +│ ├── Collabora/OnlyO. │ Office document editing +│ └── Calendar/Contacts │ Full collaboration suite +└───────────────────────┘ +``` + +**Message flow**: + +1. User sends message in a Nextcloud Talk conversation +2. Talk server checks if a bot is registered for that conversation +3. Talk server sends webhook POST to bot endpoint with HMAC-SHA256 signature +4. Bot verifies signature using shared secret +5. Bot checks access control (Nextcloud user ID allowlists) +6. Entity resolution: Nextcloud user ID resolved to entity via `entity-helper.sh` +7. Layer 0 logging: user message logged as immutable interaction +8. Context loading: entity profile + conversation summary + recent interactions +9. Bot dispatches entity-aware prompt to runner via `runner-helper.sh` +10. Runner executes via headless dispatch +11. Bot posts response back to Talk conversation via OCS API +12. Bot adds reaction emoji (hourglass while processing, checkmark on success, X on failure) + +## Installation + +### Prerequisites + +1. **Nextcloud server** (self-hosted) — version 27+ recommended for Talk Bot API +2. **Talk app** installed and enabled (bundled with Nextcloud, or install from App Store) +3. **Admin access** to the Nextcloud instance (for OCC CLI bot registration) +4. **Node.js >= 18** or **Bun** runtime for the webhook handler +5. **Network reachability**: Bot endpoint must be reachable from the Nextcloud server (localhost, LAN, or tunneled) + +### Step 1: Ensure Talk App is Installed + +```bash +# Check if Talk is installed +sudo -u www-data php /var/www/nextcloud/occ app:list | grep spreed + +# Install Talk if not present +sudo -u www-data php /var/www/nextcloud/occ app:install spreed + +# Enable Talk if disabled +sudo -u www-data php /var/www/nextcloud/occ app:enable spreed + +# Verify version (27+ required for Bot API) +sudo -u www-data php /var/www/nextcloud/occ app:info spreed +``` + +### Step 2: Register a Bot via OCC CLI + +The Talk Bot API uses OCC CLI for bot registration. Each bot gets a name, a webhook URL, a description, and a shared secret for signature verification. + +```bash +# Register a new bot +# The command returns a JSON object with the bot ID and shared secret +sudo -u www-data php /var/www/nextcloud/occ talk:bot:install \ + "aidevops" \ + "http://localhost:8780/webhook" \ + "AI-powered DevOps assistant" \ + "YOUR_SHARED_SECRET_HERE" + +# List registered bots +sudo -u www-data php /var/www/nextcloud/occ talk:bot:list + +# Remove a bot +sudo -u www-data php /var/www/nextcloud/occ talk:bot:remove BOT_ID + +# Enable bot for a specific conversation +# (done via Talk admin settings or API) +``` + +### Step 3: Generate a Shared Secret + +```bash +# Generate a cryptographically secure secret +openssl rand -hex 32 + +# Store securely +gopass insert aidevops/nextcloud-talk/webhook-secret + +# Or via credentials.sh fallback (600 permissions) +``` + +### Step 4: Configure the Webhook Endpoint + +The bot must run an HTTP server that receives webhook POSTs from the Nextcloud server. The endpoint must: + +1. Accept POST requests with JSON body +2. Verify the `X-Nextcloud-Talk-Signature` header (HMAC-SHA256) +3. Respond with 200 OK quickly (process asynchronously) + +### Step 5: Create an App Password for API Access + +The bot needs an app password to send messages back via the OCS API: + +```bash +# Create via Nextcloud UI: +# Settings > Security > Devices & sessions > Create new app password +# Name: "aidevops-talk-bot" + +# Or via OCC (admin only): +sudo -u www-data php /var/www/nextcloud/occ user:setting BOT_USER app_password + +# Store securely +gopass insert aidevops/nextcloud-talk/app-password +``` + +### Step 6: Install Dependencies + +```bash +# Using Bun (preferred) +bun add express crypto + +# Using npm +npm install express +``` + +## Bot API Integration + +### Webhook Payload Format + +When a message is posted in a conversation where the bot is enabled, Talk sends a webhook POST: + +```json +{ + "type": "Create", + "actor": { + "type": "User", + "id": "admin", + "name": "Admin User" + }, + "object": { + "type": "Message", + "id": "42", + "name": "Hello @aidevops, can you review the latest PR?", + "content": "Hello @aidevops, can you review the latest PR?", + "mediaType": "text/markdown" + }, + "target": { + "type": "Collection", + "id": "conversation-token", + "name": "Development" + } +} +``` + +### Signature Verification + +Every webhook request includes an `X-Nextcloud-Talk-Signature` header containing an HMAC-SHA256 signature of the request body, computed with the shared secret. + +```typescript +import { createHmac } from "crypto"; + +function verifySignature(body: string, signature: string, secret: string): boolean { + const expected = createHmac("sha256", secret) + .update(body) + .digest("hex"); + // Constant-time comparison to prevent timing attacks + if (expected.length !== signature.length) return false; + let result = 0; + for (let i = 0; i < expected.length; i++) { + result |= expected.charCodeAt(i) ^ signature.charCodeAt(i); + } + return result === 0; +} +``` + +### Complete Webhook Handler + +```typescript +// nextcloud-talk-bot.ts — webhook handler for Nextcloud Talk Bot API +import express from "express"; +import { createHmac } from "crypto"; + +const PORT = 8780; +const NEXTCLOUD_URL = process.env.NEXTCLOUD_URL || "https://cloud.example.com"; +const BOT_USER = process.env.BOT_USER || "aidevops-bot"; +const APP_PASSWORD = process.env.APP_PASSWORD || ""; +const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET || ""; + +// Allowed Nextcloud user IDs +const ALLOWED_USERS = new Set(["admin", "developer1", "developer2"]); + +const app = express(); + +// Raw body needed for signature verification +app.use(express.raw({ type: "application/json" })); + +function verifySignature(body: Buffer, signature: string): boolean { + const expected = createHmac("sha256", WEBHOOK_SECRET) + .update(body) + .digest("hex"); + if (expected.length !== signature.length) return false; + let result = 0; + for (let i = 0; i < expected.length; i++) { + result |= expected.charCodeAt(i) ^ signature.charCodeAt(i); + } + return result === 0; +} + +// Send message to a Talk conversation via OCS API +async function sendMessage(conversationToken: string, message: string): Promise { + const url = `${NEXTCLOUD_URL}/ocs/v2.php/apps/spreed/api/v1/chat/${conversationToken}`; + const auth = Buffer.from(`${BOT_USER}:${APP_PASSWORD}`).toString("base64"); + + await fetch(url, { + method: "POST", + headers: { + "Content-Type": "application/json", + "OCS-APIRequest": "true", + "Authorization": `Basic ${auth}`, + }, + body: JSON.stringify({ message }), + }); +} + +// Send reaction to a message +async function sendReaction( + conversationToken: string, + messageId: string, + reaction: string, +): Promise { + const url = `${NEXTCLOUD_URL}/ocs/v2.php/apps/spreed/api/v1/reaction/${conversationToken}/${messageId}`; + const auth = Buffer.from(`${BOT_USER}:${APP_PASSWORD}`).toString("base64"); + + await fetch(url, { + method: "POST", + headers: { + "Content-Type": "application/json", + "OCS-APIRequest": "true", + "Authorization": `Basic ${auth}`, + }, + body: JSON.stringify({ reaction }), + }); +} + +app.post("/webhook", async (req, res) => { + const signature = req.headers["x-nextcloud-talk-signature"] as string; + if (!signature || !verifySignature(req.body, signature)) { + res.status(401).send("Invalid signature"); + return; + } + + // Respond immediately — process asynchronously + res.status(200).send("OK"); + + const payload = JSON.parse(req.body.toString()); + const userId = payload.actor?.id; + const messageText = payload.object?.name || ""; + const messageId = payload.object?.id; + const conversationToken = payload.target?.id; + + // Access control + if (ALLOWED_USERS.size > 0 && !ALLOWED_USERS.has(userId)) { + return; + } + + // Skip empty messages or non-text + if (!messageText.trim()) return; + + // Add processing reaction + await sendReaction(conversationToken, messageId, "👀"); + + try { + // Dispatch to runner (integrate with runner-helper.sh) + const response = await dispatchToRunner(messageText, userId, conversationToken); + + await sendMessage(conversationToken, response); + + // Success reaction + await sendReaction(conversationToken, messageId, "✅"); + } catch (error) { + await sendMessage(conversationToken, `Error: ${error.message}`); + await sendReaction(conversationToken, messageId, "❌"); + } +}); + +app.listen(PORT, () => { + console.log(`Nextcloud Talk bot listening on port ${PORT}`); +}); +``` + +### OCS API: Messaging and Conversations + +```bash +# List conversations the bot user is part of +curl -s -u "bot-user:app-password" \ + -H "OCS-APIRequest: true" \ + "https://cloud.example.com/ocs/v2.php/apps/spreed/api/v4/room" | jq + +# Send a message to a conversation +curl -s -u "bot-user:app-password" \ + -H "OCS-APIRequest: true" \ + -H "Content-Type: application/json" \ + -d '{"message":"Hello from the bot!"}' \ + "https://cloud.example.com/ocs/v2.php/apps/spreed/api/v1/chat/CONVERSATION_TOKEN" + +# Get chat messages from a conversation +curl -s -u "bot-user:app-password" \ + -H "OCS-APIRequest: true" \ + "https://cloud.example.com/ocs/v2.php/apps/spreed/api/v1/chat/CONVERSATION_TOKEN?lookIntoFuture=0&limit=50" + +# Send a reaction +curl -s -u "bot-user:app-password" \ + -H "OCS-APIRequest: true" \ + -H "Content-Type: application/json" \ + -d '{"reaction":"👍"}' \ + "https://cloud.example.com/ocs/v2.php/apps/spreed/api/v1/reaction/CONVERSATION_TOKEN/MESSAGE_ID" +``` + +### Markdown Support + +Talk supports markdown in messages. The bot can send formatted responses: + +```typescript +const formattedResponse = ` +## Analysis Results + +**Repository**: \`myproject\` +**Branch**: \`feature/auth-refactor\` + +### Issues Found + +1. **SQL injection** in \`src/db/query.ts:42\` — use parameterised queries +2. **Missing input validation** in \`src/api/users.ts:18\` + +### Recommendations + +- Add input sanitisation middleware +- Enable CSP headers + +> Overall: 2 critical, 0 warnings +`; + +await sendMessage(conversationToken, formattedResponse); +``` + +### Access Control + +```typescript +// Nextcloud user ID allowlist +const ALLOWED_USERS = new Set(["admin", "developer1", "ops-team"]); + +// Conversation allowlist (by token) +const ALLOWED_CONVERSATIONS = new Set(["abc123token", "def456token"]); + +function isAllowed(userId: string, conversationToken: string): boolean { + if (ALLOWED_USERS.size > 0 && !ALLOWED_USERS.has(userId)) { + return false; + } + if (ALLOWED_CONVERSATIONS.size > 0 && !ALLOWED_CONVERSATIONS.has(conversationToken)) { + return false; + } + return true; +} +``` + +## Security Considerations + +> **CRITICAL**: Nextcloud Talk is the PRIVACY CHAMPION for self-hosted corporate communication. This section details why it offers the strongest privacy of any team collaboration platform. + +### Self-Hosted: The Key Advantage + +This is the fundamental differentiator. All data stays on YOUR server. You control: + +- **Hardware**: Your server, your rack, your data centre (or your chosen VPS) +- **Software**: Open-source stack you can audit, modify, and rebuild +- **Network**: Your firewall rules, your VPN, your DNS +- **Backups**: Your backup schedule, your backup location, your retention policy +- **Encryption keys**: Server-side encryption keys are generated and stored on YOUR server +- **Access**: No third party has access to anything unless you explicitly grant it + +No Slack admin at Salesforce, no Microsoft engineer at Teams, no Google employee at Chat can read your messages. The software is AGPL-3.0 — you download it, you run it, you own the entire stack. + +### Encryption + +- **In transit**: TLS 1.2+ — you configure the certificate (Let's Encrypt, self-signed, or CA-issued). You control the cipher suites, the certificate rotation, the HSTS headers +- **At rest**: Nextcloud server-side encryption module encrypts files and data in your database. You control the master key. When enabled, data on disk is encrypted with AES-256-CTR +- **E2E for calls**: 1:1 video and audio calls use WebRTC with OWASP-recommended encryption (SRTP with DTLS key exchange). Media goes peer-to-peer when possible — never through Nextcloud servers +- **Message storage**: Message content is stored in your Nextcloud database (PostgreSQL/MySQL/MariaDB). If you enable server-side encryption, database contents are encrypted at rest. You control the database credentials, the connection, the backup encryption + +### Metadata + +- **Only YOUR server sees metadata**. Connection logs, access times, IP addresses, user agents — all stored on YOUR server in YOUR log files +- **No third-party metadata collection**. Unlike Slack (Salesforce collects comprehensive metadata), Teams (Microsoft telemetry), or Discord (full message analytics), no external company receives any metadata +- **You control log retention**. Set log rotation, audit trail retention, and data lifecycle policies according to YOUR requirements +- **No analytics beacons**. Nextcloud does not phone home with usage data (you can optionally enable anonymous usage statistics, but it is opt-in and sends minimal aggregate data) + +### No Third-Party Data Access + +Unlike Slack/Teams/Discord/Google Chat, NO external company has access to your messages: + +| Platform | Who can access your messages | +|----------|------------------------------| +| **Slack** | Salesforce, workspace admins (full export), law enforcement (Salesforce compliance) | +| **Microsoft Teams** | Microsoft, tenant admins (eDiscovery/compliance), law enforcement (Microsoft compliance) | +| **Discord** | Discord Inc., law enforcement, trust & safety team | +| **Google Chat** | Google, Workspace admins, law enforcement (Google compliance) | +| **Nextcloud Talk** | **Only you** — server admin of your own instance | + +Nextcloud GmbH (the company that develops Nextcloud) has ZERO access to your instance. They make the software. They do not operate it. They cannot see your data. This is architecturally guaranteed by the self-hosted model. + +### Push Notifications + +- **Nextcloud push proxy**: Nextcloud provides a push notification proxy (`push-notifications` app) that relays wake-up signals to mobile devices via FCM/APNs. Minimal metadata is sent — a notification ID, no message content. The mobile app then fetches the actual message directly from YOUR server over TLS +- **Self-hosted push proxy**: You can run your own push notification proxy (`notify_push`) to eliminate ALL third-party notification metadata. The `notify_push` binary connects directly to your Nextcloud server and pushes via WebSocket to connected clients +- **Desktop/web**: Desktop and web clients use direct WebSocket connections to your server — no push proxy needed, no third-party involvement + +### AI Training + +- **NO external AI training**. Nextcloud GmbH has ZERO access to your data. There is no data pipeline, no telemetry that includes message content, no model training on your conversations +- **Local AI integration**: Nextcloud offers optional AI features (smart search, OCR, text generation) via the `assistant` app. When configured, these use local models running ON YOUR SERVER (via llamafile, Ollama, or external API you configure). No data leaves your infrastructure unless you explicitly point it at an external API +- **Your choice**: You decide whether to enable AI features, which models to use, and where they run. The default is no AI processing at all + +### Open Source + +- **Nextcloud server**: AGPL-3.0 — fully open source, auditable, forkable +- **Talk app**: AGPL-3.0 — same license, full source available +- **Mobile apps** (Android/iOS): Open source on GitHub +- **Desktop app**: Open source, Electron-based +- **Regular security audits**: Nextcloud participates in HackerOne bug bounty programme. Independent security audits published regularly +- **Reproducible builds**: Community-verified builds available +- **You can audit every line of code** that touches your data + +### Jurisdiction + +- **YOUR jurisdiction**. The server is where you put it. Run it in Germany, Switzerland, Iceland, your own office — your choice +- **Nextcloud GmbH** is headquartered in Stuttgart, Germany, and is subject to German/EU law (strong data protection under GDPR). But they have no access to your instance — jurisdiction only matters for the software distribution, not your data +- **No CLOUD Act exposure** (unless you choose to host in the US) +- **No FISA Section 702 exposure** (unless you choose to host in the US) +- **Data sovereignty**: You have complete control over where your data physically resides + +### Compliance + +You control compliance because you control the entire stack: + +- **GDPR**: Full control over data processing, retention, deletion, portability, and consent. Nextcloud provides GDPR compliance tools (data export, right to erasure) +- **HIPAA**: Can be configured for HIPAA compliance with proper access controls, audit logging, and encryption. Several healthcare organisations run Nextcloud for this reason +- **SOC2**: Audit logs, access controls, encryption at rest/in transit — all configurable +- **ISO 27001**: Nextcloud GmbH itself is ISO 27001 certified for their development processes +- **Full audit logs**: Every file access, share, login, and admin action is logged. You control retention +- **Data retention policies**: Configure automatic deletion of old messages, files, and logs +- **User management**: LDAP/AD integration, 2FA (TOTP, WebAuthn/FIDO2), SSO (SAML, OIDC) + +### Bot-Specific Security + +- **Webhook URL**: Must be reachable from your Nextcloud server. Can be `localhost` (same machine), LAN address (internal network), or tunneled (Cloudflare Tunnel, WireGuard). No public internet exposure required +- **Webhook secret**: HMAC-SHA256 signature verification prevents forged webhook deliveries. Only your Nextcloud server knows the secret +- **App password**: Bot authenticates to OCS API with an app password — scoped, revocable, auditable. Not the user's main password +- **Bot runs in YOUR infrastructure**: The webhook handler runs on your server or your network. Bot code, logs, and temporary data never leave your control + +### Comparison + +Nextcloud Talk offers the **STRONGEST privacy of any corporate-style collaboration platform**. The only platforms with better theoretical privacy are: + +- **SimpleX** — no user identifiers at all, stateless servers, but no collaboration features (files, calendar, office) +- **Signal** — E2E everything with sealed sender, but no self-hosting, no file collaboration, no office suite + +Nextcloud Talk sits in a unique position: **corporate-grade collaboration features** (file sharing, calendar, contacts, office suite, video conferencing, task boards) combined with **self-hosted privacy** where you own and control everything. + +## aidevops Integration + +### nextcloud-talk-dispatch-helper.sh + +The helper script follows the same pattern as `matrix-dispatch-helper.sh` and `slack-dispatch-helper.sh`: + +```bash +# Setup wizard — prompts for Nextcloud URL, app password, webhook secret, conversation mappings +nextcloud-talk-dispatch-helper.sh setup + +# Map conversations to runners +nextcloud-talk-dispatch-helper.sh map "development" code-reviewer +nextcloud-talk-dispatch-helper.sh map "seo-team" seo-analyst +nextcloud-talk-dispatch-helper.sh map "operations" ops-monitor + +# List mappings +nextcloud-talk-dispatch-helper.sh mappings + +# Remove a mapping +nextcloud-talk-dispatch-helper.sh unmap "development" + +# Start/stop the webhook handler +nextcloud-talk-dispatch-helper.sh start --daemon +nextcloud-talk-dispatch-helper.sh stop +nextcloud-talk-dispatch-helper.sh status + +# Test dispatch +nextcloud-talk-dispatch-helper.sh test code-reviewer "Review src/auth.ts" + +# View logs +nextcloud-talk-dispatch-helper.sh logs +nextcloud-talk-dispatch-helper.sh logs --follow +``` + +### Runner Dispatch + +The bot dispatches to runners via `runner-helper.sh`, which handles: + +- Runner AGENTS.md (personality/instructions) +- Headless session management +- Memory namespace isolation +- Entity-aware context loading +- Run logging + +### Entity Resolution + +When a Nextcloud user sends a message, the bot resolves their Nextcloud user ID to an entity: + +- **Known user**: Match on `entity_channels` table (`channel=nextcloud-talk`, `channel_id=username`) +- **New user**: Creates entity via `entity-helper.sh create` with Nextcloud user ID linked +- **Cross-channel**: If the same person is linked on other channels (Matrix, Slack, SimpleX, email), their full profile is available +- **Profile enrichment**: Nextcloud's user API provides display name, email, groups — used to populate entity profile on first contact + +### Configuration + +`~/.config/aidevops/nextcloud-talk-bot.json` (600 permissions): + +```json +{ + "nextcloudUrl": "https://cloud.example.com", + "botUser": "aidevops-bot", + "appPassword": "", + "webhookSecret": "", + "webhookPort": 8780, + "allowedUsers": ["admin", "developer1"], + "defaultRunner": "", + "conversationMappings": { + "development": "code-reviewer", + "seo-team": "seo-analyst", + "operations": "ops-monitor" + }, + "ignoreOwnMessages": true, + "maxPromptLength": 3000, + "responseTimeout": 600, + "sessionIdleTimeout": 300 +} +``` + +**Note**: `appPassword` and `webhookSecret` should be stored via `gopass` (preferred) or in the config file with 600 permissions. Never commit credentials to version control. + +## Matterbridge Integration + +Nextcloud Talk is natively supported by [Matterbridge](https://github.com/42wim/matterbridge) via the Talk API. + +```text +Nextcloud Talk + │ + │ Talk API (via app password) + │ +Matterbridge (Go binary) + │ + ├── Matrix rooms + ├── Slack workspaces + ├── Discord channels + ├── Telegram groups + ├── Signal contacts + ├── SimpleX chats + ├── IRC channels + └── 40+ other platforms +``` + +### Matterbridge Configuration + +Add to `matterbridge.toml`: + +```toml +[nextcloud.myserver] +Server = "https://cloud.example.com" +Login = "matterbridge-bot" +Password = "app-password-here" + +## SectionJoinPart shows join/leave messages +ShowJoinPart = false +``` + +Gateway configuration: + +```toml +[[gateway]] +name = "dev-bridge" +enable = true + +[[gateway.inout]] +account = "nextcloud.myserver" +channel = "development" + +[[gateway.inout]] +account = "matrix.myserver" +channel = "#dev:matrix.example.com" +``` + +**Privacy note**: Bridging Nextcloud Talk to external platforms (Slack, Discord, Telegram) means messages from your self-hosted server will be stored on third-party infrastructure. Users should be informed that bridged conversations lose the privacy guarantees of self-hosting. Bridging to other self-hosted platforms (Matrix on your server, IRC on your network) preserves the self-hosted privacy model. See `services/communications/matterbridge.md` for full bridging considerations. + +## Limitations + +### Self-Hosted Maintenance Overhead + +Nextcloud Talk requires you to run and maintain a Nextcloud server. This includes: + +- Server provisioning and hardening +- Regular Nextcloud updates (PHP, database, app updates) +- SSL certificate management +- Backup configuration and testing +- Monitoring and alerting +- Database maintenance (PostgreSQL/MySQL tuning) + +**Mitigation**: Use managed Nextcloud hosting (e.g., Hetzner StorageShare, IONOS) or Cloudron for simplified deployment. See `services/hosting/cloudron.md`. + +### Talk Bot API Maturity + +The Talk Bot API (webhook-based) is relatively new compared to Slack's Bolt SDK or Discord's bot framework: + +- API surface is smaller — fewer interactive features +- Documentation is less comprehensive than Slack/Discord +- Community ecosystem of bots is smaller +- API may change between major Talk versions + +**Mitigation**: Pin Nextcloud and Talk versions, test upgrades in staging. + +### No Rich Interactive Components + +Talk does not support interactive UI elements in bot messages: + +- No inline buttons or action menus +- No modals or dialogs +- No dropdown selects or form inputs +- Text, markdown, reactions, and file attachments only + +Bot interaction is text-based. Use slash-command patterns or prefix commands for structured input. + +### E2E Encryption Scope + +- E2E encryption is available for **1:1 video and audio calls** (WebRTC SRTP/DTLS) +- **Group chats and text messages are NOT end-to-end encrypted** — they rely on server-side encryption at rest +- This means the server admin (you) can read all text messages in the database +- For most self-hosted deployments this is acceptable — you trust your own server + +### Performance Depends on Your Hardware + +Unlike SaaS platforms with global CDN infrastructure, Nextcloud Talk performance depends on your server: + +- Video call quality depends on server bandwidth and TURN server configuration +- Message delivery latency depends on server load and database performance +- File sharing speed depends on storage backend (local disk, S3, NFS) + +**Mitigation**: Use a dedicated TURN server (coturn), Redis for caching, and adequate hardware. + +### Mobile Push Notification Setup + +Push notifications require either: + +- Nextcloud's push proxy (minimal metadata to FCM/APNs) +- Self-hosted `notify_push` binary (eliminates all third-party notification traffic) + +Both require additional configuration beyond the base Nextcloud install. + +### Smaller Ecosystem + +Compared to Slack (2000+ apps in marketplace) or Discord (millions of bots), Nextcloud Talk has a smaller bot and integration ecosystem. Most integrations need to be built custom using the webhook API or OCS REST API. + +## Related + +- `services/communications/matrix-bot.md` — Matrix bot integration (federated, E2E encrypted, self-hostable) +- `services/communications/slack.md` — Slack bot integration (proprietary, no E2E, comprehensive API) +- `services/communications/simplex.md` — SimpleX Chat (zero-identifier messaging, strongest metadata privacy) +- `services/communications/signal.md` — Signal bot integration (E2E encrypted, phone number required) +- `services/communications/matterbridge.md` — Multi-platform chat bridging +- `scripts/entity-helper.sh` — Entity memory system (identity resolution, Layer 0/1/2) +- `scripts/runner-helper.sh` — Runner management +- `tools/security/opsec.md` — Operational security guidance +- `tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- `services/hosting/cloudron.md` — Cloudron platform for simplified Nextcloud hosting +- Nextcloud Talk docs: https://nextcloud-talk.readthedocs.io/ +- Nextcloud Talk Bot API: https://docs.nextcloud.com/server/latest/developer_manual/digging_deeper/bots.html +- Nextcloud server admin: https://docs.nextcloud.com/server/latest/admin_manual/ +- Nextcloud Talk source: https://github.com/nextcloud/spreed +- Nextcloud server source: https://github.com/nextcloud/server diff --git a/.agents/services/communications/nostr.md b/.agents/services/communications/nostr.md new file mode 100644 index 000000000..0c88df29a --- /dev/null +++ b/.agents/services/communications/nostr.md @@ -0,0 +1,506 @@ +--- +description: Nostr — decentralized relay-based protocol, bot integration via nostr-tools (TypeScript), NIP-01 events, NIP-04/NIP-44 encrypted DMs, keypair identity, censorship-resistant messaging, and limitations +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Nostr Bot Integration + + + +## Quick Reference + +- **Type**: Decentralized protocol — censorship-resistant, relay-based +- **License**: Protocol is open (no license needed), nostr-tools (MIT) +- **Bot tool**: nostr-tools (TypeScript, npm) +- **Protocol**: Nostr (Notes and Other Stuff Transmitted by Relays) — NIP-01 events over WebSocket +- **Encryption**: NIP-04 DMs (secp256k1 ECDH + AES-256-CBC), NIP-44 (improved, versioned encryption) +- **Identity**: Keypair-based (nsec/npub) — no phone, no email, no server account +- **NIPs**: https://github.com/nostr-protocol/nips (protocol specification) +- **nostr-tools**: https://github.com/nbd-wtf/nostr-tools +- **Relay list**: https://nostr.watch/ + +**Key differentiator**: Nostr requires no account creation, no phone number, no email, no server registration. Identity is a cryptographic keypair. Anyone can run a relay. No single entity can censor or deplatform a user. This makes it the strongest option for censorship-resistant, pseudonymous communication. + +**When to use Nostr over other protocols**: + +| Criterion | Nostr | SimpleX | Matrix | +|-----------|-------|---------|--------| +| Identity | Keypair (nsec/npub) | None (pairwise) | `@user:server` | +| Censorship resistance | Strongest (multi-relay) | Strong (no central server) | Moderate (federated) | +| Metadata privacy | Weak (relay sees pubkeys) | Strongest (no IDs) | Moderate | +| PII required | None | None | Optional but common | +| Bot ecosystem | Growing (nostr-tools) | Growing (WebSocket API) | Mature (SDK, bridges) | +| Message persistence | Relay-dependent | None (ephemeral) | Full history | +| Best for | Censorship resistance, pseudonymous comms | Maximum privacy, agent-to-agent | Team collaboration, bridges | + + + +## Architecture + +```text +┌──────────────────────────┐ +│ Nostr Clients │ +│ (Damus, Amethyst, Primal, │ +│ Snort, Coracle, etc.) │ +└──────────┬───────────────┘ + │ NIP-01 Events (JSON over WebSocket) + │ Signed with secp256k1 keypair + │ +┌──────────▼───────────────┐ ┌──────────────────────────┐ +│ Nostr Relays │ │ Additional Relays │ +│ (wss://relay.damus.io, │ │ (redundancy, reach) │ +│ wss://nos.lol, etc.) │ │ Self-hosted relay option │ +└──────────┬───────────────┘ └──────────────────────────┘ + │ + │ Bot subscribes to events + │ (filters: DMs to bot pubkey, + │ mentions, specific kinds) + │ +┌──────────▼───────────────┐ +│ Bot Process │ +│ (TypeScript/Bun) │ +│ │ +│ ├─ Relay pool manager │ +│ ├─ DM decryption (NIP-04/ │ +│ │ NIP-44) │ +│ ├─ Command router │ +│ ├─ Pubkey allowlist │ +│ └─ aidevops dispatch │ +└───────────────────────────┘ +``` + +**Message flow**: User encrypts DM (NIP-04/NIP-44) → signs event (secp256k1) → publishes to relays → bot subscribes with filter for kind 4 events to its pubkey → decrypts → processes → publishes encrypted response. + +## Installation + +### nostr-tools Setup (TypeScript/Bun) + +```bash +mkdir nostr-bot && cd nostr-bot +bun init -y +bun add nostr-tools +``` + +### Key Generation + +```typescript +import { generateSecretKey, getPublicKey, nip19 } from "nostr-tools" + +const sk = generateSecretKey() // Uint8Array (32 bytes) +const pk = getPublicKey(sk) // hex string +const nsec = nip19.nsecEncode(sk) // nsec1... (NEVER log in production) +const npub = nip19.npubEncode(pk) // npub1... +``` + +**Key storage**: Use `gopass` or environment variables — never commit to source control. See `.agents/tools/credentials/gopass.md`. + +```bash +gopass insert aidevops/nostr-bot/nsec # preferred +export NOSTR_BOT_NSEC="nsec1..." # alternative +``` + +### Relay Selection + +| Relay | Notes | +|-------|-------| +| `wss://relay.damus.io` | Popular, reliable | +| `wss://relay.nostr.band` | Good search/discovery | +| `wss://nos.lol` | Community relay | +| `wss://purplepag.es` | Profile/contact list relay | + +**Self-hosted relay** options: `strfry` (C++, high-performance), `nostr-rs-relay` (Rust), `nostream` (TypeScript). + +### Event Subscription + +```typescript +import { SimplePool } from "nostr-tools" + +const pool = new SimplePool() +const relays = ["wss://relay.damus.io", "wss://nos.lol", "wss://relay.nostr.band"] + +const sub = pool.subscribeMany(relays, [{ + kinds: [4], // NIP-04 encrypted DMs + "#p": [botPublicKey], // addressed to bot + since: Math.floor(Date.now() / 1000), // only new events +}], { + onevent(event) { /* decrypt and process */ }, + oneose() { console.log("Listening for new events") }, +}) +``` + +## Bot API Integration + +### NIP-01: Basic Event Structure + +```typescript +interface NostrEvent { + id: string // sha256 of serialized event + pubkey: string // hex public key of creator + created_at: number // Unix timestamp + kind: number // event type + tags: string[][] // metadata tags + content: string // event content (may be encrypted) + sig: string // secp256k1 schnorr signature +} +``` + +### Event Kinds + +| Kind | NIP | Description | +|------|-----|-------------| +| 0 | NIP-01 | User metadata (profile) | +| 1 | NIP-01 | Text note (public post) | +| 3 | NIP-02 | Contact list / follow list | +| 4 | NIP-04 | Encrypted direct message | +| 5 | NIP-09 | Event deletion request | +| 7 | NIP-25 | Reaction (like, emoji) | +| 1059 | NIP-17 | Gift-wrapped event (metadata-hiding DM) | +| 10002 | NIP-65 | Relay list metadata | +| 30023 | NIP-23 | Long-form content | + +### NIP-04: Encrypted Direct Messages + +```typescript +import { nip04, finalizeEvent } from "nostr-tools" + +// Decrypt incoming DM +async function decryptDM(event: NostrEvent, botSk: Uint8Array): Promise { + return nip04.decrypt(botSk, event.pubkey, event.content) +} + +// Send encrypted DM +async function sendDM(recipientPk: string, msg: string, botSk: Uint8Array) { + const ciphertext = await nip04.encrypt(botSk, recipientPk, msg) + return finalizeEvent({ + kind: 4, + created_at: Math.floor(Date.now() / 1000), + tags: [["p", recipientPk]], + content: ciphertext, + }, botSk) +} +``` + +### NIP-44: Versioned Encryption (Recommended) + +NIP-44 improves on NIP-04 with versioned encryption, padding (hides message length), and XChaCha20: + +```typescript +import { nip44 } from "nostr-tools" + +const convKey = nip44.v2.utils.getConversationKey(sk, recipientPubkey) +const ciphertext = nip44.v2.encrypt(message, convKey) +const plaintext = nip44.v2.decrypt(ciphertext, convKey) +``` + +### NIP-17: Gift-Wrapped DMs (Metadata-Hiding) + +NIP-17 wraps DMs in multiple layers to hide metadata from relays. Adoption is still limited but represents the future of Nostr private messaging: + +- **Seal**: Inner event encrypted to recipient, contains actual DM +- **Gift wrap**: Outer event with randomized pubkey and timestamp +- Relays see only the gift wrap — cannot determine sender, recipient, or timing + +### Basic Bot Example + +```typescript +import { SimplePool, finalizeEvent, getPublicKey, nip04, nip19 } from "nostr-tools" + +const RELAYS = ["wss://relay.damus.io", "wss://nos.lol", "wss://relay.nostr.band"] + +// Load bot key from secure storage +const botNsec = process.env.NOSTR_BOT_NSEC +if (!botNsec) throw new Error("NOSTR_BOT_NSEC not set") +const { data: botSecretKey } = nip19.decode(botNsec) as { data: Uint8Array } +const botPublicKey = getPublicKey(botSecretKey) + +// Pubkey allowlist (hex pubkeys of authorized users) +const ALLOWED_PUBKEYS = new Set([ + // "hex_pubkey_1", "hex_pubkey_2", +]) + +const pool = new SimplePool() + +pool.subscribeMany(RELAYS, [{ + kinds: [4], "#p": [botPublicKey], since: Math.floor(Date.now() / 1000), +}], { + async onevent(event) { + if (ALLOWED_PUBKEYS.size > 0 && !ALLOWED_PUBKEYS.has(event.pubkey)) return + + try { + const plaintext = await nip04.decrypt(botSecretKey, event.pubkey, event.content) + const response = await handleCommand(plaintext) + const ciphertext = await nip04.encrypt(botSecretKey, event.pubkey, response) + const reply = finalizeEvent({ + kind: 4, created_at: Math.floor(Date.now() / 1000), + tags: [["p", event.pubkey]], content: ciphertext, + }, botSecretKey) + await Promise.any(pool.publish(RELAYS, reply)) + } catch (err) { + console.error("Error processing DM:", err) + } + }, + oneose() { console.log("Listening for DMs...") }, +}) + +async function handleCommand(text: string): Promise { + const cmd = text.trim().toLowerCase() + if (cmd === "/help") return "/help — This message\n/status — System status\n/ask — Ask AI" + if (cmd === "/status") return "Bot online. Connected to " + RELAYS.length + " relays." + if (cmd.startsWith("/ask ")) return "Processing: " + text.slice(5).trim() + return 'Unknown command. Type "/help" for available commands.' +} + +process.on("SIGINT", () => { pool.close(RELAYS); process.exit(0) }) +``` + +### Relay Management + +```typescript +const pool = new SimplePool() + +// Publish to multiple relays +await Promise.any(pool.publish(relays, signedEvent)) + +// Query events +const events = await pool.querySync(relays, { kinds: [1], authors: [pubkey], limit: 10 }) +const event = await pool.get(relays, { ids: [eventId] }) + +pool.close(relays) +``` + +### Access Control + +Access control is pubkey-based. Maintain an allowlist of authorized hex pubkeys, loaded from config. If the allowlist is empty, the bot is open to all. Check `event.pubkey` against the allowlist before processing any command. + +## Security Considerations + +### Encryption + +NIP-04 DMs use secp256k1 ECDH to derive a shared secret, then AES-256-CBC to encrypt message content. The encrypted content is stored in the event's `content` field. However, the event envelope (sender pubkey, recipient pubkey tag, timestamp, event kind) is **visible to all relays**. + +NIP-44 improves on NIP-04 with versioned encryption, message padding (hides exact length), and XChaCha20 instead of AES-256-CBC. + +NIP-17 (gift-wrapping) hides metadata by wrapping events in layers — inner seal encrypted to recipient, outer gift wrap with random pubkey and randomized timestamp. Relay sees only the gift wrap. + +**Adoption status**: NIP-04 is universally supported. NIP-44 has growing support. NIP-17 is still limited. + +### Metadata Exposure + +**THIS IS THE KEY WEAKNESS OF NOSTR DMs.** + +With NIP-04 (current standard), relay operators can see: + +- **WHO** is messaging **WHOM** (sender and recipient pubkeys) +- **WHEN** messages are sent (timestamps) +- That the event **IS** a DM (kind 4) +- They **CANNOT** see message content (encrypted) + +Pubkeys are pseudonymous but can be linked to real identities if the user publishes identifying info in their profile (kind 0), links their npub publicly, uses a NIP-05 identifier on a known domain, or reuses pubkeys across contexts. + +NIP-17 gift-wrapping addresses this but adoption is limited. Until widely supported, treat Nostr DMs as having **weaker metadata privacy than SimpleX or Signal**. + +### Decentralization + +No single entity controls the network. Relays are independently operated — anyone can run one. Users choose which relays to use. A relay operator can see all events on their relay but not events on other relays. Censorship requires cooperation of ALL relays a user publishes to. + +### No PII Required + +Identity is purely keypair-based. No phone number, no email, no username registration, no server account. No SIM-swapping attacks, no email-based recovery attacks, pseudonymous by default. This is a significant privacy advantage. + +### Push Notifications + +No push notifications in the traditional sense. Clients maintain WebSocket connections or poll periodically. No Google FCM or Apple APNs metadata exposure. Some clients use optional push services with a trust trade-off. + +### AI Training Risk + +The protocol itself has no AI training risk. Public notes (kind 1) are **public by design** — anyone can read and train on them. Encrypted DMs (kind 4) are not accessible to relays. Individual relay operators set their own data policies. + +### Open Source + +Protocol specification is fully open ([NIPs repository](https://github.com/nostr-protocol/nips)). All major clients (Damus, Amethyst, Primal, Snort) and relay implementations (strfry, nostr-rs-relay, nostream) are open-source. nostr-tools is MIT licensed. No proprietary components. + +### Key Management + +- Private key (nsec) security is **critical** — if compromised, attacker can impersonate the user **permanently** +- No key rotation mechanism in the base protocol — a compromised key cannot be revoked +- NIP-46 (Nostr Connect) allows delegated signing without exposing nsec +- Hardware key storage via NIP-46 signers is the gold standard for high-value identities +- **Recommendation**: Generate a dedicated keypair for the bot, separate from any personal identity + +### Relay Trust + +You must trust relay operators not to censor events or log/correlate metadata. Use multiple relays for redundancy. Self-hosting eliminates relay trust entirely. Paid relays offer better reliability but introduce a payment metadata link. + +### Comparison with Other Protocols + +| Aspect | Nostr | SimpleX | Signal | Matrix | +|--------|-------|---------|--------|--------| +| PII required | None | None | Phone number | Optional | +| DM content privacy | Encrypted | Encrypted | Encrypted | Encrypted | +| DM metadata privacy | Weak (pubkeys visible) | Strongest (no IDs) | Good (sealed sender) | Moderate | +| Censorship resistance | Strongest | Strong | Moderate | Moderate | +| Key compromise impact | Permanent impersonation | Per-connection | Account takeover | Account takeover | +| Decentralization | Full (relay-based) | Full (no servers needed) | Centralized | Federated | + +**Summary**: Nostr is best for censorship resistance and pseudonymous public communication. For private messaging, SimpleX offers stronger metadata privacy. For bot integration where censorship resistance matters more than metadata hiding, Nostr is the superior choice. + +## aidevops Integration + +### Helper Script Pattern + +```bash +#!/usr/bin/env bash +# ~/.aidevops/agents/scripts/nostr-dispatch-helper.sh +set -euo pipefail + +nostr_dispatch() { + local sender_pubkey="$1" + local command="$2" + + if ! is_authorized "$sender_pubkey"; then + echo "Unauthorized sender: $sender_pubkey" + return 1 + fi + + case "$command" in + /status) aidevops status; return 0 ;; + /pulse) aidevops pulse; return 0 ;; + /ask\ *) aidevops research "${command#/ask }"; return 0 ;; + *) echo "Unknown command: $command"; return 1 ;; + esac +} + +is_authorized() { + local pubkey="$1" + local config_file="${HOME}/.config/aidevops/nostr-bot.json" + if [[ ! -f "$config_file" ]]; then + echo "No config found: $config_file" + return 1 + fi + if jq -e --arg pk "$pubkey" '.allowed_users[] | select(. == $pk)' "$config_file" > /dev/null 2>&1; then + return 0 + fi + return 1 +} +``` + +### Configuration + +Config path: `~/.config/aidevops/nostr-bot.json` + +```json +{ + "bot_nsec_gopass_path": "aidevops/nostr-bot/nsec", + "relays": [ + "wss://relay.damus.io", + "wss://nos.lol", + "wss://relay.nostr.band" + ], + "allowed_users": [ + "hex_pubkey_of_authorized_user_1", + "hex_pubkey_of_authorized_user_2" + ], + "self_hosted_relay": null, + "log_level": "info" +} +``` + +### Entity Resolution + +Nostr uses hex pubkeys internally and bech32-encoded npub/nsec for display: + +```typescript +import { nip19, nip05 } from "nostr-tools" + +const { data: hexPubkey } = nip19.decode("npub1...") // npub → hex +const npub = nip19.npubEncode(hexPubkey) // hex → npub + +// NIP-05 resolution (user@domain.com → pubkey) +const profile = await nip05.queryProfile("user@domain.com") +// profile.pubkey, profile.relays +``` + +### Runner Dispatch + +The bot dispatches tasks via the standard pattern: receive DM → validate sender pubkey → parse command → dispatch via `nostr-dispatch-helper.sh` → collect result → send encrypted DM response. + +## Matterbridge Integration + +**NO native Matterbridge support for Nostr.** + +Matterbridge does not include a Nostr gateway. Bridging would require a custom gateway implementation. + +### Bridging Feasibility + +| Aspect | Assessment | +|--------|------------| +| Technical feasibility | Moderate — Nostr's event model maps to messages | +| Effort | High — requires custom Matterbridge gateway plugin | +| Public notes (kind 1) | Straightforward to bridge (plain text) | +| DMs (kind 4) | Complex — requires bot to decrypt/re-encrypt | +| Identity mapping | Difficult — Nostr pubkeys don't map to usernames | +| Direction | Nostr→other is easier; other→Nostr requires signing | + +**Alternative**: Bot-level bridging — bot decrypts Nostr DM, re-sends via Matrix/SimpleX, and vice versa. Simpler but less scalable than a proper Matterbridge gateway. + +## Limitations + +### NIP-04 DM Metadata Visibility + +With NIP-04, relay operators can see sender/recipient pubkeys and timestamps for all DMs. Content is encrypted but the social graph is exposed. NIP-17 gift-wrapping addresses this but adoption is limited. + +### No Rich Interactive Elements + +No protocol-level support for inline buttons, keyboards, interactive forms, menus, typing indicators, or read receipts. Bot interaction is text-command-based only. + +### Relay Reliability + +Relays are independently operated with varying uptime, no SLA. Free relays may rate-limit or filter. **Mitigation**: Connect to multiple relays; self-host for critical bots. + +### No Guaranteed Message Delivery + +If all relays are offline or the bot is offline, messages may be lost. No delivery receipts in the base protocol. **Mitigation**: Use `since` filter on reconnect; use relays with good retention policies. + +### Key Management Complexity + +No key rotation — a compromised nsec is permanently compromised. No account recovery. NIP-46 adds complexity but improves security. **Mitigation**: Dedicated bot keypairs in gopass, rotate by creating new identity. + +### Small Ecosystem + +Significantly fewer users than mainstream messengers. Developer tooling less mature. Documentation scattered across NIPs. Rapidly evolving protocol. + +### No Voice or Video Calls + +The protocol does not include voice/video capabilities. Some clients experiment with WebRTC signaling over Nostr events but there is no standardized NIP. + +### Client Compatibility + +Different clients support different NIP subsets. A NIP-44 DM may not be readable by NIP-04-only clients. Bot should handle both for maximum compatibility. NIP-17 support is very limited. + +### Spam + +Public relays are susceptible to spam. No built-in protocol-level filtering. Relays implement their own anti-spam (proof of work, payment, allowlists). Bots should implement pubkey allowlisting. + +## Related + +- `.agents/services/communications/simplex.md` — SimpleX Chat (strongest metadata privacy) +- `.agents/services/communications/matrix-bot.md` — Matrix messaging (federated, mature ecosystem) +- `.agents/services/communications/bitchat.md` — BitChat (Bitcoin-native messaging) +- `.agents/services/communications/xmtp.md` — XMTP (Ethereum-native messaging) +- `.agents/services/communications/matterbridge.md` — Matterbridge (cross-platform bridging) +- `.agents/tools/security/opsec.md` — Operational security guidance +- `.agents/tools/credentials/gopass.md` — Secret management (for nsec storage) +- `.agents/tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- Nostr NIPs: https://github.com/nostr-protocol/nips +- nostr-tools: https://github.com/nbd-wtf/nostr-tools +- Nostr relay list: https://nostr.watch/ +- Awesome Nostr: https://github.com/aljazceru/awesome-nostr diff --git a/.agents/services/communications/signal.md b/.agents/services/communications/signal.md new file mode 100644 index 000000000..0c73eb22e --- /dev/null +++ b/.agents/services/communications/signal.md @@ -0,0 +1,608 @@ +--- +description: Signal — E2E encrypted messaging gold standard, signal-cli bot integration (JSON-RPC), registration, daemon mode, group messaging, security model, matterbridge bridging, and limitations +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Signal Bot Integration + + + +## Quick Reference + +- **Type**: E2E encrypted messaging — gold standard for privacy +- **License**: AGPL-3.0 (client + server) +- **Bot tool**: signal-cli (Java/native, GPLv3) — NOT an official bot API +- **Protocol**: Signal Protocol (Double Ratchet + X3DH) +- **Encryption**: E2E by default for ALL messages (Curve25519, AES-256, HMAC-SHA256) +- **Registration**: Requires phone number (E.164 format, e.g., `+14155551234`) +- **Daemon mode**: `signal-cli -a +NUMBER daemon --http --receive-mode=on-connection` +- **API**: JSON-RPC over HTTP + Server-Sent Events for incoming messages +- **Data**: `~/.local/share/signal-cli/` (account data, keys, message store) +- **Docs**: https://github.com/AsamK/signal-cli | https://signal.org/docs/ +- **User base**: 40M+ monthly active users (largest E2E encrypted messenger) + +**Key differentiator**: Signal is the gold standard for mainstream encrypted messaging. E2E encryption is on by default for ALL messages — no opt-in required. The Signal Protocol is the most widely adopted secure messaging protocol, also used by WhatsApp, Google Messages (RCS), and Facebook Messenger. + +**When to use Signal over SimpleX**: + +| Criterion | Signal | SimpleX | +|-----------|--------|---------| +| User identifiers | Phone number | None | +| E2E encryption | Default, all messages | Default, all messages | +| Server metadata | Minimal (sealed sender) | Stateless (memory only) | +| User base | 40M+ MAU | Growing niche | +| Bot ecosystem | signal-cli (unofficial) | WebSocket API (official) | +| Group scalability | Production-grade (1000+) | Experimental (1000+) | +| Best for | Privacy-conscious mainstream users | Maximum privacy, zero identifiers | + + + +## Architecture + +```text +┌──────────────────────┐ +│ Signal Mobile/ │ +│ Desktop App │ +│ (iOS, Android, │ +│ Linux, macOS, Win) │ +└──────────┬────────────┘ + │ Signal Protocol (E2E encrypted) + │ Double Ratchet + X3DH + │ +┌──────────▼────────────┐ +│ Signal Servers │ +│ (sealed sender, │ +│ minimal metadata, │ +│ message queuing) │ +└──────────┬────────────┘ + │ +┌──────────▼────────────┐ +│ signal-cli Daemon │ +│ (JSON-RPC over HTTP │ +│ + SSE for events) │ +└──────────┬────────────┘ + │ JSON-RPC / SSE +┌──────────▼────────────┐ +│ Bot Process │ +│ ├─ Command router │ +│ ├─ Message handler │ +│ ├─ Group handler │ +│ └─ aidevops dispatch │ +└─────────────────────────┘ +``` + +**Message flow**: + +1. Sender's app encrypts message with Signal Protocol (Double Ratchet + X3DH, Curve25519) +2. Message encrypted with AES-256-CBC, authenticated with HMAC-SHA256 +3. Sealed sender envelope hides sender identity from Signal servers +4. Signal server queues encrypted message for recipient +5. Recipient's app (or signal-cli) retrieves and decrypts +6. Server deletes message after delivery confirmation + +## Installation + +### signal-cli (Native Binary — Recommended) + +```bash +# Download latest native binary (no Java required) +# Check https://github.com/AsamK/signal-cli/releases for latest version +SIGNAL_CLI_VERSION="0.13.12" +curl -fsSLo signal-cli.tar.gz \ + "https://github.com/AsamK/signal-cli/releases/download/v${SIGNAL_CLI_VERSION}/signal-cli-${SIGNAL_CLI_VERSION}-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m).tar.gz" + +tar xf signal-cli.tar.gz +sudo mv signal-cli-${SIGNAL_CLI_VERSION}/bin/signal-cli /usr/local/bin/ +sudo mv signal-cli-${SIGNAL_CLI_VERSION}/lib/ /usr/local/lib/signal-cli/ +signal-cli --version +``` + +### signal-cli (Java) + +```bash +# Requires Java 21+ +SIGNAL_CLI_VERSION="0.13.12" +curl -fsSLo signal-cli.tar.gz \ + "https://github.com/AsamK/signal-cli/releases/download/v${SIGNAL_CLI_VERSION}/signal-cli-${SIGNAL_CLI_VERSION}.tar.gz" + +tar xf signal-cli.tar.gz +sudo mv signal-cli-${SIGNAL_CLI_VERSION} /opt/signal-cli +sudo ln -sf /opt/signal-cli/bin/signal-cli /usr/local/bin/signal-cli +signal-cli --version +``` + +### macOS / Docker + +```bash +# macOS +brew install signal-cli + +# Docker +docker run --rm -v ~/.local/share/signal-cli:/home/.local/share/signal-cli \ + registry.gitlab.com/packaging/signal-cli/signal-cli-native:latest --version +``` + +## Registration + +Signal requires a phone number for registration. Two methods are available. + +### Method 1: Link to Existing Account (Recommended for Bots) + +Link signal-cli as a secondary device to an existing Signal account. Avoids needing a separate phone number. + +```bash +# Generate linking URI — displays a QR code or URI +signal-cli link -n "aidevops-bot" + +# Scan the QR code from your Signal app: +# Signal app > Settings > Linked Devices > Link New Device +# Credentials stored in ~/.local/share/signal-cli/data/ +``` + +### Method 2: Register New Number (SMS/Voice Verification) + +```bash +# Step 1: Request verification (may need CAPTCHA from https://signalcaptchas.org/registration/generate.html) +signal-cli -a +14155551234 register --captcha "CAPTCHA_TOKEN" + +# Step 2: Verify with SMS code +signal-cli -a +14155551234 verify 123456 + +# Step 3: Set profile name (required) +signal-cli -a +14155551234 updateProfile --given-name "AI Bot" --family-name "DevOps" +``` + +### Verify Installation + +```bash +signal-cli -a +14155551234 listAccounts +signal-cli -a +14155551234 send -m "Hello from signal-cli" +14155559876 +``` + +## Daemon Mode (JSON-RPC API) + +For bot integration, run signal-cli as a persistent daemon with HTTP JSON-RPC and Server-Sent Events. + +```bash +# Basic daemon (default port 8080) +signal-cli -a +14155551234 daemon --http --receive-mode=on-connection + +# Custom port +signal-cli -a +14155551234 daemon --http=localhost:7583 --receive-mode=on-connection + +# Unix socket for local-only access +signal-cli -a +14155551234 daemon --socket=/tmp/signal-cli.socket +``` + +### Receive Modes + +| Mode | Description | Use case | +|------|-------------|----------| +| `on-connection` | Fetch messages when client connects via SSE | Real-time bots | +| `manual` | Only receive when explicitly requested | Polling-based bots | + +### systemd Service + +```ini +# /etc/systemd/system/signal-cli.service +[Unit] +Description=signal-cli daemon +After=network.target + +[Service] +Type=simple +User=signal-bot +ExecStart=/usr/local/bin/signal-cli -a +14155551234 daemon --http=localhost:7583 --receive-mode=on-connection +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +``` + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now signal-cli.service +``` + +## Bot API Integration + +### JSON-RPC Commands + +signal-cli daemon exposes a JSON-RPC 2.0 API over HTTP. + +```bash +# Send text message to individual +curl -s -X POST http://localhost:7583/api/v1/rpc \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","method":"send","params":{"recipient":["+14155559876"],"message":"Hello!"},"id":1}' + +# Send to group +curl -s -X POST http://localhost:7583/api/v1/rpc \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","method":"send","params":{"groupId":"BASE64_GROUP_ID","message":"Hello group!"},"id":2}' + +# Send with attachment +curl -s -X POST http://localhost:7583/api/v1/rpc \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","method":"send","params":{"recipient":["+14155559876"],"message":"Report","attachments":["/path/to/report.pdf"]},"id":3}' +``` + +### Receiving Messages (Server-Sent Events) + +```bash +curl -N http://localhost:7583/api/v1/events +``` + +SSE events are JSON objects: + +```json +{ + "envelope": { + "source": "+14155559876", + "sourceUuid": "a1b2c3d4-e5f6-...", + "sourceName": "Alice", + "timestamp": 1700000000000, + "dataMessage": { + "message": "/status", + "timestamp": 1700000000000, + "groupInfo": null, + "attachments": [], + "mentions": [], + "quote": null, + "reaction": null + } + } +} +``` + +### Key JSON-RPC Methods + +| Method | Description | +|--------|-------------| +| `send` | Send message (text, attachments, quotes, reactions) | +| `sendReaction` | Send emoji reaction (`emoji`, `targetAuthor`, `targetTimestamp`) | +| `sendReceipt` | Send read/delivery receipt (`type`, `targetTimestamps`) | +| `sendTyping` | Send typing indicator | +| `listGroups` | List all groups | +| `listContacts` | List all contacts | +| `getContactName` | Get contact profile name | +| `updateGroup` | Modify group settings | +| `quitGroup` | Leave a group | +| `joinGroup` | Join via group invite link | +| `updateProfile` | Update bot's profile name/avatar | +| `getUserStatus` | Check if number is registered on Signal | + +### Basic Bot Implementation (Bun/TypeScript) + +```typescript +// signal-bot.ts — minimal Signal bot using signal-cli JSON-RPC + SSE +const SIGNAL_CLI_URL = "http://localhost:7583" + +// Allowed users (E.164 phone numbers) +const ALLOWED_USERS = new Set(["+14155559876", "+14155559877"]) + +// JSON-RPC helper +async function rpc(method: string, params: Record = {}): Promise { + const resp = await fetch(`${SIGNAL_CLI_URL}/api/v1/rpc`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", method, params, id: Date.now() }), + }) + const json = await resp.json() + if (json.error) throw new Error(`RPC error: ${json.error.message}`) + return json.result +} + +// Send message to individual or group +async function sendMessage(recipient: string, message: string, groupId?: string): Promise { + const params: Record = { message } + if (groupId) { params.groupId = groupId } else { params.recipient = [recipient] } + await rpc("send", params) +} + +// SSE event listener — main loop +async function listen(): Promise { + const response = await fetch(`${SIGNAL_CLI_URL}/api/v1/events`) + if (!response.body) throw new Error("No response body from SSE endpoint") + + const reader = response.body.getReader() + const decoder = new TextDecoder() + let buffer = "" + + while (true) { + const { done, value } = await reader.read() + if (done) break + + buffer += decoder.decode(value, { stream: true }) + const lines = buffer.split("\n") + buffer = lines.pop() || "" + + for (const line of lines) { + if (!line.startsWith("data:")) continue + const data = line.slice(5).trim() + if (!data) continue + + try { + const event = JSON.parse(data) + const env = event.envelope + if (!env?.dataMessage?.message) continue + + const source = env.source + const groupId = env.dataMessage.groupInfo?.groupId + const text = env.dataMessage.message.trim() + + // Access control + if (!ALLOWED_USERS.has(source)) { + if (!groupId) await sendMessage(source, "Unauthorized.") + continue + } + + // Command routing + if (text.startsWith("/")) { + const [cmd] = text.slice(1).split(/\s+/) + switch (cmd.toLowerCase()) { + case "help": + await sendMessage(source, "/help /status /ping /whoami", groupId) + break + case "status": + await sendMessage(source, `Online. Uptime: ${process.uptime().toFixed(0)}s`, groupId) + break + case "ping": + await sendMessage(source, "pong", groupId) + break + default: + await sendMessage(source, `Unknown: /${cmd}`, groupId) + } + } + } catch (err) { + console.error("SSE parse error:", err) + } + } + } +} + +listen().catch((err) => { console.error("Bot crashed:", err); process.exit(1) }) +``` + +## Security Considerations + +> **CRITICAL**: Signal is the gold standard for mainstream encrypted messaging. This section details the full security model — essential reading for any integration. + +### Encryption + +Signal Protocol is the most widely deployed E2E encryption protocol in the world. + +- **Key agreement**: Extended Triple Diffie-Hellman (X3DH) with Curve25519 +- **Ratchet**: Double Ratchet algorithm — forward secrecy AND future secrecy (post-compromise security) +- **Message encryption**: AES-256-CBC +- **Message authentication**: HMAC-SHA256 +- **Key derivation**: HKDF +- **E2E by DEFAULT**: Every message — text, voice, video, file — is E2E encrypted. No opt-in, no "secret chat" mode. This is the single most important security property. + +### Metadata Protection + +- **Sealed sender**: Hides sender identity from Signal servers. Server knows the recipient but not the sender. +- **Server stores ONLY**: Phone number (hashed), registration date, last connection date. +- **Server does NOT store**: Message content, contacts, groups, profile info, message timestamps, delivery metadata. +- **Proven in court**: Multiple grand jury subpoenas (2016, 2021) — Signal could only provide: phone number, registration date, last connection date. Nothing else exists on their servers. + +### Push Notifications + +- Uses FCM (Android) and APNs (iOS) — push contains **no message content**, only a "new message available" signal +- Actual message fetched E2E encrypted directly from Signal servers +- Minimal metadata exposure — push service knows "a message arrived" but not content, sender, or type + +### AI Training and Data Monetization + +- Signal Foundation is a **501(c)(3) non-profit** +- Explicitly does **NOT** use any user data for AI training +- **No ads**, no tracking, no data monetization — ever +- Structurally enforced by non-profit charter, not just a policy choice + +### Open Source and Auditing + +- **Client + Server**: Open source, AGPL-3.0 (Signal-Android, Signal-iOS, Signal-Desktop, Signal-Server) +- **Regular independent security audits** by Trail of Bits, NCC Group, Cure53 +- **Reproducible builds** for Android — published APK verifiable against source +- Signal Protocol formally verified by academic researchers + +### Jurisdiction + +- **Signal Foundation**, Mountain View, California, USA +- Subject to US law (FISA, NSLs) — but minimal data to hand over makes legal pressure largely ineffective +- 2021 grand jury subpoena response is public record + +### Bot-Specific Security + +- signal-cli is unofficial but well-maintained (GPLv3, active development) +- Bot messages are **still E2E encrypted** — signal-cli implements the full Signal Protocol +- Bot's decryption keys stored locally in `~/.local/share/signal-cli/` — **secure the host machine** +- HTTP API has **no built-in authentication** — bind to localhost only, or use a reverse proxy with auth +- Run signal-cli under a dedicated system user with minimal privileges + +### Phone Number Requirement + +- Signal requires a phone number — this is the **primary privacy weakness** +- Phone numbers are personally identifiable and linkable to real identities +- Signal developing **usernames** to reduce dependency +- For bots: use a dedicated VoIP or prepaid number not linked to personal identity + +### Comparison with Other Platforms + +| Property | Signal | SimpleX | Matrix | Telegram | WhatsApp | +|----------|--------|---------|--------|----------|----------| +| E2E encryption | Default, all | Default, all | Opt-in (rooms) | Opt-in (secret chats) | Default, all | +| User identifier | Phone number | None | @user:server | Phone/username | Phone number | +| Server metadata | Minimal | None (stateless) | Full history | Full history | Moderate | +| Open source | Client + server | Client + server | Client + server | Client only | Neither | +| Non-profit | Yes (501c3) | Yes | Yes (Foundation) | No (commercial) | No (Meta) | +| AI training | Never | Never | No (Foundation) | Yes (since 2024) | Yes (Meta) | + +**Summary**: Strongest privacy of any **mainstream** messenger. Only SimpleX offers better metadata privacy (no identifiers at all), but Signal has vastly larger user base (~40M+ vs niche) and more mature ecosystem. + +## Integration with aidevops + +### Components + +| Component | File | Purpose | +|-----------|------|---------| +| Subagent doc | `.agents/services/communications/signal.md` | This file | +| Helper script | `.agents/scripts/signal-dispatch-helper.sh` | Signal bot dispatch | +| Config | `~/.config/aidevops/signal-bot.json` | Bot configuration | + +### Configuration + +```json +{ + "account": "+14155551234", + "daemon_url": "http://localhost:7583", + "allowed_users": ["+14155559876", "+14155559877"], + "allowed_groups": ["BASE64_GROUP_ID_1"], + "command_prefix": "/", + "dispatch_enabled": true +} +``` + +### Dispatch Pattern + +```bash +# signal-dispatch-helper.sh pattern +# 1. Receive message via SSE +# 2. Check access control (phone number + UUID allowlist) +# 3. Parse command +# 4. Dispatch to aidevops runner or respond directly + +# Entity resolution: +# - Phone number (+E.164) → user identity +# - Group ID (base64) → channel context +# - UUID → stable user identifier (survives number change) +``` + +### Runner Dispatch + +Signal messages can trigger aidevops task execution: + +```bash +# User sends: /run deploy staging +# Bot parses: command=run, args=["deploy", "staging"] +# Bot dispatches: aidevops runner with task context +# Bot responds: "Task dispatched. Tracking ID: t1234" +# Bot follows up: "Deploy complete. PR #567 merged." +``` + +## Matterbridge Integration + +signal-cli has native support in [Matterbridge](https://github.com/42wim/matterbridge) for bridging Signal to 40+ platforms. + +```text +Signal (via signal-cli) + │ +Matterbridge + │ + ├── Matrix rooms + ├── Telegram groups + ├── Discord channels + ├── Slack workspaces + ├── SimpleX chats + ├── IRC channels + └── 40+ other platforms +``` + +### Matterbridge Configuration + +```toml +# matterbridge.toml — Signal gateway + +[signal.mybot] +Number = "+14155551234" +SignalCLIConfig = "/home/signal-bot/.local/share/signal-cli" + +[gateway.bridge-main] +name = "main-bridge" +enable = true + + [[gateway.bridge-main.inout]] + account = "signal.mybot" + channel = "BASE64_GROUP_ID" + + [[gateway.bridge-main.inout]] + account = "matrix.mybot" + channel = "#devops:matrix.example.com" +``` + +### Key Details + +- Matterbridge uses signal-cli's dbus or JSON-RPC interface +- Bridges both DM and group messages +- Supports text, images, and file attachments +- Signal group to platform channel mapping is 1:1 per gateway +- Requires signal-cli registered and running as daemon + +**Privacy gradient**: Users who need maximum privacy use Signal directly. Users who prefer convenience use bridged platforms. Messages flow between platforms transparently via Matterbridge. + +## Limitations + +### Phone Number Required + +Signal requires a phone number for registration. This is the main privacy limitation — phone numbers are personally identifiable. Signal is developing username support but phone numbers remain mandatory for account creation. + +**Mitigation**: Use a dedicated VoIP or prepaid number for bot accounts. + +### No Official Bot API + +signal-cli is unofficial. Signal does not provide an official bot API or SDK: + +- No guaranteed API stability between signal-cli versions +- Feature parity with official apps may lag +- Breaking changes in Signal Protocol updates may require signal-cli updates + +**Mitigation**: Pin signal-cli version, test updates in staging before production. + +### Java Dependency + +signal-cli requires Java 21+ (Java distribution) or native GraalVM builds (~100MB+). + +**Mitigation**: Use native binary builds where available, or Docker containers. + +### No Rich Interactive Elements + +Signal does not support inline keyboards, buttons, interactive cards, or bot command menus. Bot interaction is limited to plain text, attachments, reactions, and quoted replies. + +### Group Admin Features Limited + +signal-cli has limited group admin capabilities compared to the official app. Advanced features (permissions, disappearing messages timer, group link management) may be unavailable. Group v2 features require signal-cli to be up-to-date. + +### Rate Limiting + +Signal enforces rate limits on message sending. No official documentation on exact limits. Implement backoff and queuing in bot code. Group messages count against rate limits per recipient. + +### Single Account Per Daemon + +Each signal-cli daemon instance handles one Signal account. Multiple bot identities require multiple daemon processes on different ports. + +### Linked Device Limitations + +When signal-cli is linked as a secondary device: depends on primary device being online periodically for key sync; if primary is removed, linked device loses access; message history before linking is unavailable. + +## Related + +- `.agents/services/communications/simplex.md` — SimpleX Chat (zero-identifier messaging, strongest metadata privacy) +- `.agents/services/communications/matrix-bot.md` — Matrix messaging integration (federated, user IDs) +- `.agents/services/communications/matterbridge.md` — Cross-platform message bridging +- `.agents/tools/security/opsec.md` — Operational security guidance +- `.agents/tools/voice/speech-to-speech.md` — Voice note transcription +- `.agents/tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- Signal Protocol specification: https://signal.org/docs/ +- signal-cli repository: https://github.com/AsamK/signal-cli +- Signal Foundation: https://signalfoundation.org/ +- Signal source code: https://github.com/signalapp/ diff --git a/.agents/services/communications/slack.md b/.agents/services/communications/slack.md new file mode 100644 index 000000000..343319714 --- /dev/null +++ b/.agents/services/communications/slack.md @@ -0,0 +1,724 @@ +--- +description: Slack Bot integration — Bolt SDK setup, Socket Mode, slash commands, interactive components, Agents API, security considerations (no E2E, AI training), Matterbridge, and aidevops dispatch +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Slack Bot Integration + + + +## Quick Reference + +- **Type**: Corporate messaging platform — no E2E encryption, workspace admin has full access +- **License**: Proprietary (Salesforce). Bot SDK: `@slack/bolt` (MIT) +- **Bot tool**: Slack Bolt SDK (TypeScript, official) +- **Protocol**: Slack API (HTTP + WebSocket Socket Mode) +- **Encryption**: TLS in transit, AES-256 at rest — NO end-to-end encryption +- **Script**: `slack-dispatch-helper.sh [setup|start|stop|status|map|unmap|mappings|test|logs]` +- **Config**: `~/.config/aidevops/slack-bot.json` (600 permissions) +- **Data**: `~/.aidevops/.agent-workspace/slack-bot/` +- **Docs**: https://api.slack.com/docs | https://slack.dev/bolt-js/ +- **App Management**: https://api.slack.com/apps + +**Quick start**: + +```bash +slack-dispatch-helper.sh setup # Interactive wizard +slack-dispatch-helper.sh map C04ABCDEF general-assistant +slack-dispatch-helper.sh start --daemon +``` + + + +## Architecture + +```text +┌──────────────────────┐ +│ Slack Workspace │ +│ │ +│ User sends message │ +│ or slash command │ +└──────────┬───────────┘ + │ + │ Socket Mode (WebSocket) OR Events API (HTTP POST) + │ (recommended — no public URL) (requires public endpoint) + │ +┌──────────▼───────────┐ ┌──────────────────────┐ +│ Bolt App (Bun/Node) │ │ aidevops Dispatch │ +│ │ │ │ +│ ├─ Event listeners │────▶│ runner-helper.sh │ +│ ├─ Slash commands │ │ → AI session │ +│ ├─ Interactive handler│◀────│ → response │ +│ ├─ Access control │ │ │ +│ └─ Entity resolution │ │ │ +└──────────┬───────────┘ └──────────────────────┘ + │ +┌──────────▼───────────┐ +│ memory.db (shared) │ +│ ├── entities │ Entity profiles +│ ├── entity_channels │ Cross-channel identity +│ ├── interactions │ Layer 0: Immutable log +│ └── conversations │ Layer 1: Context summaries +└───────────────────────┘ +``` + +**Message flow**: + +1. User sends message or slash command in a Slack channel or DM +2. Slack delivers event via Socket Mode (WebSocket) or Events API (HTTP) +3. Bolt app receives event, checks access control (workspace/channel/user allowlists) +4. Bot looks up channel-to-runner mapping +5. Entity resolution: Slack user ID (`U01ABCDEF`) resolved to entity via `entity-helper.sh` +6. Layer 0 logging: user message logged as immutable interaction +7. Context loading: entity profile + conversation summary + recent interactions +8. Bot dispatches entity-aware prompt to runner via `runner-helper.sh` +9. Runner executes via headless dispatch +10. Bot posts response back to Slack channel or thread +11. Bot adds reaction emoji (eyes while processing, checkmark on success, X on failure) + +## Installation + +### Prerequisites + +1. **Slack workspace** with admin or app installation permissions +2. **Node.js >= 18** or **Bun** runtime +3. **Slack App** created at https://api.slack.com/apps + +### Step 1: Create a Slack App + +1. Go to https://api.slack.com/apps and click **Create New App** +2. Choose **From an app manifest** (recommended) or **From scratch** +3. Select the target workspace +4. If using a manifest, paste the YAML below + +### App Manifest + +```yaml +display_information: + name: aidevops Bot + description: AI-powered DevOps assistant + background_color: "#1a1a2e" + +features: + bot_user: + display_name: aidevops + always_online: true + slash_commands: + - command: /ai + description: Send a prompt to the AI assistant + usage_hint: "[prompt]" + should_escape: false + +oauth_config: + scopes: + bot: + - app_mentions:read + - channels:history + - channels:read + - chat:write + - commands + - files:read + - files:write + - groups:history + - groups:read + - im:history + - im:read + - im:write + - reactions:read + - reactions:write + - users:read + +settings: + event_subscriptions: + bot_events: + - app_mention + - message.channels + - message.groups + - message.im + interactivity: + is_enabled: true + org_deploy_enabled: false + socket_mode_enabled: true + token_rotation_enabled: false +``` + +### Step 2: Install and Obtain Tokens + +After creating the app: + +1. **Bot Token** (`xoxb-...`): OAuth & Permissions > Install to Workspace > Copy Bot User OAuth Token +2. **App-Level Token** (`xapp-...`): Basic Information > App-Level Tokens > Generate Token with `connections:write` scope +3. **Signing Secret**: Basic Information > App Credentials > Signing Secret (for HTTP Events API only) + +Store tokens securely: + +```bash +# Via gopass (preferred) +gopass insert aidevops/slack/bot-token # xoxb-... +gopass insert aidevops/slack/app-token # xapp-... +gopass insert aidevops/slack/signing-secret # (if using Events API) + +# Or via credentials.sh fallback +# Added to ~/.config/aidevops/credentials.sh (600 permissions) +``` + +### Step 3: Socket Mode vs Events API + +| Feature | Socket Mode (recommended) | Events API | +|---------|--------------------------|------------| +| Public URL required | No | Yes | +| Connection method | WebSocket | HTTP POST | +| Token needed | App-Level Token (`xapp-`) | Signing Secret | +| Firewall-friendly | Yes (outbound only) | No (inbound HTTP) | +| Latency | Slightly lower | Standard | +| Best for | Development, internal bots | Public apps, high scale | + +**Recommendation**: Use Socket Mode for aidevops bots. No public endpoint needed, simpler setup, works behind firewalls. + +### Step 4: Install Dependencies + +```bash +# Using Bun (preferred) +bun add @slack/bolt + +# Using npm +npm install @slack/bolt +``` + +## Bot API Integration + +### Basic Bolt App + +```typescript +import { App } from "@slack/bolt"; + +const app = new App({ + token: process.env.SLACK_BOT_TOKEN, // xoxb-... + appToken: process.env.SLACK_APP_TOKEN, // xapp-... (Socket Mode) + socketMode: true, +}); + +// Listen for messages mentioning the bot +app.event("app_mention", async ({ event, say }) => { + const prompt = event.text.replace(/<@[A-Z0-9]+>/g, "").trim(); + if (!prompt) { + await say({ text: "Send me a prompt and I'll help!", thread_ts: event.ts }); + return; + } + + // Add processing reaction + await app.client.reactions.add({ + channel: event.channel, + timestamp: event.ts, + name: "eyes", + }); + + try { + // Dispatch to runner (placeholder — integrate with runner-helper.sh) + const response = await dispatchToRunner(prompt, event.user, event.channel); + + await say({ text: response, thread_ts: event.ts }); + + // Replace eyes with checkmark + await app.client.reactions.remove({ + channel: event.channel, + timestamp: event.ts, + name: "eyes", + }); + await app.client.reactions.add({ + channel: event.channel, + timestamp: event.ts, + name: "white_check_mark", + }); + } catch (error) { + await say({ text: `Error: ${error.message}`, thread_ts: event.ts }); + await app.client.reactions.add({ + channel: event.channel, + timestamp: event.ts, + name: "x", + }); + } +}); + +// Listen for DMs +app.event("message", async ({ event, say }) => { + if (event.channel_type !== "im" || event.subtype) return; + + const response = await dispatchToRunner(event.text, event.user, event.channel); + await say({ text: response, thread_ts: event.ts }); +}); + +(async () => { + await app.start(); + console.log("Slack bot running (Socket Mode)"); +})(); +``` + +### Slash Commands + +```typescript +// /ai +app.command("/ai", async ({ command, ack, respond }) => { + await ack(); // Acknowledge within 3 seconds + + const prompt = command.text; + const userId = command.user_id; + const channelId = command.channel_id; + + // Respond ephemerally first (only visible to the user) + await respond({ + response_type: "ephemeral", + text: `Processing: "${prompt}"...`, + }); + + const result = await dispatchToRunner(prompt, userId, channelId); + + // Follow-up with visible response + await respond({ + response_type: "in_channel", + text: result, + }); +}); +``` + +### Interactive Components (Buttons, Selects, Modals) + +```typescript +// Send a message with buttons +await app.client.chat.postMessage({ + channel: channelId, + text: "Choose an action:", + blocks: [ + { + type: "actions", + elements: [ + { + type: "button", + text: { type: "plain_text", text: "Run Tests" }, + action_id: "run_tests", + value: "test_suite_all", + }, + { + type: "button", + text: { type: "plain_text", text: "Deploy" }, + action_id: "deploy", + style: "primary", + value: "deploy_staging", + }, + ], + }, + ], +}); + +// Handle button clicks +app.action("run_tests", async ({ ack, body, respond }) => { + await ack(); + await respond({ text: "Running test suite...", replace_original: false }); + // Dispatch test runner +}); + +app.action("deploy", async ({ ack, body, respond }) => { + await ack(); + await respond({ text: "Starting deployment to staging...", replace_original: false }); + // Dispatch deploy runner +}); +``` + +### Thread Messaging + +```typescript +// Reply in a thread +await app.client.chat.postMessage({ + channel: channelId, + thread_ts: parentMessageTs, // Creates or continues a thread + text: "Here's the analysis...", +}); + +// Broadcast thread reply to channel +await app.client.chat.postMessage({ + channel: channelId, + thread_ts: parentMessageTs, + reply_broadcast: true, + text: "Summary posted to channel from thread.", +}); +``` + +### Reactions and Typing Indicators + +```typescript +// Add/remove reactions +await app.client.reactions.add({ channel, timestamp, name: "hourglass_flowing_sand" }); +await app.client.reactions.remove({ channel, timestamp, name: "hourglass_flowing_sand" }); + +// Typing indicator (Slack doesn't have a direct API — but the bot shows +// "typing" automatically while processing Socket Mode events) +``` + +### File Uploads + +```typescript +// Upload a file to a channel +await app.client.files.uploadV2({ + channel_id: channelId, + filename: "report.md", + content: reportContent, + title: "Analysis Report", + initial_comment: "Here's the requested report.", +}); + +// Upload from filesystem +await app.client.files.uploadV2({ + channel_id: channelId, + file: fs.createReadStream("/path/to/file.pdf"), + filename: "document.pdf", +}); +``` + +### Slack Agents and AI Apps API + +Slack provides a native Agents API (beta) for building AI assistants with streaming responses and contextual suggestions. + +```typescript +// Agent-mode: handle assistant thread events +// Requires the assistant scope and Agents API beta access +app.event("assistant_thread_started", async ({ event, say }) => { + await say({ + text: "Hello! I'm the aidevops assistant. How can I help?", + thread_ts: event.assistant_thread.thread_ts, + }); +}); + +app.event("assistant_thread_context_changed", async ({ event }) => { + // User switched channels — update context + console.log(`Context changed to channel ${event.assistant_thread.channel_id}`); +}); +``` + +See: https://api.slack.com/docs/apps/ai + +### Access Control + +```typescript +// Channel allowlist +const ALLOWED_CHANNELS = new Set(["C04ABCDEF", "C04GHIJKL"]); + +// User allowlist (optional — restrict to specific users) +const ALLOWED_USERS = new Set(["U01ADMIN", "U02DEV"]); + +function isAllowed(userId: string, channelId: string): boolean { + // If allowlists are empty, allow all + if (ALLOWED_CHANNELS.size > 0 && !ALLOWED_CHANNELS.has(channelId)) { + return false; + } + if (ALLOWED_USERS.size > 0 && !ALLOWED_USERS.has(userId)) { + return false; + } + return true; +} + +// Apply in event handlers +app.event("app_mention", async ({ event, say }) => { + if (!isAllowed(event.user, event.channel)) { + await say({ text: "Access denied.", thread_ts: event.ts }); + return; + } + // ... dispatch +}); +``` + +## Security Considerations + +**CRITICAL: Read this section carefully before deploying any bot that processes sensitive information via Slack.** + +### Encryption + +Slack provides **TLS 1.2+ in transit** and **AES-256 at rest** on Slack's servers. There is **NO end-to-end encryption**. Slack (Salesforce) has full technical access to ALL message content, including: + +- All channel messages (public and private) +- All direct messages (1:1 and group DMs) +- All file uploads +- All message edits and their history +- All deleted messages (retained server-side) + +### Workspace Admin Access + +Workspace owners and admins have broad access to message data: + +| Plan | Admin Export Capability | +|------|----------------------| +| **Free / Pro** | Admins can request data exports of public channels. Compliance exports of private channels and DMs require contacting Slack support and may notify users. | +| **Business+** | Full compliance exports of ALL messages including private channels and DMs. No user notification required. Corporate eDiscovery tools. | +| **Enterprise Grid** | Full compliance exports, DLP integration, audit logs, eDiscovery, legal holds, data residency controls. Designed for regulatory compliance — messages are searchable and exportable by design. | + +### Metadata Collection + +Slack stores comprehensive metadata beyond message content: + +- Full message history with timestamps +- Edit history (all versions retained) +- Deletion logs (deleted messages recoverable by admins) +- File upload and download records +- Reaction and emoji usage +- Read receipts (who read what, when) +- Login times, IP addresses, device information +- Channel membership and join/leave history +- Search queries +- Integration and app usage patterns + +### AI Training and Data Processing + +**CRITICAL WARNING**: Slack's terms of service and privacy policy (updated September 2023) include provisions regarding AI/ML model development: + +- **Global models**: Slack's privacy policy states that customer data including messages, content, and usage data may be used to develop and train machine learning models that improve Slack's services globally. This applies to all plans unless the workspace admin explicitly opts out. +- **Workspace admins must opt out**: The opt-out is not automatic. Admins must contact Slack or adjust settings to exclude their workspace data from global ML training. Many admins are unaware of this default. +- **Slack AI features**: Slack AI (channel summaries, search answers, thread summaries) processes message content to generate responses. These features are enabled per-workspace and process messages server-side. +- **Salesforce Einstein AI**: Salesforce's AI platform can integrate with Slack data for CRM insights, sales intelligence, and workflow automation. Data flows between Slack and Salesforce are governed by Salesforce's data processing agreements. +- **Third-party AI apps**: Apps installed from the Slack Marketplace may access message content within their granted scopes. Each app's data handling is governed by its own privacy policy, not Slack's. + +**Practical impact**: Assume that any message sent in Slack may be used to train AI models unless the workspace admin has explicitly opted out of all data sharing. + +### Push Notifications + +Push notifications are delivered via Google Firebase Cloud Messaging (FCM) for Android and Apple Push Notification Service (APNs) for iOS. By default, notification content includes a message preview, making it visible to Google or Apple during transit. Admins can configure notification content to show only "You have a new message" to reduce exposure. + +### Open Source and Auditability + +- **Slack platform**: Entirely closed source. No independent audit of Slack's data handling, encryption implementation, or access controls is possible. +- **Slack SDKs**: Open source under MIT license (`@slack/bolt`, `@slack/web-api`, etc.). Bot code is auditable; Slack's server-side processing is not. +- **No reproducible builds**: Slack clients (desktop, mobile, web) are closed source. Users cannot verify that the client matches any claimed security properties. + +### Jurisdiction and Legal + +- **Entity**: Salesforce, Inc. — headquartered in San Francisco, California, USA +- **Jurisdiction**: Subject to US federal law, including the CLOUD Act (allows US government to compel disclosure of data stored abroad), FISA Section 702, and National Security Letters +- **EU Data Residency**: Available on Enterprise Grid plans. Data can be stored in specific AWS regions (Frankfurt, etc.). Note: data residency controls where data is stored, not who can access it — Salesforce US personnel may still access EU-resident data under certain conditions +- **Government requests**: Slack publishes a transparency report. Law enforcement requests are processed per Slack's Law Enforcement Guidelines + +### Bot-Specific Security + +- Bot tokens (`xoxb-`) have scoped permissions based on OAuth scopes, but can access all channels the bot is added to +- Slack's permission model is **workspace-centric** — a bot installed in a workspace can potentially access more data than intended if OAuth scopes are broad +- App-level tokens (`xapp-`) have workspace-wide scope for connection management +- **Token rotation**: Slack supports token rotation but it is disabled by default in the app manifest. Enable for production deployments +- **Signing secrets**: When using Events API (HTTP), always verify the `X-Slack-Signature` header to prevent request forgery + +### Comparison with Other Platforms + +| Aspect | Slack | Matrix (self-hosted) | SimpleX | Signal | +|--------|-------|---------------------|---------|--------| +| E2E encryption | No | Yes (Megolm) | Yes (Double ratchet) | Yes (Signal Protocol) | +| Server access to content | Full | None (if E2E on) | None (stateless) | None (sealed sender) | +| Admin message export | Yes (all plans) | Server admin only | N/A (no server storage) | No | +| AI training default | Opt-out required | No | No | No | +| Open source server | No | Yes (Synapse) | Yes (SMP) | Partial | +| User identifiers | Workspace email | `@user:server` | None | Phone number | +| Metadata retention | Comprehensive | Moderate | Minimal | Minimal | +| Self-hostable | No | Yes | Yes | No | +| Jurisdiction | USA (Salesforce) | Self-determined | Self-determined | USA (Signal Foundation) | + +**Summary**: Slack is among the **least private** mainstream messaging platforms. No E2E encryption, full admin export capability, AI training opt-out (not opt-in), comprehensive metadata retention, and closed-source server. **Treat all Slack messages as fully observable by the employer AND Salesforce.** Use Slack for work communication where corporate oversight is expected and acceptable. Never use it for sensitive personal communication, confidential legal matters, or information that should not be accessible to the workspace owner or Salesforce. + +## aidevops Integration + +### slack-dispatch-helper.sh + +The helper script follows the same pattern as `matrix-dispatch-helper.sh`: + +```bash +# Setup wizard — prompts for tokens, workspace, channel mappings +slack-dispatch-helper.sh setup + +# Map Slack channels to runners +slack-dispatch-helper.sh map C04ABCDEF code-reviewer +slack-dispatch-helper.sh map C04GHIJKL seo-analyst + +# List mappings +slack-dispatch-helper.sh mappings + +# Remove a mapping +slack-dispatch-helper.sh unmap C04ABCDEF + +# Start/stop the bot +slack-dispatch-helper.sh start --daemon +slack-dispatch-helper.sh stop +slack-dispatch-helper.sh status + +# Test dispatch +slack-dispatch-helper.sh test code-reviewer "Review src/auth.ts" + +# View logs +slack-dispatch-helper.sh logs +slack-dispatch-helper.sh logs --follow +``` + +### Runner Dispatch + +The bot dispatches to runners via `runner-helper.sh`, which handles: + +- Runner AGENTS.md (personality/instructions) +- Headless session management +- Memory namespace isolation +- Entity-aware context loading +- Run logging + +### Entity Resolution + +When a Slack user sends a message, the bot resolves their Slack user ID to an entity: + +- **Known user**: Match on `entity_channels` table (`channel=slack`, `channel_id=U01ABCDEF`) +- **New user**: Creates entity via `entity-helper.sh create` with Slack user ID linked +- **Cross-channel**: If the same person is linked on other channels (Matrix, SimpleX, email), their full profile is available +- **Profile enrichment**: Slack's `users.info` API provides display name, email (if shared), timezone — used to populate entity profile on first contact + +### Configuration + +`~/.config/aidevops/slack-bot.json` (600 permissions): + +```json +{ + "botToken": "xoxb-...", + "appToken": "xapp-...", + "signingSecret": "", + "socketMode": true, + "allowedChannels": ["C04ABCDEF", "C04GHIJKL"], + "allowedUsers": [], + "defaultRunner": "", + "channelMappings": { + "C04ABCDEF": "code-reviewer", + "C04GHIJKL": "seo-analyst" + }, + "botPrefix": "", + "ignoreOwnMessages": true, + "maxPromptLength": 3000, + "responseTimeout": 600, + "sessionIdleTimeout": 300 +} +``` + +**Note**: `botPrefix` is empty by default because Slack bots are typically invoked via `@mention` or slash commands rather than text prefixes. Set a prefix (e.g., `!ai`) if you want prefix-based triggering in addition to mentions. + +## Matterbridge Integration + +Slack is natively supported by [Matterbridge](https://github.com/42wim/matterbridge) using the Slack Bot API. + +```text +Slack Workspace + │ + │ Slack Bot API (via bot token) + │ +Matterbridge (Go binary) + │ + ├── Matrix rooms + ├── Discord channels + ├── Telegram groups + ├── SimpleX contacts + ├── IRC channels + └── 40+ other platforms +``` + +### Matterbridge Configuration + +Add to `matterbridge.toml`: + +```toml +[slack.myworkspace] +Token = "xoxb-your-bot-token" +## Optional: restrict to specific channels +## Channels are specified in the gateway section below + +## Optional: show join/leave messages +ShowJoinPart = false + +## Optional: use thread replies +UseThread = false +``` + +Gateway configuration: + +```toml +[[gateway]] +name = "dev-bridge" +enable = true + +[[gateway.inout]] +account = "slack.myworkspace" +channel = "dev-general" + +[[gateway.inout]] +account = "matrix.myserver" +channel = "#dev:matrix.example.com" +``` + +**Privacy warning**: Bridging Slack to other platforms means messages from E2E-encrypted platforms (Matrix, SimpleX) will be stored unencrypted on Slack's servers. Users on the encrypted side should be informed that their messages will be visible to Slack, Salesforce, and workspace admins. See `services/communications/matterbridge.md` for full bridging considerations. + +## Limitations + +### No End-to-End Encryption + +Slack does not support E2E encryption. All messages are readable by Slack (Salesforce) and workspace administrators. This is a fundamental platform design choice, not a missing feature — Slack's compliance and eDiscovery capabilities depend on server-side access to message content. + +### AI Training Default-On + +Workspace data may be used for AI/ML model training unless the admin explicitly opts out. This is a policy default, not a technical limitation. Admins must take active steps to disable this. + +### Rate Limits + +| API | Rate Limit | Notes | +|-----|-----------|-------| +| Web API (most methods) | 1 request per second per method per workspace | Burst allowed, then throttled | +| `chat.postMessage` | 1 per second per channel | Higher for Enterprise Grid | +| Events API | Varies by event type | Slack may retry on 5xx | +| Socket Mode | 30,000 events per hour | Per app | +| Files API | 20 per minute | Upload/download combined | + +Bolt SDK handles rate limiting automatically with retries. + +### Free Plan Restrictions + +- **90-day message history**: Messages older than 90 days are hidden (not deleted — become visible on upgrade) +- **10 app integrations**: Maximum 10 third-party apps or custom integrations +- **No compliance exports**: Admin export limited to public channels +- **1:1 huddles only**: No group audio/video +- **5 GB file storage**: Per workspace + +### Socket Mode Requirements + +- Requires an app-level token (`xapp-`) with `connections:write` scope +- Maximum 10 concurrent Socket Mode connections per app +- Connections may be dropped and must be auto-reconnected (Bolt SDK handles this) + +### No Self-Hosting + +Slack is a SaaS-only platform. There is no self-hosted option. All data is stored on Salesforce's infrastructure (AWS). Organizations requiring full data sovereignty must use alternatives (Mattermost, Matrix, Rocket.Chat). + +### Enterprise Grid Complexity + +Enterprise Grid (multi-workspace) adds complexity: + +- Org-level tokens vs workspace-level tokens +- Cross-workspace channel sharing +- Different admin permission levels +- Separate compliance and DLP configurations + +## Related + +- `services/communications/matrix-bot.md` — Matrix bot integration (E2E encrypted, self-hostable) +- `services/communications/simplex.md` — SimpleX Chat (no identifiers, maximum privacy) +- `services/communications/matterbridge.md` — Multi-platform chat bridging +- `scripts/entity-helper.sh` — Entity memory system (identity resolution, Layer 0/1/2) +- `scripts/runner-helper.sh` — Runner management +- `tools/security/opsec.md` — Operational security guidance +- `tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- Slack Bolt SDK: https://slack.dev/bolt-js/ +- Slack API: https://api.slack.com/ +- Slack Agents API: https://api.slack.com/docs/apps/ai +- Slack Privacy Policy: https://slack.com/trust/privacy/privacy-policy diff --git a/.agents/services/communications/telegram.md b/.agents/services/communications/telegram.md new file mode 100644 index 000000000..6c8026afc --- /dev/null +++ b/.agents/services/communications/telegram.md @@ -0,0 +1,732 @@ +--- +description: Telegram Bot Integration — Bot API, grammY SDK (TypeScript/Bun), BotFather setup, long polling vs webhooks, group/DM access control, inline keyboards, forum topics, security model, Matterbridge native support, and aidevops dispatch integration +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# Telegram Bot Integration + + + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Type** | Cloud-based messaging with optional E2E (Secret Chats only — 1:1 mobile) | +| **License** | Client open-source (GPLv2); server proprietary and closed-source | +| **Bot API** | HTTP Bot API (`https://api.telegram.org/bot/`) + grammY SDK | +| **grammY SDK** | TypeScript, MIT license, 4.5k+ stars, `grammy` on npm | +| **Protocol** | MTProto 2.0 | +| **Encryption** | Server-client encryption by default; optional Secret Chats use E2E (MTProto 2.0 with DH key exchange) — Secret Chats NOT available for bots, groups, or channels | +| **Bot setup** | [@BotFather](https://t.me/BotFather) on Telegram | +| **Docs** | https://core.telegram.org/bots/api, https://grammy.dev/ | +| **File limits** | 50 MB download, 20 MB upload via Bot API; 2 GB via local Bot API server | +| **Rate limits** | ~30 messages/second globally, 1 message/second per chat (group: 20/minute) | + +**Key differentiator**: Telegram's Bot API is the most feature-rich bot platform among mainstream messengers — inline keyboards, inline queries, payments, games, stickers, forum topics, reactions, web apps, and more. However, all bot messages are server-side accessible (no E2E for bots). + +**When to use Telegram over other platforms**: + +| Criterion | Telegram | Signal/SimpleX | Slack/Discord | +|-----------|----------|-----------------|---------------| +| Bot ecosystem | Very mature, HTTP API | Growing | Mature | +| E2E encryption | Optional (Secret Chats only, not bots) | Always-on | None | +| User identifiers | Phone number + username | Phone (Signal) / None (SimpleX) | Email | +| Self-hosting | Client only (server proprietary) | Full stack | No | +| Group scalability | 200,000 members | Experimental (1000+) | Limited | +| Best for | Public bots, large communities | Maximum privacy | Team workspaces | + + + +## Architecture + +```text +┌──────────────────────────┐ +│ Telegram Mobile/Desktop │ +│ Apps (iOS, Android, │ +│ macOS, Windows, Linux, │ +│ Web) │ +└──────────┬───────────────┘ + │ MTProto 2.0 (server-client encrypted) + │ +┌──────────▼───────────────┐ +│ Telegram Cloud Servers │ +│ (proprietary, multi-DC, │ +│ messages stored server- │ +│ side in plaintext from │ +│ Telegram's perspective) │ +└──────────┬───────────────┘ + │ HTTPS Bot API + │ https://api.telegram.org/bot/ + │ +┌──────────▼───────────────┐ +│ grammY Bot Process │ +│ (TypeScript/Bun) │ +│ │ +│ ├─ Long polling / Webhook │ +│ ├─ Command router │ +│ ├─ Inline keyboard handler│ +│ ├─ Conversation middleware│ +│ ├─ File handler │ +│ └─ aidevops dispatch │ +└───────────────────────────┘ +``` + +**Message flow**: + +1. User sends message in Telegram app → MTProto 2.0 to Telegram servers +2. Telegram servers store the message and route to recipients +3. If message is directed at a bot (or bot is in a group), Telegram queues an Update +4. Bot retrieves Updates via long polling (`getUpdates`) or receives them via webhook +5. grammY processes the Update, runs middleware chain, executes handlers +6. Bot responds via HTTP POST to Bot API → Telegram servers → user's app + +**Important**: Telegram servers have full access to all message content (except Secret Chats). The Bot API is an HTTP wrapper around Telegram's internal MTProto — the bot never connects directly via MTProto. + +## Installation + +### grammY SDK (TypeScript/Bun) + +```bash +# Using bun (recommended for aidevops) +bun add grammy + +# Using npm +npm install grammy + +# Using deno +# import { Bot } from "https://deno.land/x/grammy/mod.ts"; +``` + +### BotFather Setup + +1. Open Telegram and search for `@BotFather` +2. Send `/newbot` and follow the prompts +3. Choose a name (display name) and username (must end in `bot`) +4. BotFather returns an API token: `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11` +5. Store the token securely (see [Security Considerations](#security-considerations)) + +**BotFather commands**: + +| Command | Description | +|---------|-------------| +| `/newbot` | Create a new bot | +| `/setcommands` | Set bot command menu | +| `/setdescription` | Set bot description (shown in profile) | +| `/setabouttext` | Set "About" text | +| `/setuserpic` | Set bot profile photo | +| `/setinline` | Enable inline mode | +| `/setjoingroups` | Allow/disallow adding to groups | +| `/setprivacy` | Toggle group privacy mode | +| `/mybots` | List and manage your bots | +| `/deletebot` | Delete a bot | +| `/token` | Regenerate API token | + +**Group privacy mode** (important): By default, bots in groups only receive messages that are commands (`/command`) or direct replies to the bot. To receive ALL group messages, disable privacy mode via `/setprivacy` in BotFather. This is required for bots that need to monitor all conversation. + +## Bot API Integration + +### Basic Bot Setup (grammY + Bun) + +```typescript +import { Bot, Context } from "grammy"; + +// Load token from secure storage — NEVER hardcode +const token = process.env.TELEGRAM_BOT_TOKEN; +if (!token) throw new Error("TELEGRAM_BOT_TOKEN not set"); + +const bot = new Bot(token); + +// Command handlers +bot.command("start", (ctx) => + ctx.reply("Welcome! I'm an aidevops bot. Use /help for commands.") +); + +bot.command("help", (ctx) => + ctx.reply( + "Available commands:\n" + + "/status — System status\n" + + "/ask — Ask AI\n" + + "/run — Run a command (admin only)\n" + + "/help — Show this help" + ) +); + +bot.command("status", async (ctx) => { + await ctx.reply("Checking status..."); + // Dispatch to aidevops runner +}); + +// Text message handler (non-command messages) +bot.on("message:text", (ctx) => { + // Only fires if privacy mode is disabled (for groups) + // or in DMs + console.log(`Message from ${ctx.from?.id}: ${ctx.message.text}`); +}); + +// Error handler +bot.catch((err) => { + console.error("Bot error:", err); +}); + +// Start with long polling +bot.start(); +console.log("Bot started (long polling)"); +``` + +Run with: `bun run bot.ts` + +### Long Polling vs Webhooks + +| Aspect | Long Polling | Webhooks | +|--------|-------------|----------| +| Setup | Zero config, works behind NAT | Requires public HTTPS URL | +| Latency | ~300ms (poll interval) | Near-instant | +| Scaling | Single instance only | Multiple instances OK | +| Development | Best for local dev | Best for production | +| Reliability | Handles reconnects automatically | Must handle failures | + +**Long polling** (development / simple deployments): + +```typescript +// Default — grammY handles everything +bot.start(); +``` + +**Webhooks** (production): + +```typescript +import { webhookCallback } from "grammy"; + +// With Bun's native HTTP server +Bun.serve({ + port: 8443, + fetch: webhookCallback(bot, "bun"), +}); + +// Set webhook URL (run once) +// await bot.api.setWebhook("https://bot.example.com/webhook"); +``` + +Webhook ports allowed by Telegram: 443, 80, 88, 8443. + +### Inline Keyboards + +```typescript +import { InlineKeyboard } from "grammy"; + +bot.command("menu", async (ctx) => { + const keyboard = new InlineKeyboard() + .text("System Status", "cb:status") + .text("Run Task", "cb:run") + .row() + .text("View Logs", "cb:logs") + .url("Documentation", "https://example.com/docs"); + + await ctx.reply("Choose an action:", { reply_markup: keyboard }); +}); + +// Handle callback queries from inline keyboard +bot.callbackQuery("cb:status", async (ctx) => { + await ctx.answerCallbackQuery({ text: "Checking..." }); + await ctx.editMessageText("System is operational."); +}); + +bot.callbackQuery("cb:run", async (ctx) => { + await ctx.answerCallbackQuery(); + await ctx.editMessageText("What would you like to run?"); +}); +``` + +### Key Features + +| Feature | API Method / Description | +|---------|--------------------------| +| **Commands** | `/command` — auto-suggested in command menu | +| **Inline keyboards** | Buttons attached to messages (callback queries) | +| **Reply keyboards** | Custom keyboard replacing the default one | +| **Inline mode** | Bot results in any chat via `@botname query` | +| **Stickers** | Send/receive sticker packs | +| **Voice messages** | Send/receive OGG Opus audio | +| **Files** | Documents, photos, videos, audio (see limits) | +| **Forum topics** | Thread-based discussions in supergroups | +| **Reactions** | Message reactions (emoji or custom) | +| **Payments** | Built-in payment processing | +| **Web Apps** | Mini apps embedded in Telegram UI | +| **Polls** | Create and manage polls | +| **Location** | Share and receive live locations | +| **Dice** | Animated randomized emoji (dice, darts, etc.) | + +### Forum Topics + +Telegram supergroups can enable "Topics" — thread-based discussions. Bots can create, manage, and post to specific topics: + +```typescript +// Send to a specific forum topic +await bot.api.sendMessage(chatId, "Message in topic", { + message_thread_id: topicId, +}); + +// Create a new topic +const topic = await bot.api.createForumTopic(chatId, "New Topic", { + icon_color: 0x6FB9F0, +}); +``` + +### DM vs Group Access Control + +```typescript +import { Context } from "grammy"; + +// Check if message is from a private chat (DM) +function isDM(ctx: Context): boolean { + return ctx.chat?.type === "private"; +} + +// Check if message is from a group +function isGroup(ctx: Context): boolean { + return ctx.chat?.type === "group" || ctx.chat?.type === "supergroup"; +} + +// Admin-only middleware +async function adminOnly(ctx: Context, next: () => Promise) { + const adminIds = (process.env.TELEGRAM_ADMIN_IDS ?? "").split(",").map(Number); + if (!ctx.from || !adminIds.includes(ctx.from.id)) { + await ctx.reply("Unauthorized. This command requires admin access."); + return; + } + await next(); +} + +// Apply to specific commands +bot.command("run", adminOnly, async (ctx) => { + // Only admins reach here + const command = ctx.match; // text after /run + await ctx.reply(`Running: ${command}`); +}); + +// DM-only commands +bot.command("config", async (ctx) => { + if (!isDM(ctx)) { + await ctx.reply("This command is only available in DMs."); + return; + } + // Show config options +}); +``` + +### Conversations (Multi-Step Interactions) + +grammY provides the `conversations` plugin for multi-step interactions: + +```bash +bun add @grammyjs/conversations +``` + +```typescript +import { conversations, createConversation } from "@grammyjs/conversations"; +import type { Conversation, ConversationFlavor } from "@grammyjs/conversations"; + +type BotContext = Context & ConversationFlavor; + +async function askQuestion(conversation: Conversation, ctx: BotContext) { + await ctx.reply("What would you like to ask?"); + const response = await conversation.wait(); + const question = response.message?.text; + if (!question) { + await ctx.reply("Please send a text message."); + return; + } + await ctx.reply(`Processing: "${question}"...`); + // Dispatch to AI +} + +bot.use(conversations()); +bot.use(createConversation(askQuestion)); +bot.command("ask", async (ctx) => { + await ctx.conversation.enter("askQuestion"); +}); +``` + +## Security Considerations + +### Encryption Model + +**Server-client encryption only (by default)**. Telegram uses MTProto 2.0 for transport encryption between the app and Telegram's servers. However, messages are stored on Telegram's servers and are accessible to Telegram in plaintext. + +**Secret Chats** provide E2E encryption using MTProto 2.0 with Diffie-Hellman key exchange, but with critical limitations: + +- Only available for 1:1 chats on mobile apps +- NOT available for bots +- NOT available for groups or channels +- NOT available on Telegram Desktop (except macOS native app) +- NOT available on Telegram Web +- Messages are device-specific (not synced across devices) + +**For bot integrations**: All bot messages are transmitted and stored without E2E encryption. Telegram (the company) can technically read every message a bot sends or receives. + +### Metadata Exposure + +Telegram servers have access to comprehensive metadata: + +- **Who messages whom** — full social graph of all conversations +- **When** — timestamps of all messages +- **Group memberships** — every group a user belongs to +- **IP addresses** — connection IPs (unless using Tor/VPN) +- **Phone numbers** — required for registration, linked to account +- **Device information** — app version, device model, OS +- **Online status** — last seen timestamps (unless hidden by user) +- **Location data** — if shared via live location or nearby features + +### Server Access + +Telegram (the company) has **full access to all non-Secret-Chat messages**. Their privacy policy states they do not use message data for advertising, and they claim not to read messages. However: + +- The server code is **completely proprietary and closed-source** +- No independent security audit of server-side data handling exists +- Telegram employees with server access could technically read messages +- Law enforcement requests could compel data disclosure (see Jurisdiction below) +- Telegram has disclosed user data to authorities in some cases (IP addresses, phone numbers) + +### Push Notifications + +- **Android**: Via Firebase Cloud Messaging (FCM / Google) — notification metadata visible to Google +- **iOS**: Via Apple Push Notification Service (APNs) — notification metadata visible to Apple +- Notification content is encrypted, but the fact that a notification was sent (timing, sender) is visible to Google/Apple + +### AI and Data Processing + +As of 2025, Telegram has introduced several AI-powered features: + +- Message translation +- AI-generated profile photos (Premium) +- AI chatbot features (Telegram Premium) +- Story and media AI enhancements + +It is **unclear what data these features process** and whether chat content is used for AI training. Telegram's privacy policy does not explicitly address AI training on user data. Users should assume that messages processed by AI features may be sent to third-party AI providers. + +### Open Source Status + +- **Client apps**: Open-source under GPLv2 (iOS, Android, Desktop, Web) +- **Server**: Completely **proprietary and closed-source** +- **Bot API server**: Open-source ([tdlib/telegram-bot-api](https://github.com/tdlib/telegram-bot-api)) — can be self-hosted for higher file limits +- **TDLib**: Core client library is open-source (Boost Software License) + +The proprietary server means there is **no way to independently verify** what happens to messages on Telegram's infrastructure. + +### Jurisdiction + +- Telegram FZ-LLC is registered in **Dubai, UAE** +- Previously based in various jurisdictions (Russia, UK, Singapore) +- Has historically resisted government data requests (notably Russia) +- However, legal frameworks are evolving — UAE, EU (DSA), and other jurisdictions may compel data disclosure +- In 2024, Telegram's CEO was detained in France related to platform moderation obligations +- Telegram updated its privacy policy in late 2024 to clarify cooperation with law enforcement + +### Bot-Specific Security + +- **Bots CANNOT use Secret Chats** — all bot communication is server-accessible +- **Bot tokens grant full access** to all messages the bot receives — treat tokens as critical secrets +- **Bot tokens in URLs** — the token is part of every API call URL; ensure HTTPS and log sanitization +- **Group bots** — if privacy mode is disabled, the bot receives ALL messages in the group +- **Inline bots** — can receive queries from any user in any chat +- **Webhook security** — verify the `X-Telegram-Bot-Api-Secret-Token` header; use a secret path +- **File access** — files uploaded to Telegram can be downloaded by anyone with the `file_id` (not truly private) + +### Token Management + +```bash +# Store token via aidevops secret management (gopass) +gopass insert aidevops/telegram/bot-token + +# Or via credentials.sh (600 permissions) +echo 'export TELEGRAM_BOT_TOKEN="123456:ABC..."' >> ~/.config/aidevops/credentials.sh + +# NEVER commit tokens to git +# NEVER log tokens in output +# NEVER pass tokens as CLI arguments (visible in process list) +# Regenerate via @BotFather /token if compromised +``` + +### Comparison with Other Platforms + +| Aspect | Telegram | Signal | SimpleX | Slack/Discord | +|--------|----------|--------|---------|---------------| +| Default E2E | No (server-client only) | Yes | Yes | No | +| Bot E2E | No | N/A | Yes | No | +| Server code | Proprietary | Open-source | Open-source | Proprietary | +| Metadata visible to server | All | Minimal | None | All | +| Phone required | Yes | Yes | No | No (email) | +| Data location | Dubai (UAE) | USA | User-chosen | USA | +| Independent audit | Client only | Full stack | Full stack | No | + +**Summary**: Telegram is **less private** than Signal/SimpleX (no default E2E, proprietary server, full metadata access), but **more private** than Slack/Discord/Teams (at least offers optional E2E for 1:1, client is open-source, no ads-driven data mining). For aidevops bot integrations, assume **all bot messages are readable by Telegram**. + +## aidevops Integration + +### Components + +| Component | File | Purpose | +|-----------|------|---------| +| Subagent doc | `.agents/services/communications/telegram.md` | This file | +| Helper script | `.agents/scripts/telegram-dispatch-helper.sh` | Bot lifecycle management | +| Config | `~/.config/aidevops/telegram-bot.json` | Bot configuration | + +### Helper Script Pattern + +`telegram-dispatch-helper.sh` follows the standard aidevops helper pattern: + +```bash +# Setup — configure bot token and default chat mappings +telegram-dispatch-helper.sh setup + +# Start — launch the bot process (long polling or webhook) +telegram-dispatch-helper.sh start + +# Stop — gracefully stop the bot process +telegram-dispatch-helper.sh stop + +# Status — check if bot is running and healthy +telegram-dispatch-helper.sh status + +# Map — associate a Telegram chat/group with an aidevops entity +telegram-dispatch-helper.sh map +# Example: telegram-dispatch-helper.sh map -1001234567890 project myproject + +# Unmap — remove a chat-entity association +telegram-dispatch-helper.sh unmap +``` + +### Runner Dispatch + +Bot messages can trigger aidevops runner dispatch via `runner-helper.sh`: + +```typescript +// In bot command handler +bot.command("run", adminOnly, async (ctx) => { + const command = ctx.match; + if (!command) { + await ctx.reply("Usage: /run "); + return; + } + + await ctx.reply(`Dispatching: ${command}`); + + // Dispatch via runner-helper.sh + const proc = Bun.spawn( + ["bash", "-c", `runner-helper.sh dispatch "${command}"`], + { stdout: "pipe", stderr: "pipe" } + ); + const output = await new Response(proc.stdout).text(); + await ctx.reply(`Result:\n\`\`\`\n${output.slice(0, 4000)}\n\`\`\``); +}); +``` + +### Entity Resolution + +Use `entity-helper.sh` to resolve Telegram chats to aidevops entities: + +```bash +# Resolve a Telegram chat ID to an aidevops project +entity-helper.sh resolve telegram:-1001234567890 +# Returns: project:myproject + +# Reverse lookup — find Telegram chat for an entity +entity-helper.sh lookup project:myproject telegram +# Returns: -1001234567890 +``` + +### Configuration + +`~/.config/aidevops/telegram-bot.json`: + +```json +{ + "token_source": "gopass:aidevops/telegram/bot-token", + "mode": "polling", + "webhook_url": null, + "webhook_port": 8443, + "admin_ids": [123456789], + "allowed_chats": [-1001234567890], + "entity_mappings": { + "-1001234567890": { + "type": "project", + "id": "myproject", + "commands": ["status", "run", "ask"] + } + }, + "features": { + "inline_mode": false, + "forum_topics": false, + "file_handling": true + } +} +``` + +## Matterbridge Integration + +Telegram has **native support** in Matterbridge — no adapter required (unlike SimpleX which needs a separate bridge process). + +### Configuration + +In `matterbridge.toml`: + +```toml +[telegram] + [telegram.main] + Token="YOUR_BOT_TOKEN" + # Optional: use HTML or Markdown parse mode + # MessageFormat="HTMLNick" + # UseFirstName=false + +[[gateway]] +name="project-bridge" +enable=true + + [[gateway.inout]] + account="telegram.main" + channel="-1001234567890" # Supergroup chat ID (negative number) + + [[gateway.inout]] + account="matrix.home" + channel="#project:example.com" + + # Add other platforms as needed + # [[gateway.inout]] + # account="discord.myserver" + # channel="project" +``` + +### Getting the Chat ID + +1. Add `@userinfobot` or `@getidsbot` to the group +2. It will reply with the group's chat ID +3. For supergroups, the ID is negative and starts with `-100` +4. Alternatively, check the bot's `getUpdates` response after sending a message in the group + +### Bridging Limitations + +| Limitation | Detail | +|------------|--------| +| **Formatting** | Telegram HTML/Markdown may not render identically on other platforms | +| **File size** | Bot API limits: 20 MB upload, 50 MB download — files larger than this won't bridge | +| **Stickers** | Bridged as images (PNG) — animation lost | +| **Reactions** | Not bridged by default in most Matterbridge versions | +| **Threads/Topics** | Forum topics may not map to other platforms' thread models | +| **Edits** | Message edits are bridged as new messages on some platforms | +| **Deletions** | Message deletions may not propagate across all platforms | +| **Voice messages** | Bridged as audio files — no inline playback on some platforms | + +### Bot Requirements for Bridging + +The bot used for Matterbridge must: + +1. Be added to the target group as a member +2. Have **privacy mode disabled** (via BotFather `/setprivacy` → Disabled) to see all messages +3. Have permission to send messages in the group + +## Limitations + +### No E2E Encryption for Bots + +All bot communication is server-client encrypted only. Telegram (the company) can read all bot messages. There is no workaround — the Bot API is an HTTP interface that goes through Telegram's servers. If E2E is required, use SimpleX or Signal instead. + +### Phone Number Required + +Every Telegram account requires a phone number. This creates: + +- A link between real-world identity and Telegram account +- Potential for SIM-swap attacks +- Privacy concerns for users who don't want to share phone numbers +- A barrier to anonymous bot usage + +Users can hide their phone number from other users (Settings > Privacy > Phone Number), but Telegram always has it. + +### Unofficial API Risks + +Using unofficial client libraries that connect via MTProto directly (e.g., Telethon, Pyrogram, GramJS) to automate user accounts: + +- **Violates Telegram ToS** — risk of account ban +- **Session hijacking risk** — MTProto sessions can be stolen +- **Rate limiting** — aggressive automation triggers flood waits +- **Legal risk** — may violate computer fraud laws in some jurisdictions + +Always use the official Bot API for automation. User account automation ("userbots") is unsupported and risky. + +### File Size Limits + +| Operation | Bot API | Local Bot API Server | +|-----------|---------|---------------------| +| Download files | 20 MB | 2 GB | +| Upload files | 50 MB | 2 GB | +| Upload thumbnails | 200 KB | 200 KB | + +The [Local Bot API Server](https://github.com/tdlib/telegram-bot-api) can be self-hosted to raise file limits to 2 GB. It handles file storage locally instead of proxying through Telegram's servers. + +```bash +# Build and run local Bot API server +git clone --recursive https://github.com/tdlib/telegram-bot-api.git +cd telegram-bot-api +mkdir build && cd build +cmake -DCMAKE_BUILD_TYPE=Release .. +cmake --build . --target install + +# Run (requires api_id and api_hash from https://my.telegram.org) +telegram-bot-api --api-id=YOUR_API_ID --api-hash=YOUR_API_HASH --local +``` + +### Rate Limits + +| Scope | Limit | +|-------|-------| +| Global (all chats) | ~30 messages/second | +| Per chat (private) | 1 message/second | +| Per chat (group) | 20 messages/minute | +| Bulk notifications | 30 messages/second to different chats | +| Inline query results | 50 results per query | + +grammY has built-in rate limiting via the `auto-retry` and `transformer-throttler` plugins: + +```bash +bun add @grammyjs/auto-retry @grammyjs/transformer-throttler +``` + +```typescript +import { autoRetry } from "@grammyjs/auto-retry"; +import { apiThrottler } from "@grammyjs/transformer-throttler"; + +bot.api.config.use(autoRetry()); +bot.api.config.use(apiThrottler()); +``` + +### Other Limitations + +- **No message scheduling for bots** — bots cannot schedule messages for future delivery +- **No voice/video calls** — bots cannot initiate or receive calls +- **Group admin limitations** — bots cannot promote members above their own permission level +- **Channel posting** — bots can post to channels but cannot read channel messages unless they are the channel admin +- **Search** — Bot API has no message search capability; bots must maintain their own message index +- **Message history** — bots cannot access messages sent before they were added to a group + +## Related + +- `.agents/services/communications/simplex.md` — SimpleX Chat (maximum privacy, E2E, no identifiers) +- `.agents/services/communications/matrix-bot.md` — Matrix messaging (federated, self-hostable) +- `.agents/services/communications/matterbridge.md` — Multi-platform chat bridge (Telegram native support) +- `.agents/tools/security/opsec.md` — Operational security guidance +- `.agents/tools/voice/speech-to-speech.md` — Voice message transcription +- `.agents/tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- `.agents/tools/credentials/gopass.md` — Secure credential storage +- grammY docs: https://grammy.dev/ +- Telegram Bot API: https://core.telegram.org/bots/api +- Telegram Bot FAQ: https://core.telegram.org/bots/faq +- Local Bot API Server: https://github.com/tdlib/telegram-bot-api +- Matterbridge: https://github.com/42wim/matterbridge diff --git a/.agents/services/communications/urbit.md b/.agents/services/communications/urbit.md index ea26d4a49..017791846 100644 --- a/.agents/services/communications/urbit.md +++ b/.agents/services/communications/urbit.md @@ -1,5 +1,5 @@ --- -description: Urbit — personal server OS with peer-to-peer encrypted messaging (Ames protocol), decentralized identity (Urbit ID / Azimuth), Hoon/Nock programming, HTTP API for external integration, maximum sovereignty +description: Urbit — decentralized personal server OS, bot integration via Eyre HTTP API, Ames P2P encrypted networking, Azimuth identity (Ethereum L2), graph-store messaging, maximum sovereignty, and limitations mode: subagent tools: read: true @@ -12,638 +12,703 @@ tools: task: false --- -# Urbit +# Urbit Bot Integration ## Quick Reference -- **Type**: Personal server OS with built-in P2P encrypted networking — decentralized identity, deterministic computing, sovereign infrastructure -- **License**: MIT (Vere runtime), MIT (Urbit OS / Arvo kernel) -- **Runtime**: Vere (C-based Nock interpreter) — runs a single "ship" (personal server instance) -- **Language**: Hoon (high-level, compiles to Nock) / Nock (minimal combinator VM) -- **Networking**: Ames protocol — E2E encrypted, peer-to-peer, authenticated by Urbit ID -- **Identity**: Azimuth (Ethereum L1 PKI) — hierarchical address space (~galaxy > ~star > ~planet > ~moon > ~comet) -- **HTTP API**: Eyre (HTTP server vane) — SSE for events, PUT/POST for actions (JSON via `mark` system) -- **Apps**: Groups (chat/forums), Notebook, Landscape (web UI), third-party apps via `desk` distribution -- **Repo**: [github.com/urbit/urbit](https://github.com/urbit/urbit) (runtime) | [github.com/urbit/vere](https://github.com/urbit/vere) (Vere) -- **Docs**: [docs.urbit.org](https://docs.urbit.org/) | [developers.urbit.org](https://developers.urbit.org/) -- **Network explorer**: [network.urbit.org](https://network.urbit.org/) - -**Key differentiator**: Urbit is a complete personal server OS, not just a messaging protocol. Each user runs their own deterministic computer ("ship") with a permanent cryptographic identity. All networking is E2E encrypted between ships via the Ames protocol. There are no central servers — your data lives on your ship, and you own your identity as an Ethereum NFT. This provides maximum sovereignty at the cost of requiring your own infrastructure. - -**When to use Urbit vs other protocols**: - -| Criterion | Urbit | SimpleX | Matrix | XMTP | -|-----------|-------|---------|--------|------| -| Identity model | Urbit ID (Azimuth NFT) | None | `@user:server` | Wallet/DID | -| Encryption | Ames (E2E, per-ship keys) | Double ratchet (X3DH) | Megolm (optional) | MLS + post-quantum | -| Server model | Self-hosted personal server | Stateless relays | Federated servers | Decentralized nodes | -| Programming model | Full OS (Hoon/Nock apps) | Bot WebSocket API | Client-server SDK | Agent SDK | -| Bot/API support | HTTP API (Eyre) + custom agents | WebSocket JSON API | `matrix-bot-sdk` | `@xmtp/agent-sdk` | -| Decentralization | Fully decentralized (each ship is sovereign) | Decentralized relays | Federated | Node operators | -| Best for | Maximum sovereignty, long-term personal computing | Maximum privacy | Team collaboration | Web3/agent messaging | +- **Type**: Decentralized personal server OS — maximum sovereignty, peer-to-peer encrypted +- **License**: MIT (Urbit runtime), various open-source licenses for components +- **Bot tool**: Urbit HTTP API (Eyre) — external integration via HTTP/SSE +- **Protocol**: Ames (P2P encrypted networking), Eyre (HTTP API gateway) +- **Encryption**: Ames provides E2E encryption between ships (Curve25519 + AES) +- **Identity**: Urbit ID (Azimuth) — decentralized identity on Ethereum L2 +- **Runtime**: https://github.com/urbit/urbit (Vere runtime) +- **Docs**: https://docs.urbit.org/ | https://developers.urbit.org/ +- **Azimuth**: https://azimuth.network/ + +**Key differentiator**: Urbit is not just a messaging app — it is a personal server operating system. Your ship is your computer on the network. You own your identity (an NFT on Ethereum L2), your data, your applications, and your network connections. No company controls the network. No terms of service. No deplatforming. This makes it the maximum sovereignty option — you own everything. + +**When to use Urbit over other protocols**: + +| Criterion | Urbit | Nostr | SimpleX | Matrix | +|-----------|-------|-------|---------|--------| +| Identity | Azimuth (NFT, owned) | Keypair (nsec/npub) | None (pairwise) | `@user:server` | +| Sovereignty | Maximum (own server) | High (relay-dependent) | High (no servers) | Moderate (federated) | +| Metadata privacy | Strong (P2P, no relays) | Weak (relay sees pubkeys) | Strongest (no IDs) | Moderate | +| PII required | None (NFT identity) | None | None | Optional but common | +| Bot ecosystem | Minimal (HTTP API) | Growing (nostr-tools) | Growing (WebSocket API) | Mature (SDK, bridges) | +| Persistence | Full (ship stores all data) | Relay-dependent | None (ephemeral) | Full history | +| Best for | Maximum sovereignty, personal server | Censorship resistance | Maximum privacy | Team collaboration | ## Architecture ```text -┌─────────────────────────────────────────────────────┐ -│ Urbit Ship (~sampel-palnet) │ -│ │ -│ ┌─────────────────────────────────────────────────┐ │ -│ │ Arvo (Kernel) │ │ -│ │ │ │ -│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ -│ │ │ Ames │ │ Eyre │ │ Gall │ │ │ -│ │ │ (P2P net)│ │ (HTTP) │ │ (apps) │ │ │ -│ │ ├──────────┤ ├──────────┤ ├──────────┤ │ │ -│ │ │ Behn │ │ Clay │ │ Iris │ │ │ -│ │ │ (timers) │ │ (files) │ │ (HTTP │ │ │ -│ │ │ │ │ │ │ client) │ │ │ -│ │ ├──────────┤ ├──────────┤ ├──────────┤ │ │ -│ │ │ Dill │ │ Jael │ │ Khan │ │ │ -│ │ │ (term) │ │ (keys) │ │ (threads)│ │ │ -│ │ └──────────┘ └──────────┘ └──────────┘ │ │ -│ └─────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────┐ │ -│ │ Nock (Virtual Machine) │ │ -│ │ Deterministic combinator calculus │ │ -│ └─────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────┐ │ -│ │ Vere (Runtime / Interpreter) │ │ -│ │ C implementation, manages I/O, event log, state │ │ -│ └─────────────────────────────────────────────────┘ │ -└──────────────────────┬──────────────────────────────┘ - │ - ┌────────────┼────────────┐ - │ │ │ - ┌─────▼─────┐ ┌───▼───┐ ┌─────▼─────┐ - │ Ames │ │ Eyre │ │ Other │ - │ (UDP P2P) │ │ (HTTP │ │ ships │ - │ E2E enc. │ │ API) │ │ (peers) │ - └───────────┘ └───┬───┘ └───────────┘ - │ - ┌──────▼──────┐ - │ External │ - │ Client │ - │ (Bot, Web │ - │ UI, CLI) │ - └─────────────┘ +┌──────────────────────────┐ ┌──────────────────────────┐ +│ Urbit Ship A │ │ Urbit Ship B │ +│ (your personal server) │ │ (another user's ship) │ +│ │ │ │ +│ ├─ Arvo (OS kernel) │ │ ├─ Arvo (OS kernel) │ +│ ├─ Ames (P2P networking) │◄───►│ ├─ Ames (P2P networking) │ +│ ├─ Eyre (HTTP server) │ │ ├─ Eyre (HTTP server) │ +│ ├─ Graph Store (messages) │ │ ├─ Graph Store (messages) │ +│ ├─ Groups (social app) │ │ ├─ Groups (social app) │ +│ └─ Landscape (web UI) │ │ └─ Landscape (web UI) │ +└──────────┬───────────────┘ └──────────────────────────┘ + │ + │ Eyre HTTP API (localhost) + │ Authentication via +code + │ +┌──────────▼───────────────┐ +│ Bot Process │ +│ (TypeScript/Bun) │ +│ │ +│ ├─ HTTP client (auth) │ +│ ├─ SSE subscription │ +│ │ (real-time events) │ +│ ├─ Poke handler │ +│ │ (send actions) │ +│ ├─ Scry reader │ +│ │ (read state) │ +│ ├─ Command router │ +│ └─ aidevops dispatch │ +└───────────────────────────┘ ``` -### Arvo Kernel Vanes +**Message flow**: Ship A sends message via Ames (E2E encrypted, P2P) → Ship B receives and stores in graph-store → External bot connects to Ship B's Eyre HTTP API → subscribes to graph-store updates via SSE → processes incoming messages → pokes graph-store to send responses → Ames delivers response to Ship A. -| Vane | Purpose | -|------|---------| -| **Ames** | Peer-to-peer encrypted networking (UDP-based) | -| **Eyre** | HTTP server — external API for web clients and bots | -| **Gall** | Application framework — stateful agents (apps) | -| **Behn** | Timer system | -| **Clay** | Typed filesystem with revision control | -| **Iris** | HTTP client — outbound requests | -| **Jael** | Key management and Azimuth state | -| **Khan** | Thread execution (one-off computations) | -| **Dill** | Terminal driver | -| **Lick** | IPC for external processes | +**Identity tiers**: -### Message Flow (Ship-to-Ship) +| Tier | Count | Cost | Purpose | +|------|-------|------|---------| +| Galaxy (~gal) | 256 | Very expensive | Infrastructure, governance | +| Star (~star) | ~65,536 | Moderate | Issue planets, relay | +| Planet (~planet) | ~4 billion | $10-50 USD | Personal identity | +| Comet (~comet) | 2^128 | Free | Temporary, limited | -1. Sender's Gall agent produces a message (e.g., chat post in Groups app) -2. Ames encrypts the message using the recipient ship's public key (from Azimuth/Jael) -3. Message sent as UDP packet to recipient's IP (resolved via galaxy/star infrastructure) -4. Recipient's Ames decrypts and delivers to the target Gall agent -5. Acknowledgment sent back through Ames (guaranteed delivery with retry) +## Installation + +### Urbit Runtime -### Message Flow (External Client via HTTP) +```bash +# Linux +curl -L https://urbit.org/install/linux-x86_64/latest -o urbit +chmod +x urbit +sudo mv urbit /usr/local/bin/ -1. External client authenticates with Eyre using `+code` (ship's web login code) -2. Client subscribes to SSE event stream for real-time updates -3. Client sends actions via PUT/POST with JSON payloads (poked through `mark` system) -4. Eyre routes actions to the appropriate Gall agent -5. Agent processes action, updates state, and emits events on subscribed paths +# macOS +curl -L https://urbit.org/install/mac/latest -o urbit +chmod +x urbit +sudo mv urbit /usr/local/bin/ -## Urbit ID (Azimuth) +# Verify +urbit --version +``` -### Address Hierarchy +### Obtaining an Urbit ID (Planet) -Urbit's identity system is a hierarchical address space registered on Ethereum: +Urbit identity is an NFT on Ethereum L2 (Azimuth). To get a planet: -| Type | Count | Name format | Example | Role | -|------|-------|-------------|---------|------| -| Galaxy | 256 | ~zod, ~nec | `~zod` | Network infrastructure, issue stars | -| Star | 65,536 | ~marzod | `~marzod` | Peer discovery, issue planets, software distribution | -| Planet | ~4.3 billion | ~sampel-palnet | `~sampel-palnet` | Individual users (permanent identity) | -| Moon | 2^32 per planet | ~doznec-sampel-palnet | `~doznec-sampel-palnet` | Sub-identities, devices, bots | -| Comet | 2^128 | ~random-128bit | `~dozzod-dozzod-...` | Free, temporary, untrusted | +1. **Purchase**: Buy from a star operator or marketplace (e.g., azimuth.network, urbitex.io, ~tirrel planet sales) +2. **Receive**: Some communities distribute free planets +3. **Cost**: Typically $10-50 USD +4. **Wallet**: You receive a master ticket (BIP39 mnemonic) — this controls the identity -### Key Properties +**Alternative**: Boot a comet (free, no purchase needed) for testing. Comets have long names and limited reputation but full functionality. -- **NFT-based ownership**: Planets, stars, and galaxies are ERC-721 tokens on Ethereum L1 -- **Transferable**: Identity can be sold, gifted, or transferred on-chain -- **Hierarchical sponsorship**: Planets are sponsored by stars, stars by galaxies — sponsors provide peer discovery and software updates -- **Key rotation**: Networking keys can be rotated on-chain without losing identity -- **Deterministic names**: Ship names are derived from their Azimuth point number via a phonemic encoding +### Booting a Ship -### Identity for Bots +```bash +# Boot a planet (first time — requires planet name and key file) +urbit -w sampel-palnet -k /path/to/keyfile.key -For bot integration, use a **moon** (sub-identity of your planet): +# Boot a comet (free, no identity purchase needed) +urbit -c mycomet -- Free to create (no Ethereum transaction needed) -- Inherits networking from parent planet -- Disposable — can be destroyed and recreated -- Identified as `~moon-name-parent-planet` +# Resume an existing ship +urbit mycomet/ -```text -# In your ship's dojo (CLI) -|moon ~bot-name +# Boot with specific Ames port +urbit -p 34567 mycomet/ ``` -This creates a moon keyfile that can boot a separate ship instance for the bot. +**First boot** takes several minutes (OTA download of latest Arvo OS). Subsequent boots are fast. -## Hoon / Nock Programming +### Eyre HTTP API Configuration -### Nock +Eyre (the HTTP server vane) runs by default on port 8080 (or next available). To configure: -Nock is the lowest layer — a minimal combinator calculus with 12 opcodes. All Urbit computation reduces to Nock. You rarely write Nock directly, but understanding it helps debug: +```hoon +:: In dojo (Urbit's command line): +:: Set HTTP port +|pass [%e %set-host-port ~ `8080] -- **Deterministic**: Same input always produces same output -- **Functional**: No side effects in the VM itself -- **Minimal**: Entire spec fits on a napkin (12 rules) +:: Get authentication code (needed for API access) ++code +:: Returns something like: lidlut-tabwed-pillex-ridlup +``` -### Hoon +The `+code` output is the authentication password for the HTTP API. Store it securely. -Hoon is the high-level language that compiles to Nock: +### Networking (Ames) -```hoon -:: Basic Hoon syntax examples -:: -:: Comments start with :: -:: Runes are two-character digraphs (e.g., |= is "bartis") -:: -|= n=@ud :: gate (function) taking an unsigned decimal -^- @ud :: return type annotation -?: =(n 0) :: conditional (if n == 0) - 1 :: then: return 1 -(mul n $(n (dec n))) :: else: n * recurse(n-1) +Ames is the P2P networking protocol. Ships communicate directly when possible, falling back to galaxy/star infrastructure for NAT traversal. + +```bash +# Boot with specific Ames port (useful for port forwarding) +urbit -p 34567 mycomet/ + +# Port forwarding for direct P2P connections (recommended) +# Forward UDP port 34567 on your router to the ship's host ``` -**Key concepts**: +**NAT traversal**: If direct P2P is not possible, Ames routes through the ship's sponsor (star) or a galaxy. This adds latency but works without port forwarding. -| Concept | Description | -|---------|-------------| -| **Runes** | Two-character operators (e.g., `\|=` gate, `%-` function call, `^-` type cast) | -| **Cores** | Fundamental code unit — a battery (code) paired with a payload (data) | -| **Gates** | Functions — a core with a single arm `$` | -| **Arms** | Named computations within a core | -| **Faces** | Named bindings (like variable names) | -| **Molds** | Types — functions that validate/normalize data | -| **Marks** | File types with conversion rules (like MIME types for Urbit) | -| **Subjects** | The environment/context available to an expression | +## Bot API Integration -### Gall Agents +### Authentication -Gall agents are stateful applications — the primary way to build on Urbit: +All Eyre API requests require authentication via a session cookie: -```hoon -:: Minimal Gall agent skeleton -:: -/+ default-agent -|% -+$ state-0 [%0 messages=(list @t)] --- -%- agent:dbug -^- agent:gall -|_ =bowl:gall -+* this . - def ~(. (default-agent this %|) bowl) -:: -++ on-init `this(state [%0 ~]) -++ on-save !>(state) -++ on-load |=(old=vase `this(state !<(state-0 old))) -++ on-poke - |= [=mark =vase] - ?+ mark (on-poke:def mark vase) - %noun - =/ msg !<(@t vase) - `this(messages [msg messages.state]) - == -++ on-watch on-watch:def -++ on-leave on-leave:def -++ on-peek on-peek:def -++ on-agent on-agent:def -++ on-arvo on-arvo:def -++ on-fail on-fail:def --- +```typescript +// Authenticate and get session cookie +async function authenticate(shipUrl: string, code: string): Promise { + const response = await fetch(`${shipUrl}/~/login`, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: `password=${code}`, + redirect: "manual", + }) + + const setCookie = response.headers.get("set-cookie") + if (!setCookie) throw new Error("Authentication failed — no cookie returned") + const match = setCookie.match(/urbauth-[^=]+=([^;]+)/) + if (!match) throw new Error("Authentication failed — invalid cookie format") + return setCookie.split(";")[0] // Return full cookie header value +} ``` -**Agent arms** (lifecycle hooks): - -| Arm | Purpose | -|-----|---------| -| `on-init` | Called once when agent is first started | -| `on-save` | Serialize state for persistence | -| `on-load` | Deserialize state (handles upgrades) | -| `on-poke` | Handle incoming actions (from local or remote) | -| `on-watch` | Handle subscription requests | -| `on-leave` | Handle subscription cancellations | -| `on-peek` | Handle scry (read-only state queries) | -| `on-agent` | Handle responses from other agents | -| `on-arvo` | Handle kernel responses | -| `on-fail` | Handle crash recovery | +### SSE Subscriptions (Real-Time Events) -### Learning Curve +Subscribe to ship events via Server-Sent Events: -Hoon has a steep learning curve: +```typescript +// Subscribe to graph-store updates via SSE +function subscribeToUpdates( + shipUrl: string, + cookie: string, + onEvent: (data: unknown) => void +): EventSource { + // First, open an SSE channel + const channelId = `bot-${Date.now()}` + const sseUrl = `${shipUrl}/~/channel/${channelId}` + + // Subscribe by sending a poke/subscribe action + fetch(sseUrl, { + method: "PUT", + headers: { + "Content-Type": "application/json", + Cookie: cookie, + }, + body: JSON.stringify([ + { + id: 1, + action: "subscribe", + ship: "sampel-palnet", // your ship name (without ~) + app: "graph-store", + path: "/updates", + }, + ]), + }) + + // Then listen for events + const es = new EventSource(sseUrl, { + // Note: EventSource doesn't natively support cookies + // Use a library like eventsource that supports headers + }) + + es.onmessage = (event) => { + const data = JSON.parse(event.data) + // ACK the event to prevent replay + fetch(sseUrl, { + method: "PUT", + headers: { + "Content-Type": "application/json", + Cookie: cookie, + }, + body: JSON.stringify([ + { id: data.id + 1, action: "ack", "event-id": data.id }, + ]), + }) + onEvent(data) + } + + return es +} +``` -- Unfamiliar syntax (runes instead of keywords) -- Unique type system (molds, faces, subjects) -- Functional paradigm with no mutable state -- Limited tooling (no mainstream IDE support, basic LSP) -- Small community for help/examples +### Pokes (Sending Actions) -**Realistic timeline**: Expect 2-4 weeks of dedicated study to write basic Gall agents. The [Hoon School](https://docs.urbit.org/courses/hoon-school) and [App School](https://docs.urbit.org/courses/app-school) courses are the recommended path. +Poke an app to trigger an action (e.g., send a message): -## HTTP API (Eyre) +```typescript +// Poke graph-store to send a message +async function sendMessage( + shipUrl: string, + cookie: string, + channelId: string, + recipientShip: string, + message: string +): Promise { + const now = Date.now() + const index = `/${now}` + + await fetch(`${shipUrl}/~/channel/${channelId}`, { + method: "PUT", + headers: { + "Content-Type": "application/json", + Cookie: cookie, + }, + body: JSON.stringify([ + { + id: now, + action: "poke", + ship: "sampel-palnet", // your ship name + app: "graph-store", + mark: "graph-update-3", + json: { + "add-nodes": { + resource: { + ship: "sampel-palnet", + name: `dm-inbox`, + }, + nodes: { + [index]: { + post: { + author: "sampel-palnet", + index, + "time-sent": now, + contents: [{ text: message }], + hash: null, + signatures: [], + }, + children: null, + }, + }, + }, + }, + }, + ]), + }) +} +``` -### Authentication +### Scry Endpoints (Reading State) -Every ship has a web login code accessible from the dojo: +Scry is a read-only interface for querying ship state: -```text -+code +```typescript +// Read ship state via scry +async function scry( + shipUrl: string, + cookie: string, + app: string, + path: string +): Promise { + const response = await fetch( + `${shipUrl}/~/scry/${app}${path}.json`, + { headers: { Cookie: cookie } } + ) + if (!response.ok) throw new Error(`Scry failed: ${response.status}`) + return response.json() +} + +// Examples: +// Get all DM conversations +const dms = await scry(url, cookie, "graph-store", "/keys") + +// Get messages from a specific graph +const messages = await scry(url, cookie, "graph-store", + "/graph/~sampel-palnet/dm-inbox/node/subset/kith/lone/newest/count/20") + +// Get contact list +const contacts = await scry(url, cookie, "contact-store", "/all") ``` -This returns a code like `lidlut-tabwed-pillex-ridlur`. Use it to authenticate HTTP sessions. +### Thread Execution -```bash -# Authenticate and get session cookie -curl -i -X POST http://localhost:8080/~/login \ - -d "password=lidlut-tabwed-pillex-ridlur" +Threads allow complex multi-step operations: -# Response includes set-cookie header with urbauth-~sampel-palnet= +```typescript +// Execute a thread (spider) +async function runThread( + shipUrl: string, + cookie: string, + desk: string, + threadName: string, + inputMark: string, + outputMark: string, + body: unknown +): Promise { + const response = await fetch( + `${shipUrl}/spider/${desk}/${inputMark}/${threadName}/${outputMark}.json`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + Cookie: cookie, + }, + body: JSON.stringify(body), + } + ) + return response.json() +} ``` -### Subscribing to Events (SSE) +### Access Control -```bash -# Subscribe to a Gall agent's path via Server-Sent Events -curl -N -H "Cookie: urbauth-~sampel-palnet=" \ - http://localhost:8080/~/channel/my-channel-1234 \ - --data '[{"id":1,"action":"subscribe","ship":"sampel-palnet","app":"chat-store","path":"/mailbox/~sampel-palnet/general"}]' +Access control is ship-based. Maintain an allowlist of ship names (e.g., `~zod`, `~sampel-palnet`) that are authorized to interact with the bot. Check the author field of incoming messages against this allowlist. + +```typescript +const ALLOWED_SHIPS = new Set([ + // "~sampel-palnet", + // "~zod", +]) + +function isAuthorized(ship: string): boolean { + if (ALLOWED_SHIPS.size === 0) return true // open to all if empty + return ALLOWED_SHIPS.has(ship) +} ``` -### Sending Actions (Poke) +### Basic Bot Example -```bash -# Poke a Gall agent with a JSON action -curl -X PUT http://localhost:8080/~/channel/my-channel-1234 \ - -H "Cookie: urbauth-~sampel-palnet=" \ - -H "Content-Type: application/json" \ - -d '[{ - "id": 2, - "action": "poke", - "ship": "sampel-palnet", - "app": "chat-hook", - "mark": "chat-action", - "json": { - "message": { - "path": "/~sampel-palnet/general", - "envelope": { - "uid": "0v1.abcde.fghij", - "number": 1, - "author": "~sampel-palnet", - "when": 1234567890000, - "letter": {"text": "Hello from external client"} +```typescript +import { EventSourcePlus } from "event-source-plus" + +const SHIP_URL = "http://localhost:8080" +const SHIP_NAME = "sampel-palnet" +const SHIP_CODE = process.env.URBIT_SHIP_CODE +if (!SHIP_CODE) throw new Error("URBIT_SHIP_CODE not set") + +// Authenticate +const loginResp = await fetch(`${SHIP_URL}/~/login`, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: `password=${SHIP_CODE}`, + redirect: "manual", +}) +const cookie = loginResp.headers.get("set-cookie")?.split(";")[0] +if (!cookie) throw new Error("Login failed") + +const channelId = `bot-${Date.now()}` +const channelUrl = `${SHIP_URL}/~/channel/${channelId}` + +// Subscribe to graph-store updates +await fetch(channelUrl, { + method: "PUT", + headers: { "Content-Type": "application/json", Cookie: cookie }, + body: JSON.stringify([{ + id: 1, + action: "subscribe", + ship: SHIP_NAME, + app: "graph-store", + path: "/updates", + }]), +}) + +// Listen for events via SSE +const sse = new EventSourcePlus(channelUrl, { + headers: { Cookie: cookie }, +}) + +sse.listen({ + onMessage(event) { + try { + const data = JSON.parse(event.data) + + // ACK the event + fetch(channelUrl, { + method: "PUT", + headers: { "Content-Type": "application/json", Cookie: cookie }, + body: JSON.stringify([{ + id: Date.now(), + action: "ack", + "event-id": data.id, + }]), + }) + + // Process graph-store updates + if (data.json?.["add-nodes"]) { + const nodes = data.json["add-nodes"].nodes + for (const [index, node] of Object.entries(nodes)) { + const post = (node as { post: { author: string; contents: { text?: string }[] } }).post + if (post.author !== SHIP_NAME) { + const textContent = post.contents + .filter((c: { text?: string }) => c.text) + .map((c: { text?: string }) => c.text) + .join(" ") + console.log(`Message from ~${post.author}: ${textContent}`) + // Handle command and send response... + } } } + } catch (err) { + console.error("Error processing event:", err) } - }]' + }, +}) + +console.log(`Bot listening on ${SHIP_URL} as ~${SHIP_NAME}`) ``` -### Channel Protocol +## Security Considerations -Eyre uses a channel-based protocol for bidirectional communication: +### Decentralization -| Action | Method | Description | -|--------|--------|-------------| -| `poke` | PUT | Send action to a Gall agent | -| `subscribe` | PUT | Subscribe to an agent's event path | -| `unsubscribe` | PUT | Cancel a subscription | -| `ack` | PUT | Acknowledge received events (prevents replay) | -| `delete` | DELETE | Close the channel | +**FULLY decentralized.** Each user runs their own server (ship). No central servers, no company controlling the network. Your ship is YOUR computer. The Urbit Foundation steers development but has no control over running ships. No single entity can censor, deplatform, or surveil users. -Events arrive via SSE on the channel URL. Each event has an incrementing ID that must be acknowledged. +### Encryption -### Scry (Read-Only Queries) +Ames protocol provides E2E encryption between ships using Curve25519 key exchange and AES symmetric encryption. All inter-ship communication is encrypted by default — there is no unencrypted mode. This is transport-level and content-level encryption combined in one protocol. -```bash -# Read agent state without side effects -curl -H "Cookie: urbauth-~sampel-palnet=" \ - http://localhost:8080/~/scry/chat-store/mailbox/~sampel-palnet/general.json -``` +### Metadata Privacy -Scry is a synchronous, read-only query into agent state — useful for polling current state without subscribing. +No central entity collects metadata. Your ship knows who you communicate with, but no third party does. Network topology is peer-to-peer — when direct connections are possible, no intermediary sees any metadata at all. -### JavaScript/TypeScript Client +**Caveat**: When NAT traversal is required, galaxy/star infrastructure routes initial connections. These nodes could theoretically log connection metadata (which ships are communicating) but cannot read content. Self-hosting a star eliminates this concern entirely. -The `@urbit/http-api` package provides a typed client: +### Identity -```bash -npm install @urbit/http-api -``` +Urbit ID (Azimuth) is decentralized identity on Ethereum L2. Identity is an NFT — you own it cryptographically. No phone number, no email, no central authority can revoke it. -```typescript -import Urbit from "@urbit/http-api"; - -// Connect to ship -const api = await Urbit.authenticate({ - ship: "sampel-palnet", - url: "http://localhost:8080", - code: "lidlut-tabwed-pillex-ridlur", -}); - -// Subscribe to events -api.subscribe({ - app: "chat-store", - path: "/mailbox/~sampel-palnet/general", - event: (data) => { - console.log("New event:", data); - }, - err: (error) => { - console.error("Subscription error:", error); - }, - quit: () => { - console.log("Subscription closed"); - }, -}); - -// Poke (send action) -await api.poke({ - app: "chat-hook", - mark: "chat-action", - json: { - message: { - path: "/~sampel-palnet/general", - envelope: { - uid: "0v1.abcde.fghij", - number: 1, - author: "~sampel-palnet", - when: Date.now(), - letter: { text: "Hello from TypeScript" }, - }, - }, - }, -}); +| Tier | Count | Acquisition | Reputation | +|------|-------|-------------|------------| +| Galaxy | 256 | Purchased (very expensive) | Highest — infrastructure nodes | +| Star | ~65,536 | Purchased (moderate) | High — issues planets | +| Planet | ~4 billion | Purchased ($10-50) | Standard — personal identity | +| Comet | 2^128 | Free (self-generated) | Lowest — temporary, often filtered | -// Scry (read state) -const state = await api.scry({ app: "chat-store", path: "/keys" }); -``` +### Push Notifications -## Installation +No push notifications. Ships are always-on servers — they receive messages directly via Ames. No Google FCM or Apple APNs dependency. No metadata leakage to push notification providers. -### Binary (Recommended) +### AI Training Risk -```bash -# Linux (x86_64) -curl -L https://urbit.org/install/linux-x86_64/latest -o urbit -chmod +x urbit -./urbit +No AI training risk from the protocol or network. Your data lives on your ship. No external company has access to your messages, contacts, or application data. There is no cloud service harvesting user data. -# macOS (Apple Silicon) -curl -L https://urbit.org/install/macos-aarch64/latest -o urbit -chmod +x urbit -./urbit +### Open Source -# macOS (Intel) -curl -L https://urbit.org/install/macos-x86_64/latest -o urbit -chmod +x urbit -./urbit -``` +Fully open-source: -### Boot a Ship +| Component | License | Repository | +|-----------|---------|------------| +| Vere (runtime) | MIT | https://github.com/urbit/urbit | +| Arvo (OS) | MIT | https://github.com/urbit/urbit | +| Azimuth (identity) | MIT | https://github.com/urbit/azimuth | +| Landscape (web UI) | MIT | https://github.com/tloncorp/landscape | +| Groups (social app) | MIT | https://github.com/tloncorp/tlon-apps | -```bash -# Boot a comet (free, temporary identity) -./urbit -c mycomet +Active development community. All core applications are open-source and auditable. -# Boot a planet (requires keyfile from Bridge) -./urbit -w sampel-palnet -k sampel-palnet-1.key +### Sovereignty -# Boot a moon (requires parent planet's dojo) -# On parent: |moon ~bot-name -# Then: ./urbit -w bot-name-sampel-palnet -k bot-name-sampel-palnet-1.key -``` +**Maximum digital sovereignty.** You own your identity, your data, your server, your network connections. No terms of service, no content moderation by a platform, no deplatforming risk. Your ship is a general-purpose computer that you control completely. -### Docker +This exceeds even Nostr's sovereignty model — Nostr users depend on relay operators for message delivery and persistence. Urbit ships store their own data and communicate directly. -```bash -docker run -d \ - --name urbit \ - -p 8080:80 \ - -v urbit-data:/urbit \ - tloncorp/urbit:latest -``` +### Key Management -### Hosting Providers +- **Master ticket**: BIP39 mnemonic that controls the Urbit ID — this is the root credential +- **Management proxy**: Can configure networking keys, set spawn proxy, etc. +- **Networking keys**: Used for Ames encryption — can be rotated (breach) +- **Breach**: Resetting networking keys. Disrupts all existing connections — contacts must reconnect. Use only when keys are compromised. +- **Hardware wallet**: Azimuth supports hardware wallet storage for master ticket (Trezor, Ledger) +- **Recommendation**: Store master ticket in hardware wallet or gopass. Use management proxy for day-to-day operations. -For users who do not want to self-host: +```bash +# Store Urbit credentials securely +gopass insert aidevops/urbit-bot/ship-code # +code for Eyre API +gopass insert aidevops/urbit-bot/master-ticket # Master ticket (CRITICAL) +``` -| Provider | Description | -|----------|-------------| -| [Tlon](https://tlon.io/) | Official hosting by Urbit's primary developer | -| [Native Planet](https://www.nativeplanet.io/) | Dedicated Urbit hardware appliances | -| [Red Horizon](https://redhorizon.com/) | Cloud hosting | +### Practical Concerns -### Dojo (CLI) +- Galaxy/star infrastructure routes Ames connections for NAT traversal — these nodes could log connection metadata (which ships communicate) but not content +- Self-hosting a star mitigates this entirely +- Ship must be running 24/7 to receive messages — requires always-on infrastructure +- Breach (key reset) is disruptive — all peers must re-establish connections -Once booted, the dojo is the ship's command-line interface: +### Comparison with Other Protocols -```text -:: Check ship identity -our +| Aspect | Urbit | Nostr | SimpleX | Signal | Matrix | +|--------|-------|-------|---------|--------|--------| +| PII required | None (NFT identity) | None | None | Phone number | Optional | +| Sovereignty | Maximum (own server) | High | High | Low (central) | Moderate | +| DM content privacy | Encrypted (Ames) | Encrypted (NIP-04) | Encrypted (double ratchet) | Encrypted | Encrypted | +| Metadata privacy | Strong (P2P) | Weak (relay sees pubkeys) | Strongest (no IDs) | Good (sealed sender) | Moderate | +| Censorship resistance | Maximum (no deps) | Strong (multi-relay) | Strong | Moderate | Moderate | +| Data ownership | You own everything | Relay-dependent | Ephemeral | Company holds | Server holds | +| Decentralization | Full (P2P) | Full (relay-based) | Full (no servers) | Centralized | Federated | -:: Check web login code -+code +**Summary**: Maximum sovereignty and privacy — you own everything. More sovereign than even Nostr (which depends on relay operators). The trade-off is complexity: running a personal server requires technical skill and always-on infrastructure. Best for users who prioritize sovereignty above convenience. -:: Install an app from a distribution ship -|install ~paldev %pals +## aidevops Integration -:: List installed desks (app packages) -+vats +### Helper Script Pattern -:: Send a chat message (via Groups app) -:groups &groups-action [%channel-post [~sampel-palnet %general] ...] +```bash +#!/usr/bin/env bash +# ~/.aidevops/agents/scripts/urbit-dispatch-helper.sh +set -euo pipefail + +urbit_dispatch() { + local sender_ship="$1" + local command="$2" + + if ! is_authorized "$sender_ship"; then + echo "Unauthorized ship: $sender_ship" + return 1 + fi + + case "$command" in + /status) aidevops status; return 0 ;; + /pulse) aidevops pulse; return 0 ;; + /ask\ *) aidevops research "${command#/ask }"; return 0 ;; + *) echo "Unknown command: $command"; return 1 ;; + esac +} + +is_authorized() { + local ship="$1" + local config_file="${HOME}/.config/aidevops/urbit-bot.json" + if [[ ! -f "$config_file" ]]; then + echo "No config found: $config_file" + return 1 + fi + if jq -e --arg s "$ship" '.allowed_ships[] | select(. == $s)' "$config_file" > /dev/null 2>&1; then + return 0 + fi + return 1 +} ``` -## Limitations - -### Steep Learning Curve - -Urbit's programming model (Hoon/Nock) is unlike any mainstream language. The rune-based syntax, subject-oriented programming, and unique type system require significant investment to learn. This is the primary barrier to building custom integrations. +### Configuration + +Config path: `~/.config/aidevops/urbit-bot.json` + +```json +{ + "ship_name": "sampel-palnet", + "ship_url": "http://localhost:8080", + "ship_code_gopass_path": "aidevops/urbit-bot/ship-code", + "allowed_ships": [ + "~zod", + "~sampel-palnet" + ], + "log_level": "info" +} +``` -### Niche Ecosystem +### Entity Resolution -- Small developer community compared to Matrix, SimpleX, or XMTP -- Limited third-party libraries and tooling -- Few production-grade bot frameworks -- IDE support is minimal (basic syntax highlighting, experimental LSP) +Urbit uses ship names (phonemic base — pronounceable syllable pairs): -### Limited Bot Tooling +| Format | Example | Usage | +|--------|---------|-------| +| Planet | `~sampel-palnet` | Standard identity | +| Star | `~sampel` | Infrastructure | +| Galaxy | `~zod` | Top-level infrastructure | +| Comet | `~doznec-marzod-...` (long) | Temporary identity | -There is no dedicated "bot SDK" equivalent to SimpleX's WebSocket API or XMTP's Agent SDK. Building a bot requires either: +Ship names are resolved via Ames networking. No external DNS or registry lookup needed — the Azimuth PKI on Ethereum provides the mapping from ship name to networking keys. -1. **Custom Gall agent** (Hoon) — full native integration but requires learning Hoon -2. **HTTP API bridge** (any language) — connect via Eyre, but limited to what the HTTP API exposes -3. **Airlock libraries** — `@urbit/http-api` (JS/TS), `urbit-api` (Python), `urbit-q` (Rust) — thin HTTP wrappers +### Runner Dispatch -### Performance +The bot dispatches tasks via the standard pattern: receive message via SSE subscription → validate sender ship name → parse command → dispatch via `urbit-dispatch-helper.sh` → collect result → poke graph-store to send response. -- Nock interpretation is slower than native code (jet system mitigates for common operations) -- Ship boot and OTA updates can be slow (minutes to hours for large state) -- Memory usage grows with state size — ships with years of data can consume significant RAM +## Matterbridge Integration -### Infrastructure Requirements +**NO native Matterbridge support for Urbit.** -Each ship is a long-running process that needs: +Matterbridge does not include an Urbit gateway. There is no community-maintained bridge. -- Persistent storage (event log grows over time) -- Stable network connectivity (for Ames peer-to-peer communication) -- Port forwarding or hosting provider for inbound connections -- Regular OTA updates (applied automatically but can be disruptive) +### Bridging Feasibility -### App Ecosystem Maturity +| Aspect | Assessment | +|--------|------------| +| Technical feasibility | Moderate — Eyre HTTP API can be used as integration point | +| Effort | High — requires custom Matterbridge gateway plugin | +| Messages (graph-store) | Possible via SSE subscription + poke | +| DMs | Complex — requires bot ship as relay | +| Identity mapping | Difficult — ship names don't map to usernames on other platforms | +| Direction | Urbit→other via SSE events; other→Urbit via pokes | +| Complexity | Higher than most — Urbit's data model (graph-store) is unique | -- Groups (chat/forums) is the primary social app — functional but less polished than mainstream alternatives -- App distribution is decentralized (install from other ships) but discovery is limited -- No push notifications in the traditional sense — ships are always-on servers +**Alternative**: Bot-level bridging — bot subscribes to Urbit graph-store via SSE, re-sends messages to Matrix/SimpleX, and vice versa. Simpler than a Matterbridge gateway but less scalable. -### Ethereum Dependency +## Limitations -Planet acquisition requires an Ethereum transaction (or receiving one from a star owner). Gas costs fluctuate. Layer 2 solutions (Naive Rollups) reduce cost but add complexity. +### Steep Learning Curve -## Security Considerations +Urbit has its own programming language (Hoon), virtual machine (Nock), operating system (Arvo), and networking protocol (Ames). Understanding the system requires learning fundamentally different computing concepts. The documentation assumes familiarity with functional programming. -### Threat Model +### Niche Ecosystem -Urbit protects against: +Small user base compared to mainstream messaging platforms. Limited third-party tooling. Community is technically sophisticated but small. Finding developers with Urbit experience is difficult. -- **Server compromise**: No central servers — each ship is independently operated -- **Identity theft**: Urbit ID keys are on Ethereum; identity cannot be forged without private key compromise -- **Network surveillance**: Ames encrypts all ship-to-ship traffic end-to-end -- **Metadata collection**: No third-party servers to collect metadata — your star sees connection requests but not message content -- **Platform deplatforming**: Identity is an NFT you own; no platform can revoke it +### Requires Always-On Server -Urbit does **not** protect against: +Your ship must be running 24/7 to receive messages and stay synchronized with the network. This requires always-on infrastructure — a VPS, dedicated server, or always-on home machine. Hosting providers (e.g., Tlon, Red Horizon, Escape Pod) offer managed hosting. -- **Device/host compromise**: Ship state on disk contains all data in the clear (event log) -- **Ethereum key compromise**: Attacker with master ticket can transfer identity and rotate keys -- **Star-level censorship**: A malicious star could refuse to route for its planets (mitigated by star transfer) -- **Traffic analysis**: Ames traffic patterns (timing, volume) are visible to network observers -- **Galaxy/star collusion**: The top of the hierarchy has structural power over routing and updates +### Urbit ID Cost -### Privacy Assessment +Planets (the standard identity tier) cost money — typically $10-50 USD as an NFT purchase. This is a barrier to entry compared to free identity systems. Comets (free) work for testing but have long names and may be filtered by some ships. -| Property | Rating | Notes | -|----------|--------|-------| -| E2E encryption | Strong | Ames encrypts all inter-ship communication | -| Decentralization | Maximum | Each user runs their own server | -| Identity sovereignty | Maximum | Urbit ID is an NFT — user owns it outright | -| Metadata protection | Strong | No central servers collecting metadata | -| Anonymity | Weak | Urbit IDs are pseudonymous but persistent and linkable | -| Infrastructure sovereignty | Maximum | Self-hosted; no dependency on any platform | -| Censorship resistance | Strong | Hierarchical sponsorship is the main vector; star transfer mitigates | +### Limited Bot Tooling -### Operational Security +No official bot SDK. The Eyre HTTP API is the only external integration point. Bot developers must work with raw HTTP requests, SSE streams, and Urbit's unique data structures (graph-store marks, nouns). No equivalent to Telegram's Bot API or Discord's SDK. -- Store master ticket (Ethereum key) in cold storage — it controls identity transfer -- Use management proxy for routine key operations -- Run ship on trusted infrastructure (VPS you control, or dedicated hardware) -- Keep ship updated — OTA updates include security patches -- Use moons for bot/service identities to isolate from primary planet -- Back up ship pier (data directory) regularly — loss means loss of all state +### HTTP API as Only External Interface -## Integration with aidevops +Eyre (HTTP server) is the sole integration point for external processes. There is no WebSocket API, no gRPC, no message queue interface. All bot communication must go through HTTP requests and SSE subscriptions. -### Current Status +### Performance -Urbit integration requires either a custom Hoon agent or an HTTP API bridge via Eyre. There is no turnkey bot framework. The HTTP API (`@urbit/http-api`) is the most practical path for external integration. +Nock VM (Urbit's execution environment) has overhead compared to native code. Operations that would be instant on a traditional server may take noticeable time on Urbit. This is improving with each runtime release but remains a factor. -### Potential Architecture +### Breaking Changes (Kelvin Versioning) -```text -┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ Urbit Ship │ │ HTTP Bridge │ │ aidevops Runner │ -│ (~sampel-palnet) │ │ (Node.js/Bun) │ │ │ -│ │ │ │ │ runner-helper.sh │ -│ Groups app / │────▶│ 1. SSE subscribe │────▶│ → AI session │ -│ custom agent │ │ 2. Parse message │ │ → response │ -│ │◀────│ 3. Dispatch │◀────│ │ -│ Chat channel │ │ 4. Poke reply │ │ │ -└──────────────────┘ └──────────────────┘ └──────────────────┘ -``` +Urbit uses kelvin versioning — version numbers count DOWN toward zero (stability). This means the system is still evolving and breaking changes can occur between versions. OTA updates are automatic — ships update themselves, which can break bot integrations. -### Bot via HTTP API (TypeScript) +### No Mobile App with Push Notifications -```typescript -import Urbit from "@urbit/http-api"; - -const api = await Urbit.authenticate({ - ship: "sampel-palnet", - url: "http://localhost:8080", - code: "lidlut-tabwed-pillex-ridlur", // from +code in dojo -}); - -// Subscribe to chat messages -api.subscribe({ - app: "groups", - path: "/groups/~sampel-palnet/my-group/channels/chat/~sampel-palnet/general/posts", - event: async (data) => { - // Parse incoming message - const message = extractMessage(data); - - // Dispatch to aidevops runner - const result = await dispatchToRunner(message); - - // Reply via poke - await api.poke({ - app: "channels", - mark: "channel-action", - json: buildReply(result), - }); - }, -}); -``` +No native mobile app with push notifications. The Tlon app exists but relies on the ship being accessible — there is no push notification infrastructure equivalent to Apple APNs or Google FCM. Users must keep the app connected or check manually. -### Use Cases for aidevops +### NAT Traversal Dependency -| Scenario | Value | -|----------|-------| -| Sovereign AI assistant | AI bot running on user's own ship — no third-party servers | -| Private group automation | Automate tasks in Urbit Groups channels | -| Decentralized dispatch | Use Ames for agent-to-agent communication without central infrastructure | -| Identity-verified commands | Urbit ID provides cryptographic authentication for command authorization | -| Long-term personal computing | Ship persists indefinitely — bot state survives across years | +Direct P2P connections require proper port forwarding. Without it, Ames routes through galaxy/star infrastructure, adding latency and introducing a metadata exposure point (the relay node sees which ships are communicating). -### Matterbridge Integration +### No Voice or Video Calls -Urbit does not have a native Matterbridge adapter. A custom adapter could be built using `@urbit/http-api` and Matterbridge's REST API, following the same pattern as the SimpleX adapter. This would bridge Urbit Groups channels to Matrix, Telegram, Discord, and other platforms. +No protocol-level support for voice or video calls. Some experimental integrations exist but nothing standardized. The focus is on text-based communication and application hosting. ## Related -- `services/communications/simplex.md` — SimpleX Chat (zero-knowledge, no identifiers) -- `services/communications/matrix-bot.md` — Matrix bot integration (federated) -- `services/communications/xmtp.md` — XMTP (Web3 messaging, wallet identity) -- `services/communications/bitchat.md` — Bitchat (Bluetooth mesh, offline) -- `services/communications/matterbridge.md` — Multi-platform chat bridge -- `tools/security/opsec.md` — Operational security guidance +- `.agents/services/communications/nostr.md` — Nostr (decentralized, relay-based, censorship-resistant) +- `.agents/services/communications/simplex.md` — SimpleX Chat (strongest metadata privacy) +- `.agents/services/communications/matrix-bot.md` — Matrix messaging (federated, mature ecosystem) +- `.agents/services/communications/bitchat.md` — BitChat (Bitcoin-native messaging) +- `.agents/services/communications/xmtp.md` — XMTP (Ethereum-native messaging) +- `.agents/services/communications/matterbridge.md` — Matterbridge (cross-platform bridging) +- `.agents/tools/security/opsec.md` — Operational security guidance +- `.agents/tools/credentials/gopass.md` — Secret management (for ship code storage) +- `.agents/tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns - Urbit Docs: https://docs.urbit.org/ -- Urbit Developer Docs: https://developers.urbit.org/ -- Urbit GitHub: https://github.com/urbit -- Azimuth (Urbit ID): https://azimuth.network/ -- Hoon School: https://docs.urbit.org/courses/hoon-school -- App School: https://docs.urbit.org/courses/app-school -- HTTP API Reference: https://docs.urbit.org/system/kernel/eyre/reference/external-api-ref +- Urbit Developers: https://developers.urbit.org/ +- Azimuth (Identity): https://azimuth.network/ +- Urbit GitHub: https://github.com/urbit/urbit +- Tlon (primary developer): https://tlon.io/ diff --git a/.agents/services/communications/whatsapp.md b/.agents/services/communications/whatsapp.md new file mode 100644 index 000000000..eb280f8da --- /dev/null +++ b/.agents/services/communications/whatsapp.md @@ -0,0 +1,740 @@ +--- +description: WhatsApp bot integration via Baileys (unofficial) — E2E encrypted messaging owned by Meta, Signal Protocol encryption but extensive metadata harvesting +mode: subagent +tools: + read: true + write: false + edit: false + bash: true + glob: false + grep: false + webfetch: false + task: false +--- + +# WhatsApp Bot Integration + + + +## Quick Reference + +- **Type**: E2E encrypted messaging owned by Meta — content encrypted but metadata extensively harvested +- **License**: Baileys (MIT, unofficial), WhatsApp Business API (proprietary, official) +- **Bot tool**: Baileys (TypeScript, 10k+ stars, unofficial WhatsApp Web API) +- **Protocol**: Signal Protocol (same as Signal app) +- **Encryption**: E2E by default (Signal Protocol — Curve25519, AES-256, HMAC-SHA256) +- **Docs**: https://github.com/WhiskeySockets/Baileys | https://developers.facebook.com/docs/whatsapp/ +- **Config**: `~/.config/aidevops/whatsapp-bot.json` (600 permissions) +- **Helper**: `whatsapp-dispatch-helper.sh [setup|start|stop|status|test|logs]` + +**Key differentiator**: WhatsApp has the largest messenger user base globally (2B+ users). Message content is E2E encrypted using the Signal Protocol — the same protocol used by Signal. However, Meta collects extensive metadata (contacts, timing, groups, device info, usage patterns) and uses it for ad targeting across Meta platforms. Think of it as: "your letters are sealed, but the postal service photographs every envelope and sells that data." + +**When to use WhatsApp over other messengers**: + +| Criterion | WhatsApp | Signal | SimpleX | Matrix | +|-----------|----------|--------|---------|--------| +| User base | 2B+ (largest) | ~40M | Small | ~100M | +| Content encryption | Signal Protocol | Signal Protocol | Double ratchet | Megolm/Olm | +| Metadata privacy | Poor (Meta harvests) | Good (minimal) | Excellent (none) | Moderate | +| User identifiers | Phone number | Phone number | None | `@user:server` | +| Bot ecosystem | Unofficial (Baileys) + official Business API | Minimal | Growing | Mature | +| Open source client | No | Yes | Yes (AGPL-3.0) | Yes | +| Best for | Reaching existing users | Privacy-conscious users | Maximum privacy | Federation, bridges | + + + +## Architecture + +```text +┌──────────────────────┐ +│ WhatsApp Mobile App │ +│ (iOS / Android) │ +│ │ +│ User sends message │ +└──────────┬───────────┘ + │ Signal Protocol (E2E encrypted) + │ +┌──────────▼───────────┐ ┌──────────────────────┐ +│ WhatsApp Servers │ │ Meta Metadata │ +│ (Meta infrastructure) │ │ Collection │ +│ │ │ │ +│ Cannot read content │ │ Records: who, when, │ +│ (E2E encrypted) │ │ how often, groups, │ +│ │ │ contacts, device, │ +│ Routes messages only │ │ IP, location │ +└──────────┬───────────┘ └──────────────────────┘ + │ +┌──────────▼───────────┐ +│ WhatsApp Web │ +│ Multi-Device Protocol │ +│ (no phone needed │ +│ after initial link) │ +└──────────┬───────────┘ + │ WebSocket + │ +┌──────────▼───────────┐ +│ Baileys Library │ +│ (TypeScript) │ +│ │ +│ Unofficial WA Web API │ +│ Handles encryption, │ +│ session management, │ +│ message parsing │ +└──────────┬───────────┘ + │ +┌──────────▼───────────┐ +│ Bot Process │ +│ (TypeScript/Bun) │ +│ │ +│ ├─ Command router │ +│ ├─ Message handler │ +│ ├─ Media handler │ +│ ├─ Access control │ +│ └─ aidevops dispatch │ +└────────────────────────┘ +``` + +**Message flow**: + +1. User sends message via WhatsApp mobile app +2. Message encrypted with Signal Protocol (Curve25519, AES-256, HMAC-SHA256) +3. Encrypted message routed through Meta's servers (content unreadable to Meta) +4. Meta records metadata: sender, recipient, timestamp, group, device info, IP +5. Multi-device protocol delivers to linked WhatsApp Web session +6. Baileys receives via WebSocket, decrypts using stored session keys +7. Bot process handles message, dispatches to aidevops runner +8. Response sent back through Baileys → WhatsApp servers → recipient + +## Installation + +### Prerequisites + +- **Node.js** >= 18 or **Bun** >= 1.0 +- **WhatsApp account** with active phone number +- A phone to scan QR code during initial setup (only needed once) + +### Baileys Library Setup + +```bash +# Using Bun (recommended) +bun add @whiskeysockets/baileys + +# Using npm +npm install @whiskeysockets/baileys + +# Additional dependencies +bun add qrcode-terminal # QR code display in terminal +bun add pino # Logger (required by Baileys) +``` + +### QR Code Authentication + +Baileys authenticates by emulating WhatsApp Web. On first run, you scan a QR code with your phone's WhatsApp app to link the device. + +```typescript +import makeWASocket, { + DisconnectReason, + useMultiFileAuthState, +} from "@whiskeysockets/baileys"; +import { Boom } from "@hapi/boom"; +import qrcode from "qrcode-terminal"; + +async function connectToWhatsApp() { + // Auth state persisted to filesystem — survives restarts + const { state, saveCreds } = await useMultiFileAuthState("auth_info_baileys"); + + const sock = makeWASocket({ + auth: state, + printQRInTerminal: true, // Display QR in terminal + }); + + // Save credentials whenever they update + sock.ev.on("creds.update", saveCreds); + + // Handle connection state changes + sock.ev.on("connection.update", (update) => { + const { connection, lastDisconnect, qr } = update; + + if (qr) { + // QR code displayed — scan with phone + // WhatsApp app > Settings > Linked Devices > Link a Device + console.log("Scan QR code with WhatsApp mobile app"); + } + + if (connection === "close") { + const reason = (lastDisconnect?.error as Boom)?.output?.statusCode; + if (reason !== DisconnectReason.loggedOut) { + // Reconnect on non-logout disconnections + connectToWhatsApp(); + } else { + console.log("Logged out — delete auth_info_baileys/ and restart"); + } + } + + if (connection === "open") { + console.log("Connected to WhatsApp"); + } + }); + + return sock; +} +``` + +### Multi-Device Support + +After the initial QR code scan, the bot runs independently — the phone does not need to stay online. WhatsApp's multi-device protocol syncs encryption keys across linked devices. + +**Limitations**: + +- Maximum 4 linked devices per WhatsApp account +- Phone must remain registered (active SIM / WhatsApp account) +- If the phone's WhatsApp is uninstalled, all linked devices are disconnected +- Linked devices are automatically unlinked after 14 days of phone inactivity + +### Session Persistence + +Auth state is stored in the `auth_info_baileys/` directory. This directory contains: + +- Session encryption keys +- Device registration data +- Pre-keys and identity keys + +**Back up this directory securely** — it grants full access to the WhatsApp account. Set file permissions to 700: + +```bash +chmod 700 auth_info_baileys/ +``` + +## Bot API Integration + +### Message Handling + +```typescript +import makeWASocket, { useMultiFileAuthState } from "@whiskeysockets/baileys"; + +async function startBot() { + const { state, saveCreds } = await useMultiFileAuthState("auth_info_baileys"); + const sock = makeWASocket({ auth: state, printQRInTerminal: true }); + sock.ev.on("creds.update", saveCreds); + + // Listen for incoming messages + sock.ev.on("messages.upsert", async ({ messages, type }) => { + if (type !== "notify") return; // Only process new messages + + for (const msg of messages) { + // Skip own messages + if (msg.key.fromMe) continue; + + // Extract sender and chat info + const chatId = msg.key.remoteJid!; // JID of chat (DM or group) + const isGroup = chatId.endsWith("@g.us"); + const sender = isGroup ? msg.key.participant! : chatId; + const pushName = msg.pushName || "Unknown"; // Display name + + // Extract text content + const text = + msg.message?.conversation || + msg.message?.extendedTextMessage?.text || + ""; + + if (!text) continue; // Skip non-text messages + + console.log(`[${isGroup ? "GROUP" : "DM"}] ${pushName}: ${text}`); + + // Command routing + if (text.startsWith("/")) { + await handleCommand(sock, chatId, sender, text); + } + } + }); + + return sock; +} + +async function handleCommand( + sock: ReturnType, + chatId: string, + sender: string, + text: string, +) { + const [command, ...args] = text.slice(1).split(" "); + const prompt = args.join(" "); + + switch (command) { + case "help": + await sock.sendMessage(chatId, { + text: "Available commands:\n/help - Show this message\n/ask - Ask AI\n/status - System status", + }); + break; + + case "ask": + if (!prompt) { + await sock.sendMessage(chatId, { text: "Usage: /ask " }); + return; + } + // Dispatch to aidevops runner + await sock.sendMessage(chatId, { + text: "Processing your request...", + }); + // runner-helper.sh dispatch would go here + break; + + case "status": + await sock.sendMessage(chatId, { text: "Bot is running." }); + break; + + default: + await sock.sendMessage(chatId, { + text: `Unknown command: /${command}`, + }); + } +} + +startBot(); +``` + +### Media Support + +Baileys supports sending and receiving various media types: + +```typescript +// Send image +await sock.sendMessage(chatId, { + image: { url: "./photo.jpg" }, // or Buffer + caption: "Image caption", +}); + +// Send video +await sock.sendMessage(chatId, { + video: { url: "./video.mp4" }, + caption: "Video caption", + gifPlayback: false, // true for GIF-style playback +}); + +// Send audio (voice note) +await sock.sendMessage(chatId, { + audio: { url: "./audio.ogg" }, + mimetype: "audio/ogg; codecs=opus", + ptt: true, // true = voice note, false = audio file +}); + +// Send document +await sock.sendMessage(chatId, { + document: { url: "./report.pdf" }, + mimetype: "application/pdf", + fileName: "report.pdf", +}); + +// Send sticker +await sock.sendMessage(chatId, { + sticker: { url: "./sticker.webp" }, +}); + +// Download received media +import { downloadMediaMessage } from "@whiskeysockets/baileys"; +const buffer = await downloadMediaMessage(msg, "buffer", {}); +``` + +### Reactions, Read Receipts, and Polls + +```typescript +// Send reaction +await sock.sendMessage(chatId, { + react: { text: "👍", key: msg.key }, +}); + +// Remove reaction +await sock.sendMessage(chatId, { + react: { text: "", key: msg.key }, +}); + +// Mark message as read +await sock.readMessages([msg.key]); + +// Send poll (WhatsApp polls) +await sock.sendMessage(chatId, { + poll: { + name: "What should we deploy?", + values: ["Frontend", "Backend", "Both", "Neither"], + selectableCount: 1, // single-select (use higher for multi-select) + }, +}); + +// Send status/stories broadcast +await sock.sendMessage("status@broadcast", { + text: "System maintenance at 3 AM UTC", +}); +``` + +### Access Control + +```typescript +// Allowlist-based access control +const ALLOWED_USERS = new Set([ + "44123456789@s.whatsapp.net", // Phone number JID format + "44987654321@s.whatsapp.net", +]); + +const ADMIN_USERS = new Set(["44123456789@s.whatsapp.net"]); + +function isAllowed(sender: string): boolean { + // Empty allowlist = allow all + if (ALLOWED_USERS.size === 0) return true; + return ALLOWED_USERS.has(sender); +} + +function isAdmin(sender: string): boolean { + return ADMIN_USERS.has(sender); +} + +// In message handler +sock.ev.on("messages.upsert", async ({ messages, type }) => { + if (type !== "notify") return; + for (const msg of messages) { + if (msg.key.fromMe) continue; + const sender = msg.key.remoteJid?.endsWith("@g.us") + ? msg.key.participant! + : msg.key.remoteJid!; + + if (!isAllowed(sender)) { + console.log(`Blocked message from unauthorized user: ${sender}`); + continue; + } + // Process message... + } +}); +``` + +### Group Management + +```typescript +// Get group metadata +const groupMeta = await sock.groupMetadata(groupId); +console.log(groupMeta.subject); // Group name +console.log(groupMeta.participants); // Member list + +// Check if bot is admin +const botJid = sock.user?.id; +const botParticipant = groupMeta.participants.find( + (p) => p.id === botJid, +); +const isBotAdmin = botParticipant?.admin === "admin" || botParticipant?.admin === "superadmin"; + +// Only respond to mentions in groups (optional) +const mentionedJids = msg.message?.extendedTextMessage?.contextInfo?.mentionedJid || []; +if (isGroup && !mentionedJids.includes(botJid!)) { + return; // Only respond when @mentioned in groups +} +``` + +## Security Considerations + +### Encryption: What IS Protected + +WhatsApp uses the **Signal Protocol** for end-to-end encryption — the same protocol used by Signal. This is genuinely strong encryption: + +- **Key exchange**: Extended Triple Diffie-Hellman (X3DH) with Curve25519 +- **Message encryption**: AES-256 in CBC mode +- **Message authentication**: HMAC-SHA256 +- **Forward secrecy**: Double Ratchet algorithm — compromise of current keys does not reveal past messages +- **Key verification**: QR code / 60-digit security code for manual verification + +**What Meta cannot read**: Message text, images, videos, voice notes, documents, and call audio are end-to-end encrypted. Meta's servers transport ciphertext they cannot decrypt. + +### Metadata: What IS Harvested + +**This is the critical privacy issue with WhatsApp.** Despite strong content encryption, Meta collects extensive metadata: + +| Metadata Category | What Meta Collects | Used For | +|-------------------|--------------------|----------| +| **Social graph** | Who you message, how often, when | Ad targeting, "People You May Know" | +| **Group data** | Group names, participants, activity | Interest profiling, social mapping | +| **Phone contacts** | All contacts uploaded (even non-WhatsApp users) | Cross-platform identity linking | +| **Device info** | Phone model, OS, carrier, battery level, signal strength | Device fingerprinting | +| **IP addresses** | Connection IPs, approximate location | Location-based ad targeting | +| **Usage patterns** | Session duration, frequency, feature usage | Engagement profiling | +| **Profile data** | Photo, status, about text, last seen | Identity enrichment | +| **Business interactions** | Messages to/from business accounts | Commerce targeting | +| **Payments** | Transaction details (where available) | Financial profiling | +| **Registration** | Phone number, verification timestamps | Core identity | + +**The analogy**: Your letters are sealed with the strongest encryption available, but the postal service photographs every envelope — recording sender, recipient, timestamp, weight, and frequency — and sells that data to advertisers. + +### Server Access + +- **Content**: Meta **CANNOT** read message content (E2E encryption is mathematically enforced) +- **Metadata**: Meta **HAS** and **USES** all metadata listed above for ad targeting across Facebook, Instagram, and WhatsApp +- **Backups**: Cloud backups (Google Drive / iCloud) are **NOT** E2E encrypted by default. WhatsApp offers optional E2E encrypted backups — users must explicitly enable this. Unencrypted backups are accessible to Google/Apple and law enforcement. + +### Push Notifications + +- iOS: Push notifications via Apple Push Notification service (APNs) +- Android: Push notifications via Firebase Cloud Messaging (FCM / Google) +- Notification metadata (sender, timing) is visible to Apple/Google +- Message content is not included in push payloads (only notification trigger) + +### AI Training and Data Use + +**CRITICAL WARNING**: Meta's privacy policy explicitly permits using WhatsApp metadata for AI model training and advertising: + +- WhatsApp metadata feeds Meta's advertising algorithms across all Meta platforms +- Meta has integrated AI features into WhatsApp (Meta AI chatbot) that process conversations users opt into +- WhatsApp Business API messages may be processed by Meta's AI systems for business insights +- Meta's terms of service allow them to update data usage policies with notice but without requiring explicit consent +- Business accounts interacting via the official Business API have additional data shared with Meta for "business messaging quality" and analytics + +### Open Source Status + +- **Client**: CLOSED source — no independent audit of client-side behavior +- **Server**: CLOSED source — no verification of server-side data handling +- **Protocol**: Signal Protocol is open source and independently audited, but WhatsApp's implementation is unverifiable +- **Baileys**: MIT-licensed OPEN source reverse-engineering of WhatsApp Web protocol — community-maintained, not endorsed by Meta + +### Jurisdiction + +- **Meta Platforms, Inc.** — headquartered in Menlo Park, California, USA +- **Meta Platforms Ireland Ltd** — data controller for EU/EEA users +- Subject to GDPR in the EU, but Meta has been fined repeatedly (e.g., EUR 225M in 2021, EUR 1.2B in 2023) for privacy violations +- Subject to US CLOUD Act — US government can compel data disclosure +- WhatsApp has cooperated with law enforcement by providing metadata (not message content) + +### Bot-Specific Risks + +| Risk | Severity | Detail | +|------|----------|--------| +| **Account ban** | HIGH | WhatsApp actively detects and bans unofficial API usage (Baileys). Detection methods include behavioral analysis, API call patterns, and protocol version checks. Bans are permanent for the phone number. | +| **Phone number exposure** | MEDIUM | Bot requires a real phone number. This phone number is visible to all contacts and group members. | +| **Business API cost** | LOW | Official WhatsApp Business API requires Meta business verification, monthly fees, and per-message pricing. Avoids ban risk but grants Meta more data access. | +| **Session hijacking** | HIGH | The `auth_info_baileys/` directory contains full session credentials. Anyone with access can impersonate the WhatsApp account. Secure with 700 permissions and encrypted backups. | +| **Rate limiting** | MEDIUM | WhatsApp has aggressive anti-spam detection. Sending too many messages too quickly triggers temporary or permanent bans. | + +### Comparison: Content Security vs Metadata Privacy + +| Messenger | Content Security | Metadata Privacy | Open Source | Overall Privacy | +|-----------|-----------------|-------------------|-------------|-----------------| +| Signal | Excellent (Signal Protocol) | Good (minimal collection) | Yes | Excellent | +| SimpleX | Excellent (Double Ratchet) | Excellent (no identifiers) | Yes (AGPL-3.0) | Best available | +| WhatsApp | Excellent (Signal Protocol) | **Poor** (Meta harvests) | No | **Poor overall** | +| Matrix | Good (Megolm/Olm) | Moderate (server-dependent) | Yes | Good (self-hosted) | +| Telegram | Moderate (MTProto, not default E2E) | Poor (phone number, server-side) | Partial (client only) | Moderate | + +**Bottom line**: WhatsApp's content encryption is as strong as Signal's. But metadata privacy is among the worst of mainstream messengers because Meta's entire business model depends on harvesting this data for advertising. Use WhatsApp when you need to reach users who are already on it — not when privacy is the primary requirement. + +## aidevops Integration + +### Helper Script + +`whatsapp-dispatch-helper.sh` follows the same pattern as `matrix-dispatch-helper.sh` and `simplex-helper.sh`: + +```bash +# Setup (interactive wizard) +whatsapp-dispatch-helper.sh setup + +# Start bot (foreground) +whatsapp-dispatch-helper.sh start + +# Start bot (daemon) +whatsapp-dispatch-helper.sh start --daemon + +# Stop bot +whatsapp-dispatch-helper.sh stop + +# Check status +whatsapp-dispatch-helper.sh status + +# Test dispatch +whatsapp-dispatch-helper.sh test "Ask a question" + +# View logs +whatsapp-dispatch-helper.sh logs +whatsapp-dispatch-helper.sh logs --follow +``` + +### Configuration + +`~/.config/aidevops/whatsapp-bot.json` (600 permissions): + +```json +{ + "authDir": "~/.aidevops/.agent-workspace/whatsapp-bot/auth_info_baileys", + "allowedUsers": [ + "44123456789@s.whatsapp.net" + ], + "adminUsers": [ + "44123456789@s.whatsapp.net" + ], + "botPrefix": "/", + "defaultRunner": "general", + "groupMappings": { + "120363012345678901@g.us": "code-reviewer" + }, + "ignoreOwnMessages": true, + "maxPromptLength": 3000, + "responseTimeout": 600, + "sessionIdleTimeout": 300, + "respondToMentionsOnly": true +} +``` + +### Runner Dispatch + +The bot dispatches to aidevops runners via `runner-helper.sh`: + +```bash +# Create runners for WhatsApp chats +runner-helper.sh create general \ + --description "General AI assistant for WhatsApp" + +runner-helper.sh create code-reviewer \ + --description "Code review and security analysis" +``` + +### Entity Resolution + +WhatsApp users are resolved to entities via `entity-helper.sh`: + +- **Channel**: `whatsapp` +- **Channel ID**: Phone number JID (e.g., `44123456789@s.whatsapp.net`) +- **Display name**: Push name from WhatsApp profile +- **Cross-channel**: Can link to same entity on Matrix, SimpleX, email + +### Session State Management + +Baileys auth state is stored at `~/.aidevops/.agent-workspace/whatsapp-bot/auth_info_baileys/`. This directory must be: + +- Persisted across bot restarts (contains session keys) +- Backed up securely (grants full account access) +- Set to 700 permissions +- Never committed to version control + +Conversation sessions follow the same Layer 0/1/2 model as the Matrix bot, stored in the shared `memory.db`. + +## Matterbridge Integration + +Matterbridge has native WhatsApp support via [whatsmeow](https://github.com/tulir/whatsmeow) (Go WhatsApp library, similar to Baileys but in Go). + +```text +WhatsApp (whatsmeow) + │ +Matterbridge + │ + ├── Matrix rooms + ├── SimpleX contacts + ├── Telegram groups + ├── Discord channels + ├── Slack workspaces + ├── IRC channels + └── 40+ other platforms +``` + +### Matterbridge Configuration + +`matterbridge.toml`: + +```toml +[whatsapp.mywhatsapp] +# No token needed — QR code auth on first run +# Matterbridge will display QR in terminal + +[[gateway]] +name = "whatsapp-bridge" +enable = true + +[[gateway.inout]] +account = "whatsapp.mywhatsapp" +channel = "120363012345678901@g.us" # WhatsApp group JID + +[[gateway.inout]] +account = "matrix.mymatrix" +channel = "#bridged-room:example.com" +``` + +**Key details**: + +- Uses whatsmeow (Go) — more stable than Baileys for bridging +- Same QR code auth flow as Baileys +- Same account ban risk as any unofficial WhatsApp API usage +- Bridges text, images, videos, documents +- Does not bridge reactions, polls, or voice notes +- See `services/communications/matterbridge.md` for full Matterbridge setup + +## Limitations + +### Unofficial API (Account Ban Risk) + +Baileys and whatsmeow are unofficial reverse-engineered libraries. WhatsApp's Terms of Service prohibit automated or bulk messaging via unofficial clients. Meta actively detects and permanently bans accounts using unofficial APIs. There is no appeal process for bans. + +**Mitigation**: Use a dedicated phone number for the bot (not your personal number). Accept that the account may be banned at any time. + +### Phone Number Required + +Every WhatsApp account requires a phone number. Unlike SimpleX (no identifiers) or Matrix (email optional), there is no way to use WhatsApp without a phone number. This phone number is visible to all contacts. + +### No Official Bot API for Personal Accounts + +The official WhatsApp Business API is only available for business accounts with Meta business verification. Personal accounts have no official bot API — Baileys is the only option, with all the ban risks that entails. + +### WhatsApp Business API Costs + +The official Business API has per-conversation pricing: + +| Category | Approximate Cost (varies by country) | +|----------|--------------------------------------| +| Marketing | $0.05 - $0.15 per conversation | +| Utility | $0.03 - $0.08 per conversation | +| Authentication | $0.02 - $0.06 per conversation | +| Service | Free (first 1000/month), then $0.03+ | + +Plus Business Solution Provider (BSP) fees if using a third-party platform. + +### File Size Limits + +| Media Type | Maximum Size | +|------------|-------------| +| Image | 16 MB | +| Video | 16 MB | +| Audio | 16 MB | +| Document | 100 MB | +| Sticker | 500 KB (static), 500 KB (animated) | + +### Rate Limiting + +WhatsApp has aggressive anti-spam detection: + +- Sending too many messages in a short period triggers warnings or bans +- New accounts have lower sending limits +- Business API has defined rate limits (varies by tier) +- Unofficial API usage has unpredictable limits — no documented thresholds + +### Multi-Device Limitations + +- Maximum 4 linked devices per account (1 phone + 4 companions) +- Linked devices are unlinked after 14 days of phone inactivity +- Broadcast lists and status updates have device-specific limitations +- Some features may not be available on linked devices + +### No Federation + +WhatsApp is a centralized, closed platform. There is no federation, no self-hosted servers, no alternative clients (officially). All traffic routes through Meta's infrastructure. You cannot run your own WhatsApp server. + +### Group Limitations + +- Maximum 1024 members per group +- Community groups: up to 5000 members across linked groups +- Admin-only messaging available but reduces bot utility +- No threaded conversations (all messages in single timeline) + +## Related + +- `.agents/services/communications/simplex.md` — SimpleX (maximum privacy, no identifiers) +- `.agents/services/communications/matrix-bot.md` — Matrix bot integration (federated, self-hosted) +- `.agents/services/communications/matterbridge.md` — Matterbridge cross-platform bridging +- `.agents/services/communications/bitchat.md` — BitChat (Bitcoin-native messaging) +- `.agents/services/communications/xmtp.md` — XMTP (Ethereum-native messaging) +- `.agents/tools/security/opsec.md` — Operational security guidance +- `.agents/tools/voice/speech-to-speech.md` — Voice note transcription +- `.agents/tools/ai-assistants/headless-dispatch.md` — Headless AI dispatch patterns +- Baileys GitHub: https://github.com/WhiskeySockets/Baileys +- WhatsApp Business API: https://developers.facebook.com/docs/whatsapp/ +- Signal Protocol: https://signal.org/docs/ +- whatsmeow (Go library): https://github.com/tulir/whatsmeow diff --git a/.agents/subagent-index.toon b/.agents/subagent-index.toon index 48c5ee139..0c2f26154 100644 --- a/.agents/subagent-index.toon +++ b/.agents/subagent-index.toon @@ -72,7 +72,7 @@ services/accessibility/,Unified web and email accessibility auditing - WCAG comp services/hosting/,Hosting providers - DNS and cloud servers and local dev,local-hosting|localhost|hostinger|hetzner|cloudflare|cloudflare-platform|cloudron|closte services/networking/,Networking - mesh VPN and secure device connectivity,tailscale|netbird services/email/,Email services - transactional email deliverability testing and autonomous mission communication,ses|email-agent|email-health-check|email-testing|email-delivery-test|email-design-test|email-delivery-testing|email-design-testing -services/communications/,Communications - SMS voice Matrix bot Discord bot multi-platform chat bridging Bluetooth mesh and Web3 messaging,twilio|telfon|matrix-bot|discord|matterbridge|simplex|bitchat|xmtp +services/communications/,Communications - SMS voice Matrix bot multi-platform chat bridging encrypted messaging Bluetooth mesh and Web3 messaging,twilio|telfon|matrix-bot|matterbridge|simplex|signal|telegram|whatsapp|imessage|nostr|slack|discord|google-chat|msteams|nextcloud-talk|urbit|bitchat|xmtp services/crm/,CRM integration - contact management,fluentcrm services/analytics/,Website analytics - GA4 reporting,google-analytics services/monitoring/,Error monitoring and debugging,sentry|socket diff --git a/.agents/tools/security/opsec.md b/.agents/tools/security/opsec.md index 057347a7b..da2f8eb43 100644 --- a/.agents/tools/security/opsec.md +++ b/.agents/tools/security/opsec.md @@ -70,19 +70,54 @@ AI agents that process untrusted content (web pages, MCP tool outputs, user uplo ## Platform Trust Matrix -### Messaging Platforms - -| Platform | E2E Default | Metadata | Training on data | Server location | Self-host | -|----------|-------------|----------|-----------------|-----------------|-----------| -| **SimpleX** | Yes (all) | Minimal (no user IDs) | No | Self-hostable | Yes | -| **Signal** | Yes (all) | Phone number required | No | US (Open Whisper) | Partial (server) | -| **Matrix/Element** | Optional (E2E rooms) | Room membership visible to server | No | Self-hostable | Yes | -| **iMessage** | Yes (Apple-to-Apple) | iCloud backup breaks E2E | No | Apple (US) | No | -| **WhatsApp** | Yes (messages) | Metadata to Meta | No | Meta (US) | No | -| **Telegram** | No (default) | All to Telegram | No | Dubai/US | No | -| **Discord** | No | All to Discord | Yes (ToS) | US | No | -| **Slack** | No | All to Salesforce | Yes (Enterprise AI) | US | No | -| **Teams** | No | All to Microsoft | Yes (M365 Copilot) | US/EU | Partial | +### Messaging Platforms — Privacy Comparison + +| Platform | E2E Default | E2E Scope | Metadata Exposure | Phone/Email Required | Push Notification Privacy | AI Training Policy | Open Source | Self-Hostable | Bot API Maturity | +|----------|-------------|-----------|-------------------|---------------------|--------------------------|-------------------|-------------|---------------|-----------------| +| **SimpleX** | Yes | All messages | Minimal (no user IDs, stateless relays) | No | Self-hosted push proxy available | None — non-profit, no data access | Client + server + protocol | Yes (SMP + XFTP) | Growing (WebSocket) | +| **Signal** | Yes | All messages | Minimal (sealed sender, phone hash only) | Phone number | Minimal (no content in push) | None — 501(c)(3) non-profit | Client + server | Partial (server) | Unofficial (signal-cli) | +| **Matrix/Element** | Optional | Per-room (Megolm) | Room membership visible to homeserver | Optional | Depends on homeserver config | None — protocol is open | Client + server + protocol | Yes (Synapse/Dendrite) | Mature (SDK, bridges) | +| **Nextcloud Talk** | Partial | 1:1 calls (WebRTC) | Your server only — no third party | Nextcloud account | Self-hosted push proxy | None — you own the server | Client + server (AGPL-3.0) | Yes (full stack) | Growing (webhook) | +| **XMTP** | Yes | All messages | Wallet address (pseudonymous) | No (wallet-based) | Varies by client | None — protocol is open | Protocol + SDK | Partial (nodes) | Growing | +| **Bitchat** | Yes | All messages | Bitcoin identity (pseudonymous) | No | None (P2P) | None — protocol is open | Full stack | Yes (P2P) | Experimental | +| **Nostr** | Partial | DMs only (NIP-04/44) | Pubkeys + timestamps visible to relays | No (keypair only) | None (client polling) | None from protocol — relay-dependent | Protocol + clients | Yes (relays) | Growing | +| **Urbit** | Yes | All inter-ship | Ship-to-ship only — no central metadata | No (Urbit ID) | None (always-on ship) | None — fully sovereign | Runtime + OS (MIT) | Yes (personal server) | Experimental | +| **iMessage** | Yes | Apple-to-Apple only | Apple sees metadata; iCloud backup risk | Apple ID | APNs (Apple sees metadata) | No (Apple policy) | Closed source | No | Unofficial (BlueBubbles) | +| **Telegram** | No | Secret Chats only (not bots/groups) | Telegram sees all non-Secret-Chat data | Phone number | FCM/APNs (metadata exposed) | Unclear — AI features exist | Client only (GPLv2) | No | Official (Bot API) | +| **WhatsApp** | Yes | Message content only | Extensive metadata to Meta (social graph, usage, device) | Phone number | FCM/APNs (metadata exposed) | Yes — Meta uses metadata for AI/ads | Closed source | No | Unofficial (Baileys) | +| **Slack** | No | None | Full access by Salesforce + workspace admins | Email | FCM/APNs (content in preview) | Yes — default ON, admin must opt out | Closed source | No | Official (Bolt SDK) | +| **Discord** | No | None | Full access by Discord Inc. | Email | FCM/APNs (content in preview) | Yes — data used for AI features | Closed source | No | Official (discord.js) | +| **Google Chat** | No | None | Full access by Google + workspace admins | Google account | FCM (Google sees everything) | Yes — Gemini processes chat data | Closed source | No | Official (Chat API) | +| **MS Teams** | No | None | Full access by Microsoft + tenant admins | M365 account | WNS/FCM/APNs | Yes — Copilot processes chat data | Closed source | No | Official (Bot Framework) | + +### Privacy Tiers — Threat Model Recommendations + +| Threat Tier | Recommended Platforms | Avoid | +|-------------|----------------------|-------| +| **T1** (data brokers) | Any E2E platform + VPN | Unencrypted email, SMS | +| **T2** (platform operator) | Signal, SimpleX, Matrix (self-hosted), Nextcloud Talk | Slack, Discord, Teams, Google Chat | +| **T3** (network observer) | SimpleX, Signal + Mullvad VPN, Nostr + Tor | Any platform without E2E | +| **T4** (nation-state) | SimpleX (no identifiers), Urbit (sovereign), Nostr (censorship-resistant) | Any platform requiring phone/email, any closed-source server | +| **T5** (physical access) | SimpleX (disappearing messages) + full-disk encryption | Any platform with cloud backups enabled | + +### AI Training Risk Summary + +Platforms that use or may use your data for AI training: + +| Platform | AI Training Status | What's Processed | How to Opt Out | +|----------|-------------------|-----------------|----------------| +| **Slack** | Default ON | All messages for AI/ML models | Workspace admin must email Slack to opt out | +| **Discord** | Active | Messages for AI features (summaries, Clyde) | User settings > Privacy > toggle off | +| **Google Chat** | Active (Gemini) | Chat content for Gemini AI | Workspace admin disables Gemini features | +| **MS Teams** | Active (Copilot) | Chat content for Copilot | Tenant admin configures Copilot access | +| **WhatsApp** | Metadata only | Metadata for ad targeting + AI; Business API messages for AI | Cannot opt out of metadata collection | +| **Telegram** | Unclear | Unknown — AI features exist (translation, etc.) | No known opt-out | +| **Signal** | Never | Nothing | N/A — non-profit, no data access | +| **SimpleX** | Never | Nothing | N/A — no data access possible | +| **Nextcloud Talk** | Never (self-hosted) | Nothing leaves your server | N/A — you control everything | +| **Matrix** | Never (protocol) | Nothing (self-hosted homeserver) | N/A — you control the server | +| **Nostr** | Never (protocol) | Nothing from protocol | N/A — relay operators set own policies | +| **Urbit** | Never | Nothing | N/A — fully sovereign | ### SimpleX vs Matrix Comparison @@ -267,10 +302,35 @@ sudo fwupdmgr update ## Related +### Security Tools + - `tools/security/prompt-injection-defender.md` — Prompt injection defense for AI agents and agentic apps -- `services/communications/simplex.md` — SimpleX install, bot API, self-hosted servers - `tools/security/tirith.md` — Terminal command security guard - `tools/credentials/encryption-stack.md` — gopass, SOPS, gocryptfs - `tools/credentials/gopass.md` — Secret management - `tools/browser/browser-automation.md` — Playwright, CamoFox integration -- `services/communications/matterbridge.md` — Multi-platform chat bridging (security warnings) + +### Communications — Privacy-First (recommended for T2-T4) + +- `services/communications/simplex.md` — SimpleX: zero-knowledge, no user IDs, E2E everything +- `services/communications/signal.md` — Signal: gold standard E2E, minimal metadata, non-profit +- `services/communications/matrix-bot.md` — Matrix: self-hosted, federated, E2E per-room +- `services/communications/nextcloud-talk.md` — Nextcloud Talk: self-hosted, you own everything +- `services/communications/nostr.md` — Nostr: decentralized, censorship-resistant, keypair identity +- `services/communications/urbit.md` — Urbit: maximum sovereignty, personal server OS +- `services/communications/xmtp.md` — XMTP: wallet-based E2E messaging +- `services/communications/bitchat.md` — Bitchat: Bitcoin-identity P2P messaging + +### Communications — Mainstream (T1 only, AI training risks) + +- `services/communications/telegram.md` — Telegram: no default E2E, server-side storage, unclear AI policy +- `services/communications/whatsapp.md` — WhatsApp: E2E content but extensive Meta metadata harvesting +- `services/communications/imessage.md` — iMessage: Apple E2E, iCloud backup risk, closed source +- `services/communications/slack.md` — Slack: no E2E, AI training default-on, full admin access +- `services/communications/discord.md` — Discord: no E2E, AI features process content +- `services/communications/google-chat.md` — Google Chat: no E2E, Gemini processes chat data +- `services/communications/msteams.md` — MS Teams: no E2E, Copilot processes chat data + +### Bridging + +- `services/communications/matterbridge.md` — Multi-platform chat bridging (security warnings — bridging reduces privacy to the weakest link)