Skip to content
Closed
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
998c3ea
feat(cli): Add PMOVES Agent SDK CLI Wizard & Rebrand Crush to PMOVES …
POWERFULMOVES Dec 29, 2025
0201b2f
feat: Geometric framework upgrade with CHIT integration
POWERFULMOVES Dec 31, 2025
cae15f8
feat(geometry): GEOMETRY BUS integration, CHIT services & BoTZ env va…
POWERFULMOVES Dec 18, 2025
b3ff4ff
feat(geometry-bus): CHIT mathematical integration with persona visual…
POWERFULMOVES Dec 22, 2025
4dd0544
feat(gateway): add consciousness demo endpoint for CGP generation
Dec 12, 2025
fb93005
fix: merge PMOVES.AI-Edition-Hardened into feat/hardened-chit-geometry
Jan 2, 2026
e7e5bbf
feat(architecture): Dynamic port pattern and services catalog update
Jan 4, 2026
a7f3fb1
fix(compose): Fix invalid nested port expansion patterns
Jan 4, 2026
2e78b77
fix(hardened): Fix invidious env-tier-media anchor after rebase
Jan 5, 2026
c87005f
security: Fix 3 critical security vulnerabilities
Jan 6, 2026
fa1fefc
fix(security): Apply 3 critical security fixes to feat/submodule-rest…
Jan 6, 2026
ee07b1d
fix(sql): Correct policy name for shape_index table
Jan 6, 2026
4530029
Merge remote-tracking branch 'origin/feat/submodule-restructure-merge…
Jan 6, 2026
be02476
feat(chit-evoswarm): Restore CHIT, Evoswarm, Flute, Geometry Bus, and…
Jan 6, 2026
c0a3dc2
fix(chit/docker): Complete CHIT v2 secrets management and docker fixes
Jan 6, 2026
580aa48
fix(compose): Add tier anchors and dynamic port allocation to services
Jan 7, 2026
2d26138
feat(chit): Add credential sync and multi-tier secrets support
Jan 7, 2026
5838e78
fix(security): Fix credential sync and error handling issues
Jan 7, 2026
3c7da8c
fix(security): Remove tracked env.tier-* files from repository
Jan 7, 2026
f630ac2
Merge feat/hardened-chit-geometry into PMOVES.AI-Edition-Hardened
Jan 7, 2026
87f56f0
Merge feat/hardened-agent-sdk-cli into PMOVES.AI-Edition-Hardened
Jan 7, 2026
09f1c16
Merge feat/hardened-dynamic-ports into PMOVES.AI-Edition-Hardened
Jan 7, 2026
c45af66
fix(security): Fix CodeQL alerts on PR #477
Jan 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
434 changes: 434 additions & 0 deletions .claude/context/evoswarm.md

Large diffs are not rendered by default.

258 changes: 258 additions & 0 deletions .claude/context/flute-gateway.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
# Flute-Gateway: Voice Communication Layer

**Reference documentation for the flute-gateway multimodal voice service.**

Based on PR #332 (Pipecat integration) - December 2025.

---

## Overview

Flute-Gateway provides a unified voice communication layer with:
- **Prosodic TTS** - Natural speech synthesis with intelligent pausing
- **Real-time streaming** - WebSocket-based audio I/O
- **Multiple backends** - VibeVoice, Ultimate-TTS-Studio, Whisper
- **Pipecat integration** - Pipeline-based audio processing

---

## Service Endpoints

| Port | Protocol | Purpose |
|------|----------|---------|
| 8055 | HTTP | REST API endpoints |
| 8056 | WebSocket | Real-time audio streaming |

---

## API Reference

### Health Check
```bash
curl http://localhost:8055/healthz
```

**Response:**
```json
{
"status": "healthy",
"providers": {
"vibevoice": true,
"whisper": true,
"ultimate_tts": true
},
"nats": "connected",
"supabase": "connected"
}
```

### Prosodic Text Analysis
```bash
curl -X POST http://localhost:8055/v1/voice/analyze/prosodic \
-H "Content-Type: application/json" \
-d '{"text": "Hello, this is a test."}'
```

**Response:**
```json
{
"chunks": [
{
"text": "Hello, this",
"boundary_after": "NONE",
"pause_ms": 0,
"is_first": true,
"is_final": false,
"estimated_syllables": 3
},
{
"text": "is a test.",
"boundary_after": "SENTENCE",
"pause_ms": 350,
"is_first": false,
"is_final": true,
"estimated_syllables": 3
}
],
"total_chunks": 2,
"estimated_ttfs_benefit": "~80% faster TTFS"
}
```

### Prosodic TTS Synthesis
```bash
curl -X POST http://localhost:8055/v1/voice/synthesize/prosodic \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FLUTE_API_KEY" \
-d '{
"text": "Hello, this is a test.",
"voice": "default",
"format": "wav"
}'
```

### Create Voice Session (WebSocket)
```bash
curl -X POST http://localhost:8055/v1/sessions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FLUTE_API_KEY" \
-d '{
"mode": "duplex",
"tts_engine": "ultimate-tts",
"stt_engine": "whisper"
}'
```

**Response:**
```json
{
"session_id": "sess_abc123",
"websocket_url": "ws://localhost:8056/sessions/sess_abc123",
"expires_at": "2025-12-19T23:00:00Z"
}
```

---

## Prosodic Synthesis

Prosodic synthesis improves time-to-first-speech (TTFS) by:
1. Analyzing text for natural break points
2. Chunking at breath boundaries and sentence ends
3. Streaming audio as each chunk completes

**Boundary Types:**
| Type | Pause (ms) | Description |
|------|------------|-------------|
| `NONE` | 0 | No pause, continue speaking |
| `BREATH` | 100-150 | Natural breath pause |
| `CLAUSE` | 200-250 | Comma or clause boundary |
| `SENTENCE` | 300-400 | Period, question, exclamation |

---

## WebSocket Protocol

### Connection
```javascript
const ws = new WebSocket('ws://localhost:8056/sessions/sess_abc123');

ws.onopen = () => console.log('Connected');
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
// Handle audio data or transcription
};
```

### Message Types

**Audio Input (STT):**
```json
{
"type": "audio",
"data": "<base64-encoded-audio>",
"format": "pcm16",
"sample_rate": 16000
}
```

**Text Input (TTS):**
```json
{
"type": "text",
"text": "Hello, world!",
"voice": "default"
}
```

**Audio Output:**
```json
{
"type": "audio_chunk",
"data": "<base64-encoded-audio>",
"chunk_index": 0,
"is_final": false
}
```

**Transcription Output:**
```json
{
"type": "transcription",
"text": "Hello, world!",
"is_final": true,
"confidence": 0.95
}
```

---

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `FLUTE_API_KEY` | (none) | API authentication key |
| `FLUTE_HTTP_PORT` | 8055 | HTTP API port |
| `FLUTE_WS_PORT` | 8056 | WebSocket port |
| `ULTIMATE_TTS_URL` | `http://host.docker.internal:7861` | TTS backend |
| `VIBEVOICE_URL` | `http://host.docker.internal:3000` | Alternative TTS |
| `WHISPER_URL` | `http://ffmpeg-whisper:8078` | STT backend |
| `DEFAULT_VOICE_PROVIDER` | `vibevoice` | Default TTS provider |

---

## Integration with Ultimate-TTS-Studio

Flute-Gateway connects to Ultimate-TTS-Studio for high-quality synthesis:

```
┌─────────────────┐ HTTP ┌──────────────────────┐
│ Flute-Gateway │──────────────▶│ Ultimate-TTS-Studio │
│ (Port 8055) │ │ (Port 7861) │
└─────────────────┘ └──────────────────────┘
│
│ WebSocket (8056)
│
▼
Client Audio I/O
```

---

## Slash Commands

| Command | Description |
|---------|-------------|
| `/tts:synthesize` | Generate TTS audio |
| `/tts:status` | Check TTS service health |
| `/tts:voices` | List available voices |
| `/pipecat:status` | Check Pipecat layer health |
| `/pipecat:connect` | Create voice session |

---

## Troubleshooting

### Service Not Responding
```bash
# Check container status
docker ps --filter "name=flute"

# View logs
docker logs pmoves-flute-gateway-1 --tail 50
```

### WebSocket Connection Failed
- Verify port 8056 is exposed
- Check CORS settings if connecting from browser
- Ensure session exists before connecting

### TTS Quality Issues
- Try different TTS engine (ultimate-tts vs vibevoice)
- Adjust voice parameters
- Check GPU availability for Ultimate-TTS

---

*Last Updated: December 2025*
*PR Reference: #332 (Pipecat Integration)*
Loading
Loading