TetraHome/Tetra-AI-HA-Integration
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

readme and version floor update

Browse Source
This commit is contained in:
techartdev
2026-02-20 21:52:53 +02:00
parent e5510cd6ab
commit 726ceb3ef6
3 changed files with 122 additions and 303 deletions
Download Patch File Download Diff File Expand all files Collapse all files
IMPLEMENTATION_PLAN.md
+118 -302
View File
@@ -1,355 +1,171 @@
# OpenClaw Integration — Implementation Plan
## 1. Overview
## 1. Scope and current state
This document describes the architecture and phased implementation plan for the
**OpenClaw Integration** — a native Home Assistant custom integration that acts as
a satellite companion to the existing **OpenClaw Assistant** addon.
This plan reflects the actual implementation status as of release `0.1.34`.
### Goals
### Project goals
| # | Goal | Priority |
|---|------|----------|
| 1 | Chat card (text, streaming, files, voice) embedded in HA dashboard | P0 |
| 2 | Sensor / binary-sensor entities for status, model, sessions | P0 |
| 3 | Bidirectional addon ↔ integration communication (zero manual setup) | P0 |
| 4 | Native HA conversation agent (Assist / Voice PE) | P1 |
| 5 | Service calls & events for automations | P1 |
| 6 | Media player entity for TTS output | P2 |
| # | Goal | Priority | Status |
|---|------|----------|--------|
| 1 | Stable HA integration with addon auto-discovery | P0 | ✅ Done |
| 2 | Sensors + binary sensor for gateway visibility | P0 | ✅ Done |
| 3 | Native Assist conversation agent | P0 | ✅ Done |
| 4 | Lovelace chat card (text + voice) | P0 | ✅ Done |
| 5 | Voice provider choice (`browser` / `assist_stt`) | P0 | ✅ Done |
| 6 | Production polish and broad compatibility hardening | P1 | 🚧 In progress |
| 7 | Optional native media player/TTS routing entity | P2 | ⏳ Not started |
---
## 2. Existing Addon — Key Facts
## 2. Architecture (implemented)
| Property | Value |
|----------|-------|
| Slug | `openclaw_assistant_dev` |
| Gateway port | `18789` (configurable via `gateway_port`) |
| Auth | Token-based (`gateway.auth.token` in `openclaw.json`) |
| Config file | `/config/.openclaw/openclaw.json` (addon container) |
| OpenAI-compatible endpoint | `POST /v1/chat/completions` (opt-in via `enable_openai_api`) |
| Gateway bind | `127.0.0.1` (loopback) or `0.0.0.0` (lan) |
| Node process | `openclaw gateway run` |
| Ingress port | `48099` (nginx) |
### 2.1 Core backend flow
- Integration communicates with OpenClaw gateway over HTTP.
- Primary chat endpoint: `POST /v1/chat/completions` (OpenAI-compatible).
- Coordinator uses lightweight connectivity checks and model probing with graceful fallback.
- Services and conversation agent share response extraction logic for modern OpenAI-compatible payloads.
### 2.2 Frontend flow
- Card sends chat requests via HA services/websocket (`openclaw.send_message`).
- Card receives replies through HA event subscription (`openclaw_message_received`).
- Backend keeps in-memory history and exposes websocket history sync (`openclaw/get_history`).
- Settings endpoint (`openclaw/get_settings`) provides integration-level card options.
### 2.3 Voice modes
- `browser` provider:
- Uses `SpeechRecognition` / `webkitSpeechRecognition`.
- Supports manual mic and continuous voice mode (wake-word flow).
- `assist_stt` provider:
- Captures local mic audio and sends to HA STT API (`/api/stt/<provider>`).
- Manual one-shot transcription flow.
- Provider metadata negotiation for language/sample rate/channels to reduce `415` failures.
---
## 3. Architecture
## 3. Repository map (current)
```
┌──────────────────────────────────────────────────────────┐
│ Home Assistant Core │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │
│ │ Sensors / │ │ Conversation │ │ Services / │ │
│ │ Binary │ │ Agent │ │ Events │ │
│ │ Sensors │ │ (Assist/VPE) │ │ │ │
│ └──────┬───────┘ └──────┬───────┘ └───────┬────────┘ │
│ │ │ │ │
│ └────────┬────────┴───────────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ OpenClawAPI │ (api.py — HTTP client) │
│ │ Client │ │
│ └───────┬────────┘ │
│ │ HTTP / SSE │
├──────────────────┼────────────────────────────────────────┤
│ │ │
│ ┌───────────────▼──────────────────────────────┐ │
│ │ OpenClaw Gateway (addon) │ │
│ │ │ │
│ │ /v1/chat/completions (SSE streaming) │ │
│ │ /api/status (JSON) │ │
│ │ /api/sessions (JSON) │ │
│ │ /api/models (JSON) │ │
│ └──────────────────────────────────────────────┘ │
│ │
│ OpenClaw Assistant Addon Container │
└───────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
│ Lovelace Dashboard │
│ │
│ ┌────────────────────────────────────────┐ │
│ │ openclaw-chat-card │ │
│ │ ┌──────────────────────────────────┐ │ │
│ │ │ Message history (markdown) │ │ │
│ │ │ Typing indicator │ │ │
│ │ │ File / image attachments │ │ │
│ │ │ Voice input button │ │ │
│ │ │ Voice mode toggle │ │ │
│ │ └──────────────────────────────────┘ │ │
│ └────────────────────────────────────────┘ │
│ ▲ │
│ │ HA WebSocket API │
│ │ (openclaw.send_message service) │
│ │ + event subscriptions │
└───────┼─────────────────────────────────────┘
│
▼
Home Assistant Core (services / events)
```
### 3.1 Auto-Discovery (Zero-Config)
The integration discovers the addon with **no manual setup** using two mechanisms:
1. **Supervisor API** — at config-flow time the integration calls
`GET /addons/openclaw_assistant_dev/info` via the HA Supervisor client.
This gives us: addon state, network ports, options.
2. **Shared filesystem** — the addon mounts `/addon_configs/<slug>` which maps
to `/config` inside the container. The integration can read
`openclaw.json` to get the gateway auth token, mode, and port.
**Net effect:** the user clicks "Add Integration → OpenClaw" and everything
connects automatically — no token or URL entry required.
### 3.2 Communication Protocol
| Direction | Transport | Endpoint / Mechanism |
|-----------|-----------|----------------------|
| Integration → Chat | HTTP POST + SSE stream | `/v1/chat/completions` (OpenAI compat) |
| Integration → Status | HTTP GET (polled every 30 s) | `/api/status` |
| Integration → Sessions | HTTP GET (polled every 60 s) | `/api/sessions` |
| Integration → Models | HTTP GET (on startup + hourly) | `/v1/models` |
| Addon → HA (events) | HA REST API (long-lived token) | `POST /api/events/openclaw_*` |
| Frontend → Integration | HA WebSocket API | service calls + event subscriptions |
### 3.3 Latency Budget (Voice Mode)
| Step | Target | Notes |
|------|--------|-------|
| STT (browser/HA) | ≤ 500 ms | WebSpeech API or Whisper via HA |
| Network to OpenClaw | ≤ 100 ms | localhost or LAN |
| OpenClaw inference | ≤ 1500 ms | First token via SSE stream |
| TTS | ≤ 500 ms | HA `tts.speak` or browser SpeechSynthesis |
| **Total** | **≤ 2600 ms** | Under 3 s target |
---
## 4. File Structure
```
openclaw_integration/
├── IMPLEMENTATION_PLAN.md ← this file
├── hacs.json ← HACS metadata
OpenClawHomeAssistantIntegration/
├── IMPLEMENTATION_PLAN.md
├── CHANGELOG.md
├── README.md
│
├── hacs.json
├── custom_components/
│ └── openclaw/
│ ├── __init__.py ← integration setup, coordinator
│ ├── manifest.json ← HA integration manifest
│ ├── config_flow.py ← auto-discovery config flow
│ ├── const.py ← constants, defaults
│ ├── api.py ← OpenClaw gateway HTTP client
│ ├── coordinator.py ← DataUpdateCoordinator for polling
│ ├── sensor.py ← sensor entities
│ ├── binary_sensor.py ← binary sensor entities
│ ├── conversation.py ← native conversation agent
│ ├── services.py ← service call handlers
│ ├── services.yaml ← service definitions
│ ├── strings.json ← English UI strings
│ └── translations/
│ └── en.json ← English translations
│
└── www/
└── openclaw-chat-card.js ← Lovelace custom card (Lit)
│ ├── __init__.py
│ ├── api.py
│ ├── binary_sensor.py
│ ├── config_flow.py
│ ├── const.py
│ ├── conversation.py
│ ├── coordinator.py
│ ├── exposure.py
│ ├── manifest.json
│ ├── sensor.py
│ ├── services.yaml
│ ├── strings.json
│ ├── translations/en.json
│ └── www/openclaw-chat-card.js
└── www/openclaw-chat-card.js
```
---
## 5. Phased Implementation
## 4. Implemented milestones
### Phase 1 — Foundation (MVP)
### Milestone A — Integration foundation
**Goal:** Integration installs, auto-discovers addon, exposes sensors.
- Config flow and options flow.
- Gateway API client and coordinator.
- Sensor + binary sensor platforms.
- HACS-ready packaging.
| Task | File(s) | Description |
|------|---------|-------------|
| 1.1 | `manifest.json` | Integration metadata, dependencies |
| 1.2 | `const.py` | Domain, default ports, config keys |
| 1.3 | `api.py` | `OpenClawApiClient` — HTTP client to gateway |
| 1.4 | `config_flow.py` | Auto-discovery via Supervisor + shared config |
| 1.5 | `coordinator.py` | `DataUpdateCoordinator` polling status/sessions/model |
| 1.6 | `__init__.py` | Integration setup, platforms, coordinator init |
| 1.7 | `sensor.py` | `openclaw_status`, `openclaw_last_activity`, `openclaw_session_count`, `openclaw_model` |
| 1.8 | `binary_sensor.py` | `openclaw_connected` |
| 1.9 | `strings.json`, `translations/en.json` | UI text |
| 1.10 | `hacs.json` | HACS configuration |
### Milestone B — Assist + automations
**Acceptance criteria:**
- `Add Integration → OpenClaw` works with zero config
- All 5 entities appear and update
- Integration reconnects gracefully when addon restarts
- Conversation agent integration.
- Services: `openclaw.send_message`, `openclaw.clear_history`.
- Event emission: `openclaw_message_received`.
- Compatibility fix for intent response handling across HA versions.
### Phase 2 — Services, Events & Conversation Agent
### Milestone C — Chat card reliability
**Goal:** Automations can send/receive messages; Assist pipeline works.
- Auto resource registration and cleanup of duplicate/legacy card resources.
- Versioned card URL strategy to avoid stale cache issues.
- Robust response parsing for multiple OpenAI-compatible response shapes.
- Backend chat history sync to restore UI state after navigation/reload.
| Task | File(s) | Description |
|------|---------|-------------|
| 2.1 | `services.yaml`, `services.py` | `openclaw.send_message`, `openclaw.clear_history` |
| 2.2 | `__init__.py` (update) | Fire `openclaw_message_received` HA event on response |
| 2.3 | `conversation.py` | Register as `conversation` agent |
| 2.4 | `api.py` (update) | SSE streaming support (`async for` over response chunks) |
| 2.5 | Testing | Verify Assist Text/Voice pipeline end-to-end |
### Milestone D — Voice hardening
**Acceptance criteria:**
- `openclaw.send_message` callable from automations
- `openclaw_message_received` event fires and can trigger automations
- OpenClaw appears in Assist agent picker
- Voice PE works: wake word → STT → OpenClaw → TTS → speaker
### Phase 3 — Chat Card (Frontend)
**Goal:** Full chat UI as a Lovelace custom card.
| Task | File(s) | Description |
|------|---------|-------------|
| 3.1 | `openclaw-chat-card.js` | Base card shell (Lit element, card config) |
| 3.2 | (continued) | Message history with timestamps, markdown rendering |
| 3.3 | (continued) | Streaming response display (typing indicator) |
| 3.4 | (continued) | File/image attachment support (upload via service) |
| 3.5 | (continued) | Voice input (WebSpeech/MediaRecorder → send audio) |
| 3.6 | (continued) | Voice mode toggle (continuous listen → auto-respond with TTS) |
| 3.7 | `__init__.py` (update) | Register card as Lovelace resource |
| 3.8 | Card editor | Visual card configuration editor |
**Acceptance criteria:**
- Card renders chat history with markdown
- Real-time streaming of AI responses
- File upload works in both directions
- Voice input captures and sends audio
- Voice mode maintains continuous conversation
### Phase 4 — Media Player & Polish
**Goal:** Native TTS output, wake word integration, production hardening.
| Task | File(s) | Description |
|------|---------|-------------|
| 4.1 | `media_player.py` | Media player entity for TTS output routing |
| 4.2 | | Wake word detection integration |
| 4.3 | | WebSocket connection management (token refresh, reconnect) |
| 4.4 | | Error handling, rate limiting, connection pooling |
| 4.5 | | Performance profiling (voice latency budget) |
| 4.6 | | Documentation and HACS submission |
- Wake word and always voice mode options.
- Brave guard with explicit allow override option.
- Language normalization and preferred Assist pipeline language handling.
- Improved TTS voice/language selection.
- Multi-pending response handling to prevent stuck typing state.
- `assist_stt` provider with negotiated STT metadata.
- `AudioWorkletNode` capture first, with `ScriptProcessorNode` fallback.
---
## 6. Entity Reference
## 5. Current configuration surface
### Sensors
### Integration options
| Entity ID | Class | State | Attributes |
|-----------|-------|-------|------------|
| `sensor.openclaw_status` | — | `online` / `offline` / `processing` | `gateway_version`, `uptime` |
| `sensor.openclaw_last_activity` | `timestamp` | ISO 8601 datetime | `last_message_preview` |
| `sensor.openclaw_session_count` | — | integer (active count) | `sessions` (list of IDs) |
| `sensor.openclaw_model` | — | model name string | `provider`, `context_window` |
- Prompt/context behavior options.
- Tool-call execution toggle.
- Wake word + always voice mode.
- Brave web speech override.
- Voice provider selector (`browser` or `assist_stt`).
### Binary Sensors
### Card-level behavior
| Entity ID | Class | State |
|-----------|-------|-------|
| `binary_sensor.openclaw_connected` | `connectivity` | `on` / `off` |
### Services
| Service | Fields | Description |
|---------|--------|-------------|
| `openclaw.send_message` | `message` (str), `session_id` (str, optional), `attachments` (list, optional) | Send a message to OpenClaw |
| `openclaw.clear_history` | `session_id` (str, optional) | Clear conversation history |
### Events
| Event | Data | Description |
|-------|------|-------------|
| `openclaw_message_received` | `message`, `session_id`, `model`, `timestamp` | Fired when OpenClaw sends a response |
- Can consume integration settings from websocket settings endpoint.
- Optional card config overrides remain available.
---
## 7. Key Technical Decisions
## 6. Remaining roadmap (next iterations)
### 7.1 Why HTTP+SSE instead of WebSocket?
### R1 — Voice provider validation and UX polish (next)
- The addon already exposes an **OpenAI-compatible** HTTP endpoint with SSE streaming
- No additional gateway code needed on the addon side
- SSE is simpler to manage (no bi-directional state, automatic reconnect)
- WebSocket can be added later for real-time push features
1. Validate `assist_stt` against multiple HA STT providers/languages.
2. Improve surfaced error details when `/api/stt/<provider>` returns non-200.
3. Add optional manual recording duration setting for `assist_stt`.
4. Add provider-specific status text for easier user troubleshooting.
### 7.2 Why Supervisor API for discovery?
### R2 — Compatibility and resilience
- Available in all HAOS / Supervised installs (target audience)
- Provides addon state, network config, and options without filesystem access
- Filesystem fallback (`/addon_configs/`) handles edge cases
1. Continue hardening for older/newer HA Core API differences.
2. Add broader runtime checks for changed Assist pipeline payload shapes.
3. Expand fallback behavior when pipeline metadata is unavailable.
### 7.3 Why register as a conversation agent?
### R3 — Optional enhancements (after stabilization)
- Native Assist integration = works with Voice PE, S3 satellite, etc.
- No need for third-party HACS integrations (Extended OpenAI Conversation)
- Event-driven: HA handles STT/TTS pipeline, we just process text
### 7.4 Frontend card communication
- Card uses HA WebSocket API (already authenticated)
- Calls `openclaw.send_message` service
- Subscribes to `openclaw_message_received` events
- No separate WebSocket to gateway needed (avoids CORS, auth issues)
1. Evaluate media-player based TTS routing entity.
2. Explore optional continuous flow for HA STT/TTS pipeline mode.
3. Add automated tests around settings websocket payload and card voice state transitions.
---
## 8. Challenges & Mitigations
## 7. Verification checklist before each release
| Challenge | Mitigation |
|-----------|------------|
| Gateway not reachable (loopback bind) | Integration runs in same host; `127.0.0.1` works. Document `lan` mode for remote setups. |
| Token rotation / mismatch | Re-read `openclaw.json` on connection failure; config flow stores token in HA config entry |
| Voice latency > 3s | Use SSE streaming (first token fast), browser-side SpeechSynthesis for TTS, Whisper locally for STT |
| Addon not installed | Config flow gracefully fails with a message directing user to install the addon first |
| HACS distribution | Standard HACS custom integration + Lovelace card resources |
| Large file uploads | Chunk uploads through service call; gateway handles multipart |
1. Bump and sync versions in:
- `custom_components/openclaw/manifest.json`
- `custom_components/openclaw/__init__.py` (`_CARD_URL`)
- `www/openclaw-chat-card.js` loader shim
2. Restart HA and hard-refresh dashboard.
3. Confirm browser console reports the expected card version.
4. Validate both providers:
- `browser`: manual + continuous mode
- `assist_stt`: manual transcription flow
5. Update `CHANGELOG.md` and `README.md` for any new option/behavior.
---
## 9. Dependencies
## 8. Minimum supported environment
### Python (integration)
| Package | Purpose | In HA? |
|---------|---------|--------|
| `aiohttp` | HTTP client for gateway API | ✅ built-in |
| `homeassistant` | HA core APIs | ✅ built-in |
### JavaScript (frontend card)
| Library | Purpose | Bundled? |
|---------|---------|----------|
| `lit` | Web component framework | ✅ available via HA |
| `marked` | Markdown rendering | Bundle in card JS |
No additional pip dependencies required — the integration uses only HA built-ins.
---
## 10. Testing Strategy
| Level | Tool | Coverage |
|-------|------|----------|
| Unit | `pytest` + `pytest-homeassistant-custom-component` | API client, coordinator, config flow |
| Integration | HA dev container with addon mock | End-to-end entity updates, service calls |
| Frontend | Manual + Playwright | Card rendering, streaming, voice |
---
## 11. Timeline Estimate
| Phase | Duration | Milestone |
|-------|----------|-----------|
| Phase 1 | 1–2 weeks | Sensors visible, auto-discovery works |
| Phase 2 | 1–2 weeks | Conversation agent in Assist, services/events |
| Phase 3 | 2–3 weeks | Chat card MVP (text + streaming + files) |
| Phase 4 | 2–3 weeks | Voice mode, media player, polish |
**Total estimated: 6–10 weeks to full feature set.**
- Home Assistant Core: `2025.1.0+` (declared HACS minimum; chosen conservatively for broad compatibility).
- Browser: modern Chromium/Firefox/Safari for card UI; voice capability depends on browser APIs and permissions.