Voice Assistant Pipeline¶
Overview¶
The Tab5 functions as a local voice satellite for Home Assistant. It runs wake-word detection entirely on-device, streams audio only on demand, and gives real-time visual feedback through an icon on the UI.
The pipeline involves these stages in sequence: local wake-word model → I2S audio capture → Home Assistant Voice pipeline (Wyoming STT → conversation agent → Wyoming TTS) → ES8388 DAC playback.
The conversation agent is the one of the Home Assistant voice assistant in use: two modes of the screen, Domo and Discu, pick which assistant answers (Assistant mode selection). Any conversation agent of Home Assistant will do.
Wake-word detection¶
The device runs micro_wake_word — ESPHome's TensorFlow Lite Micro integration. The model (okay_nabu) runs continuously in the background, processing audio frames from the I2S microphone. This runs on-device, in the ESP32-P4's local memory. No audio is transmitted over the network during idle listening.
When the model detects the wake phrase with sufficient confidence, it fires an internal event that ESPHome forwards to the Home Assistant Voice pipeline.
Resource cost: the micro_wake_word model runs at roughly 5–8% CPU on the ESP32-P4 at 360 MHz, continuously. This is acceptable, but it means the remaining ~92% must cover LVGL rendering, I2S handling, and Wi-Fi. On this hardware, that's comfortable. On an ESP32-S3, headroom would be tighter.
Wake-word detection can be toggled via a UI button (tab5_wake_word_active switch entity). When off, the device acts as a manual push-to-talk panel only.
Second wake word: "Stop" (roller shutter)¶
A second microWakeWord model (Stop) is declared alongside okay_nabu in tab5-assist.yaml, but it is only armed (micro_wake_word.enable_model) while the roller shutter is moving (volet_en_mouvement global, pushed by HA) and disarmed as soon as the movement ends. Saying "Stop" while the shutter moves triggers the stop script directly on the device — no wake phrase, no pipeline round-trip, no network latency. Two gotchas already burned into this setup: ESPHome only auto-enables the first declared model (hence the explicit enable/disable), and the detected wake_word string is "Stop" with a capital S.
Interrupting a running reply¶
Tapping the mic icon while TTS is playing stops the pipeline (voice_assistant.stop, which Home Assistant follows) and the speaker, aborts the current session and immediately restarts listening (tab5_vocal_interrupt_and_listen, tab5-scripts.yaml). This is the firmware-side answer to barge-in: while the Assist pipeline is in its responding phase the microphone belongs to the pipeline and the wake word is inactive, so a voice "stop" cannot work there — the tap can.
Audio capture¶
| Parameter | Value |
|---|---|
| Sample rate | 16 kHz |
| Bit depth | 16-bit |
| Channels | Mono |
| Interface | I2S |
| GPIO DIN | GPIO 28 |
| GPIO BCLK | GPIO 27 |
| GPIO LRCLK | GPIO 29 |
16 kHz / 16-bit mono is the standard format for speech processing pipelines. Home Assistant's Whisper-based STT expects this format.
Gain staging — take gain in the analog stage, not the digital one. All the input gain is set on the ES7210 preamp (mic_gain: 30dB, adjustable 0–37.5 dB), before the converter, and both gain_factor values (wake word and voice assistant) are left at 1. ESPHome's gain_factor is an integer multiply followed by a hard clamp (microphone_source.cpp): it amplifies the converter's noise floor exactly as much as the voice — so it buys no signal-to-noise — and it clips peaks into square waves, which is what Whisper transcribes worst. Moving 6 dB from digital to analog on 2026-08-05 kept the same output level and the same clipping threshold, but put the amplification where it actually adds signal. noise_suppression_level is set to 0 for the same reason: it is applied by HA (MicroVadSpeexEnhancer in assist_pipeline/pipeline.py), but a denoiser distorts speech and Whisper is trained on natural audio. Raise it to 1–2 only if recognition degrades in a noisy room.
Once the wake-word fires, audio is streamed in real time over the existing Wi-Fi connection to the HA Voice pipeline. The pipeline handles STT (Whisper), intent processing (Conversation agent), and TTS (Piper or cloud). The result comes back as an audio stream routed to the media_player entity on the Tab5.
Audio playback — ES8388¶
TTS responses play through the ES8388 DAC → amplifier → built-in speaker. The media player entity on the Tab5 is a standard ESPHome I2S media player component.
Boot sequence: the amplifier enable switch is activated after the backlight and the media_player volume have been set (and only then does the device wait for the HA API connection). Enabling the amplifier before the I2S clock is stable produces an audible pop. The on_boot sequence in tab5-ha-hmi.yaml enforces this order.
Pushes during TTS: Home Assistant no longer paces its pushes (the 1 s pauses between blocks were removed on 2026-10-01, see docs/architecture.md). A full push is 8 bulk calls of a few hundred bytes; outside a reconnection it runs every 10 minutes.
Playback format — this is a Home Assistant setting, not an output setting. The announcement_pipeline block of the media_player declares the preferred format advertised to HA, i.e. what HA transcodes the TTS to before sending it. It must therefore match the TTS engine, not the DAC:
| Parameter | Value | Why |
|---|---|---|
| Format | FLAC | Lossless, ~half the bytes of raw WAV over Wi-Fi. ESPHome's default, and what the official Home Assistant Voice PE firmware uses |
| Sample rate | 24000 Hz | The higher of the two engines now in service: "Domotique" stays on local Piper (fr_FR-upmc-medium, 22050 Hz), "Discussion LLM" moved to HA Cloud / AlainNeural (24000 Hz) on 2026-08-05. At 24000 the cloud is native and Piper is merely upsampled, which discards no content; asking for 22050 would do the reverse and cut the cloud above 11 kHz. Asking for 16 kHz — the original setting — silently threw away everything above 8 kHz, sibilants included. Revisit whenever the TTS engines change |
| Channels | Mono | One speaker |
The I2S speaker in primary mode accepts 16000–48000 Hz and reconfigures its clock to whatever stream arrives, so the sample_rate: 48000 on the speaker: component is only a default, never a constraint on the pipeline.
Playback buffer: buffer_duration: 500ms on the speaker (the ESPHome default). Do not lower it. TTS is streamed over HTTP from HA while it is still being generated, over Wi-Fi carried by the C6 co-processor on SDIO, and when this ring buffer runs dry the speaker task does not pause the stream — it pads the DMA buffer with silence. Every network hiccup becomes an audible micro-gap.
Pipeline states and visual feedback¶
The microphone icon in the UI changes color to reflect the current pipeline state:
| State | Icon color | Meaning |
|---|---|---|
| Idle — wake word listening | Grey | Background detection running |
| Wake word detected | Green | Actively capturing speech |
| Processing | Orange | HA pipeline analyzing the command |
| Speaking | Blue | TTS audio playing back |
| Error / not understood | Red | Pipeline returned no result |
| Wake word disabled | Dim grey | Detection switched off |
ESPHome's voice assistant component fires callbacks (on_listening, on_stt_end, on_tts_start, on_end, on_error) declared in tab5-assist.yaml; each one calls assist_set_pipeline_state() (tab5_assist.cpp), which sets the icon, its UIColor. colour (the active palette) and the status label from a single table. There is no separate state variable — the icon color is the state indicator.
Assistant mode selection¶
Two buttons, on the home page and in the assistant popup, pick the Home Assistant voice assistant that answers:
- Domo (Home Assistant icon) — Home Assistant's preferred assistant. To control the home, its conversation agent is « Home Assistant » (devices, sensors, automations).
- Discu (robot icon) — the assistant picked in « Tab5 · pipeline de discussion », with any conversation agent: a language model online, a local Ollama, another integration. In this mode, each request opens the assistant popup.
Setting them up in Home Assistant: Tablet settings → Voice assistant: the two modes.
The author's Discu goes through his own engine, vromvrom-engine, for three reasons: - a fast home shortcut: short home commands are recognised and run directly, without going through an LLM; - an LLM with dedicated agents depending on the request (specialists); - replacing the chosen LLM when it is unavailable, or for a smarter one for instance, local or in the cloud.
This engine is still a rough draft, which the author manages only through AI: it is not a prerequisite, nor something he can recommend. Any Home Assistant conversation agent will do.
How it works: the mode is stored in the conversation_mode global bool (kept across restarts); both pairs of buttons show it with a coloured border. On the HA side, the tablet's pipeline select (« Assistant », select.m5stack_tab5_home_assistant_hmi_assistant with the default device name) holds the pipeline in use. The tablet does not set it itself: it sends an esphome.tab5_mode_assistant event (option preferred or Discussion LLM) on each tap, and at each start once Home Assistant is connected, and packages/tab5_evenements.yaml selects on the select of that tablet the preferred pipeline, or for Discussion the one chosen in « Tab5 · pipeline de discussion » (up to 3.2, a pipeline named exactly « Discussion LLM »), only if it exists (ADR-0025). That is why a hand change of « Assistant » does not last. « Aucun » in the list selects the preferred pipeline, and the screen slots automation then reports the « discussion » zone as absent: the buttons are hidden and the tablet goes back to Domo (tab5-zones.yaml). The same package stops the tablet's media player when a reply is interrupted (esphome.tab5_voix_stop) and speaks the announcements (esphome.tab5_annonce: appointments, « Volet arrêté »). No « allow Home Assistant actions » option is needed.
→ Broader context: related_projects.md