Docs: speech-to-text and text-to-speech recipes for a panel's mic and speaker

faster-whisper and piper as plain Function nodes, the mic -> STT -> logic ->
TTS -> speaker wiring, the Modules manifest they need (uv pip compile, av<19)
and measured sizes, memory and CPU times. Modules docs now say uv pip sync
installs no dependencies.
This commit is contained in:
2026-10-15 17:01:25 +02:00
parent 56fa34c2c9
commit d3b52bd7d6
3 changed files with 134 additions and 6 deletions
+128 -3
View File
@@ -155,6 +155,130 @@ and the result plays there too, just smaller. The node needs `ffmpeg` on the
[`WITH_FFMPEG=true`](../reference/configuration.md#image-options) in `.env`.
Otherwise run the node on a [remote worker](workers.md) that has it.
### Speech to and from a panel
openpanel publishes a 16 kHz mono WAV on its `mic` port after its wake word,
and plays any `audio` reference sent to its `speaker` port (WAV, MP3 or AAC).
Two Function nodes close that loop with local models.
[faster-whisper](https://github.com/SYSTRAN/faster-whisper) turns speech into
text and [piper](https://github.com/OHF-Voice/piper1-gpl) turns text into
speech.
The listening node has an `audio` input port `audio` and a `str` output port
`text`:
```python
from functools import cache
from pathlib import Path
from faster_whisper import WhisperModel
import fluksio
MODELS = Path.home() / ".cache" / "voice"
@cache
def _model(size):
# Downloaded on the first call, then held by the worker between calls.
return WhisperModel(
size, device="cpu", compute_type="int8", download_root=str(MODELS)
)
def process(audio, model="base", language="en"):
# vad_filter skips silence, which Whisper would otherwise hear as "You".
segments, _ = _model(model).transcribe(
fluksio.load_artifact(audio), language=language or None, vad_filter=True
)
return {"text": " ".join(segment.text.strip() for segment in segments)}
```
The speaking node has a `str` input port `text` and an `audio` output port
`speech`:
```python
import io
import wave
from functools import cache
from pathlib import Path
from piper import PiperVoice
from piper.download_voices import download_voice
import fluksio
MODELS = Path.home() / ".cache" / "voice"
@cache
def _voice(name):
# Downloaded on the first call, then held by the worker between calls.
MODELS.mkdir(parents=True, exist_ok=True)
download_voice(name, MODELS)
return PiperVoice.load(MODELS / f"{name}.onnx")
def process(text, voice="en_US-lessac-medium"):
buffer = io.BytesIO()
with wave.open(buffer, "wb") as wav:
_voice(voice).synthesize_wav(text, wav)
speech = fluksio.save_artifact(buffer.getvalue(), "speech.wav", "audio/wav")
return {"speech": speech}
```
`model`, `language` and `voice` are settings. `tiny` is faster than `base` and
less accurate. An empty `language` lets Whisper guess it for each clip. `voice`
takes any name from the [piper voices](https://huggingface.co/rhasspy/piper-voices),
such as `de_DE-thorsten-medium`.
To wire the loop in a flow `home`:
1. Declare a [flow input](../concepts/flows.md#flow-inputs) `heard` of type
`audio` with origin `panel`.
2. Under the panel's [I/O](../interface/dashboards.md#the-devices-own-io), wire
`mic` as `in` to `home.heard` and `speaker` as `out` to `home.reply`.
3. Bind the listening node's `audio` to `heard` and its `text` to `said`.
4. Have your own nodes read `said` and publish an answer as a `str`, such as
`answer`.
5. Bind the speaking node's `text` to `answer` and its `speech` to `reply`.
Binding the speaking node's `text` straight to `said` makes the panel repeat
what it heard, which tests the loop before there is any logic in it.
Both packages go into the node's environment, as described in
[Packages](#packages). A managed environment installs its manifest with
`uv pip sync`, which skips dependencies, so paste the fully resolved list into
**Modules**:
```sh
printf 'faster-whisper\nav<19\npiper-tts\n' | uv pip compile --python-version 3.13 -
```
The Docker image runs Python 3.13. `av<19` holds PyAV back, since faster-whisper
1.2.1 cannot open files with PyAV 19. In your own venv or on a remote worker,
`pip install faster-whisper "av<19" piper-tts` is enough.
The models land in `~/.cache/voice` on the machine running the node. The Docker
image keeps its home directory inside the container, so a recreated container
downloads them again. Point `MODELS` at a directory under `/data` to keep them
on the volume. An [attached worker](workers.md) starts a process per call and
loads the model each time.
What it costs, measured on a 2016 server CPU (Xeon E5-2683 v4):
| | Size | Memory | 3 s of speech, one core | All cores |
|---|---|---|---|---|
| Packages | 150 MB download, 460 MB installed | | | |
| Whisper `tiny` | 78 MB | 300 MB | 1.6 s | 0.7 s |
| Whisper `base` | 148 MB | 375 MB | 2.9 s | 1.2 s |
| piper `en_US-lessac-medium` | 63 MB | 250 MB | 0.6 s | 0.3 s |
Each model downloads once, on its node's first call. Loading it takes about a
second for Whisper and four seconds for piper. Memory is the worker's peak while
it holds the model. These figures come from piper's own speech fed back into
Whisper. A room microphone is harder, on `tiny` most of all.
## Printing
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
@@ -220,13 +344,14 @@ environment. `NODE_VENV=managed` always requests one; the Docker image uses
this setting.
In managed mode, the Modules manifest is installed with `uv pip sync` and
versioned with the flows. Applying it refreshes local workers. The API accepts
the same manifest:
versioned with the flows. Sync installs only the packages listed, not their
dependencies, so list those too, for example as resolved by `uv pip compile`.
Applying it refreshes local workers. The API accepts the same manifest:
```sh
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"requirements": "numpy>=2\npandas\n"}'
-d '{"requirements": "numpy>=2\n"}'
```
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
+2 -1
View File
@@ -420,7 +420,8 @@ to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#makin
The device decides what the panel's [I/O](#the-devices-own-io) port names
mean. openpanel serves `gpio.<n>`, `backlight`, `touch`, `mic` and `speaker`,
lists each one's `config` in its README and ignores any other name. A browser
screen serves none of them.
screen serves none of them. The [speech recipe](../code/nodes.md#speech-to-and-from-a-panel)
turns `mic` into text and text into `speaker`.
### The device's own I/O
+4 -2
View File
@@ -33,8 +33,10 @@ packages and Python version. There are two setups:
- **Managed environment:** Docker uses a separate node environment. A
superuser edits **Requirements** and presses **Apply**. Packages are synced
with `uv pip sync`; deleting a requirement removes it. The manifest is
versioned with the flows, and successful installs refresh the workers.
with `uv pip sync`; deleting a requirement removes it. Sync installs only
what is listed, so list the dependencies too, for example the output of
`uv pip compile`. The manifest is versioned with the flows, and successful
installs refresh the workers.
- **Adopted environment:** a fresh pip installation inside your own virtual
environment can use that environment for nodes too. Modules is read-only.
Install dependencies there with pip or uv, then sync or restart Fluksio.