From d3b52bd7d6709600a75feb7a3a958d2e6ad74361 Mon Sep 17 00:00:00 2001 From: stroblme Date: Thu, 15 Oct 2026 17:01:25 +0200 Subject: [PATCH] Docs: speech-to-text and text-to-speech recipes for a panel's mic and speaker faster-whisper and piper as plain Function nodes, the mic -> STT -> logic -> TTS -> speaker wiring, the Modules manifest they need (uv pip compile, av<19) and measured sizes, memory and CPU times. Modules docs now say uv pip sync installs no dependencies. --- docs/code/nodes.md | 131 ++++++++++++++++++++++++++++++++++- docs/interface/dashboards.md | 3 +- docs/interface/operations.md | 6 +- 3 files changed, 134 insertions(+), 6 deletions(-) diff --git a/docs/code/nodes.md b/docs/code/nodes.md index cc86c81..f65c9a3 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -155,6 +155,130 @@ and the result plays there too, just smaller. The node needs `ffmpeg` on the [`WITH_FFMPEG=true`](../reference/configuration.md#image-options) in `.env`. Otherwise run the node on a [remote worker](workers.md) that has it. +### Speech to and from a panel + +openpanel publishes a 16 kHz mono WAV on its `mic` port after its wake word, +and plays any `audio` reference sent to its `speaker` port (WAV, MP3 or AAC). +Two Function nodes close that loop with local models. +[faster-whisper](https://github.com/SYSTRAN/faster-whisper) turns speech into +text and [piper](https://github.com/OHF-Voice/piper1-gpl) turns text into +speech. + +The listening node has an `audio` input port `audio` and a `str` output port +`text`: + +```python +from functools import cache +from pathlib import Path + +from faster_whisper import WhisperModel + +import fluksio + +MODELS = Path.home() / ".cache" / "voice" + + +@cache +def _model(size): + # Downloaded on the first call, then held by the worker between calls. + return WhisperModel( + size, device="cpu", compute_type="int8", download_root=str(MODELS) + ) + + +def process(audio, model="base", language="en"): + # vad_filter skips silence, which Whisper would otherwise hear as "You". + segments, _ = _model(model).transcribe( + fluksio.load_artifact(audio), language=language or None, vad_filter=True + ) + return {"text": " ".join(segment.text.strip() for segment in segments)} +``` + +The speaking node has a `str` input port `text` and an `audio` output port +`speech`: + +```python +import io +import wave +from functools import cache +from pathlib import Path + +from piper import PiperVoice +from piper.download_voices import download_voice + +import fluksio + +MODELS = Path.home() / ".cache" / "voice" + + +@cache +def _voice(name): + # Downloaded on the first call, then held by the worker between calls. + MODELS.mkdir(parents=True, exist_ok=True) + download_voice(name, MODELS) + return PiperVoice.load(MODELS / f"{name}.onnx") + + +def process(text, voice="en_US-lessac-medium"): + buffer = io.BytesIO() + with wave.open(buffer, "wb") as wav: + _voice(voice).synthesize_wav(text, wav) + speech = fluksio.save_artifact(buffer.getvalue(), "speech.wav", "audio/wav") + return {"speech": speech} +``` + +`model`, `language` and `voice` are settings. `tiny` is faster than `base` and +less accurate. An empty `language` lets Whisper guess it for each clip. `voice` +takes any name from the [piper voices](https://huggingface.co/rhasspy/piper-voices), +such as `de_DE-thorsten-medium`. + +To wire the loop in a flow `home`: + +1. Declare a [flow input](../concepts/flows.md#flow-inputs) `heard` of type + `audio` with origin `panel`. +2. Under the panel's [I/O](../interface/dashboards.md#the-devices-own-io), wire + `mic` as `in` to `home.heard` and `speaker` as `out` to `home.reply`. +3. Bind the listening node's `audio` to `heard` and its `text` to `said`. +4. Have your own nodes read `said` and publish an answer as a `str`, such as + `answer`. +5. Bind the speaking node's `text` to `answer` and its `speech` to `reply`. + +Binding the speaking node's `text` straight to `said` makes the panel repeat +what it heard, which tests the loop before there is any logic in it. + +Both packages go into the node's environment, as described in +[Packages](#packages). A managed environment installs its manifest with +`uv pip sync`, which skips dependencies, so paste the fully resolved list into +**Modules**: + +```sh +printf 'faster-whisper\nav<19\npiper-tts\n' | uv pip compile --python-version 3.13 - +``` + +The Docker image runs Python 3.13. `av<19` holds PyAV back, since faster-whisper +1.2.1 cannot open files with PyAV 19. In your own venv or on a remote worker, +`pip install faster-whisper "av<19" piper-tts` is enough. + +The models land in `~/.cache/voice` on the machine running the node. The Docker +image keeps its home directory inside the container, so a recreated container +downloads them again. Point `MODELS` at a directory under `/data` to keep them +on the volume. An [attached worker](workers.md) starts a process per call and +loads the model each time. + +What it costs, measured on a 2016 server CPU (Xeon E5-2683 v4): + +| | Size | Memory | 3 s of speech, one core | All cores | +|---|---|---|---|---| +| Packages | 150 MB download, 460 MB installed | | | | +| Whisper `tiny` | 78 MB | 300 MB | 1.6 s | 0.7 s | +| Whisper `base` | 148 MB | 375 MB | 2.9 s | 1.2 s | +| piper `en_US-lessac-medium` | 63 MB | 250 MB | 0.6 s | 0.3 s | + +Each model downloads once, on its node's first call. Loading it takes about a +second for Whisper and four seconds for piper. Memory is the worker's peak while +it holds the model. These figures come from piper's own speech fed back into +Whisper. A room microphone is harder, on `tiny` most of all. + ## Printing `print` and `fluksio.logger.info(...)` are captured in the node logs. The @@ -220,13 +344,14 @@ environment. `NODE_VENV=managed` always requests one; the Docker image uses this setting. In managed mode, the Modules manifest is installed with `uv pip sync` and -versioned with the flows. Applying it refreshes local workers. The API accepts -the same manifest: +versioned with the flows. Sync installs only the packages listed, not their +dependencies, so list those too, for example as resolved by `uv pip compile`. +Applying it refreshes local workers. The API accepts the same manifest: ```sh curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \ -H 'Content-Type: application/json' \ - -d '{"requirements": "numpy>=2\npandas\n"}' + -d '{"requirements": "numpy>=2\n"}' ``` Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an diff --git a/docs/interface/dashboards.md b/docs/interface/dashboards.md index 86053db..c3a0bb8 100644 --- a/docs/interface/dashboards.md +++ b/docs/interface/dashboards.md @@ -420,7 +420,8 @@ to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#makin The device decides what the panel's [I/O](#the-devices-own-io) port names mean. openpanel serves `gpio.`, `backlight`, `touch`, `mic` and `speaker`, lists each one's `config` in its README and ignores any other name. A browser -screen serves none of them. +screen serves none of them. The [speech recipe](../code/nodes.md#speech-to-and-from-a-panel) +turns `mic` into text and text into `speaker`. ### The device's own I/O diff --git a/docs/interface/operations.md b/docs/interface/operations.md index d332425..8a57d22 100644 --- a/docs/interface/operations.md +++ b/docs/interface/operations.md @@ -33,8 +33,10 @@ packages and Python version. There are two setups: - **Managed environment:** Docker uses a separate node environment. A superuser edits **Requirements** and presses **Apply**. Packages are synced - with `uv pip sync`; deleting a requirement removes it. The manifest is - versioned with the flows, and successful installs refresh the workers. + with `uv pip sync`; deleting a requirement removes it. Sync installs only + what is listed, so list the dependencies too, for example the output of + `uv pip compile`. The manifest is versioned with the flows, and successful + installs refresh the workers. - **Adopted environment:** a fresh pip installation inside your own virtual environment can use that environment for nodes too. Modules is read-only. Install dependencies there with pip or uv, then sync or restart Fluksio.