Docs: speech-to-text and text-to-speech recipes for a panel's mic and speaker
faster-whisper and piper as plain Function nodes, the mic -> STT -> logic -> TTS -> speaker wiring, the Modules manifest they need (uv pip compile, av<19) and measured sizes, memory and CPU times. Modules docs now say uv pip sync installs no dependencies.
This commit is contained in:
+128
-3
@@ -155,6 +155,130 @@ and the result plays there too, just smaller. The node needs `ffmpeg` on the
|
|||||||
[`WITH_FFMPEG=true`](../reference/configuration.md#image-options) in `.env`.
|
[`WITH_FFMPEG=true`](../reference/configuration.md#image-options) in `.env`.
|
||||||
Otherwise run the node on a [remote worker](workers.md) that has it.
|
Otherwise run the node on a [remote worker](workers.md) that has it.
|
||||||
|
|
||||||
|
### Speech to and from a panel
|
||||||
|
|
||||||
|
openpanel publishes a 16 kHz mono WAV on its `mic` port after its wake word,
|
||||||
|
and plays any `audio` reference sent to its `speaker` port (WAV, MP3 or AAC).
|
||||||
|
Two Function nodes close that loop with local models.
|
||||||
|
[faster-whisper](https://github.com/SYSTRAN/faster-whisper) turns speech into
|
||||||
|
text and [piper](https://github.com/OHF-Voice/piper1-gpl) turns text into
|
||||||
|
speech.
|
||||||
|
|
||||||
|
The listening node has an `audio` input port `audio` and a `str` output port
|
||||||
|
`text`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from functools import cache
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
|
||||||
|
import fluksio
|
||||||
|
|
||||||
|
MODELS = Path.home() / ".cache" / "voice"
|
||||||
|
|
||||||
|
|
||||||
|
@cache
|
||||||
|
def _model(size):
|
||||||
|
# Downloaded on the first call, then held by the worker between calls.
|
||||||
|
return WhisperModel(
|
||||||
|
size, device="cpu", compute_type="int8", download_root=str(MODELS)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def process(audio, model="base", language="en"):
|
||||||
|
# vad_filter skips silence, which Whisper would otherwise hear as "You".
|
||||||
|
segments, _ = _model(model).transcribe(
|
||||||
|
fluksio.load_artifact(audio), language=language or None, vad_filter=True
|
||||||
|
)
|
||||||
|
return {"text": " ".join(segment.text.strip() for segment in segments)}
|
||||||
|
```
|
||||||
|
|
||||||
|
The speaking node has a `str` input port `text` and an `audio` output port
|
||||||
|
`speech`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import io
|
||||||
|
import wave
|
||||||
|
from functools import cache
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from piper import PiperVoice
|
||||||
|
from piper.download_voices import download_voice
|
||||||
|
|
||||||
|
import fluksio
|
||||||
|
|
||||||
|
MODELS = Path.home() / ".cache" / "voice"
|
||||||
|
|
||||||
|
|
||||||
|
@cache
|
||||||
|
def _voice(name):
|
||||||
|
# Downloaded on the first call, then held by the worker between calls.
|
||||||
|
MODELS.mkdir(parents=True, exist_ok=True)
|
||||||
|
download_voice(name, MODELS)
|
||||||
|
return PiperVoice.load(MODELS / f"{name}.onnx")
|
||||||
|
|
||||||
|
|
||||||
|
def process(text, voice="en_US-lessac-medium"):
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
with wave.open(buffer, "wb") as wav:
|
||||||
|
_voice(voice).synthesize_wav(text, wav)
|
||||||
|
speech = fluksio.save_artifact(buffer.getvalue(), "speech.wav", "audio/wav")
|
||||||
|
return {"speech": speech}
|
||||||
|
```
|
||||||
|
|
||||||
|
`model`, `language` and `voice` are settings. `tiny` is faster than `base` and
|
||||||
|
less accurate. An empty `language` lets Whisper guess it for each clip. `voice`
|
||||||
|
takes any name from the [piper voices](https://huggingface.co/rhasspy/piper-voices),
|
||||||
|
such as `de_DE-thorsten-medium`.
|
||||||
|
|
||||||
|
To wire the loop in a flow `home`:
|
||||||
|
|
||||||
|
1. Declare a [flow input](../concepts/flows.md#flow-inputs) `heard` of type
|
||||||
|
`audio` with origin `panel`.
|
||||||
|
2. Under the panel's [I/O](../interface/dashboards.md#the-devices-own-io), wire
|
||||||
|
`mic` as `in` to `home.heard` and `speaker` as `out` to `home.reply`.
|
||||||
|
3. Bind the listening node's `audio` to `heard` and its `text` to `said`.
|
||||||
|
4. Have your own nodes read `said` and publish an answer as a `str`, such as
|
||||||
|
`answer`.
|
||||||
|
5. Bind the speaking node's `text` to `answer` and its `speech` to `reply`.
|
||||||
|
|
||||||
|
Binding the speaking node's `text` straight to `said` makes the panel repeat
|
||||||
|
what it heard, which tests the loop before there is any logic in it.
|
||||||
|
|
||||||
|
Both packages go into the node's environment, as described in
|
||||||
|
[Packages](#packages). A managed environment installs its manifest with
|
||||||
|
`uv pip sync`, which skips dependencies, so paste the fully resolved list into
|
||||||
|
**Modules**:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
printf 'faster-whisper\nav<19\npiper-tts\n' | uv pip compile --python-version 3.13 -
|
||||||
|
```
|
||||||
|
|
||||||
|
The Docker image runs Python 3.13. `av<19` holds PyAV back, since faster-whisper
|
||||||
|
1.2.1 cannot open files with PyAV 19. In your own venv or on a remote worker,
|
||||||
|
`pip install faster-whisper "av<19" piper-tts` is enough.
|
||||||
|
|
||||||
|
The models land in `~/.cache/voice` on the machine running the node. The Docker
|
||||||
|
image keeps its home directory inside the container, so a recreated container
|
||||||
|
downloads them again. Point `MODELS` at a directory under `/data` to keep them
|
||||||
|
on the volume. An [attached worker](workers.md) starts a process per call and
|
||||||
|
loads the model each time.
|
||||||
|
|
||||||
|
What it costs, measured on a 2016 server CPU (Xeon E5-2683 v4):
|
||||||
|
|
||||||
|
| | Size | Memory | 3 s of speech, one core | All cores |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Packages | 150 MB download, 460 MB installed | | | |
|
||||||
|
| Whisper `tiny` | 78 MB | 300 MB | 1.6 s | 0.7 s |
|
||||||
|
| Whisper `base` | 148 MB | 375 MB | 2.9 s | 1.2 s |
|
||||||
|
| piper `en_US-lessac-medium` | 63 MB | 250 MB | 0.6 s | 0.3 s |
|
||||||
|
|
||||||
|
Each model downloads once, on its node's first call. Loading it takes about a
|
||||||
|
second for Whisper and four seconds for piper. Memory is the worker's peak while
|
||||||
|
it holds the model. These figures come from piper's own speech fed back into
|
||||||
|
Whisper. A room microphone is harder, on `tiny` most of all.
|
||||||
|
|
||||||
## Printing
|
## Printing
|
||||||
|
|
||||||
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
|
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
|
||||||
@@ -220,13 +344,14 @@ environment. `NODE_VENV=managed` always requests one; the Docker image uses
|
|||||||
this setting.
|
this setting.
|
||||||
|
|
||||||
In managed mode, the Modules manifest is installed with `uv pip sync` and
|
In managed mode, the Modules manifest is installed with `uv pip sync` and
|
||||||
versioned with the flows. Applying it refreshes local workers. The API accepts
|
versioned with the flows. Sync installs only the packages listed, not their
|
||||||
the same manifest:
|
dependencies, so list those too, for example as resolved by `uv pip compile`.
|
||||||
|
Applying it refreshes local workers. The API accepts the same manifest:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
|
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
|
||||||
-H 'Content-Type: application/json' \
|
-H 'Content-Type: application/json' \
|
||||||
-d '{"requirements": "numpy>=2\npandas\n"}'
|
-d '{"requirements": "numpy>=2\n"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
|
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
|
||||||
|
|||||||
@@ -420,7 +420,8 @@ to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#makin
|
|||||||
The device decides what the panel's [I/O](#the-devices-own-io) port names
|
The device decides what the panel's [I/O](#the-devices-own-io) port names
|
||||||
mean. openpanel serves `gpio.<n>`, `backlight`, `touch`, `mic` and `speaker`,
|
mean. openpanel serves `gpio.<n>`, `backlight`, `touch`, `mic` and `speaker`,
|
||||||
lists each one's `config` in its README and ignores any other name. A browser
|
lists each one's `config` in its README and ignores any other name. A browser
|
||||||
screen serves none of them.
|
screen serves none of them. The [speech recipe](../code/nodes.md#speech-to-and-from-a-panel)
|
||||||
|
turns `mic` into text and text into `speaker`.
|
||||||
|
|
||||||
### The device's own I/O
|
### The device's own I/O
|
||||||
|
|
||||||
|
|||||||
@@ -33,8 +33,10 @@ packages and Python version. There are two setups:
|
|||||||
|
|
||||||
- **Managed environment:** Docker uses a separate node environment. A
|
- **Managed environment:** Docker uses a separate node environment. A
|
||||||
superuser edits **Requirements** and presses **Apply**. Packages are synced
|
superuser edits **Requirements** and presses **Apply**. Packages are synced
|
||||||
with `uv pip sync`; deleting a requirement removes it. The manifest is
|
with `uv pip sync`; deleting a requirement removes it. Sync installs only
|
||||||
versioned with the flows, and successful installs refresh the workers.
|
what is listed, so list the dependencies too, for example the output of
|
||||||
|
`uv pip compile`. The manifest is versioned with the flows, and successful
|
||||||
|
installs refresh the workers.
|
||||||
- **Adopted environment:** a fresh pip installation inside your own virtual
|
- **Adopted environment:** a fresh pip installation inside your own virtual
|
||||||
environment can use that environment for nodes too. Modules is read-only.
|
environment can use that environment for nodes too. Modules is read-only.
|
||||||
Install dependencies there with pip or uv, then sync or restart Fluksio.
|
Install dependencies there with pip or uv, then sync or restart Fluksio.
|
||||||
|
|||||||
Reference in New Issue
Block a user