Voice nodes (openpanel V4): stt, tts, wren reply/unhandled, socket publish
- `stt`: one realtime transcription session per utterance on the speech
server's url, opened at `listening` true or the first chunk, committed at
false; a whole clip goes to /v1/audio/transcriptions. Returns at once and
publishes `partial` and `text` when the server answers.
- `tts`: one /v1/audio/speech call per sentence, cached by voice and text in
node state; a reference is marked `append` while earlier ones still play.
- wren: `reply` says what was applied, `unhandled` carries the prompt when
nothing was. The install hint names the git pin (PyPI's `wren` is another
package).
- ai/client: transcribe, speak and transcribe_stream beside stream_completion.
- A device socket publishes scalar `in` ports as {"type": "publish"} text,
in order with its binary frames.
- Pipeline.publish walks only what its values reach: a node downstream of a
consumer that published nothing no longer runs again (wren re-acted on
every microphone chunk).
This commit is contained in:
+6
-2
@@ -286,8 +286,12 @@ that family, and the whole message at most `MAX_FRAME_BYTES`. The frame is
|
||||
held in memory and published in one step, named after the message and credited
|
||||
to its device node. A frame that does not
|
||||
qualify is dropped, the reason logged once per socket, and the socket stays
|
||||
open. With `http` it uploads with `PUT /artifacts?volatile=true&name=<message>`
|
||||
and publishes with `POST /messages/<message>`. A device that is unpaired or
|
||||
open. A value of one of its `in` ports wired as `bool`, `int`, `float` or `str`
|
||||
goes as the text message `{"type": "publish", "name": "<message>", "value":
|
||||
...}`, taken in turn with the frames, so a microphone's `listening` false
|
||||
arrives after its last chunk. With `http` it uploads with
|
||||
`PUT /artifacts?volatile=true&name=<message>` and publishes with
|
||||
`POST /messages/<message>`. A device that is unpaired or
|
||||
deleted while connected is closed with 1008, at the latest with its next frame.
|
||||
|
||||
### Live view
|
||||
|
||||
+4
-1
@@ -179,7 +179,10 @@ and ARM, so nothing has to be installed into the image.
|
||||
|
||||
openpanel publishes a 16 kHz mono WAV on its `mic` port after its wake word,
|
||||
and plays any `audio` reference sent to its `speaker` port (WAV, MP3 or AAC).
|
||||
Two Function nodes close that loop with local models.
|
||||
The [Speech to text](../reference/node-types.md#speech-to-text) and
|
||||
[Text to speech](../reference/node-types.md#text-to-speech) nodes close that
|
||||
loop through a speech server on a GPU. Without one, two Function nodes close it
|
||||
with local models.
|
||||
[faster-whisper](https://github.com/SYSTRAN/faster-whisper) turns speech into
|
||||
text and [piper](https://github.com/OHF-Voice/piper1-gpl) turns text into
|
||||
speech.
|
||||
|
||||
@@ -514,8 +514,10 @@ to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#makin
|
||||
The device decides what its [port](#the-devices-own-ports) names
|
||||
mean. openpanel [reports](#what-a-device-reports) the ones it serves, such as
|
||||
`gpio.<n>`, `backlight`, `touch`, `mic` and `speaker`, with the settings each
|
||||
one reads, and ignores any other name. A browser screen serves none of them. The [speech recipe](../code/nodes.md#speech-to-and-from-a-panel)
|
||||
turns `mic` into text and text into `speaker`.
|
||||
one reads, and ignores any other name. A browser screen serves none of them. The
|
||||
[Speech to text](../reference/node-types.md#speech-to-text) and
|
||||
[Text to speech](../reference/node-types.md#text-to-speech) nodes turn `mic`
|
||||
into text and text into `speaker`, and `caption` shows what is being heard.
|
||||
|
||||
### The device's own ports
|
||||
|
||||
|
||||
@@ -344,7 +344,9 @@ use a [batch flow](../concepts/runs.md) and a Function node.
|
||||
|
||||
[Wren](https://github.com/stroblme/wren) is an optional local language model
|
||||
for device commands. Install it in the engine's environment with
|
||||
`pip install wren`; the node reports a missing dependency if it is absent.
|
||||
`pip install git+https://github.com/stroblme/wren@eae4d6ff9ded68e36624972fd3f4a8f8d210b0c2`.
|
||||
The `wren` package on PyPI is an unrelated one. Without the model the node
|
||||
reports a missing dependency.
|
||||
|
||||
| Input | Type | Notes |
|
||||
|---|---|---|
|
||||
@@ -353,7 +355,8 @@ for device commands. Install it in the engine's environment with
|
||||
| `outdoor` | `float` or `int` | °C; 20 without it, clamped to −5…30 |
|
||||
| `dark` | `bool` | without it, dark before 07:00 and from 18:00 |
|
||||
|
||||
The outputs are `actions`, a list of records, and any number of **targets**. A
|
||||
The outputs are `actions`, a list of records, `reply`, `unhandled`, and any
|
||||
number of **targets**. A
|
||||
target is an output named `<area>_<kind>` and bound to the message its device
|
||||
listens on: `living_room_setpoint` bound to `home.setpoint` is the living
|
||||
room's heating. The wiring is the allowlist, so the model can write nothing
|
||||
@@ -393,6 +396,10 @@ Each record in `actions` contains `message`, `value`, `applied`, `delay_s`,
|
||||
`call`, and `confidence`. Unwired calls have an empty `message` and are not
|
||||
applied. Keep `apply=false` while checking how the model interprets commands.
|
||||
|
||||
`reply` is one sentence saying what was applied, such as "Kitchen light on."
|
||||
or "Garage light on in 5 minutes.". When nothing was applied, `unhandled`
|
||||
carries the prompt instead, for something else to answer.
|
||||
|
||||
A call with a delay ("in 10m") is held by the engine's work queue, like the
|
||||
Debounce node's second value: it survives a restart where the queue is in
|
||||
Redis, and not otherwise.
|
||||
@@ -408,6 +415,57 @@ They are reported but cannot receive writes unless wired.
|
||||
|
||||
The chat's [Control tab](../code/agents.md#control) sends commands by hand.
|
||||
|
||||
### Speech to text
|
||||
|
||||
**`stt`** — turn a microphone's audio into text, through a speech server.
|
||||
|
||||
| Input | Type | Notes |
|
||||
|---|---|---|
|
||||
| `audio` | `audio` | 16 kHz mono PCM16 chunks (`audio/L16`), or a whole clip such as a WAV |
|
||||
| `listening` | `bool` | true while the device listens; false ends the utterance |
|
||||
| `url` | `str` | the speech server, from a [Service](#service) node; read, never waited for |
|
||||
|
||||
| Output | Type | Notes |
|
||||
|---|---|---|
|
||||
| `partial` | `str` | the words agreed on so far, as they come, then the final text |
|
||||
| `text` | `str` | what was said, once per utterance |
|
||||
|
||||
The server is any that speaks OpenAI's audio API, such as `parrot serve`. Each
|
||||
utterance is one realtime transcription session (`{url}/v1/realtime`), opened
|
||||
when `listening` turns true or at the first chunk and committed when it turns
|
||||
false. A clip is sent whole to `{url}/v1/audio/transcriptions`. The node
|
||||
returns at once and publishes when the server answers, since a device's next
|
||||
chunk waits for it. An utterance that fails, or gets no chunk for 10 seconds,
|
||||
publishes nothing and is logged.
|
||||
|
||||
`listening` has to hold a value before a chunk wakes the node. Declare it as a
|
||||
flow input with the initial value `false` when a device may never send it,
|
||||
such as one that sends whole clips.
|
||||
|
||||
### Text to speech
|
||||
|
||||
**`tts`** — speak text as audio for a speaker, through a speech server.
|
||||
|
||||
| Input | Type | Notes |
|
||||
|---|---|---|
|
||||
| `text` | `str` | what to say, a whole reply or one sentence at a time |
|
||||
| `url` | `str` | the speech server, as for Speech to text |
|
||||
|
||||
| Output | Type | Notes |
|
||||
|---|---|---|
|
||||
| `speech` | `audio` | one WAV reference per sentence |
|
||||
|
||||
| Setting | Default | Notes |
|
||||
|---|---|---|
|
||||
| `voice` | `af_heart` | one of Kokoro's voices, such as `bm_george` |
|
||||
|
||||
Each sentence is one call to `{url}/v1/audio/speech` and published as soon as
|
||||
it is spoken. A sentence spoken before in the same voice is reused rather than
|
||||
spoken again, for the last 256 sentences. A reference carries `"append": true`
|
||||
while the sentences before it are still playing, which queues it on a panel's
|
||||
`speaker`. The first sentence of a reply carries none and cuts off whatever
|
||||
was playing.
|
||||
|
||||
## Devices
|
||||
|
||||
### Device
|
||||
|
||||
Reference in New Issue
Block a user