nodes.py had grown to 2k lines holding every integration behind a single
blanket mypy exemption. It is now a package split by the outside world
each node talks to, so the exemption shrinks to the four integration
modules; base and mlp are type-checked, which turned up a dozen missing
annotations.
The senders opened a fresh connection — and, in the MQTT case, a fresh
thread pool and event loop — for every single message. HTTP senders now
share one pooled client, and a publisher holds one broker connection for
its lifetime, fed from a bounded queue that drops the oldest value when
the broker cannot keep up.
An HTTP sender also no longer trips over a JSON reply that is not an
object: outputs are keyed by port, so a bare scalar is a valid reply with
nothing to publish.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
A node's subscription, schedule or poll loop was a bare asyncio task: one
that raised outside its own retry handling was simply gone, and the node
went on being listed as running while nothing listened any more.
Those loops now run under a supervisor that restarts them with a growing
delay and quarantines a flow that burns through five restarts in five
minutes — a flow crash-looping every second is worse than one that is
visibly stopped, and the dashboard can now say which.
The MQTT subscription loses its private five-second reconnect in the
process: one backoff policy per socket, and it belongs to the supervisor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
The API image ran four uvicorn workers, and each one built a full flow
controller — four sets of MQTT subscriptions, cron ticks and webhooks.
Runs one worker now; scaling out is the worker split, not more processes.
Adds a loop-lag watchdog and a deep /utils/health/ that fails when the
event loop is wedged or Redis is unreachable, the two failure modes a
process-alive check never sees. Autoheal restarts on that signal, behind
a compose profile because it mounts the Docker socket.
The private user-seeding routes now need an explicit opt-in rather than
just ENVIRONMENT=local, so a deployment that kept the default never
exposes them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
