Run python nodes out of process, with modules of their own

User code no longer execs in the engine. A pool of persistent worker
subprocesses speaks one JSON object per line; the controller installs a
proxy as the node's function, so every execution path funnels through it
and the pipeline is untouched. A crash costs one subprocess, a per-node
timeout is a kill, and cancelling from the canvas is that same kill.

The workers run a venv of the user's own on the data volume, filled from
a pip manifest versioned beside the flows. Applying it retires the
workers and rebuilds, so a package lands without restarting the engine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
2026-08-16 21:43:36 +02:00
co-authored by Claude Fable 5
parent 979c9d3c1f
commit f300c43f3a
27 changed files with 1536 additions and 46 deletions
+14
View File
@@ -11,6 +11,12 @@ should reopen it.
## Open
### To be sorted
- INFRA: merge the philosophy statement at the beginning of vision.md into the rest of the document. Dissolve the decision dates and fold the decisions into a clean structure
- BUG/UI mobile friendly support is degraded: 1) toolbar in the "Flows" viewport extend mobile viewport width 2) position of nodes should never be static; always adjust such that there are as few as possible overlaps and direction is left to right (desktop) or top to bottom (mobile) 3) Dashboard view is not mobile friendly at all; as dashboard design is infeasible on mobile, render all widgets in a vertically stacked order. This allows to inspect each widget and make changes. Layout changes are not a feature on mobile 4) the home view is not responsive; all items shown there should re-order on mobile such that no scrollbars appear
### Connector write paths
Needs someone watching the real hardware, so it is not a background task. This
@@ -30,6 +36,14 @@ is what M4 still waits on, together with porting the flows.
- CHORE/FLOW: `_to_messages` keeps its `if not retval: return None` guard ahead of the new type check, so a falsy non-dict return (`0`, `""`, `[]`) is still silently "no output" rather than the named error. Deliberate for now; worth a decision.
- CHORE/FLOW: `WorkItem.kind == "node"` ("executes exactly one node") was documented but never implemented. If a run-one-node item is wanted, it still needs writing.
### Out-of-process nodes and modules
- BUG/API: `POST /flows/{name}/nodes/{node_id}/trigger` answers 500 when the node's code raises, because the inline trigger path runs `Node.__call__` rather than `Pipeline._execute_node` and nothing catches it. Predates the worker pool, which only made it easier to hit; the person waiting on the response should get the node's error, not a stack trace in the server log.
- CHORE/FLOW: `PythonWorkerPool._running` is keyed by node id and last-wins, so two concurrent runs of one node mean cancel kills the newest. Key by run id once M5's run records exist.
- CHORE/FLOW: `compile_check` sends the *draft* source under the running node's cache key, so the worker recompiles the published source on its next call. Correct, but one wasted compile per save on a busy node.
- FEAT/API: `POST /modules/apply` rebuilds the whole pipeline so a node that could not import its package stops being red. That resubscribes every MQTT node in the deployment; a targeted rebuild of the flows that actually failed to load would be gentler.
- CHORE/FLOW: a node's return value now round-trips through JSON, so tuples arrive downstream as lists and anything non-JSON is an explicit error. That is the message contract, but flows written before this may notice.
### Dashboard follow-ups
- BUG/UI: ensure dashboard wallpanel (read-only) links hot reload automatically on dashboard changes