Run python nodes out of process, with modules of their own

User code no longer execs in the engine. A pool of persistent worker
subprocesses speaks one JSON object per line; the controller installs a
proxy as the node's function, so every execution path funnels through it
and the pipeline is untouched. A crash costs one subprocess, a per-node
timeout is a kill, and cancelling from the canvas is that same kill.

The workers run a venv of the user's own on the data volume, filled from
a pip manifest versioned beside the flows. Applying it retires the
workers and rebuilds, so a package lands without restarting the engine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
2026-08-16 21:43:36 +02:00
co-authored by Claude Fable 5
parent 979c9d3c1f
commit f300c43f3a
27 changed files with 1536 additions and 46 deletions
+10 -3
View File
@@ -49,6 +49,11 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
- [x] Secrets/credentials store for node integrations managed via the API/UI
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
`.env` bootstrap-only
- [x] Python modules for node code, managed from the UI: a pip manifest versioned
with the flows, installed with `uv pip sync` into a venv of the user's own on
the data volume. The worker processes run that interpreter, so an install
takes effect without restarting the engine and can never shadow the app's
own packages
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
a polling coordinator that deduplicates, `x-secret` parameters the editor
renders as a secret picker, and health reporting. Connectors are installed
@@ -121,9 +126,11 @@ Rust, optimised for throughput. Executes nodes and distributes them across worke
- [ ] Parallel invocation of stateless nodes over independent input sets, to
keep I/O delay minimal (stateful I/O nodes keep serializing via the
`synchronous` mechanism)
- [ ] Run user Python nodes out of process. One occupies a worker thread until it
returns today, so a runaway node cannot be bounded by a timeout nor cancelled
from the canvas — both fall out of the isolation
- [x] Run user Python nodes out of process: a pool of persistent worker subprocesses
speaking one JSON object per line, entered through a proxy the controller
installs as the node's function, so every execution path funnels through it
unchanged. A crash costs one subprocess, a per-node timeout is a kill, and
cancelling from the canvas is that same kill on request
- [ ] Extract node execution from the Python prototype into a Rust engine
- [ ] Worker distribution and load balancing across capable devices
- [ ] Input/output validation at the node boundary