Run python nodes out of process, with modules of their own
User code no longer execs in the engine. A pool of persistent worker subprocesses speaks one JSON object per line; the controller installs a proxy as the node's function, so every execution path funnels through it and the pipeline is untouched. A crash costs one subprocess, a per-node timeout is a kill, and cancelling from the canvas is that same kill. The workers run a venv of the user's own on the data volume, filled from a pip manifest versioned beside the flows. Applying it retires the workers and rebuilds, so a package lands without restarting the engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
+10
-3
@@ -49,6 +49,11 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
|
||||
- [x] Secrets/credentials store for node integrations managed via the API/UI
|
||||
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
|
||||
`.env` bootstrap-only
|
||||
- [x] Python modules for node code, managed from the UI: a pip manifest versioned
|
||||
with the flows, installed with `uv pip sync` into a venv of the user's own on
|
||||
the data volume. The worker processes run that interpreter, so an install
|
||||
takes effect without restarting the engine and can never shadow the app's
|
||||
own packages
|
||||
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
|
||||
a polling coordinator that deduplicates, `x-secret` parameters the editor
|
||||
renders as a secret picker, and health reporting. Connectors are installed
|
||||
@@ -121,9 +126,11 @@ Rust, optimised for throughput. Executes nodes and distributes them across worke
|
||||
- [ ] Parallel invocation of stateless nodes over independent input sets, to
|
||||
keep I/O delay minimal (stateful I/O nodes keep serializing via the
|
||||
`synchronous` mechanism)
|
||||
- [ ] Run user Python nodes out of process. One occupies a worker thread until it
|
||||
returns today, so a runaway node cannot be bounded by a timeout nor cancelled
|
||||
from the canvas — both fall out of the isolation
|
||||
- [x] Run user Python nodes out of process: a pool of persistent worker subprocesses
|
||||
speaking one JSON object per line, entered through a proxy the controller
|
||||
installs as the node's function, so every execution path funnels through it
|
||||
unchanged. A crash costs one subprocess, a per-node timeout is a kill, and
|
||||
cancelling from the canvas is that same kill on request
|
||||
- [ ] Extract node execution from the Python prototype into a Rust engine
|
||||
- [ ] Worker distribution and load balancing across capable devices
|
||||
- [ ] Input/output validation at the node boundary
|
||||
|
||||
Reference in New Issue
Block a user