Run node code on the venv Fluksio was installed into

The workflow this serves: make a venv, install what you work with, then `pip
install fluksio` into the same one. Building a second environment beside it
was exactly wrong — the packages the nodes need are already here, and the
Modules screen was asking for them a second time.

`NODE_VENV=auto` (the default) adopts that venv. It declines in the three
cases where adopting would be wrong: `managed` says otherwise, a managed venv
already exists and may hold packages somebody installed on purpose, or the
engine is not running from a venv at all. The images set `managed`, since the
venv in them holds the app and nothing of anybody else's.

An adopted venv is never written to. `uv pip sync` makes a venv hold exactly
the manifest, so pointed at somebody's own environment it uninstalls their
work and the engine with it — `sync()` refuses outright and `reconcile()`
returns before it can be called at startup, which is where that would have
happened first. The Modules screen lists what is installed and drops its
editor; `pip` is how that environment changes.

`fluksio serve` now names the interpreter node code runs on, which is the
thing a data scientist most needs to know at that moment. `fluksio-worker`
already defaulted `--python` to its own interpreter, so a GPU box works the
same way — that was only ever undocumented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
2026-08-24 10:35:13 +02:00
co-authored by Claude Fable 5
parent 68fa5527b1
commit fea57064f9
15 changed files with 357 additions and 69 deletions
+56 -18
View File
@@ -63,13 +63,54 @@ export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
## Your packages are already there
## Tell it about your packages
If you installed Fluksio into the environment you work in — the venv that
already has torch or numpy in it — that is the environment your nodes run on.
Nothing to declare, nothing to install twice:
Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
environment Fluksio itself is installed in — so a pin of yours can never
collide with one of ours. That venv starts empty, so the first thing to do is
say what your script imports:
```sh
python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas # what you were going to install anyway
pip install fluksio # and then this
fluksio serve
```
`fluksio serve` says which interpreter it settled on:
```text
Nodes /home/you/research/.venv/bin/python
your environment, adopted. Add packages with pip.
```
That venv is yours. Add a package the way you added the rest — `pip install
scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they
pick it up. The Modules screen lists what is installed and stays read-only,
because the alternative would be Fluksio deciding what belongs in an
environment it did not make.
!!! note "Your pins and ours share a site-packages"
The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same — pin it back, or keep Fluksio somewhere separate with
the venv of its own below.
### A venv of Fluksio's own
Sometimes you want the isolation instead: a shared installation, a container,
or an environment too precious to let a node's dependency near. Set
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
```text
Nodes /home/you/.fluksio/user-venv/bin/python
a venv of its own; the Modules screen installs into it.
```
Then the Modules screen is how packages get in — a pip manifest, installed
with `uv pip sync` and versioned alongside your flows, so what a run imported
is recorded with what it ran:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
@@ -77,26 +118,23 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
It is a pip manifest, installed with `uv pip sync`, and it is versioned
alongside your flows — so what a run imported is recorded with what it ran.
Adding a package takes effect immediately; nothing restarts.
Adding a package takes effect immediately; nothing restarts. The Docker image
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
nothing of yours — so a container is always this case.
??? note "Already have a venv you would rather not duplicate?"
!!! tip "A GPU box works the same way"
Attach it as a worker instead of reinstalling into it. Mint a token, then
point the agent at your existing interpreter:
`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with.
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name": "laptop"}'
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
--token "$WORKER_TOKEN" --labels gpu
```
Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).
Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand