Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run node code on the venv Fluksio was installed into

Browse Source 
The workflow this serves: make a venv, install what you work with, then `pip
install fluksio` into the same one. Building a second environment beside it
was exactly wrong — the packages the nodes need are already here, and the
Modules screen was asking for them a second time.

`NODE_VENV=auto` (the default) adopts that venv. It declines in the three
cases where adopting would be wrong: `managed` says otherwise, a managed venv
already exists and may hold packages somebody installed on purpose, or the
engine is not running from a venv at all. The images set `managed`, since the
venv in them holds the app and nothing of anybody else's.

An adopted venv is never written to. `uv pip sync` makes a venv hold exactly
the manifest, so pointed at somebody's own environment it uninstalls their
work and the engine with it — `sync()` refuses outright and `reconcile()`
returns before it can be called at startup, which is where that would have
happened first. The Modules screen lists what is installed and drops its
editor; `pip` is how that environment changes.

`fluksio serve` now names the interpreter node code runs on, which is the
thing a data scientist most needs to know at that moment. `fluksio-worker`
already defaulted `--python` to its own interpreter, so a GPU box works the
same way — that was only ever undocumented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
stroblme
2026-08-24 10:35:13 +02:00
co-authored by Claude Fable 5
parent 68fa5527b1
commit fea57064f9
15 changed files with 357 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/cli.md
+1
View File
@@ -205,6 +205,7 @@ directory. The ones you are most likely to touch:
|---|---|---|
| `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this |
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
docs/code/nodes.md
+8
View File
@@ -182,6 +182,14 @@ flows. An install takes effect immediately; nothing restarts.
### Your own code as a package
!!! tip "If Fluksio is installed in the venv you work in, skip this"
Node code then runs on that environment, so your project and everything it
imports are already importable — see
[Getting started: data science](../getting-started/data-science.md). What
follows is for a Fluksio with a venv of its own, which is what a container
always has.
A manifest line can name a directory, so the project you already have is
installable like any other dependency:
docs/getting-started/data-science.md
+56 -18
View File
@@ -63,13 +63,54 @@ export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
## Your packages are already there
## Tell it about your packages
If you installed Fluksio into the environment you work in — the venv that
already has torch or numpy in it — that is the environment your nodes run on.
Nothing to declare, nothing to install twice:
Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
environment Fluksio itself is installed in — so a pin of yours can never
collide with one of ours. That venv starts empty, so the first thing to do is
say what your script imports:
```sh
python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas # what you were going to install anyway
pip install fluksio # and then this
fluksio serve
```
`fluksio serve` says which interpreter it settled on:
```text
Nodes /home/you/research/.venv/bin/python
your environment, adopted. Add packages with pip.
```
That venv is yours. Add a package the way you added the rest — `pip install
scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they
pick it up. The Modules screen lists what is installed and stays read-only,
because the alternative would be Fluksio deciding what belongs in an
environment it did not make.
!!! note "Your pins and ours share a site-packages"
The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same — pin it back, or keep Fluksio somewhere separate with
the venv of its own below.
### A venv of Fluksio's own
Sometimes you want the isolation instead: a shared installation, a container,
or an environment too precious to let a node's dependency near. Set
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
```text
Nodes /home/you/.fluksio/user-venv/bin/python
a venv of its own; the Modules screen installs into it.
```
Then the Modules screen is how packages get in — a pip manifest, installed
with `uv pip sync` and versioned alongside your flows, so what a run imported
is recorded with what it ran:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
@@ -77,26 +118,23 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
It is a pip manifest, installed with `uv pip sync`, and it is versioned
alongside your flows — so what a run imported is recorded with what it ran.
Adding a package takes effect immediately; nothing restarts.
Adding a package takes effect immediately; nothing restarts. The Docker image
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
nothing of yours — so a container is always this case.
??? note "Already have a venv you would rather not duplicate?"
!!! tip "A GPU box works the same way"
Attach it as a worker instead of reinstalling into it. Mint a token, then
point the agent at your existing interpreter:
`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with.
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name": "laptop"}'
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
--token "$WORKER_TOKEN" --labels gpu
```
Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).
Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
docs/reference/configuration.md
+12
View File
@@ -22,10 +22,22 @@ Anything already exported wins over the file.
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
| `NODE_VENV` | `auto` | which interpreter node code runs on — see below |
Set `DATA_DIR` and the rest follow. Set one explicitly and it wins — which is
what the container images do to pin everything onto `/data`.
`NODE_VENV` is the exception, being about an environment rather than a path:
| Value | What node code runs on |
|---|---|
| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. |
| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. |
| a path | that interpreter, or that venv, whatever it is. |
An installation that already has a managed venv keeps it on upgrade under
`auto`, because it may hold packages somebody installed on purpose.
!!! warning "The four files that must be on persistent storage"
`secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written