Run node code on the venv Fluksio was installed into

The workflow this serves: make a venv, install what you work with, then `pip
install fluksio` into the same one. Building a second environment beside it
was exactly wrong — the packages the nodes need are already here, and the
Modules screen was asking for them a second time.

`NODE_VENV=auto` (the default) adopts that venv. It declines in the three
cases where adopting would be wrong: `managed` says otherwise, a managed venv
already exists and may hold packages somebody installed on purpose, or the
engine is not running from a venv at all. The images set `managed`, since the
venv in them holds the app and nothing of anybody else's.

An adopted venv is never written to. `uv pip sync` makes a venv hold exactly
the manifest, so pointed at somebody's own environment it uninstalls their
work and the engine with it — `sync()` refuses outright and `reconcile()`
returns before it can be called at startup, which is where that would have
happened first. The Modules screen lists what is installed and drops its
editor; `pip` is how that environment changes.

`fluksio serve` now names the interpreter node code runs on, which is the
thing a data scientist most needs to know at that moment. `fluksio-worker`
already defaulted `--python` to its own interpreter, so a GPU box works the
same way — that was only ever undocumented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
2026-08-24 10:35:13 +02:00
co-authored by Claude Fable 5
parent 68fa5527b1
commit fea57064f9
15 changed files with 357 additions and 69 deletions
+6
View File
@@ -20,6 +20,12 @@ WORKDIR /app/
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment
ENV PATH="/app/.venv/bin:$PATH"
# /app/.venv holds the app and nothing of anybody else's, so there is nothing
# here to adopt: node code gets a venv of its own on the data volume, which is
# what the Modules screen installs into. A `pip install fluksio` into an
# environment somebody already works in is the case that adopts instead.
ENV NODE_VENV=managed
# Install dependencies
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#intermediate-layers
RUN --mount=type=cache,target=/root/.cache/uv \
+13
View File
@@ -191,11 +191,24 @@ def cmd_serve(args: argparse.Namespace) -> int:
import uvicorn
from fluksio.flow import modules
from fluksio.main import app
config = cloud_config.load()
_say(f"Fluksio {fluksio.__version__} — data in {data_dir}")
_say(f" API http://{args.host}:{args.port}{'/api/v1'}")
# Which environment node code imports from is the thing a data scientist
# most needs to know at this moment, and the answer differs depending on
# how Fluksio was installed. Saying it costs one line.
if modules.adopted() is not None:
_say(f" Nodes {modules.venv_python()}")
_say(" your environment, adopted. Add packages with pip.")
else:
# Not `venv_python()`: the managed venv is built when the engine comes
# up, which is after this prints, and until then that would answer with
# whatever interpreter happens to be running this.
_say(f" Nodes {modules.venv_dir() / 'bin' / 'python'}")
_say(" a venv of its own; the Modules screen installs into it.")
if config is not None:
_say(f" Portal {config.portal_url}, installation {config.installation_id}")
_say(" The dashboard is served by the portal; nothing is served here.")
+6
View File
@@ -70,6 +70,12 @@ class Settings(BaseSettings):
# alerting is: where a screen hangs is the deployment's concern rather than
# any one dashboard's.
PANELS_FILE: Path = Path("flow-data/panels.json")
# Which interpreter node code runs on. "auto" adopts the venv the engine
# was installed into, when it was installed into one and there is no venv
# of its own to lose — which is the `pip install fluksio` beside your own
# packages case. "managed" always builds a separate one, which is what a
# container wants. A path names an interpreter outright.
NODE_VENV: str = "auto"
# The MCP endpoint, and the OAuth server agents authenticate against. Off
# until someone asks for it: it opens client registration to the network.
MCP_ENABLED: bool = False
+81 -3
View File
@@ -9,6 +9,15 @@ and the worker processes that import them need nothing from the app.
``uv pip sync`` rather than install, so a line taken out of the manifest is
uninstalled. The manifest is written only after a sync succeeds, which is all
the rollback a failed resolve needs.
None of that applies to an *adopted* venv. A data scientist makes a venv,
installs what they work with, and then installs Fluksio into it too — at which
point building a second environment beside it is exactly wrong: the packages
the nodes need are already here. So when the engine is running from a venv of
somebody else's, node code runs on it. That venv is theirs: `uv pip sync` is
never pointed at it, because sync means "hold exactly this" and would uninstall
their work along with the engine. It is read-only here, and `pip` is how they
change it.
"""
from __future__ import annotations
@@ -51,6 +60,39 @@ def uv_bin() -> str:
return shutil.which("uv") or "uv"
def adopted() -> Path | None:
"""The venv this engine was installed into, when node code should use it.
``None`` means the managed venv — one the engine builds and owns. The three
ways that is the answer, in order:
* ``NODE_VENV=managed`` says so. A container sets this: its venv holds the
app and nothing of anybody's, so there is nothing to adopt.
* There is already a managed venv. It may have packages in it that
somebody installed on purpose, and an upgrade must not take them away.
* The engine is not running from a venv at all.
Anything else — ``pip install fluksio`` into the environment you already
work in — is the case this exists for.
"""
setting = settings.NODE_VENV.strip()
if setting == "managed":
return None
if setting and setting != "auto":
return Path(setting)
if (VENV_DIR / "bin" / "python").exists():
return None
prefix = Path(sys.prefix)
if prefix == Path(sys.base_prefix) or prefix == VENV_DIR:
return None
return prefix
def venv_dir() -> Path:
"""Whichever venv node code runs from, adopted or managed."""
return adopted() or VENV_DIR
def venv_python() -> str:
"""The interpreter node code runs on.
@@ -58,6 +100,12 @@ def venv_python() -> str:
could not build one still runs python nodes, it just cannot add packages
to them.
"""
root = adopted()
if root is not None:
# NODE_VENV may name a venv or the interpreter inside one; both are
# useful things to be handed, and they are told apart by shape.
inner = root / "bin" / "python"
return str(inner if inner.exists() or root.is_dir() else root)
path = VENV_DIR / "bin" / "python"
return str(path) if path.exists() else sys.executable
@@ -73,6 +121,8 @@ def _digest(requirements: str) -> str:
def ensure_venv() -> None:
"""Create the venv if it is missing or its interpreter has gone."""
if adopted() is not None:
return
if (VENV_DIR / "bin" / "python").exists():
return
VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
@@ -100,6 +150,16 @@ def sync(requirements: str) -> tuple[bool, str]:
other — the caller answers 400 with what came back, which is the only thing
a person can act on.
"""
if adopted() is not None:
# The one thing this must never do. `uv pip sync` makes a venv hold
# exactly the manifest, so pointed at somebody's own environment it
# uninstalls their packages — and the engine with them.
return False, (
"These packages are not Fluksio's to install: node code runs on "
f"{venv_python()}, the environment Fluksio itself was installed "
"into. Install into it with pip or uv, or set NODE_VENV=managed "
"for a venv the engine owns."
)
manifest = ""
try:
ensure_venv()
@@ -143,6 +203,12 @@ def reconcile(store: FlowStore) -> None:
import error each time they execute.
"""
try:
root = adopted()
if root is not None:
# Nothing to reconcile: the environment is somebody else's, and
# bringing it "in line with the manifest" would empty it.
logger.info("Node code runs on the adopted venv at %s", root)
return
# First, and unconditionally: the workers are started against this
# interpreter, and it has to be the venv's one before anything is
# installed into it, not after.
@@ -162,15 +228,20 @@ def reconcile(store: FlowStore) -> None:
def info(store: FlowStore) -> ModulesInfo:
"""What is installed, what was asked for, and whether the two agree."""
requirements = store.read_requirements()
config = VENV_DIR / "pyvenv.cfg"
root = venv_dir()
is_adopted = adopted() is not None
version = ""
config = root / "pyvenv.cfg"
if config.exists():
for line in config.read_text().splitlines():
if line.startswith("version"):
version = line.split("=", 1)[1].strip()
if not version:
version = ".".join(str(part) for part in sys.version_info[:3])
packages: list[ModulePackage] = []
site = sorted(VENV_DIR.glob("lib/python*/site-packages"))
site = sorted(root.glob("lib/python*/site-packages"))
if site:
packages = sorted(
(
@@ -182,13 +253,20 @@ def info(store: FlowStore) -> ModulesInfo:
return ModulesInfo(
python_version=version,
venv_path=str(VENV_DIR),
venv_path=str(root),
requirements=requirements,
packages=packages,
# An adopted venv is never "out of step": the manifest does not
# describe it, so there is nothing for it to disagree with.
applied=(
True
if is_adopted
else (
_marker().read_text() == _digest(requirements)
if _marker().exists()
# Nothing asked for and nothing installed is already in step.
else not requirements.strip()
)
),
adopted=is_adopted,
)
+4
View File
@@ -242,6 +242,10 @@ class ModulesInfo(BaseModel):
packages: list[ModulePackage] = Field(default_factory=list)
#: Whether what is installed matches the manifest.
applied: bool = False
#: True when node code runs on the venv Fluksio was installed into rather
#: than one the engine built. That venv belongs to whoever made it, so the
#: manifest does not describe it and nothing here installs into it.
adopted: bool = False
class ApplyRequest(BaseModel):
+5
View File
@@ -31,6 +31,11 @@ os.environ.setdefault("EMAILS_FROM_EMAIL", "noreply@example.com")
# builds a TestClient — and so a lifespan — per test module. Tests that want the
# endpoint mount it themselves.
os.environ["MCP_ENABLED"] = "false"
# The suite runs from a venv of its own, which "auto" would adopt as if it were
# a data scientist's — so node code would run on the checkout's environment and
# the module tests would have nothing of their own to build. The tests that
# cover adoption ask for it explicitly.
os.environ["NODE_VENV"] = "managed"
# The private seeding endpoints are opt-in; the suite is one of the two places
# (with the dev stack) where they are meant to work.
os.environ["PRIVATE_API_ENABLED"] = "true"
+87
View File
@@ -75,3 +75,90 @@ def test_a_manifest_that_does_not_resolve_leaves_the_venv_alone(venv: Path):
assert output
# The marker still describes the manifest that actually installed.
assert (venv / ".applied").read_text() == modules._digest("")
# ---------------------------------------------------------------------------
# Adopting the venv Fluksio was installed into
#
# The workflow this exists for: make a venv, install what you work with, then
# `pip install fluksio` into the same one. Building a second environment beside
# it would leave node code unable to import the packages that are the point.
# ---------------------------------------------------------------------------
@pytest.fixture
def their_venv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Pretend the engine is running from a venv somebody else made."""
directory = tmp_path / "research"
(directory / "bin").mkdir(parents=True)
(directory / "bin" / "python").write_text("")
monkeypatch.setattr(modules.sys, "prefix", str(directory))
monkeypatch.setattr(modules.sys, "base_prefix", str(tmp_path / "usr"))
# Undoes the suite-wide "managed" in tests/__init__.py.
monkeypatch.setattr(modules.settings, "NODE_VENV", "auto")
return directory
def test_a_venv_of_your_own_is_adopted(venv: Path, their_venv: Path):
assert modules.adopted() == their_venv
assert modules.venv_python() == str(their_venv / "bin" / "python")
def test_an_adopted_venv_is_never_synced(venv: Path, their_venv: Path):
"""The whole hazard: sync means "hold exactly this", so it would empty it."""
ok, output = modules.sync("numpy>=2")
assert ok is False
assert "not Fluksio's to install" in output
# Nothing was built beside it either.
assert not venv.exists()
def test_reconcile_leaves_an_adopted_venv_alone(
venv: Path, their_venv: Path, tmp_path: Path
):
store = FlowStore(tmp_path / "flows")
store.write_requirements("numpy>=2\n")
modules.reconcile(store)
assert not venv.exists()
assert modules.venv_python() == str(their_venv / "bin" / "python")
def test_a_venv_the_engine_already_owns_is_kept(venv: Path, their_venv: Path):
"""An upgrade must not take away packages somebody installed on purpose."""
(venv / "bin").mkdir(parents=True)
(venv / "bin" / "python").write_text("")
assert modules.adopted() is None
assert modules.venv_python() == str(venv / "bin" / "python")
def test_managed_is_what_a_container_asks_for(
venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(modules.settings, "NODE_VENV", "managed")
assert modules.adopted() is None
def test_an_interpreter_can_be_named_outright(
venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(modules.settings, "NODE_VENV", "/opt/conda/bin/python")
assert modules.venv_python() == "/opt/conda/bin/python"
def test_info_describes_an_adopted_venv(venv: Path, their_venv: Path, tmp_path: Path):
store = FlowStore(tmp_path / "flows")
store.write_requirements("numpy>=2\n")
information = modules.info(store)
assert information.adopted is True
assert information.venv_path == str(their_venv)
assert information.python_version
# The manifest does not describe it, so it cannot be out of step with it.
assert information.applied is True
+1
View File
@@ -205,6 +205,7 @@ directory. The ones you are most likely to touch:
|---|---|---|
| `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this |
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
+8
View File
@@ -182,6 +182,14 @@ flows. An install takes effect immediately; nothing restarts.
### Your own code as a package
!!! tip "If Fluksio is installed in the venv you work in, skip this"
Node code then runs on that environment, so your project and everything it
imports are already importable — see
[Getting started: data science](../getting-started/data-science.md). What
follows is for a Fluksio with a venv of its own, which is what a container
always has.
A manifest line can name a directory, so the project you already have is
installable like any other dependency:
+56 -18
View File
@@ -63,13 +63,54 @@ export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
## Your packages are already there
## Tell it about your packages
If you installed Fluksio into the environment you work in — the venv that
already has torch or numpy in it — that is the environment your nodes run on.
Nothing to declare, nothing to install twice:
Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
environment Fluksio itself is installed in — so a pin of yours can never
collide with one of ours. That venv starts empty, so the first thing to do is
say what your script imports:
```sh
python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas # what you were going to install anyway
pip install fluksio # and then this
fluksio serve
```
`fluksio serve` says which interpreter it settled on:
```text
Nodes /home/you/research/.venv/bin/python
your environment, adopted. Add packages with pip.
```
That venv is yours. Add a package the way you added the rest — `pip install
scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they
pick it up. The Modules screen lists what is installed and stays read-only,
because the alternative would be Fluksio deciding what belongs in an
environment it did not make.
!!! note "Your pins and ours share a site-packages"
The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same — pin it back, or keep Fluksio somewhere separate with
the venv of its own below.
### A venv of Fluksio's own
Sometimes you want the isolation instead: a shared installation, a container,
or an environment too precious to let a node's dependency near. Set
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
```text
Nodes /home/you/.fluksio/user-venv/bin/python
a venv of its own; the Modules screen installs into it.
```
Then the Modules screen is how packages get in — a pip manifest, installed
with `uv pip sync` and versioned alongside your flows, so what a run imported
is recorded with what it ran:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
@@ -77,26 +118,23 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
It is a pip manifest, installed with `uv pip sync`, and it is versioned
alongside your flows — so what a run imported is recorded with what it ran.
Adding a package takes effect immediately; nothing restarts.
Adding a package takes effect immediately; nothing restarts. The Docker image
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
nothing of yours — so a container is always this case.
??? note "Already have a venv you would rather not duplicate?"
!!! tip "A GPU box works the same way"
Attach it as a worker instead of reinstalling into it. Mint a token, then
point the agent at your existing interpreter:
`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with.
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name": "laptop"}'
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
--token "$WORKER_TOKEN" --labels gpu
```
Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).
Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
+12
View File
@@ -22,10 +22,22 @@ Anything already exported wins over the file.
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
| `NODE_VENV` | `auto` | which interpreter node code runs on — see below |
Set `DATA_DIR` and the rest follow. Set one explicitly and it wins — which is
what the container images do to pin everything onto `/data`.
`NODE_VENV` is the exception, being about an environment rather than a path:
| Value | What node code runs on |
|---|---|
| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. |
| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. |
| a path | that interpreter, or that venv, whatever it is. |
An installation that already has a managed venv keeps it on upgrade under
`auto`, because it may hold packages somebody installed on purpose.
!!! warning "The four files that must be on persistent storage"
`secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written
+5
View File
@@ -1521,6 +1521,11 @@ export const ModulesInfoSchema = {
type: 'boolean',
title: 'Applied',
default: false
},
adopted: {
type: 'boolean',
title: 'Adopted',
default: false
}
},
type: 'object',
+1
View File
@@ -583,6 +583,7 @@ export type ModulesInfo = {
requirements?: string;
packages?: Array<ModulePackage>;
applied?: boolean;
adopted?: boolean;
};
export type NewPassword = {
+22 -5
View File
@@ -62,19 +62,23 @@ function Modules() {
const requirements = draft ?? stored
const packages = data?.packages ?? []
const dirty = requirements !== stored
// Fluksio was installed into an environment somebody else made, and node
// code runs on it. Installing from here would mean `uv pip sync`, which
// holds a venv to exactly one list — and would uninstall their work.
const adopted = data?.adopted ?? false
return (
<div className="grid gap-6">
<div className="grid gap-1">
<h1 className="text-2xl">Modules</h1>
<p className="text-sm text-muted-foreground">
The Python packages your function nodes can import. They are installed
into an environment of their own, separate from the engine's, so a
version you pin here is the one your code gets. The list is kept with
your flows, so a rebuilt deployment installs the same set again.
{adopted
? "The Python packages your function nodes can import. Fluksio was installed into an environment you already had, so that is the one your nodes run on — and it is yours to change, with pip or uv rather than from here."
: "The Python packages your function nodes can import. They are installed into an environment of their own, separate from the engine's, so a version you pin here is the one your code gets. The list is kept with your flows, so a rebuilt deployment installs the same set again."}
</p>
</div>
{adopted ? null : (
<section className="grid gap-3">
<h2 className={SECTION}>Requirements</h2>
<textarea
@@ -112,14 +116,27 @@ function Modules() {
</pre>
) : null}
</section>
)}
<section className="grid gap-3">
<h2 className={SECTION}>Installed</h2>
<p className="text-sm text-muted-foreground">
<p className="text-sm text-muted-foreground" data-testid="modules-venv">
{data
? `Python ${data.python_version || "unknown"} at ${data.venv_path}`
: "Reading the environment…"}
</p>
{adopted ? (
<p
className="text-sm text-muted-foreground"
data-testid="modules-adopted"
>
Adopted this is the environment Fluksio itself is installed in.
Add a package the way you added the rest:{" "}
<span className="font-mono">pip install </span>, then sync or
restart so the workers pick it up. For a venv the engine owns
instead, set <span className="font-mono">NODE_VENV=managed</span>.
</p>
) : null}
{packages.length === 0 ? (
<p className="text-sm text-muted-foreground">
Nothing installed yet node code has the standard library.
+9 -2
View File
@@ -4,7 +4,11 @@ Point it at the engine on the box with the GPU and it dials in::
pip install fluksio-worker
fluksio-worker --url wss://api.example.com/api/v1/workers/attach \\
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu --python /opt/venv/bin/python
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
Install it into the environment the training code already runs in and node
code runs on that: ``--python`` defaults to the interpreter this was started
with. Point it elsewhere only when the two are meant to differ.
It connects *out*, so the engine needs no route back and nothing has to expose
Redis. What it then does is what the engine's own worker pool does: hold a few
@@ -274,7 +278,10 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument(
"--python",
default=sys.executable,
help="the interpreter node code runs on; point it at the venv with torch",
help=(
"the interpreter node code runs on (default: the one running this, "
"so installing into the venv with torch in it is enough)"
),
)
parser.add_argument("--parallel", type=int, default=1)
parser.add_argument("--artifact-url", default="")