Run node code on the venv Fluksio was installed into
The workflow this serves: make a venv, install what you work with, then `pip install fluksio` into the same one. Building a second environment beside it was exactly wrong — the packages the nodes need are already here, and the Modules screen was asking for them a second time. `NODE_VENV=auto` (the default) adopts that venv. It declines in the three cases where adopting would be wrong: `managed` says otherwise, a managed venv already exists and may hold packages somebody installed on purpose, or the engine is not running from a venv at all. The images set `managed`, since the venv in them holds the app and nothing of anybody else's. An adopted venv is never written to. `uv pip sync` makes a venv hold exactly the manifest, so pointed at somebody's own environment it uninstalls their work and the engine with it — `sync()` refuses outright and `reconcile()` returns before it can be called at startup, which is where that would have happened first. The Modules screen lists what is installed and drops its editor; `pip` is how that environment changes. `fluksio serve` now names the interpreter node code runs on, which is the thing a data scientist most needs to know at that moment. `fluksio-worker` already defaulted `--python` to its own interpreter, so a GPU box works the same way — that was only ever undocumented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
@@ -20,6 +20,12 @@ WORKDIR /app/
|
||||
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
|
||||
# /app/.venv holds the app and nothing of anybody else's, so there is nothing
|
||||
# here to adopt: node code gets a venv of its own on the data volume, which is
|
||||
# what the Modules screen installs into. A `pip install fluksio` into an
|
||||
# environment somebody already works in is the case that adopts instead.
|
||||
ENV NODE_VENV=managed
|
||||
|
||||
# Install dependencies
|
||||
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#intermediate-layers
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
|
||||
@@ -191,11 +191,24 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
||||
|
||||
import uvicorn
|
||||
|
||||
from fluksio.flow import modules
|
||||
from fluksio.main import app
|
||||
|
||||
config = cloud_config.load()
|
||||
_say(f"Fluksio {fluksio.__version__} — data in {data_dir}")
|
||||
_say(f" API http://{args.host}:{args.port}{'/api/v1'}")
|
||||
# Which environment node code imports from is the thing a data scientist
|
||||
# most needs to know at this moment, and the answer differs depending on
|
||||
# how Fluksio was installed. Saying it costs one line.
|
||||
if modules.adopted() is not None:
|
||||
_say(f" Nodes {modules.venv_python()}")
|
||||
_say(" your environment, adopted. Add packages with pip.")
|
||||
else:
|
||||
# Not `venv_python()`: the managed venv is built when the engine comes
|
||||
# up, which is after this prints, and until then that would answer with
|
||||
# whatever interpreter happens to be running this.
|
||||
_say(f" Nodes {modules.venv_dir() / 'bin' / 'python'}")
|
||||
_say(" a venv of its own; the Modules screen installs into it.")
|
||||
if config is not None:
|
||||
_say(f" Portal {config.portal_url}, installation {config.installation_id}")
|
||||
_say(" The dashboard is served by the portal; nothing is served here.")
|
||||
|
||||
@@ -70,6 +70,12 @@ class Settings(BaseSettings):
|
||||
# alerting is: where a screen hangs is the deployment's concern rather than
|
||||
# any one dashboard's.
|
||||
PANELS_FILE: Path = Path("flow-data/panels.json")
|
||||
# Which interpreter node code runs on. "auto" adopts the venv the engine
|
||||
# was installed into, when it was installed into one and there is no venv
|
||||
# of its own to lose — which is the `pip install fluksio` beside your own
|
||||
# packages case. "managed" always builds a separate one, which is what a
|
||||
# container wants. A path names an interpreter outright.
|
||||
NODE_VENV: str = "auto"
|
||||
# The MCP endpoint, and the OAuth server agents authenticate against. Off
|
||||
# until someone asks for it: it opens client registration to the network.
|
||||
MCP_ENABLED: bool = False
|
||||
|
||||
@@ -9,6 +9,15 @@ and the worker processes that import them need nothing from the app.
|
||||
``uv pip sync`` rather than install, so a line taken out of the manifest is
|
||||
uninstalled. The manifest is written only after a sync succeeds, which is all
|
||||
the rollback a failed resolve needs.
|
||||
|
||||
None of that applies to an *adopted* venv. A data scientist makes a venv,
|
||||
installs what they work with, and then installs Fluksio into it too — at which
|
||||
point building a second environment beside it is exactly wrong: the packages
|
||||
the nodes need are already here. So when the engine is running from a venv of
|
||||
somebody else's, node code runs on it. That venv is theirs: `uv pip sync` is
|
||||
never pointed at it, because sync means "hold exactly this" and would uninstall
|
||||
their work along with the engine. It is read-only here, and `pip` is how they
|
||||
change it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -51,6 +60,39 @@ def uv_bin() -> str:
|
||||
return shutil.which("uv") or "uv"
|
||||
|
||||
|
||||
def adopted() -> Path | None:
|
||||
"""The venv this engine was installed into, when node code should use it.
|
||||
|
||||
``None`` means the managed venv — one the engine builds and owns. The three
|
||||
ways that is the answer, in order:
|
||||
|
||||
* ``NODE_VENV=managed`` says so. A container sets this: its venv holds the
|
||||
app and nothing of anybody's, so there is nothing to adopt.
|
||||
* There is already a managed venv. It may have packages in it that
|
||||
somebody installed on purpose, and an upgrade must not take them away.
|
||||
* The engine is not running from a venv at all.
|
||||
|
||||
Anything else — ``pip install fluksio`` into the environment you already
|
||||
work in — is the case this exists for.
|
||||
"""
|
||||
setting = settings.NODE_VENV.strip()
|
||||
if setting == "managed":
|
||||
return None
|
||||
if setting and setting != "auto":
|
||||
return Path(setting)
|
||||
if (VENV_DIR / "bin" / "python").exists():
|
||||
return None
|
||||
prefix = Path(sys.prefix)
|
||||
if prefix == Path(sys.base_prefix) or prefix == VENV_DIR:
|
||||
return None
|
||||
return prefix
|
||||
|
||||
|
||||
def venv_dir() -> Path:
|
||||
"""Whichever venv node code runs from, adopted or managed."""
|
||||
return adopted() or VENV_DIR
|
||||
|
||||
|
||||
def venv_python() -> str:
|
||||
"""The interpreter node code runs on.
|
||||
|
||||
@@ -58,6 +100,12 @@ def venv_python() -> str:
|
||||
could not build one still runs python nodes, it just cannot add packages
|
||||
to them.
|
||||
"""
|
||||
root = adopted()
|
||||
if root is not None:
|
||||
# NODE_VENV may name a venv or the interpreter inside one; both are
|
||||
# useful things to be handed, and they are told apart by shape.
|
||||
inner = root / "bin" / "python"
|
||||
return str(inner if inner.exists() or root.is_dir() else root)
|
||||
path = VENV_DIR / "bin" / "python"
|
||||
return str(path) if path.exists() else sys.executable
|
||||
|
||||
@@ -73,6 +121,8 @@ def _digest(requirements: str) -> str:
|
||||
|
||||
def ensure_venv() -> None:
|
||||
"""Create the venv if it is missing or its interpreter has gone."""
|
||||
if adopted() is not None:
|
||||
return
|
||||
if (VENV_DIR / "bin" / "python").exists():
|
||||
return
|
||||
VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -100,6 +150,16 @@ def sync(requirements: str) -> tuple[bool, str]:
|
||||
other — the caller answers 400 with what came back, which is the only thing
|
||||
a person can act on.
|
||||
"""
|
||||
if adopted() is not None:
|
||||
# The one thing this must never do. `uv pip sync` makes a venv hold
|
||||
# exactly the manifest, so pointed at somebody's own environment it
|
||||
# uninstalls their packages — and the engine with them.
|
||||
return False, (
|
||||
"These packages are not Fluksio's to install: node code runs on "
|
||||
f"{venv_python()}, the environment Fluksio itself was installed "
|
||||
"into. Install into it with pip or uv, or set NODE_VENV=managed "
|
||||
"for a venv the engine owns."
|
||||
)
|
||||
manifest = ""
|
||||
try:
|
||||
ensure_venv()
|
||||
@@ -143,6 +203,12 @@ def reconcile(store: FlowStore) -> None:
|
||||
import error each time they execute.
|
||||
"""
|
||||
try:
|
||||
root = adopted()
|
||||
if root is not None:
|
||||
# Nothing to reconcile: the environment is somebody else's, and
|
||||
# bringing it "in line with the manifest" would empty it.
|
||||
logger.info("Node code runs on the adopted venv at %s", root)
|
||||
return
|
||||
# First, and unconditionally: the workers are started against this
|
||||
# interpreter, and it has to be the venv's one before anything is
|
||||
# installed into it, not after.
|
||||
@@ -162,15 +228,20 @@ def reconcile(store: FlowStore) -> None:
|
||||
def info(store: FlowStore) -> ModulesInfo:
|
||||
"""What is installed, what was asked for, and whether the two agree."""
|
||||
requirements = store.read_requirements()
|
||||
config = VENV_DIR / "pyvenv.cfg"
|
||||
root = venv_dir()
|
||||
is_adopted = adopted() is not None
|
||||
|
||||
version = ""
|
||||
config = root / "pyvenv.cfg"
|
||||
if config.exists():
|
||||
for line in config.read_text().splitlines():
|
||||
if line.startswith("version"):
|
||||
version = line.split("=", 1)[1].strip()
|
||||
if not version:
|
||||
version = ".".join(str(part) for part in sys.version_info[:3])
|
||||
|
||||
packages: list[ModulePackage] = []
|
||||
site = sorted(VENV_DIR.glob("lib/python*/site-packages"))
|
||||
site = sorted(root.glob("lib/python*/site-packages"))
|
||||
if site:
|
||||
packages = sorted(
|
||||
(
|
||||
@@ -182,13 +253,20 @@ def info(store: FlowStore) -> ModulesInfo:
|
||||
|
||||
return ModulesInfo(
|
||||
python_version=version,
|
||||
venv_path=str(VENV_DIR),
|
||||
venv_path=str(root),
|
||||
requirements=requirements,
|
||||
packages=packages,
|
||||
# An adopted venv is never "out of step": the manifest does not
|
||||
# describe it, so there is nothing for it to disagree with.
|
||||
applied=(
|
||||
_marker().read_text() == _digest(requirements)
|
||||
if _marker().exists()
|
||||
# Nothing asked for and nothing installed is already in step.
|
||||
else not requirements.strip()
|
||||
True
|
||||
if is_adopted
|
||||
else (
|
||||
_marker().read_text() == _digest(requirements)
|
||||
if _marker().exists()
|
||||
# Nothing asked for and nothing installed is already in step.
|
||||
else not requirements.strip()
|
||||
)
|
||||
),
|
||||
adopted=is_adopted,
|
||||
)
|
||||
|
||||
@@ -242,6 +242,10 @@ class ModulesInfo(BaseModel):
|
||||
packages: list[ModulePackage] = Field(default_factory=list)
|
||||
#: Whether what is installed matches the manifest.
|
||||
applied: bool = False
|
||||
#: True when node code runs on the venv Fluksio was installed into rather
|
||||
#: than one the engine built. That venv belongs to whoever made it, so the
|
||||
#: manifest does not describe it and nothing here installs into it.
|
||||
adopted: bool = False
|
||||
|
||||
|
||||
class ApplyRequest(BaseModel):
|
||||
|
||||
@@ -31,6 +31,11 @@ os.environ.setdefault("EMAILS_FROM_EMAIL", "noreply@example.com")
|
||||
# builds a TestClient — and so a lifespan — per test module. Tests that want the
|
||||
# endpoint mount it themselves.
|
||||
os.environ["MCP_ENABLED"] = "false"
|
||||
# The suite runs from a venv of its own, which "auto" would adopt as if it were
|
||||
# a data scientist's — so node code would run on the checkout's environment and
|
||||
# the module tests would have nothing of their own to build. The tests that
|
||||
# cover adoption ask for it explicitly.
|
||||
os.environ["NODE_VENV"] = "managed"
|
||||
# The private seeding endpoints are opt-in; the suite is one of the two places
|
||||
# (with the dev stack) where they are meant to work.
|
||||
os.environ["PRIVATE_API_ENABLED"] = "true"
|
||||
|
||||
@@ -75,3 +75,90 @@ def test_a_manifest_that_does_not_resolve_leaves_the_venv_alone(venv: Path):
|
||||
assert output
|
||||
# The marker still describes the manifest that actually installed.
|
||||
assert (venv / ".applied").read_text() == modules._digest("")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Adopting the venv Fluksio was installed into
|
||||
#
|
||||
# The workflow this exists for: make a venv, install what you work with, then
|
||||
# `pip install fluksio` into the same one. Building a second environment beside
|
||||
# it would leave node code unable to import the packages that are the point.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def their_venv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
"""Pretend the engine is running from a venv somebody else made."""
|
||||
directory = tmp_path / "research"
|
||||
(directory / "bin").mkdir(parents=True)
|
||||
(directory / "bin" / "python").write_text("")
|
||||
monkeypatch.setattr(modules.sys, "prefix", str(directory))
|
||||
monkeypatch.setattr(modules.sys, "base_prefix", str(tmp_path / "usr"))
|
||||
# Undoes the suite-wide "managed" in tests/__init__.py.
|
||||
monkeypatch.setattr(modules.settings, "NODE_VENV", "auto")
|
||||
return directory
|
||||
|
||||
|
||||
def test_a_venv_of_your_own_is_adopted(venv: Path, their_venv: Path):
|
||||
assert modules.adopted() == their_venv
|
||||
assert modules.venv_python() == str(their_venv / "bin" / "python")
|
||||
|
||||
|
||||
def test_an_adopted_venv_is_never_synced(venv: Path, their_venv: Path):
|
||||
"""The whole hazard: sync means "hold exactly this", so it would empty it."""
|
||||
ok, output = modules.sync("numpy>=2")
|
||||
|
||||
assert ok is False
|
||||
assert "not Fluksio's to install" in output
|
||||
# Nothing was built beside it either.
|
||||
assert not venv.exists()
|
||||
|
||||
|
||||
def test_reconcile_leaves_an_adopted_venv_alone(
|
||||
venv: Path, their_venv: Path, tmp_path: Path
|
||||
):
|
||||
store = FlowStore(tmp_path / "flows")
|
||||
store.write_requirements("numpy>=2\n")
|
||||
|
||||
modules.reconcile(store)
|
||||
|
||||
assert not venv.exists()
|
||||
assert modules.venv_python() == str(their_venv / "bin" / "python")
|
||||
|
||||
|
||||
def test_a_venv_the_engine_already_owns_is_kept(venv: Path, their_venv: Path):
|
||||
"""An upgrade must not take away packages somebody installed on purpose."""
|
||||
(venv / "bin").mkdir(parents=True)
|
||||
(venv / "bin" / "python").write_text("")
|
||||
|
||||
assert modules.adopted() is None
|
||||
assert modules.venv_python() == str(venv / "bin" / "python")
|
||||
|
||||
|
||||
def test_managed_is_what_a_container_asks_for(
|
||||
venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setattr(modules.settings, "NODE_VENV", "managed")
|
||||
|
||||
assert modules.adopted() is None
|
||||
|
||||
|
||||
def test_an_interpreter_can_be_named_outright(
|
||||
venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch
|
||||
):
|
||||
monkeypatch.setattr(modules.settings, "NODE_VENV", "/opt/conda/bin/python")
|
||||
|
||||
assert modules.venv_python() == "/opt/conda/bin/python"
|
||||
|
||||
|
||||
def test_info_describes_an_adopted_venv(venv: Path, their_venv: Path, tmp_path: Path):
|
||||
store = FlowStore(tmp_path / "flows")
|
||||
store.write_requirements("numpy>=2\n")
|
||||
|
||||
information = modules.info(store)
|
||||
|
||||
assert information.adopted is True
|
||||
assert information.venv_path == str(their_venv)
|
||||
assert information.python_version
|
||||
# The manifest does not describe it, so it cannot be out of step with it.
|
||||
assert information.applied is True
|
||||
|
||||
@@ -205,6 +205,7 @@ directory. The ones you are most likely to touch:
|
||||
|---|---|---|
|
||||
| `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this |
|
||||
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
||||
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
||||
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
||||
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
|
||||
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
|
||||
|
||||
@@ -182,6 +182,14 @@ flows. An install takes effect immediately; nothing restarts.
|
||||
|
||||
### Your own code as a package
|
||||
|
||||
!!! tip "If Fluksio is installed in the venv you work in, skip this"
|
||||
|
||||
Node code then runs on that environment, so your project and everything it
|
||||
imports are already importable — see
|
||||
[Getting started: data science](../getting-started/data-science.md). What
|
||||
follows is for a Fluksio with a venv of its own, which is what a container
|
||||
always has.
|
||||
|
||||
A manifest line can name a directory, so the project you already have is
|
||||
installable like any other dependency:
|
||||
|
||||
|
||||
@@ -63,13 +63,54 @@ export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
|
||||
|
||||
While you are experimenting, the interactive schema at
|
||||
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
|
||||
## Your packages are already there
|
||||
|
||||
## Tell it about your packages
|
||||
If you installed Fluksio into the environment you work in — the venv that
|
||||
already has torch or numpy in it — that is the environment your nodes run on.
|
||||
Nothing to declare, nothing to install twice:
|
||||
|
||||
Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
|
||||
environment Fluksio itself is installed in — so a pin of yours can never
|
||||
collide with one of ours. That venv starts empty, so the first thing to do is
|
||||
say what your script imports:
|
||||
```sh
|
||||
python -m venv .venv && . .venv/bin/activate
|
||||
pip install torch numpy pandas # what you were going to install anyway
|
||||
pip install fluksio # and then this
|
||||
fluksio serve
|
||||
```
|
||||
|
||||
`fluksio serve` says which interpreter it settled on:
|
||||
|
||||
```text
|
||||
Nodes /home/you/research/.venv/bin/python
|
||||
your environment, adopted. Add packages with pip.
|
||||
```
|
||||
|
||||
That venv is yours. Add a package the way you added the rest — `pip install
|
||||
scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they
|
||||
pick it up. The Modules screen lists what is installed and stays read-only,
|
||||
because the alternative would be Fluksio deciding what belongs in an
|
||||
environment it did not make.
|
||||
|
||||
!!! note "Your pins and ours share a site-packages"
|
||||
|
||||
The cost of not having two environments: a package the engine depends on
|
||||
is one you can now upgrade out from under it. In practice this is what
|
||||
everybody does with every other tool in the venv, and the answer when it
|
||||
bites is the same — pin it back, or keep Fluksio somewhere separate with
|
||||
the venv of its own below.
|
||||
|
||||
### A venv of Fluksio's own
|
||||
|
||||
Sometimes you want the isolation instead: a shared installation, a container,
|
||||
or an environment too precious to let a node's dependency near. Set
|
||||
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
|
||||
|
||||
```text
|
||||
Nodes /home/you/.fluksio/user-venv/bin/python
|
||||
a venv of its own; the Modules screen installs into it.
|
||||
```
|
||||
|
||||
Then the Modules screen is how packages get in — a pip manifest, installed
|
||||
with `uv pip sync` and versioned alongside your flows, so what a run imported
|
||||
is recorded with what it ran:
|
||||
|
||||
```sh
|
||||
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
|
||||
@@ -77,26 +118,23 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
|
||||
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
|
||||
```
|
||||
|
||||
It is a pip manifest, installed with `uv pip sync`, and it is versioned
|
||||
alongside your flows — so what a run imported is recorded with what it ran.
|
||||
Adding a package takes effect immediately; nothing restarts.
|
||||
Adding a package takes effect immediately; nothing restarts. The Docker image
|
||||
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
|
||||
nothing of yours — so a container is always this case.
|
||||
|
||||
??? note "Already have a venv you would rather not duplicate?"
|
||||
!!! tip "A GPU box works the same way"
|
||||
|
||||
Attach it as a worker instead of reinstalling into it. Mint a token, then
|
||||
point the agent at your existing interpreter:
|
||||
`pip install fluksio-worker` into the environment the training code runs
|
||||
in, and node code runs on it: `--python` defaults to the interpreter the
|
||||
agent was started with.
|
||||
|
||||
```sh
|
||||
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"name": "laptop"}'
|
||||
|
||||
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
|
||||
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
|
||||
--token "$WORKER_TOKEN" --labels gpu
|
||||
```
|
||||
|
||||
Then mark the node `"device": "local"` and it runs on that interpreter. It
|
||||
is the same mechanism that sends a node to a GPU box, and it is worth
|
||||
knowing about early — see [Remote workers](../code/workers.md).
|
||||
Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).
|
||||
|
||||
## Say which functions are nodes
|
||||
|
||||
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
|
||||
|
||||
@@ -22,10 +22,22 @@ Anything already exported wins over the file.
|
||||
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
|
||||
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
|
||||
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
|
||||
| `NODE_VENV` | `auto` | which interpreter node code runs on — see below |
|
||||
|
||||
Set `DATA_DIR` and the rest follow. Set one explicitly and it wins — which is
|
||||
what the container images do to pin everything onto `/data`.
|
||||
|
||||
`NODE_VENV` is the exception, being about an environment rather than a path:
|
||||
|
||||
| Value | What node code runs on |
|
||||
|---|---|
|
||||
| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. |
|
||||
| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. |
|
||||
| a path | that interpreter, or that venv, whatever it is. |
|
||||
|
||||
An installation that already has a managed venv keeps it on upgrade under
|
||||
`auto`, because it may hold packages somebody installed on purpose.
|
||||
|
||||
!!! warning "The four files that must be on persistent storage"
|
||||
|
||||
`secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written
|
||||
|
||||
@@ -1521,6 +1521,11 @@ export const ModulesInfoSchema = {
|
||||
type: 'boolean',
|
||||
title: 'Applied',
|
||||
default: false
|
||||
},
|
||||
adopted: {
|
||||
type: 'boolean',
|
||||
title: 'Adopted',
|
||||
default: false
|
||||
}
|
||||
},
|
||||
type: 'object',
|
||||
|
||||
@@ -583,6 +583,7 @@ export type ModulesInfo = {
|
||||
requirements?: string;
|
||||
packages?: Array<ModulePackage>;
|
||||
applied?: boolean;
|
||||
adopted?: boolean;
|
||||
};
|
||||
|
||||
export type NewPassword = {
|
||||
|
||||
@@ -62,64 +62,81 @@ function Modules() {
|
||||
const requirements = draft ?? stored
|
||||
const packages = data?.packages ?? []
|
||||
const dirty = requirements !== stored
|
||||
// Fluksio was installed into an environment somebody else made, and node
|
||||
// code runs on it. Installing from here would mean `uv pip sync`, which
|
||||
// holds a venv to exactly one list — and would uninstall their work.
|
||||
const adopted = data?.adopted ?? false
|
||||
|
||||
return (
|
||||
<div className="grid gap-6">
|
||||
<div className="grid gap-1">
|
||||
<h1 className="text-2xl">Modules</h1>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
The Python packages your function nodes can import. They are installed
|
||||
into an environment of their own, separate from the engine's, so a
|
||||
version you pin here is the one your code gets. The list is kept with
|
||||
your flows, so a rebuilt deployment installs the same set again.
|
||||
{adopted
|
||||
? "The Python packages your function nodes can import. Fluksio was installed into an environment you already had, so that is the one your nodes run on — and it is yours to change, with pip or uv rather than from here."
|
||||
: "The Python packages your function nodes can import. They are installed into an environment of their own, separate from the engine's, so a version you pin here is the one your code gets. The list is kept with your flows, so a rebuilt deployment installs the same set again."}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<section className="grid gap-3">
|
||||
<h2 className={SECTION}>Requirements</h2>
|
||||
<textarea
|
||||
className="min-h-[160px] w-full rounded-md border border-border bg-card p-3 font-mono text-sm shadow-e1 outline-none focus-visible:border-primary"
|
||||
spellCheck={false}
|
||||
aria-label="Requirements"
|
||||
data-testid="requirements"
|
||||
placeholder={"requests==2.32.3\npandas>=2.2"}
|
||||
value={requirements}
|
||||
onChange={(event) => setDraft(event.target.value)}
|
||||
/>
|
||||
<div className="flex items-center gap-3">
|
||||
<Button
|
||||
disabled={apply.isPending}
|
||||
data-testid="apply-modules"
|
||||
onClick={() => apply.mutate(requirements)}
|
||||
>
|
||||
<Package />
|
||||
{apply.isPending ? "Installing" : "Apply"}
|
||||
</Button>
|
||||
<span className="text-sm text-muted-foreground">
|
||||
{dirty
|
||||
? "Not applied yet."
|
||||
: data?.applied
|
||||
? "Installed and in step."
|
||||
: "What is installed does not match this list."}
|
||||
</span>
|
||||
</div>
|
||||
{output ? (
|
||||
<pre
|
||||
className="max-h-64 overflow-auto rounded-md border border-border bg-card p-3 font-mono text-xs shadow-e1"
|
||||
data-testid="modules-output"
|
||||
>
|
||||
{output}
|
||||
</pre>
|
||||
) : null}
|
||||
</section>
|
||||
{adopted ? null : (
|
||||
<section className="grid gap-3">
|
||||
<h2 className={SECTION}>Requirements</h2>
|
||||
<textarea
|
||||
className="min-h-[160px] w-full rounded-md border border-border bg-card p-3 font-mono text-sm shadow-e1 outline-none focus-visible:border-primary"
|
||||
spellCheck={false}
|
||||
aria-label="Requirements"
|
||||
data-testid="requirements"
|
||||
placeholder={"requests==2.32.3\npandas>=2.2"}
|
||||
value={requirements}
|
||||
onChange={(event) => setDraft(event.target.value)}
|
||||
/>
|
||||
<div className="flex items-center gap-3">
|
||||
<Button
|
||||
disabled={apply.isPending}
|
||||
data-testid="apply-modules"
|
||||
onClick={() => apply.mutate(requirements)}
|
||||
>
|
||||
<Package />
|
||||
{apply.isPending ? "Installing" : "Apply"}
|
||||
</Button>
|
||||
<span className="text-sm text-muted-foreground">
|
||||
{dirty
|
||||
? "Not applied yet."
|
||||
: data?.applied
|
||||
? "Installed and in step."
|
||||
: "What is installed does not match this list."}
|
||||
</span>
|
||||
</div>
|
||||
{output ? (
|
||||
<pre
|
||||
className="max-h-64 overflow-auto rounded-md border border-border bg-card p-3 font-mono text-xs shadow-e1"
|
||||
data-testid="modules-output"
|
||||
>
|
||||
{output}
|
||||
</pre>
|
||||
) : null}
|
||||
</section>
|
||||
)}
|
||||
|
||||
<section className="grid gap-3">
|
||||
<h2 className={SECTION}>Installed</h2>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
<p className="text-sm text-muted-foreground" data-testid="modules-venv">
|
||||
{data
|
||||
? `Python ${data.python_version || "unknown"} at ${data.venv_path}`
|
||||
: "Reading the environment…"}
|
||||
</p>
|
||||
{adopted ? (
|
||||
<p
|
||||
className="text-sm text-muted-foreground"
|
||||
data-testid="modules-adopted"
|
||||
>
|
||||
Adopted — this is the environment Fluksio itself is installed in.
|
||||
Add a package the way you added the rest:{" "}
|
||||
<span className="font-mono">pip install …</span>, then sync or
|
||||
restart so the workers pick it up. For a venv the engine owns
|
||||
instead, set <span className="font-mono">NODE_VENV=managed</span>.
|
||||
</p>
|
||||
) : null}
|
||||
{packages.length === 0 ? (
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Nothing installed yet — node code has the standard library.
|
||||
|
||||
@@ -4,7 +4,11 @@ Point it at the engine on the box with the GPU and it dials in::
|
||||
|
||||
pip install fluksio-worker
|
||||
fluksio-worker --url wss://api.example.com/api/v1/workers/attach \\
|
||||
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu --python /opt/venv/bin/python
|
||||
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
|
||||
|
||||
Install it into the environment the training code already runs in and node
|
||||
code runs on that: ``--python`` defaults to the interpreter this was started
|
||||
with. Point it elsewhere only when the two are meant to differ.
|
||||
|
||||
It connects *out*, so the engine needs no route back and nothing has to expose
|
||||
Redis. What it then does is what the engine's own worker pool does: hold a few
|
||||
@@ -274,7 +278,10 @@ def main(argv: list[str] | None = None) -> int:
|
||||
parser.add_argument(
|
||||
"--python",
|
||||
default=sys.executable,
|
||||
help="the interpreter node code runs on; point it at the venv with torch",
|
||||
help=(
|
||||
"the interpreter node code runs on (default: the one running this, "
|
||||
"so installing into the venv with torch in it is enough)"
|
||||
),
|
||||
)
|
||||
parser.add_argument("--parallel", type=int, default=1)
|
||||
parser.add_argument("--artifact-url", default="")
|
||||
|
||||
Reference in New Issue
Block a user