diff --git a/backend/Dockerfile b/backend/Dockerfile index e122c3a..50ce40e 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -20,6 +20,12 @@ WORKDIR /app/ # Ref: https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment ENV PATH="/app/.venv/bin:$PATH" +# /app/.venv holds the app and nothing of anybody else's, so there is nothing +# here to adopt: node code gets a venv of its own on the data volume, which is +# what the Modules screen installs into. A `pip install fluksio` into an +# environment somebody already works in is the case that adopts instead. +ENV NODE_VENV=managed + # Install dependencies # Ref: https://docs.astral.sh/uv/guides/integration/docker/#intermediate-layers RUN --mount=type=cache,target=/root/.cache/uv \ diff --git a/backend/fluksio/cli.py b/backend/fluksio/cli.py index b4143f6..25a4e94 100644 --- a/backend/fluksio/cli.py +++ b/backend/fluksio/cli.py @@ -191,11 +191,24 @@ def cmd_serve(args: argparse.Namespace) -> int: import uvicorn + from fluksio.flow import modules from fluksio.main import app config = cloud_config.load() _say(f"Fluksio {fluksio.__version__} — data in {data_dir}") _say(f" API http://{args.host}:{args.port}{'/api/v1'}") + # Which environment node code imports from is the thing a data scientist + # most needs to know at this moment, and the answer differs depending on + # how Fluksio was installed. Saying it costs one line. + if modules.adopted() is not None: + _say(f" Nodes {modules.venv_python()}") + _say(" your environment, adopted. Add packages with pip.") + else: + # Not `venv_python()`: the managed venv is built when the engine comes + # up, which is after this prints, and until then that would answer with + # whatever interpreter happens to be running this. + _say(f" Nodes {modules.venv_dir() / 'bin' / 'python'}") + _say(" a venv of its own; the Modules screen installs into it.") if config is not None: _say(f" Portal {config.portal_url}, installation {config.installation_id}") _say(" The dashboard is served by the portal; nothing is served here.") diff --git a/backend/fluksio/core/config.py b/backend/fluksio/core/config.py index 35723db..6f02c38 100644 --- a/backend/fluksio/core/config.py +++ b/backend/fluksio/core/config.py @@ -70,6 +70,12 @@ class Settings(BaseSettings): # alerting is: where a screen hangs is the deployment's concern rather than # any one dashboard's. PANELS_FILE: Path = Path("flow-data/panels.json") + # Which interpreter node code runs on. "auto" adopts the venv the engine + # was installed into, when it was installed into one and there is no venv + # of its own to lose — which is the `pip install fluksio` beside your own + # packages case. "managed" always builds a separate one, which is what a + # container wants. A path names an interpreter outright. + NODE_VENV: str = "auto" # The MCP endpoint, and the OAuth server agents authenticate against. Off # until someone asks for it: it opens client registration to the network. MCP_ENABLED: bool = False diff --git a/backend/fluksio/flow/modules.py b/backend/fluksio/flow/modules.py index 6a9ed5c..b14aea4 100644 --- a/backend/fluksio/flow/modules.py +++ b/backend/fluksio/flow/modules.py @@ -9,6 +9,15 @@ and the worker processes that import them need nothing from the app. ``uv pip sync`` rather than install, so a line taken out of the manifest is uninstalled. The manifest is written only after a sync succeeds, which is all the rollback a failed resolve needs. + +None of that applies to an *adopted* venv. A data scientist makes a venv, +installs what they work with, and then installs Fluksio into it too — at which +point building a second environment beside it is exactly wrong: the packages +the nodes need are already here. So when the engine is running from a venv of +somebody else's, node code runs on it. That venv is theirs: `uv pip sync` is +never pointed at it, because sync means "hold exactly this" and would uninstall +their work along with the engine. It is read-only here, and `pip` is how they +change it. """ from __future__ import annotations @@ -51,6 +60,39 @@ def uv_bin() -> str: return shutil.which("uv") or "uv" +def adopted() -> Path | None: + """The venv this engine was installed into, when node code should use it. + + ``None`` means the managed venv — one the engine builds and owns. The three + ways that is the answer, in order: + + * ``NODE_VENV=managed`` says so. A container sets this: its venv holds the + app and nothing of anybody's, so there is nothing to adopt. + * There is already a managed venv. It may have packages in it that + somebody installed on purpose, and an upgrade must not take them away. + * The engine is not running from a venv at all. + + Anything else — ``pip install fluksio`` into the environment you already + work in — is the case this exists for. + """ + setting = settings.NODE_VENV.strip() + if setting == "managed": + return None + if setting and setting != "auto": + return Path(setting) + if (VENV_DIR / "bin" / "python").exists(): + return None + prefix = Path(sys.prefix) + if prefix == Path(sys.base_prefix) or prefix == VENV_DIR: + return None + return prefix + + +def venv_dir() -> Path: + """Whichever venv node code runs from, adopted or managed.""" + return adopted() or VENV_DIR + + def venv_python() -> str: """The interpreter node code runs on. @@ -58,6 +100,12 @@ def venv_python() -> str: could not build one still runs python nodes, it just cannot add packages to them. """ + root = adopted() + if root is not None: + # NODE_VENV may name a venv or the interpreter inside one; both are + # useful things to be handed, and they are told apart by shape. + inner = root / "bin" / "python" + return str(inner if inner.exists() or root.is_dir() else root) path = VENV_DIR / "bin" / "python" return str(path) if path.exists() else sys.executable @@ -73,6 +121,8 @@ def _digest(requirements: str) -> str: def ensure_venv() -> None: """Create the venv if it is missing or its interpreter has gone.""" + if adopted() is not None: + return if (VENV_DIR / "bin" / "python").exists(): return VENV_DIR.parent.mkdir(parents=True, exist_ok=True) @@ -100,6 +150,16 @@ def sync(requirements: str) -> tuple[bool, str]: other — the caller answers 400 with what came back, which is the only thing a person can act on. """ + if adopted() is not None: + # The one thing this must never do. `uv pip sync` makes a venv hold + # exactly the manifest, so pointed at somebody's own environment it + # uninstalls their packages — and the engine with them. + return False, ( + "These packages are not Fluksio's to install: node code runs on " + f"{venv_python()}, the environment Fluksio itself was installed " + "into. Install into it with pip or uv, or set NODE_VENV=managed " + "for a venv the engine owns." + ) manifest = "" try: ensure_venv() @@ -143,6 +203,12 @@ def reconcile(store: FlowStore) -> None: import error each time they execute. """ try: + root = adopted() + if root is not None: + # Nothing to reconcile: the environment is somebody else's, and + # bringing it "in line with the manifest" would empty it. + logger.info("Node code runs on the adopted venv at %s", root) + return # First, and unconditionally: the workers are started against this # interpreter, and it has to be the venv's one before anything is # installed into it, not after. @@ -162,15 +228,20 @@ def reconcile(store: FlowStore) -> None: def info(store: FlowStore) -> ModulesInfo: """What is installed, what was asked for, and whether the two agree.""" requirements = store.read_requirements() - config = VENV_DIR / "pyvenv.cfg" + root = venv_dir() + is_adopted = adopted() is not None + version = "" + config = root / "pyvenv.cfg" if config.exists(): for line in config.read_text().splitlines(): if line.startswith("version"): version = line.split("=", 1)[1].strip() + if not version: + version = ".".join(str(part) for part in sys.version_info[:3]) packages: list[ModulePackage] = [] - site = sorted(VENV_DIR.glob("lib/python*/site-packages")) + site = sorted(root.glob("lib/python*/site-packages")) if site: packages = sorted( ( @@ -182,13 +253,20 @@ def info(store: FlowStore) -> ModulesInfo: return ModulesInfo( python_version=version, - venv_path=str(VENV_DIR), + venv_path=str(root), requirements=requirements, packages=packages, + # An adopted venv is never "out of step": the manifest does not + # describe it, so there is nothing for it to disagree with. applied=( - _marker().read_text() == _digest(requirements) - if _marker().exists() - # Nothing asked for and nothing installed is already in step. - else not requirements.strip() + True + if is_adopted + else ( + _marker().read_text() == _digest(requirements) + if _marker().exists() + # Nothing asked for and nothing installed is already in step. + else not requirements.strip() + ) ), + adopted=is_adopted, ) diff --git a/backend/fluksio/flow/schemas.py b/backend/fluksio/flow/schemas.py index b8e0ed6..5ffb677 100644 --- a/backend/fluksio/flow/schemas.py +++ b/backend/fluksio/flow/schemas.py @@ -242,6 +242,10 @@ class ModulesInfo(BaseModel): packages: list[ModulePackage] = Field(default_factory=list) #: Whether what is installed matches the manifest. applied: bool = False + #: True when node code runs on the venv Fluksio was installed into rather + #: than one the engine built. That venv belongs to whoever made it, so the + #: manifest does not describe it and nothing here installs into it. + adopted: bool = False class ApplyRequest(BaseModel): diff --git a/backend/tests/__init__.py b/backend/tests/__init__.py index dce203b..70e55f0 100644 --- a/backend/tests/__init__.py +++ b/backend/tests/__init__.py @@ -31,6 +31,11 @@ os.environ.setdefault("EMAILS_FROM_EMAIL", "noreply@example.com") # builds a TestClient — and so a lifespan — per test module. Tests that want the # endpoint mount it themselves. os.environ["MCP_ENABLED"] = "false" +# The suite runs from a venv of its own, which "auto" would adopt as if it were +# a data scientist's — so node code would run on the checkout's environment and +# the module tests would have nothing of their own to build. The tests that +# cover adoption ask for it explicitly. +os.environ["NODE_VENV"] = "managed" # The private seeding endpoints are opt-in; the suite is one of the two places # (with the dev stack) where they are meant to work. os.environ["PRIVATE_API_ENABLED"] = "true" diff --git a/backend/tests/flow/test_modules.py b/backend/tests/flow/test_modules.py index 6c7fdda..1d3d991 100644 --- a/backend/tests/flow/test_modules.py +++ b/backend/tests/flow/test_modules.py @@ -75,3 +75,90 @@ def test_a_manifest_that_does_not_resolve_leaves_the_venv_alone(venv: Path): assert output # The marker still describes the manifest that actually installed. assert (venv / ".applied").read_text() == modules._digest("") + + +# --------------------------------------------------------------------------- +# Adopting the venv Fluksio was installed into +# +# The workflow this exists for: make a venv, install what you work with, then +# `pip install fluksio` into the same one. Building a second environment beside +# it would leave node code unable to import the packages that are the point. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def their_venv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """Pretend the engine is running from a venv somebody else made.""" + directory = tmp_path / "research" + (directory / "bin").mkdir(parents=True) + (directory / "bin" / "python").write_text("") + monkeypatch.setattr(modules.sys, "prefix", str(directory)) + monkeypatch.setattr(modules.sys, "base_prefix", str(tmp_path / "usr")) + # Undoes the suite-wide "managed" in tests/__init__.py. + monkeypatch.setattr(modules.settings, "NODE_VENV", "auto") + return directory + + +def test_a_venv_of_your_own_is_adopted(venv: Path, their_venv: Path): + assert modules.adopted() == their_venv + assert modules.venv_python() == str(their_venv / "bin" / "python") + + +def test_an_adopted_venv_is_never_synced(venv: Path, their_venv: Path): + """The whole hazard: sync means "hold exactly this", so it would empty it.""" + ok, output = modules.sync("numpy>=2") + + assert ok is False + assert "not Fluksio's to install" in output + # Nothing was built beside it either. + assert not venv.exists() + + +def test_reconcile_leaves_an_adopted_venv_alone( + venv: Path, their_venv: Path, tmp_path: Path +): + store = FlowStore(tmp_path / "flows") + store.write_requirements("numpy>=2\n") + + modules.reconcile(store) + + assert not venv.exists() + assert modules.venv_python() == str(their_venv / "bin" / "python") + + +def test_a_venv_the_engine_already_owns_is_kept(venv: Path, their_venv: Path): + """An upgrade must not take away packages somebody installed on purpose.""" + (venv / "bin").mkdir(parents=True) + (venv / "bin" / "python").write_text("") + + assert modules.adopted() is None + assert modules.venv_python() == str(venv / "bin" / "python") + + +def test_managed_is_what_a_container_asks_for( + venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch +): + monkeypatch.setattr(modules.settings, "NODE_VENV", "managed") + + assert modules.adopted() is None + + +def test_an_interpreter_can_be_named_outright( + venv: Path, their_venv: Path, monkeypatch: pytest.MonkeyPatch +): + monkeypatch.setattr(modules.settings, "NODE_VENV", "/opt/conda/bin/python") + + assert modules.venv_python() == "/opt/conda/bin/python" + + +def test_info_describes_an_adopted_venv(venv: Path, their_venv: Path, tmp_path: Path): + store = FlowStore(tmp_path / "flows") + store.write_requirements("numpy>=2\n") + + information = modules.info(store) + + assert information.adopted is True + assert information.venv_path == str(their_venv) + assert information.python_version + # The manifest does not describe it, so it cannot be out of step with it. + assert information.applied is True diff --git a/docs/code/cli.md b/docs/code/cli.md index 0b3d4ba..f63b848 100644 --- a/docs/code/cli.md +++ b/docs/code/cli.md @@ -205,6 +205,7 @@ directory. The ones you are most likely to touch: |---|---|---| | `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this | | `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL | +| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter | | `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart | | `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links | | `ENVIRONMENT` | `local` | `production` closes the interactive API schema | diff --git a/docs/code/nodes.md b/docs/code/nodes.md index d44fb7b..84c8998 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -182,6 +182,14 @@ flows. An install takes effect immediately; nothing restarts. ### Your own code as a package +!!! tip "If Fluksio is installed in the venv you work in, skip this" + + Node code then runs on that environment, so your project and everything it + imports are already importable — see + [Getting started: data science](../getting-started/data-science.md). What + follows is for a Fluksio with a venv of its own, which is what a container + always has. + A manifest line can name a directory, so the project you already have is installable like any other dependency: diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index b213c28..8372243 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -63,13 +63,54 @@ export TOKEN=$(jq -r .token ~/.config/fluksio/client.json) While you are experimenting, the interactive schema at is the fastest way to see what is available. +## Your packages are already there -## Tell it about your packages +If you installed Fluksio into the environment you work in — the venv that +already has torch or numpy in it — that is the environment your nodes run on. +Nothing to declare, nothing to install twice: -Node code runs in `~/.fluksio/user-venv`, deliberately separate from the -environment Fluksio itself is installed in — so a pin of yours can never -collide with one of ours. That venv starts empty, so the first thing to do is -say what your script imports: +```sh +python -m venv .venv && . .venv/bin/activate +pip install torch numpy pandas # what you were going to install anyway +pip install fluksio # and then this +fluksio serve +``` + +`fluksio serve` says which interpreter it settled on: + +```text + Nodes /home/you/research/.venv/bin/python + your environment, adopted. Add packages with pip. +``` + +That venv is yours. Add a package the way you added the rest — `pip install +scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they +pick it up. The Modules screen lists what is installed and stays read-only, +because the alternative would be Fluksio deciding what belongs in an +environment it did not make. + +!!! note "Your pins and ours share a site-packages" + + The cost of not having two environments: a package the engine depends on + is one you can now upgrade out from under it. In practice this is what + everybody does with every other tool in the venv, and the answer when it + bites is the same — pin it back, or keep Fluksio somewhere separate with + the venv of its own below. + +### A venv of Fluksio's own + +Sometimes you want the isolation instead: a shared installation, a container, +or an environment too precious to let a node's dependency near. Set +`NODE_VENV=managed` and Fluksio builds and owns one under the data directory: + +```text + Nodes /home/you/.fluksio/user-venv/bin/python + a venv of its own; the Modules screen installs into it. +``` + +Then the Modules screen is how packages get in — a pip manifest, installed +with `uv pip sync` and versioned alongside your flows, so what a run imported +is recorded with what it ran: ```sh curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ @@ -77,26 +118,23 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ -d "{\"requirements\": $(jq -Rs . < requirements.txt)}" ``` -It is a pip manifest, installed with `uv pip sync`, and it is versioned -alongside your flows — so what a run imported is recorded with what it ran. -Adding a package takes effect immediately; nothing restarts. +Adding a package takes effect immediately; nothing restarts. The Docker image +sets `NODE_VENV=managed` for itself, because the venv in it holds the app and +nothing of yours — so a container is always this case. -??? note "Already have a venv you would rather not duplicate?" +!!! tip "A GPU box works the same way" - Attach it as a worker instead of reinstalling into it. Mint a token, then - point the agent at your existing interpreter: + `pip install fluksio-worker` into the environment the training code runs + in, and node code runs on it: `--python` defaults to the interpreter the + agent was started with. ```sh - curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' -d '{"name": "laptop"}' - fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \ - --token "$WORKER_TOKEN" --labels local --python "$(which python)" + --token "$WORKER_TOKEN" --labels gpu ``` - Then mark the node `"device": "local"` and it runs on that interpreter. It - is the same mechanism that sends a node to a GPU box, and it is worth - knowing about early — see [Remote workers](../code/workers.md). + Then give the node `device="gpu"` — see [Remote workers](../code/workers.md). + ## Say which functions are nodes A **flow** is a graph of nodes. A **batch flow** is one that runs on demand diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 66b757d..b77b1f5 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -22,10 +22,22 @@ Anything already exported wins over the file. | `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings | | `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens | | `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any | +| `NODE_VENV` | `auto` | which interpreter node code runs on — see below | Set `DATA_DIR` and the rest follow. Set one explicitly and it wins — which is what the container images do to pin everything onto `/data`. +`NODE_VENV` is the exception, being about an environment rather than a path: + +| Value | What node code runs on | +|---|---| +| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. | +| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. | +| a path | that interpreter, or that venv, whatever it is. | + +An installation that already has a managed venv keeps it on upgrade under +`auto`, because it may hold packages somebody installed on purpose. + !!! warning "The four files that must be on persistent storage" `secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written diff --git a/frontend/src/client/schemas.gen.ts b/frontend/src/client/schemas.gen.ts index 7890747..437c21a 100644 --- a/frontend/src/client/schemas.gen.ts +++ b/frontend/src/client/schemas.gen.ts @@ -1521,6 +1521,11 @@ export const ModulesInfoSchema = { type: 'boolean', title: 'Applied', default: false + }, + adopted: { + type: 'boolean', + title: 'Adopted', + default: false } }, type: 'object', diff --git a/frontend/src/client/types.gen.ts b/frontend/src/client/types.gen.ts index a1d1d06..afd7b43 100644 --- a/frontend/src/client/types.gen.ts +++ b/frontend/src/client/types.gen.ts @@ -583,6 +583,7 @@ export type ModulesInfo = { requirements?: string; packages?: Array; applied?: boolean; + adopted?: boolean; }; export type NewPassword = { diff --git a/frontend/src/routes/_layout/modules.tsx b/frontend/src/routes/_layout/modules.tsx index d06991c..6094875 100644 --- a/frontend/src/routes/_layout/modules.tsx +++ b/frontend/src/routes/_layout/modules.tsx @@ -62,64 +62,81 @@ function Modules() { const requirements = draft ?? stored const packages = data?.packages ?? [] const dirty = requirements !== stored + // Fluksio was installed into an environment somebody else made, and node + // code runs on it. Installing from here would mean `uv pip sync`, which + // holds a venv to exactly one list — and would uninstall their work. + const adopted = data?.adopted ?? false return (

Modules

- The Python packages your function nodes can import. They are installed - into an environment of their own, separate from the engine's, so a - version you pin here is the one your code gets. The list is kept with - your flows, so a rebuilt deployment installs the same set again. + {adopted + ? "The Python packages your function nodes can import. Fluksio was installed into an environment you already had, so that is the one your nodes run on — and it is yours to change, with pip or uv rather than from here." + : "The Python packages your function nodes can import. They are installed into an environment of their own, separate from the engine's, so a version you pin here is the one your code gets. The list is kept with your flows, so a rebuilt deployment installs the same set again."}

-
-

Requirements

-