Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Modules resolve the manifest with uv pip compile before syncing, so listed packages bring their dependencies

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WgBfkm4rCmtd1T4vHWjEPx
This commit is contained in:
stroblme
2026-10-15 17:06:20 +02:00
co-authored by Claude Opus 5.5
parent 356b0331bf
commit 50e6c033f1
4 changed files with 98 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/fluksio/flow/modules.py
+47 -27
View File
@@ -6,9 +6,12 @@ into a venv on the data volume rather than into the engine's environment: a
pin here can never shadow — or be shadowed by — what the app itself runs on,
and the worker processes that import them need nothing from the app.
``uv pip sync`` rather than install, so a line taken out of the manifest is
uninstalled. The manifest is written only after a sync succeeds, which is all
the rollback a failed resolve needs.
The manifest is resolved with ``uv pip compile`` first, so a line naming a
package brings its dependencies along, and a manifest that already pins
everything resolves to itself. The result goes to ``uv pip sync`` rather than
install, so a line taken out of the manifest is uninstalled. The manifest is
written only after a sync succeeds, which is all the rollback a failed resolve
needs.
None of that applies to an *adopted* venv. A data scientist makes a venv,
installs what they work with, and then installs Fluksio into it too — at which
@@ -179,35 +182,52 @@ def sync(requirements: str) -> tuple[bool, str]:
def _sync(requirements: str) -> tuple[bool, str]:
"""The install itself, under `_sync_lock`."""
manifest = ""
"""The resolve and install, under `_sync_lock`."""
try:
ensure_venv()
with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False) as handle:
handle.write(requirements)
manifest = handle.name
result = subprocess.run(
# Empty means empty: without the flag uv refuses to clear a venv,
# so deleting the last line would leave the package installed.
[
uv_bin(),
"pip",
"sync",
"--allow-empty-requirements",
"--python",
venv_python(),
manifest,
],
capture_output=True,
text=True,
timeout=SYNC_TIMEOUT,
)
python = venv_python()
with tempfile.TemporaryDirectory() as directory:
manifest = Path(directory, "requirements.in")
lock = Path(directory, "requirements.txt")
manifest.write_text(requirements)
for command in (
# The emit flags carry the manifest's index, find-links and
# binary options into the lock, which is all sync reads.
[
uv_bin(),
"pip",
"compile",
"--quiet",
"--emit-index-url",
"--emit-find-links",
"--emit-build-options",
"--python",
python,
str(manifest),
"--output-file",
str(lock),
],
# Empty means empty: without the flag uv refuses to clear a
# venv, so deleting the last line would leave the package
# installed.
[
uv_bin(),
"pip",
"sync",
"--allow-empty-requirements",
"--python",
python,
str(lock),
],
):
result = subprocess.run(
command, capture_output=True, text=True, timeout=SYNC_TIMEOUT
)
if result.returncode != 0:
break
except (OSError, subprocess.SubprocessError) as exc:
detail = str(getattr(exc, "stderr", None) or exc).strip()
return False, f"uv could not run: {detail}"
finally:
if manifest:
Path(manifest).unlink(missing_ok=True)
output = (result.stdout + result.stderr).strip()
if result.returncode == 0:
backend/tests/flow/test_modules.py
+35
View File
@@ -2,6 +2,7 @@
import shutil
import subprocess
import zipfile
from pathlib import Path
import pytest
@@ -77,6 +78,40 @@ def test_a_manifest_that_does_not_resolve_leaves_the_venv_alone(venv: Path):
assert (venv / ".applied").read_text() == modules._digest("")
def _wheel(directory: Path, name: str, requires: str = "") -> None:
"""A wheel with nothing in it but a name and, optionally, one dependency."""
info = f"{name}-1.0.dist-info"
metadata = f"Metadata-Version: 2.1\nName: {name}\nVersion: 1.0\n"
if requires:
metadata += f"Requires-Dist: {requires}\n"
with zipfile.ZipFile(directory / f"{name}-1.0-py3-none-any.whl", "w") as wheel:
wheel.writestr(f"{name}/__init__.py", "")
wheel.writestr(f"{info}/METADATA", metadata)
wheel.writestr(
f"{info}/WHEEL",
"Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py3-none-any\n",
)
wheel.writestr(f"{info}/RECORD", "")
def test_a_listed_package_brings_its_dependencies(
venv: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
):
wheels = tmp_path / "wheels"
wheels.mkdir()
_wheel(wheels, "fluksio_top", requires="fluksio_dep")
_wheel(wheels, "fluksio_dep")
monkeypatch.setenv("UV_OFFLINE", "1")
ok, output = modules.sync(f"--find-links {wheels}\nfluksio-top\n")
assert ok, output
imported = subprocess.run(
[modules.venv_python(), "-c", "import fluksio_dep"], check=False
)
assert imported.returncode == 0
# ---------------------------------------------------------------------------
# Adopting the venv Fluksio was installed into
#
docs/code/nodes.md
+12 -12
View File
@@ -247,17 +247,17 @@ Binding the speaking node's `text` straight to `said` makes the panel repeat
what it heard, which tests the loop before there is any logic in it.
Both packages go into the node's environment, as described in
[Packages](#packages). A managed environment installs its manifest with
`uv pip sync`, which skips dependencies, so paste the fully resolved list into
**Modules**:
[Packages](#packages). In a managed environment, list them in **Modules**:
```sh
printf 'faster-whisper\nav<19\npiper-tts\n' | uv pip compile --python-version 3.13 -
```text
faster-whisper
av<19
piper-tts
```
The Docker image runs Python 3.13. `av<19` holds PyAV back, since faster-whisper
1.2.1 cannot open files with PyAV 19. In your own venv or on a remote worker,
`pip install faster-whisper "av<19" piper-tts` is enough.
`av<19` holds PyAV back, since faster-whisper 1.2.1 cannot open files with
PyAV 19. In your own venv or on a remote worker,
`pip install faster-whisper "av<19" piper-tts` does the same.
The models land in `~/.cache/voice` on the machine running the node. The Docker
image keeps its home directory inside the container, so a recreated container
@@ -343,10 +343,10 @@ already has a managed one. Without an existing venv, it creates a managed
environment. `NODE_VENV=managed` always requests one; the Docker image uses
this setting.
In managed mode, the Modules manifest is installed with `uv pip sync` and
versioned with the flows. Sync installs only the packages listed, not their
dependencies, so list those too, for example as resolved by `uv pip compile`.
Applying it refreshes local workers. The API accepts the same manifest:
In managed mode, the Modules manifest is resolved with `uv pip compile`,
installed with `uv pip sync` and versioned with the flows. Listing a package
brings its dependencies along. Applying it refreshes local workers. The API
accepts the same manifest:
```sh
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
docs/interface/operations.md
+4 -5
View File
@@ -32,11 +32,10 @@ the flow repository. Using references keeps credentials out of saved flows.
packages and Python version. There are two setups:
- **Managed environment:** Docker uses a separate node environment. A
superuser edits **Requirements** and presses **Apply**. Packages are synced
with `uv pip sync`; deleting a requirement removes it. Sync installs only
what is listed, so list the dependencies too, for example the output of
`uv pip compile`. The manifest is versioned with the flows, and successful
installs refresh the workers.
superuser edits **Requirements** and presses **Apply**. The list is resolved
with `uv pip compile` and synced with `uv pip sync`, so each package brings
its dependencies and deleting a requirement removes it. The manifest is
versioned with the flows, and successful installs refresh the workers.
- **Adopted environment:** a fresh pip installation inside your own virtual
environment can use that environment for nodes too. Modules is read-only.
Install dependencies there with pip or uv, then sync or restart Fluksio.