From e5be1880fd29b6874439dce4fc1b0309a7f31999 Mon Sep 17 00:00:00 2001 From: stroblme Date: Sat, 22 Aug 2026 10:31:14 +0200 Subject: [PATCH] Docs: a node can import the project you already have MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "What it can import is what the Modules screen installed" was true and read as a wall: every example is a self-contained file, and a function that imports half a repository looks unsupported. It is not — the manifest is handed to `uv pip sync` verbatim, so `-e /home/you/repo` installs the project you already have and a node becomes a three-line wrapper over it. The code stays in your own repository, under your own version control, importing its own siblings. Two caveats that are easy to lose an afternoon to, so both are written down: an editable install reaches the venv without a reinstall but not into a worker that already imported it — the engine's workers are long-lived, and Apply is what retires them, while an attached worker starts a process per call and is always fresh. And the path is resolved on whichever machine runs the node, while the manifest is committed to the flow repository, so a laptop path means nothing in a container. Verified rather than assumed: `uv pip sync` takes `-e`, the shim body loads through the real `load_function`, an edit is live in a fresh process and stale in a reused one. Co-Authored-By: Claude Opus 5 (1M context) --- docs/code/nodes.md | 48 +++++++++++++++++++++++++++- docs/getting-started/data-science.md | 43 +++++++++++++++++++++++++ 2 files changed, 90 insertions(+), 1 deletion(-) diff --git a/docs/code/nodes.md b/docs/code/nodes.md index 6f6ad33..aedcb9b 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -28,7 +28,8 @@ through a port it declared. **Nothing else is importable from the engine.** Node code runs in a separate process, on a separate interpreter, with none of Fluksio's own modules on its path. What it can import is what the [Modules](../interface/operations.md) -screen installed. +screen installed — which can include [your own project](#your-own-code-as-a-package), +so a node need not be a self-contained file. **A node is a pure function of its inputs.** No context object, no global store, no handle to reach for. A running total or a debounce timer has a @@ -179,6 +180,51 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ It is a pip manifest installed with `uv pip sync`, versioned alongside your flows. An install takes effect immediately; nothing restarts. +### Your own code as a package + +A manifest line can name a directory, so the project you already have is +installable like any other dependency: + +```text +-e /home/you/my-research +numpy>=2 +``` + +A node body then imports it, and the logic stays where it already lives — in +your repository, under your own version control, importing its own siblings: + +```python +from myresearch.train import fit + + +def process(lr, epochs): + return fit(lr, epochs) +``` + +That is the whole of it. The node is three lines, `myresearch` can be as many +modules as it likes, and nothing was copied. + +!!! warning "Editable, but not live" + + `-e` means edits reach the venv without reinstalling — but a node's process + already holds the imported module in memory. The engine's workers are + long-lived, so a change to your code is picked up when they are retired, + which is what **Apply** does. Pressing it after an edit is the loop. + + A [worker](workers.md) you attach yourself is the exception: it starts a + process per call, so it reads your code fresh every run. If you are + iterating on the code many times an hour, point one at your own + interpreter — `fluksio-worker --python "$(which python)"` — and mark the + node with its label. + +!!! note "The path is a deployment detail" + + It is resolved on whichever machine runs the node, and the manifest is + committed to the flow repository — so an absolute path from your laptop + means nothing inside a container or on a GPU box. Those need their own + install of the same project; a VCS requirement + (`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not. + ## A worked example The repository ships a small supervised fit as a seedable demo — three nodes, diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index 155ee0e..0fb4ab9 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -140,6 +140,49 @@ function is invisible to all three. checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a small reference. Nothing changes about how you write the file. +### If your code does not fit in one file + +The example above is a single self-contained file, which most real projects are +not. If the function you want as a node imports half your repository, do not +move it — install the repository into the venv the nodes run on, by adding one +line to the manifest you just applied: + +```text +-e /home/you/my-research +``` + +Now the node is a wrapper over what you already have, and your code stays in +your own repository, under your own version control, importing its own +siblings as it always did: + +```python +"""The node. The training lives in the project, where it belongs.""" + +from myresearch.train import fit + + +def process(lr, epochs): + return fit(lr, epochs) +``` + +A generator still works through the wrapper — `yield from fit(...)` — so the +per-epoch metrics arrive exactly as before. + +!!! warning "Apply after you edit" + + The engine's workers are long-lived and hold your imported modules in + memory, so a change to `myresearch` is picked up when they are retired — + which is what **Apply** does. If you are editing many times an hour, attach + your own interpreter as a worker instead: it starts a process per call and + therefore reads your code fresh every run. + + ```sh + fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \ + --token "$WORKER_TOKEN" --labels local --python "$(which python)" + ``` + + Then mark the node `"device": "local"`. + !!! note "Where a `yield` cannot reach" If the number comes from inside somebody else's callback — Keras, Lightning,