diff --git a/docs/code/nodes.md b/docs/code/nodes.md index 6f6ad33..aedcb9b 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -28,7 +28,8 @@ through a port it declared. **Nothing else is importable from the engine.** Node code runs in a separate process, on a separate interpreter, with none of Fluksio's own modules on its path. What it can import is what the [Modules](../interface/operations.md) -screen installed. +screen installed — which can include [your own project](#your-own-code-as-a-package), +so a node need not be a self-contained file. **A node is a pure function of its inputs.** No context object, no global store, no handle to reach for. A running total or a debounce timer has a @@ -179,6 +180,51 @@ curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ It is a pip manifest installed with `uv pip sync`, versioned alongside your flows. An install takes effect immediately; nothing restarts. +### Your own code as a package + +A manifest line can name a directory, so the project you already have is +installable like any other dependency: + +```text +-e /home/you/my-research +numpy>=2 +``` + +A node body then imports it, and the logic stays where it already lives — in +your repository, under your own version control, importing its own siblings: + +```python +from myresearch.train import fit + + +def process(lr, epochs): + return fit(lr, epochs) +``` + +That is the whole of it. The node is three lines, `myresearch` can be as many +modules as it likes, and nothing was copied. + +!!! warning "Editable, but not live" + + `-e` means edits reach the venv without reinstalling — but a node's process + already holds the imported module in memory. The engine's workers are + long-lived, so a change to your code is picked up when they are retired, + which is what **Apply** does. Pressing it after an edit is the loop. + + A [worker](workers.md) you attach yourself is the exception: it starts a + process per call, so it reads your code fresh every run. If you are + iterating on the code many times an hour, point one at your own + interpreter — `fluksio-worker --python "$(which python)"` — and mark the + node with its label. + +!!! note "The path is a deployment detail" + + It is resolved on whichever machine runs the node, and the manifest is + committed to the flow repository — so an absolute path from your laptop + means nothing inside a container or on a GPU box. Those need their own + install of the same project; a VCS requirement + (`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not. + ## A worked example The repository ships a small supervised fit as a seedable demo — three nodes, diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index 155ee0e..0fb4ab9 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -140,6 +140,49 @@ function is invisible to all three. checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a small reference. Nothing changes about how you write the file. +### If your code does not fit in one file + +The example above is a single self-contained file, which most real projects are +not. If the function you want as a node imports half your repository, do not +move it — install the repository into the venv the nodes run on, by adding one +line to the manifest you just applied: + +```text +-e /home/you/my-research +``` + +Now the node is a wrapper over what you already have, and your code stays in +your own repository, under your own version control, importing its own +siblings as it always did: + +```python +"""The node. The training lives in the project, where it belongs.""" + +from myresearch.train import fit + + +def process(lr, epochs): + return fit(lr, epochs) +``` + +A generator still works through the wrapper — `yield from fit(...)` — so the +per-epoch metrics arrive exactly as before. + +!!! warning "Apply after you edit" + + The engine's workers are long-lived and hold your imported modules in + memory, so a change to `myresearch` is picked up when they are retired — + which is what **Apply** does. If you are editing many times an hour, attach + your own interpreter as a worker instead: it starts a process per call and + therefore reads your code fresh every run. + + ```sh + fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \ + --token "$WORKER_TOKEN" --labels local --python "$(which python)" + ``` + + Then mark the node `"device": "local"`. + !!! note "Where a `yield` cannot reach" If the number comes from inside somebody else's callback — Keras, Lightning,