Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a Python SDK: flows declared in your own repository

Browse Source 
A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.

`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.

- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
  generator, an HTTP client and sync. Standard library only at import, so
  `from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
  repository's commit; `POST /modules/refresh` retires the workers without an
  install, which every sync calls — a worker holds the imported package in
  memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
  instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
  the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
stroblme
2026-08-23 20:16:08 +02:00
co-authored by Claude Fable 5
parent 775d151307
commit a38e2745eb
35 changed files with 2693 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/cli.md
+61
View File
@@ -11,6 +11,10 @@ There is a second, smaller distribution — `fluksio-worker` — for a machine t
should only *run nodes* for an engine elsewhere. It has none of the engine in
it. See [Remote workers](workers.md).
The command is two things at once: `serve`, `enroll` and `worker` *are* an
installation, while `login`, `sync`, `run` and `runs` talk to one that may be
anywhere.
## `fluksio serve`
Runs the engine.
@@ -100,6 +104,63 @@ fluksio worker --url wss://api.example.com/api/v1/workers/attach \
See [Remote workers](workers.md).
## Talking to an engine
The four commands below are the client half: they run wherever you work, and
address an engine over its API rather than being one.
### `fluksio login`
```sh
fluksio login --url http://127.0.0.1:8000
```
Asks for an email and password, and keeps the token it gets in
`~/.config/fluksio/client.json` (`$XDG_CONFIG_HOME` is honoured). Everything
below reads it from there, or from `FLUKSIO_URL` and `FLUKSIO_TOKEN`, or from
its own `--url` and `--token`.
### `fluksio sync`
```sh
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
```
Imports what you name, collects the flows the decorators declared, and uploads
each one with a generated import shim per node. A directory that is a package
is walked; a dotted name is imported as it stands; nothing is loaded from a
file path, because the shim has to import the same way.
| Flag | What it does |
|---|---|
| `--dry-run` | print the flow documents and shims, upload nothing |
| `--no-publish` | leave the upload as a draft |
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
Every sync retires the engine's workers, including one that had nothing to
upload — a worker holds your package in memory, so an edit to it is invisible
until the process goes. See
[Getting started: data science](../getting-started/data-science.md).
### `fluksio run`
```sh
fluksio run train --lr 0.05 --seed 7 [--wait]
```
Submits a run. Flags that are not its own are the flow's inputs, typed by what
the flow declares them as. `--wait` blocks until the run finishes and exits
non-zero if it failed.
### `fluksio runs`
```sh
fluksio runs [--flow train] [--limit 20]
```
The runs an engine has recorded, newest first: id, status, flow, duration, the
commit of the repository it came from, and its parameters.
## What lives in the data directory
```text
docs/code/nodes.md
+9
View File
@@ -204,6 +204,15 @@ def process(lr, epochs):
That is the whole of it. The node is three lines, `myresearch` can be as many
modules as it likes, and nothing was copied.
!!! tip "You can have those three lines written for you"
Decorate `fit` with `@node(...)` where it is defined, say which nodes make
a flow with `Flow(...)`, and `fluksio sync` generates the body above —
along with the flow document, so there is nothing to PUT by hand. The
declaration lives beside the function it describes and is checked against
its signature. See
[Getting started: data science](../getting-started/data-science.md).
!!! warning "Editable, but not live"
`-e` means edits reach the venv without reinstalling — but a node's process
docs/getting-started/data-science.md
+264 -124
View File
@@ -44,14 +44,21 @@ notices.
orchestrator spends before it does anything. Leave it in a `tmux` window,
or write a small `systemd --user` unit for it.
## Get a token
## Log in
Everything below is the HTTP API. Grab a token once:
```sh
fluksio login --url http://127.0.0.1:8000
```
It asks for the email and password printed above and keeps the token in
`~/.config/fluksio/client.json`, so nothing below needs credentials again.
Everything the commands do is the HTTP API, and some of this page shows it
directly. For that, grab the same token as a shell variable:
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
-d "username=admin@example.com&password=k3Qm-8vTpLdX" | jq -r .access_token)
export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
```
While you are experimenting, the interactive schema at
@@ -90,17 +97,18 @@ Adding a package takes effect immediately; nothing restarts.
Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).
## Wrap your training script
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
from parameters to a result, which is what an experiment is. Your existing
script becomes the body of a node.
from parameters to a result, which is what an experiment is. A node is one of
your own functions — it stays in your repository, imported by its siblings as
it always was.
Say your script looks roughly like this:
Say your project looks roughly like this:
```python
def train(lr, epochs):
# myresearch/train.py
def fit(dataset, lr, epochs=25):
model = build_model()
for epoch in range(epochs):
loss = step(model, lr)
@@ -109,17 +117,27 @@ def train(lr, epochs):
return loss
```
Two changes turn it into a node:
Two changes turn it into a node, and neither of them moves it:
```python
"""Fit the model. A generator, so numbers escape while it is still running."""
# myresearch/train.py
import fluksio
from fluksio import Port, node
def process(lr, epochs):
model = build_model()
for epoch in range(int(epochs)):
@node(
requires=["dataset", Port("lr", "float")],
provides=[
Port("loss", "float", stream=True),
Port("weights", "artifact"),
Port("final_loss", "float"),
],
timeout=600,
)
def fit(dataset, lr, epochs=25):
"""Fit the model. A generator, so numbers escape while it is running."""
model = build_model(fluksio.load_artifact(dataset))
for epoch in range(epochs):
loss = step(model, lr)
yield {"loss": loss} # ← published now, on the loss port
torch.save(model.state_dict(), "weights.pt")
@@ -140,48 +158,23 @@ function is invisible to all three.
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
small reference. Nothing changes about how you write the file.
### If your code does not fit in one file
The decorator returns the function untouched, so `fit(dataset, 0.05)` in a
test, a notebook or a debugger is exactly what it was before. Outside a run
`fluksio.emit` does nothing rather than failing, so a progress report never
stops your own code from running.
The example above is a single self-contained file, which most real projects are
not. If the function you want as a node imports half your repository, do not
move it — install the repository into the venv the nodes run on, by adding one
line to the manifest you just applied:
### What is declared, and what is not
```text
-e /home/you/my-research
```
Now the node is a wrapper over what you already have, and your code stays in
your own repository, under your own version control, importing its own
siblings as it always did:
```python
"""The node. The training lives in the project, where it belongs."""
from myresearch.train import fit
def process(lr, epochs):
return fit(lr, epochs)
```
A generator still works through the wrapper — `yield from fit(...)` — so the
per-epoch metrics arrive exactly as before.
!!! warning "Apply after you edit"
The engine's workers are long-lived and hold your imported modules in
memory, so a change to `myresearch` is picked up when they are retired —
which is what **Apply** does. If you are editing many times an hour, attach
your own interpreter as a worker instead: it starts a process per call and
therefore reads your code fresh every run.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
```
Then mark the node `"device": "local"`.
- **Ports are declared, never inferred.** `requires` names the messages the
node reads; each arrives as the parameter of the same name. A bare string is
the shorthand — `"dataset"` takes its type from whoever provides it, so a
message's type is written down exactly once.
- **Parameters with a default that are not ports become settings.** `epochs`
above is stored with the flow and tunable on the canvas without touching
this file.
- **The declaration is checked against the function.** A port with no matching
parameter, or a parameter that is neither port nor setting, is an error when
the module is imported — not when the node is first called.
!!! note "Where a `yield` cannot reach"
@@ -195,92 +188,172 @@ per-epoch metrics arrive exactly as before.
)])
```
## Create the flow
## Say which nodes make a flow
There is no scaffolding command yet, so a flow is created by PUTting its
definition. That is a fifteen-line script you run once:
Membership is a list, not a directory layout: the functions can live wherever
they already do.
```python
"""Create the `train` flow. Run once; edit it in the canvas afterwards."""
# myresearch/pipeline.py
from fluksio import Flow, Port, use
import httpx
from myresearch.data import augment, prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit
API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
"/login/access-token",
data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"
api.put("/flows/train", json={
"name": "train",
"title": "Model training",
# Batch: nothing is activated, nothing fires until a run asks.
"mode": "batch",
# Its inputs are the run's parameters, with the values a run gets when it
# names none.
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
],
# What a run reports as its result.
"outputs": ["final_loss", "weights"],
"nodes": [{
"id": "train",
"type": "python",
"title": "Fit the model",
# An *idle* timeout once the node streams: this is how long it may go
# quiet, not how long it may run.
"timeout": 600,
"requires": [
{"name": "lr", "dtype": "float"},
{"name": "epochs", "dtype": "int"},
],
"provides": [
# `stream` says this port publishes repeatedly during one execution.
{"name": "loss", "dtype": "float", "stream": True},
{"name": "final_loss", "dtype": "float"},
{"name": "weights", "dtype": "artifact"},
],
}],
}).raise_for_status()
api.put("/flows/train/nodes/train/source",
json={"code": open("train_node.py").read()}).raise_for_status()
version = api.get("/flows/train", params={"draft": True}).json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
print("published")
train = Flow(
"train",
nodes=[prepare, fit, evaluate],
inputs=[Port("lr", "float", initial=0.01)],
outputs=["score", "final_loss"],
)
```
Two things worth noticing. Ports are declared, not inferred — `process(lr,
epochs)` gets its arguments from the ports of the same name, and the types are
checked on every value. And saving writes a *draft*; `publish` is what the
engine picks up. That separation is what lets you edit a flow that is running.
`inputs` are the run's parameters, with the value a run gets when it names
none; `outputs` are what a run reports as its result.
Nodes are connected because one **provides** a message another **requires** —
never because one imported the other. Importing `fit` into a second flow means
"the same code", not "wired to it".
Which is how the same function serves two flows, rewired and reconfigured for
each:
```python
finetune = Flow(
"finetune",
nodes=[
prepare,
augment,
use(fit, wire={"dataset": "augmented"}, epochs=3),
evaluate,
],
inputs=[Port("lr", "float", initial=1e-4)],
outputs=["score"],
)
```
`use(fn, ...)` is one use of a node in one flow: `wire` binds a port to a
different message, keyword arguments override settings, and `id=` tells two
uses of one function apart. `train` above is unaffected by any of it.
## Sync it
```sh
fluksio sync myresearch
```
That imports the package, checks every declaration, and uploads each flow with
a generated body per node:
```python
# generated by fluksio sync from myresearch/train.py — edit that file instead
import sys
_REPO = '/home/you/my-research'
if _REPO not in sys.path:
sys.path.insert(0, _REPO)
from myresearch.train import fit
def process(dataset, lr, **settings):
return (yield from fit(dataset=dataset, lr=lr, **settings))
```
So the flow store still holds a complete, runnable, git-versioned definition —
the body simply happens to import rather than duplicate. Your code stays in
your repository, under your version control.
The upload is stamped with that repository's commit, and every run records it
alongside the store's own. `--dry-run` prints all of this and uploads nothing;
`--no-publish` leaves it as a draft.
!!! warning "Sync after you edit"
The engine's workers are long-lived and hold your imported modules in
memory, so an edit to `myresearch` is invisible until they are retired —
which is what every `fluksio sync` does, including one that has nothing to
upload. If you are editing many times an hour, attach your own interpreter
as a worker instead: it starts a process per call and therefore reads your
code fresh every run.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
```
Then give the node `device="local"`.
!!! note "The repository path is a deployment detail"
The generated body puts your repository on `sys.path` by absolute path,
which is right for the machine you synced from and meaningless in a
container. For anything else, install the package where the node runs —
`-e /home/you/my-research` in the module manifest, or
`myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not.
### Which hardware a node runs on
`device` picks the worker, the same way an executor does elsewhere:
```python
@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer")
def fit(dataset, lr, epochs=25):
...
```
`device_policy="require"` (the default) waits for a worker carrying that label;
`"prefer"` runs it locally when none is attached, which is what you want while
the GPU box is not switched on. See [Remote workers](../code/workers.md).
### What the canvas does with a synced flow
Its node bodies are generated, so the editor shows them read-only and says
where the real code is. Everything else behaves as usual — but ports and
settings changed there are overwritten by the next sync, which is the point of
the repository being the source of truth. A node whose code you edit on the
canvas makes the next sync stop and say so rather than discarding your edit;
`--force` overrides that.
## Run it
```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.003, "epochs": 200}, "seed": 7}'
fluksio run train --lr 0.003 --seed 7
fluksio runs --flow train
```
It answers immediately with a queued run — training is measured in hours, so
nothing waits for it. A parameter you did not declare, or one of the wrong
type, is refused with a 422 before anything executes.
`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
parameter you did not declare, or one of the wrong type, is refused before
anything executes. It answers immediately with a queued run — training is
measured in hours, so nothing waits for it unless you pass `--wait`.
Then, whenever you like:
From Python, the flow you declared is also the handle to its runs:
```python
from myresearch.pipeline import train
run = train.submit(lr=0.003, seed=7).wait()
print(run.status, run.result)
print(run.metrics("train.loss")[-1]) # the whole series is kept
open("weights.pt", "wb").write(run.download("train.weights"))
```
Or over HTTP, which is what both of those are:
```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.003}, "seed": 7}'
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
```
The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, and every artifact it produced. That is
the answer to "what was the learning rate on the run that got 94%?".
long each node took, what it logged, every artifact it produced, and both
commits — the flow store's and your repository's. That is the answer to "what
was the learning rate on the run that got 94%?".
## Sweep it
@@ -309,6 +382,71 @@ curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
which answers in exactly the shape a chart widget draws.
## What sync does, at the API level
Nothing here is privileged: `sync` is a client, and a flow is a document you
can PUT yourself. The whole of it, for the one-node case:
```python
"""Create the `train` flow by hand. What `fluksio sync` automates."""
import httpx
API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
"/login/access-token",
data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"
api.put("/flows/train", json={
"name": "train",
"title": "Model training",
# Batch: nothing is activated, nothing fires until a run asks.
"mode": "batch",
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
],
"outputs": ["final_loss", "weights"],
"nodes": [{
"id": "train",
"type": "python",
"title": "Fit the model",
# An *idle* timeout once the node streams: this is how long it may go
# quiet, not how long it may run.
"timeout": 600,
"requires": [
{"name": "lr", "dtype": "float"},
{"name": "epochs", "dtype": "int"},
],
"provides": [
# `stream` says this port publishes repeatedly during one execution.
{"name": "loss", "dtype": "float", "stream": True},
{"name": "final_loss", "dtype": "float"},
{"name": "weights", "dtype": "artifact"},
],
}],
}).raise_for_status()
api.put("/flows/train/nodes/train/source",
json={"code": open("train_node.py").read()}).raise_for_status()
version = api.get("/flows/train").json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
```
Two things worth noticing, because the decorators only move where they are
said. Ports are declared, not inferred — `process(lr, epochs)` gets its
arguments from the ports of the same name, and the types are checked on every
value. And saving writes a *draft*; `publish` is what the engine picks up.
That separation is what lets you edit a flow that is running.
A flow uploaded this way carries no `origin`, which is what tells the canvas —
and the next `fluksio sync` — that it was not generated. See
[The API](../code/api.md).
## Small scripts you are just playing with
The same machinery, minus the ceremony. If what you want is "keep a record of
@@ -328,9 +466,10 @@ For quick iteration, keep the flow small (one node is fine), keep the engine
running, and submit from wherever you are working:
```python
import httpx
run = httpx.post(f"{API}/runs/flows/train", json={"params": {"lr": lr}},
headers=auth).json()
from myresearch.pipeline import train
for lr in (0.001, 0.003, 0.01):
train.submit(lr=lr)
```
A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.
@@ -360,6 +499,7 @@ is the same stack.
artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
and may not do
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs` in full
- [Remote workers](../code/workers.md) — send the training node to the GPU box
and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this