281 lines
11 KiB
Markdown
281 lines
11 KiB
Markdown
# Runs: pipelines that finish
|
|
|
|
A **live flow** reacts to events. A **batch flow** starts with parameters,
|
|
runs its graph once and returns a result. Each execution is a **run**, with
|
|
its own state, parameters, node records, metrics and artifacts.
|
|
|
|
Use runs for experiments, reports and other jobs with a finish line. For a
|
|
complete first example, start with the
|
|
[data science guide](../getting-started/data-science.md).
|
|
|
|
## A batch flow
|
|
|
|
In Python, `Flow(...)` defaults to `mode="batch"`. Declare `inputs` for
|
|
run parameters and `outputs` for the final result. On the canvas, choose batch
|
|
mode and define the same inputs and outputs.
|
|
|
|
A batch flow does not need activation and does not open subscriptions,
|
|
schedules or webhooks. A run executes each node once after its dependencies
|
|
finish. Missing parameters use their declared initial values.
|
|
|
|
Non-streaming ports cannot use `interval` in a batch flow: there is no timer
|
|
to deliver a delayed value. Streaming ports can use it to reduce display
|
|
updates while keeping every metric point.
|
|
|
|
## Submitting
|
|
|
|
In **Runs → New run**, choose a flow, enter its inputs and press **Run**.
|
|
**Use draft** runs the working copy without publishing it.
|
|
|
|
From a terminal:
|
|
|
|
```sh
|
|
fluksio run train --lr 0.1 --epochs 20 --wait
|
|
```
|
|
|
|
Or submit through the API:
|
|
|
|
```sh
|
|
curl -X POST "$FLUKSIO/runs/flows/train" \
|
|
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
|
-d '{"params": {"lr": 0.1, "epochs": 20}, "seed": 7}'
|
|
```
|
|
|
|
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN`
|
|
is an access token. The path takes the flow's id, here a flow whose id is
|
|
its key, as for one made before ids existed. See
|
|
[The HTTP API](../code/api.md) for authentication.
|
|
|
|
Submission returns immediately with status `queued`. Read
|
|
`GET /api/v1/runs/{id}` for status and results. Unknown parameters or wrong
|
|
types return 422 before execution.
|
|
|
|
The API body also accepts `"draft": true`, `"no_cache": true` and an
|
|
`idempotency_key`. Reusing an idempotency key returns the original submission
|
|
instead of creating another run. Drafts are read when execution starts, so
|
|
edits made while a run is queued can affect it.
|
|
|
|
A seed is recorded on the run and fills a declared input named `seed`.
|
|
Your node code must use that input to seed its random number generator.
|
|
|
|
### Sweeps
|
|
|
|
A sweep submits every combination of input values as one group:
|
|
|
|
```sh
|
|
fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait
|
|
```
|
|
|
|
These four runs share a `group_id` and execute concurrently within the
|
|
engine's limits. Each has its own state, so one run cannot overwrite another's
|
|
messages. Vary seeds with `--param seed=1,2,3` if the flow declares that input.
|
|
|
|
The API equivalent is `POST /api/v1/runs/flows/{flow id}/sweep` with a
|
|
`runs` list of parameter dictionaries and optional seeds. Filter history
|
|
with `GET /api/v1/runs?group=<group-id>`.
|
|
|
|
## Producing values before you are finished
|
|
|
|
Training curves and progress values leave through output ports. A Python
|
|
generator publishes them with `yield`; callbacks can use `fluksio.emit`.
|
|
Declare the port with `stream: true`. See
|
|
[Writing node code](../code/nodes.md#producing-values-over-time) for examples.
|
|
|
|
Every numeric emission is stored under its flow-qualified message name,
|
|
such as `train.loss`, with a step counter and timestamp. Retrieve it with:
|
|
|
|
```text
|
|
GET /api/v1/runs/{id}/metrics?name=train.loss
|
|
GET /api/v1/runs/series/compare?ids=<a>,<b>&metric=train.loss
|
|
```
|
|
|
|
A streaming port's `interval` reduces updates sent to the canvas, while the
|
|
run history keeps every point. In a run, intermediate emissions do not start
|
|
downstream nodes; those wait for the node's final result.
|
|
|
|
A node's `timeout` measures silence. Each yield or emit resets it, so a node
|
|
can run for hours while reporting progress. No timeout is imposed by default.
|
|
|
|
## Artifacts
|
|
|
|
Use `fluksio.save_artifact` for checkpoints, datasets, plots and other bytes.
|
|
Return its reference through an `artifact` port. Another node calls
|
|
`fluksio.load_artifact` to obtain a local path.
|
|
|
|
Identical bytes share one stored copy. Returned artifacts are listed on the
|
|
run and can be downloaded with:
|
|
|
|
```sh
|
|
fluksio artifacts <run-id>
|
|
fluksio artifacts <run-id> train.weights -o weights.json
|
|
```
|
|
|
|
Use the exact message name shown by the list command. Emitted media frames
|
|
are transient; return a frame you need to keep as a run artifact.
|
|
|
|
## Stage caching
|
|
|
|
Before executing a Python node, the engine checks for a successful previous
|
|
execution with matching source, settings, port declarations and input values.
|
|
On a hit, it restores the outputs and artifacts and records the node as
|
|
`cached`.
|
|
|
|
For code-defined flows, sync also records digests of the project modules
|
|
reachable from each node. Editing those modules invalidates the corresponding
|
|
stage. Static import tracking does not follow computed import names or track
|
|
installed dependency versions; use `--no-cache` after changes it cannot see.
|
|
|
|
Artifact inputs are compared by content digest. Secret settings contribute
|
|
their references rather than secret values. External state is not part of a
|
|
cache key, so disable caching on a node that reads changing data or performs
|
|
side effects:
|
|
|
|
```python
|
|
@node(provides=Port("reading", "float"), cache=False)
|
|
def read_sensor():
|
|
return get_current_reading()
|
|
```
|
|
|
|
Use `fluksio run ... --no-cache`, `fluksio sweep ... --no-cache`, or
|
|
`"no_cache": true` in an API submission to execute every node.
|
|
|
|
Only Python nodes are cached. Built-in and connector nodes are not. The cache
|
|
uses run history; outputs too large to store there are not cacheable, and
|
|
missing artifact bytes make an entry a miss. An entry written before the cache
|
|
key last changed shape can never match again, so the engine drops it from the
|
|
cache and keeps the node's recorded outputs.
|
|
|
|
A cache hit does not replay emissions. Its metrics are read from the original
|
|
run, so comparisons still show the recorded curve. Deleting that original run
|
|
leaves cached results available but removes their curve. The run's page then
|
|
names the deleted run each lost curve went with.
|
|
|
|
## Objects that cannot be serialized
|
|
|
|
Live models, data loaders and compiled functions cannot cross a JSON message
|
|
boundary. Keep stages that share live objects in one node, or save a checkpoint
|
|
artifact and rebuild the object in the next node.
|
|
|
|
## Running a node somewhere else
|
|
|
|
Attach a [remote worker](../code/workers.md) to the engine and declare the
|
|
worker's label as the node's `worker`. `worker_policy="require"` waits for
|
|
that label; `"prefer"` uses the local engine when none is attached.
|
|
|
|
The selected machine needs your imported packages. Code-defined flows also
|
|
need your project installed or available there: generated import shims travel
|
|
with the call, but your package does not. Workers connect outward to the
|
|
engine; see the worker guide for tokens and resource requests.
|
|
|
|
## Durability
|
|
|
|
Queued runs are stored in the database and picked up when the engine restarts.
|
|
Running jobs refresh a lease. If the engine dies and the lease expires, those
|
|
runs become `abandoned` rather than being executed again automatically.
|
|
|
|
Retry a finished run or the unfinished results of a sweep:
|
|
|
|
```sh
|
|
fluksio retry <run-id>
|
|
fluksio retry --group <group-id>
|
|
```
|
|
|
|
A retry creates a new run with the previous inputs, seed and group, linked
|
|
to its parent. It uses the flow's current definition, and can reuse completed
|
|
cached stages. Cancel active runs before retrying them.
|
|
|
|
## Looking at what ran
|
|
|
|
The **Runs** screen filters by flow, status or sweep. Each run shows its
|
|
parameters, per-node timings and logs, traceback, metrics, artifacts and
|
|
result. Its inputs are listed as the flow declared them when the run was
|
|
submitted, and the ones it took from the flow are marked as defaults.
|
|
|
|
Select two or more runs to compare curves and the parameters that differ.
|
|
Plot against step, elapsed seconds or another metric from the same runs.
|
|
Comparisons have shareable URLs.
|
|
|
|
Deleting a flow keeps its run history, and the artifacts those runs recorded
|
|
stay on disk. You can still inspect those runs, but cannot retry them until
|
|
the flow exists again. Tick **Also delete its runs** in the delete dialog to
|
|
clear them with the flow, or delete history separately after active runs
|
|
finish or are cancelled.
|
|
|
|
### Taking it into a dataframe
|
|
|
|
Export metrics as a long table and results as a row per run:
|
|
|
|
```sh
|
|
fluksio export metrics --flow train -o curves.csv
|
|
fluksio export runs --flow train --status ok -o results.csv
|
|
```
|
|
|
|
Both include run IDs. Result rows also include input values, final numeric
|
|
outputs, status, duration and code identifiers. Nested values use dotted paths,
|
|
such as `metric.final_metrics.train_loss`. A row names the flow by its id in
|
|
`flow` and by its key in `flow_name`. History stays with the id, so a flow
|
|
keeps its runs whatever its key is later, and a new flow that takes the key
|
|
of a deleted one starts without them. Metric names keep the key, as in
|
|
`train.loss`.
|
|
|
|
Python returns the same rows directly. `flow` takes the flow's id or key,
|
|
as everywhere in the client:
|
|
|
|
```python
|
|
from fluksio.sdk.client import Client
|
|
|
|
client = Client()
|
|
curves = client.export_metrics(flow="train")
|
|
results = client.export_runs(flow="train", status="ok")
|
|
```
|
|
|
|
Pass either list to `pandas.DataFrame`. The
|
|
[CLI reference](../code/cli.md#fluksio-export) covers selection filters,
|
|
JSON Lines and Parquet.
|
|
|
|
### Bringing in runs made elsewhere
|
|
|
|
Import historical runs without executing them:
|
|
|
|
```python
|
|
from fluksio.sdk.client import Client
|
|
|
|
Client().import_runs([{
|
|
"flow": "train",
|
|
"external_id": "previous-study:run-42",
|
|
"params": {"lr": 0.05, "epochs": 10},
|
|
"status": "ok",
|
|
"created_at": "2026-03-01T10:00:00Z",
|
|
"result": {"score": 0.94},
|
|
}])
|
|
```
|
|
|
|
The flow must exist, and parameters are checked against its inputs. The
|
|
Python client takes the flow's id or key for `flow`. Over the HTTP API an
|
|
entry's `flow` is the flow's id. Repeated imports of the same
|
|
`external_id` reuse the existing record. Imported runs
|
|
have no code commit or digest because the engine did not execute them.
|
|
Entries may also carry metric points and base64-encoded artifacts; see the
|
|
API schema for their fields.
|
|
|
|
### A dashboard, read against runs
|
|
|
|
Choose **Open in dashboard** from a run or comparison to view those results
|
|
with a dashboard's widgets. Charts draw one line per run, scalar widgets use
|
|
the first run, and controls are disabled.
|
|
|
|
If the flow has no dashboard, the same menu can create one from its streaming
|
|
ports and outputs. The API equivalent is
|
|
`POST /api/v1/dashboards/from-flow/{flow id}`. The generated dashboard is
|
|
published and can be edited like any other dashboard.
|
|
|
|
To keep a chart showing recent experiments, set its source to **Runs** and
|
|
select the latest runs of a flow, a sweep or specific run IDs.
|
|
|
|
## See also
|
|
|
|
- [Getting started: data science](../getting-started/data-science.md)
|
|
- [Writing node code](../code/nodes.md)
|
|
- [Remote workers](../code/workers.md)
|
|
- [The HTTP API](../code/api.md)
|