Add a Python SDK: flows declared in your own repository

A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.

`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.

- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
  generator, an HTTP client and sync. Standard library only at import, so
  `from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
  repository's commit; `POST /modules/refresh` retires the workers without an
  install, which every sync calls — a worker holds the imported package in
  memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
  instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
  the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
2026-08-23 20:16:08 +02:00
co-authored by Claude Fable 5
parent 775d151307
commit a38e2745eb
35 changed files with 2693 additions and 142 deletions
+13
View File
@@ -276,6 +276,18 @@ class MetricSink:
)
def origin_commit(flow: FlowDef) -> str:
"""The user repository's commit, for a flow declared in code elsewhere.
Marked dirty when the tree had uncommitted changes at sync, because then
the hash names something other than what actually ran.
"""
origin = flow.origin
if origin is None or not origin.commit:
return ""
return f"{origin.commit}-dirty" if origin.dirty else origin.commit
class RunService:
"""Accepts runs, drives them, and writes down what they did."""
@@ -360,6 +372,7 @@ class RunService:
flow=flow.name,
flow_version=flow.version,
commit=self.controller.store.head(),
origin_commit=origin_commit(flow),
params=params,
params_digest=digest_of(params, seed),
seed=seed,
+31
View File
@@ -77,6 +77,30 @@ class NodeDef(BaseModel):
return _validate_name(value)
class FlowOrigin(BaseModel):
"""Where a flow was declared, when that was somewhere other than here.
A flow drawn on the canvas has no origin: the store is where it lives. One
stamped with this was declared with the decorators in somebody's own
repository and put here by ``fluksio sync``, so the node bodies below it
are generated imports and the code they run is versioned twice — once here
and once there. Its presence is what makes a flow code-defined.
Deliberately no timestamp. The store commits every change it is given, so
when a flow was last synced is a fact its own history already holds — and
one that would otherwise change on every sync, making an unchanged upload
look like a new version of the flow.
"""
kind: Literal["python"] = "python"
#: The repository root on the machine that ran ``sync``.
repo: str = ""
#: Its commit, and whether the tree had uncommitted changes at the time —
#: a run stamped with a dirty commit names code that was never stored.
commit: str = ""
dirty: bool = False
class FlowInput(BaseModel):
"""A message the flow starts with rather than computes."""
@@ -108,6 +132,13 @@ class FlowDef(BaseModel):
"means every message the flow ends up holding."
),
)
origin: FlowOrigin | None = Field(
default=None,
description=(
"Set when the flow was declared in code elsewhere and uploaded by "
"`fluksio sync`. Absent for a flow drawn on the canvas."
),
)
@field_validator("name")
@classmethod