Add a Python SDK: flows declared in your own repository

A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.

`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.

- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
  generator, an HTTP client and sync. Standard library only at import, so
  `from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
  repository's commit; `POST /modules/refresh` retires the workers without an
  install, which every sync calls — a worker holds the imported package in
  memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
  instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
  the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
2026-08-23 20:16:08 +02:00
co-authored by Claude Fable 5
parent 775d151307
commit a38e2745eb
35 changed files with 2693 additions and 142 deletions
+34 -1
View File
@@ -21,6 +21,7 @@ from fluksio.api.deps import (
from fluksio.flow import modules
from fluksio.flow.events import event_bus
from fluksio.flow.schemas import ApplyRequest, ApplyResult, ModulesInfo
from fluksio.models import Message
router = APIRouter(
prefix="/modules", tags=["modules"], dependencies=[Depends(get_current_user)]
@@ -70,6 +71,38 @@ async def apply_modules(
# Retire the workers first, so the rebuild compiles every node against the
# packages that were just installed — a node that could not import one is
# the reason this was called, and it stays red until it is built again.
await _retire(controller, pool)
return ApplyResult(ok=True, output=output)
@router.post("/refresh", response_model=Message)
async def refresh_modules(
controller: FlowControllerDep,
pool: WorkerPoolDep,
user: CurrentUser,
) -> Any:
"""Retire the workers without installing anything.
A node that imports the caller's own package holds it in `sys.modules` for
as long as the process lives, so editing that package changes nothing a
running worker can see — recompiling the node would not help either, since
the import returns the module already there. Retiring the processes is the
whole of it, and `fluksio sync` asks for it after every upload.
"""
event_bus.publish(
{
"type": "audit",
"action": "refreshed workers",
"flow": "",
"user": user.email,
"ts": time.time(),
}
)
await _retire(controller, pool)
return Message(message="Workers retired; the next call imports afresh")
async def _retire(controller: Any, pool: Any) -> None:
"""Send the worker processes away and rebuild whatever was broken."""
pool.respawn_all()
await controller.reload_failed_flows()
return ApplyResult(ok=True, output=output)