stroblmeandClaude Fable 5
a38e2745eb
A data scientist keeps their code where it is and decorates it: `@node` declares a function's ports beside the function, `Flow(name, nodes=[...])` says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds one for a single flow. `fluksio sync` uploads the document plus a generated import shim per node, so the store still holds a complete, runnable, git-versioned definition while the code it imports stays theirs. `fluksio login|run|runs` and `flow.submit().wait()` are the client half, over the run endpoints that already existed. Runs record the user repository's commit beside the store's, so "what code produced this number" is answerable on the side that now holds the code. - `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim generator, an HTTP client and sync. Standard library only at import, so `from fluksio import node` in a training script pulls in no engine. - `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the repository's commit; `POST /modules/refresh` retires the workers without an install, which every sync calls — a worker holds the imported package in memory, so an edit to it is invisible until the process goes. - The canvas shows a generated body read-only and names the repository to edit instead; a body edited there stops the next sync rather than being discarded. - The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since the shim imports a module whose first line declares them. - `examples/myresearch` is the worked example, `make sync-example` uploads it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
73 lines
2.6 KiB
Python
73 lines
2.6 KiB
Python
"""The Fluksio engine, and the decorators that declare flows in your own code.
|
|
|
|
Two audiences, one import name. In a research repository, ``from fluksio
|
|
import Port, node, Flow`` is the authoring API: it says which of your existing
|
|
functions are nodes and which nodes make up a flow, and ``fluksio sync``
|
|
uploads what it finds. Those names come from :mod:`fluksio.sdk`, which imports
|
|
nothing but the standard library.
|
|
|
|
Inside a node, ``import fluksio`` is not this package at all: the worker
|
|
installs a reporter of its own under that name before any node code runs, so
|
|
what a node gets is :mod:`fluksio_worker.worker_main`'s ``emit`` and
|
|
``save_artifact`` — and inert copies of the decorators, since a module the
|
|
node imports may well declare them at its top. The stubs below stand in the
|
|
same place everywhere else, and say so rather than failing as a missing
|
|
attribute.
|
|
|
|
Nothing is imported from the rest of the package here. Every ``from fluksio.x
|
|
import y`` in the engine passes through this module, and an import cycle or a
|
|
second of start-up cost would both begin here.
|
|
"""
|
|
|
|
import logging
|
|
from importlib.metadata import PackageNotFoundError
|
|
from importlib.metadata import version as _version
|
|
from typing import Any
|
|
|
|
from fluksio.sdk import Flow, Port, node, use
|
|
|
|
try:
|
|
__version__ = _version("fluksio")
|
|
except PackageNotFoundError: # pragma: no cover - a checkout that was never installed
|
|
__version__ = "0.0.0+unknown"
|
|
|
|
__all__ = [
|
|
"Flow",
|
|
"Port",
|
|
"emit",
|
|
"load_artifact",
|
|
"node",
|
|
"save_artifact",
|
|
"use",
|
|
]
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
_OUTSIDE = (
|
|
"fluksio.{name}() only works inside a node: the worker running it installs "
|
|
"the real one. There is nothing to {verb} out here."
|
|
)
|
|
|
|
|
|
def emit(**ports: Any) -> None:
|
|
"""Publish values on a node's declared output ports, mid-run.
|
|
|
|
Outside a node there is nowhere to publish to, and this does nothing on
|
|
purpose: a node function called directly — in a test, in a notebook, under
|
|
a debugger — is ordinary Python, and having it die on a progress report
|
|
would defeat the point of the code staying yours.
|
|
"""
|
|
logger.debug("fluksio.emit(%s) outside a node", ", ".join(ports))
|
|
|
|
|
|
def save_artifact(
|
|
source: Any, name: str = "", media_type: str = "application/octet-stream"
|
|
) -> dict[str, Any]:
|
|
"""Put bytes in the artifact store and return a reference to them."""
|
|
raise RuntimeError(_OUTSIDE.format(name="save_artifact", verb="save to"))
|
|
|
|
|
|
def load_artifact(ref: dict[str, Any]) -> str:
|
|
"""Fetch what an artifact reference points at, and return a path to it."""
|
|
raise RuntimeError(_OUTSIDE.format(name="load_artifact", verb="load from"))
|