20 Commits

Include renames

Author	SHA1	Message	Date
stroblmeandClaude Opus 5	68d2565054	Say at startup when a flow wants a card, and record one seed rather than two ... Three things the first pass left. `serve` now names the flows asking for a GPU when the engine has none declared. The placer already warned, but into the log, where a fresh install that forgot `--gpus` does not read it — and the cost of missing it is GPU nodes running concurrently, which is what the declaration exists to prevent. The seed was the one field an export still had to coalesce: `--seed 1` filled the run-level column and left `param.seed` blank, while a declared seed filled the parameter and left the column blank. It is resolved like every other input now, and the column carries the seed the run actually used however it arrived — including when a parameter outranks the run's own, where the column used to report the one that lost. And the docs say plainly that declaring the card is what buys the worker retirement: a node that imports jax without `resources={"gpus": 1}` never gets CUDA_VISIBLE_DEVICES, so nothing marks its worker as one holding a card. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc	2026-08-29 15:18:42 +02:00
stroblmeandClaude Opus 5	53b49e5f68	Retire the GPU workers when a run that held a card finishes ... A worker that has run a jax node keeps holding the GPU after the run: XLA preallocates most of the VRAM at import and never releases it, so the next process OOMs on preallocation while a warm idle worker sits on the card. Pools are kept warm on purpose — a library reads its environment at import, so a warm worker cannot be re-told — but the end of a run is a point where the memory should go back, and the environments carrying a GPU assignment are exactly the pools that ran on one. Idle ones go now, busy ones when they return. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc	2026-08-29 13:53:11 +02:00
stroblmeandClaude Opus 5	c050a7a52c	Record the inputs a run actually starts from, not only the ones passed ... A run submitted without explicit inputs recorded `params = {}`: Port initials filled the values at node level and were never written back, so an exported row had a blank `param.*` cell and the runs listing could not tell a run that took every default from one submitted with those same numbers. Declared initials are now folded in at submit, explicit values winning, and the run-level seed still wins over a declared one. `params_digest` is computed over the resolved values, so it changes shape once: a run recorded before this does not dedupe against a newer identical submit, and its stage-cache entries miss once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc	2026-08-29 13:45:49 +02:00
stroblmeandClaude Opus 5	7efa75e242	Refuse a zero concurrency limit instead of reading it as the default ... FLOW_MAX_WORKERS, FLOW_MAX_CASCADES and FLOW_MAX_RUNS are all pool sizes, so 0 says neither "none" nor "unlimited" — it is a pool that cannot be built. They are PositiveInt now, so a 0 fails at startup naming the setting rather than being swallowed by `max_cascades or MAX_CASCADES`. The consuming fallbacks take only None as "nobody said": explicit `is None` in the executor, and no clamp on RunService.parallel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1moruzue2kTJd3uVisgNk	2026-08-28 20:00:08 +02:00
stroblmeandClaude Opus 5	91ef2bbe9a	Key a node on the code it reaches, not on the repository around it ... `sync` follows each node function's imports through the project's own modules — stopping at the standard library, at anything installed, and at Fluksio itself, whose checkout would otherwise be most of every digest — and records the file list with what it hashed to. The engine hashes those files again when the run is claimed, so the fingerprint is live rather than a snapshot, and falls back to what sync recorded when it cannot see them: a remote worker's runs used to share one empty digest, and therefore one key. Three things follow. Editing a helper a node calls into re-runs that node, as before. Editing something the node never reaches no longer re-runs anything — a notebook two directories away was invalidating every arm. And `Run.code_digest` is now the hash of its nodes' digests, so it is neither looser nor tighter than "the code behind these numbers", which is what makes it worth joining an exported table on. `sync` says so too: it compares the per-node digest against the stored one, so a helper edit prints `train: updated (flow, fit)` instead of `unchanged`. The digest is read when the document is built rather than when the flow is declared, so a second `sync()` in one process sees an edit between them. Also: `fluksio runs` shows only the inputs that differ from what the flow declares, fitted to the terminal, so a flow taking a few kB of json no longer wraps every line. Every existing cache entry misses once — the fingerprint changed shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa	2026-08-27 22:35:23 +02:00
stroblmeandClaude Opus 5	37a7df9d24	Let a sweep run more than four at a time, and name the run a failure was in ... Concurrent runs sat at 4 whatever FLOW_MAX_CASCADES said: that setting bounds cascades, and the run drivers read a hardcoded MAX_PARALLEL nobody could reach. FLOW_MAX_RUNS is the knob they read now, --max-runs/--max-cascades/--max-workers are the same three as flags on serve, and the engine says which numbers it started with — which is the only way to tell that a settings file was read. Events keep the run they happened in. The payload always carried it and the persist path dropped it, so reading one run's failures meant filtering the engine-wide list; a batch run's id reaches those events now too, since a run has no journaled item to name itself by. Also: a provisioner's 0 means "no deadline" rather than "cancel on the next reconcile", and a command that reaches no engine says how to start one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sbYeYaVgYQqm1sbx7wPdL	2026-08-27 14:17:51 +02:00
stroblmeandClaude Opus 5	40f8ad378d	Ask a cluster for a machine when nothing here will do ... Slurm is not a machine that attaches and stays; it is a queue somebody else owns. So nothing here submits a node to it. It submits a job whose payload is an ordinary worker dialling back in, and everything downstream — the protocol, the artifacts, cancellation, the books — already worked and did not have to learn what Slurm is. The alternative, which Covalent takes, is to stage a serialized call and a runner onto the login node, poll squeue and copy the result back: a second way of running a node beside the one that exists. The cost of not doing that is one assumption, that a compute node can open a connection outward. Where that is false, _payload is the single method a staged variant would replace. Clusters are configured in provisioners.json beside the alerts, since this is infrastructure an operator writes rather than anything a flow says. The script is generated with the system ssh and no new dependency, and prerun owns the environment — deliberately no pip install, because what is on a cluster is somebody's decision. One outstanding request per profile, cancelled if it never attaches and on the way out. Nothing autoscales. The run gate needed the same hook: a run held before it starts never reaches the placer's own wait, so it would have queued forever on a machine nothing had asked for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW	2026-08-27 09:09:44 +02:00
stroblmeandClaude Opus 5	6ff56533f5	Schedule a node across every machine, not just this one ... The engine answered "where does this node run" twice, in two ways that could not see each other: a device sent it to a worker carrying that label, and resources were counted against the engine's own cores. Declaring both meant the second answer won and nothing was counted at all — which the data-science getting-started page and the worked example both do. One question now, in flow/placement.py: of every machine attached, which could grant what this node asked for, and which of those has it free. The books move onto each machine — one accountant per worker, built from the inventory it reported — and the waiting moves above them, where one condition variable can be woken by a release anywhere or by a worker attaching. Locks go one way: placer, then a machine's books, never back. So a node asking for a card now finds the box that has one, rather than being clamped down to none and run here. When nothing can grant the ask at all it is still cut down and run — a flow written on a cluster has to work on a laptop — but the ceiling is one real machine now, since taking the largest of each dimension separately can describe a machine nobody has. Two things fixed on the way. A device on a connector node held every batch run of its flow forever, waiting for a worker that could never run an entry point. And `prefer` falling back to the engine skipped the books, so the fallback held nothing. The bench flow's node has taken a `params` argument that with_settings has not forwarded for some time, so the benchmark could not run at all: 62 ms median submit-to-result with this, against the 61 ms on record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW	2026-08-27 08:49:36 +02:00
stroblmeandClaude Opus 5	0ffcabfdb9	Media dtypes: image, audio and video as narrowed artifact references ... A port may now declare `image`, `audio` or `video`. Each is the artifact reference the engine already had, narrowed by the `media_type` on it, so a speech recogniser declares what it eats rather than taking any bytes at all and finding out. Bytes still never travel as a message and nothing on the wire stops being JSON: a camera publishes one reference per frame, a microphone one per chunk, and a reference may carry a `meta` dict nothing here interprets. Streaming media is therefore an ordinary streaming port — with one change to what that means. An emission used to journal an item with no payload, so downstream read whatever was current when the item was claimed; a consumer slower than its producer saw only the newest chunk and the ones between were lost. That is right for a training curve and wrong for a second of speech, so an emission now journals a `kind="emission"` item carrying its values, and the executor hands them to the nodes reading that message instead of writing them to state again. The value in state stays the latest, which is what everything else reads, and the wave is filtered by what actually changed rather than walking everything reachable. No queue serialization change — the existing `outputs` field carries it. Continuous media makes the store's missing GC a real problem, so this closes it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row records or a live message holds, spares anything written in the last hour, and stands aside entirely while a run is in flight, since a node may store a checkpoint long before it returns the reference to it. That also collects the orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns it off. Around the edges: `GET /artifacts/{digest}` serves the media type the caller passes and answers ranged requests, so a browser plays a clip rather than downloading it; `PUT` spools to disk instead of holding the whole body in memory, as does `save_artifact` given a path; a Media widget draws whatever its message points at, and a wall panel may fetch the bytes its own tiles are showing and nothing else; and a connector gets `save_artifact`, for a device whose readings are bytes. What this cannot do is live video: a frame every second or two is a glance, and the honest answer above that is the camera's own stream, which the widget takes as a URL and the browser plays from source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 23:44:55 +02:00
stroblmeandClaude Opus 5	4f3eaf950c	Report a published node that has no code of its own ... The store answers the new-node template when nothing was ever written for a node, so such a node ran — returning {} on every call, reporting active and ok, and saying nothing anywhere. Unreachable through `fluksio sync`, which writes every body before it publishes; the editor end was open. A run of a flow holding one is now refused, and the flow carries a missing_source issue so it is visible before anybody runs it. A draft is exempt: a node being written legitimately has no published body yet. The generated client is regenerated for the new issue code, which also catches up the drift left by earlier backend work (resources, code_digest, idempotency_key). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:55:11 +02:00
stroblmeandClaude Opus 5	2e82367926	Keep the file name a node gave an artifact ... A run_artifact row is keyed by the message the bytes left on, and that was also the only name it could answer with — so an `@run:` reference resolved through the row was the same bytes under a name its producer never chose. The row now records the file name beside the message name; rows written before the column answer as they always did. The fallback also checks the bytes are still in the store, which the bare digest spelling beside it has always done. A missing blob now fails at submit rather than in the middle of the run that wanted it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:45:11 +02:00
stroblmeandClaude Opus 5	4a38c6ed31	Name the code a run ran, and let an interrupted sync finish ... Three faults with one root: the stored body of a code-defined node is an import shim, and nothing that mattered was ever read from the code itself. - The run stamp could not identify what ran. The shim imports whatever is on disk when the worker starts, and an uncommitted tree stamps <commit>-dirty for every run it ever produces. Run.code_digest hashes the repository's .py files, memoized on their stat state, and it is read again when the run is actually claimed -- so a sweep queued for hours records the code each of its runs executed, not the code that was there when it was submitted. - The stage cache adopted code that was too new. The fingerprint hashed the shim, which is invariant under any edit to the imported function or anything it calls into, so a re-run was served from cache and answered without the outputs the edit added. It now carries the repo digest and the node's declared ports. Every fingerprint changes once, which invalidates the existing cache; a canvas flow has no repository and keys as before. - An interrupted sync looked like a hand-edited canvas. The engine answers a new-node template for a node with no stored body, and the template carries no marker, so the drift check read "somebody edited this" and demanded --force -- for the one state that re-running the sync is the fix for. NodeSource.missing states the fact, and sync skips those and reuses the bodies it read instead of asking for each one twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 21:27:10 +02:00
stroblmeandClaude Opus 5	1f7c6646f1	Survive a busy engine: retry, idempotent submit, resilient waiting ... A driver script died of one slow answer: httpx.ReadTimeout out of RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which cost a sweep 78 of its 84 runs. - Split the timeout (5 s connect, 120 s read): a wrong URL fails at once, and a busy engine gets longer than the slowest thing it does on purpose (a 60 s compile, a 15 s rebuild wait). - Retry idempotent calls three times on a transport error or 502/503/504. 503 is the engine's own "ask again" — it is what RebuildBusy answers. - Submit carries a key the engine stores with the run, so a retry after a timeout returns that run instead of starting a second. A sweep keys every entry, so a half-created one recreates only what is missing. - wait() and --follow tolerate five failed polls in a row; a 404 still stops at once, because that is an answer rather than a gap. - CLI says "engine not answering" and names the run still on the engine, instead of printing a traceback. - runs: clamp the params column to 80 characters; events() takes the flow/since/until the endpoint already had; RunHandle.failures answers "what killed this run" from the run's own node rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 21:19:09 +02:00
stroblmeandClaude Opus 5	3c15964364	A cached node keeps its curve, and any input can name a run's output ... A cache hit still replays no emissions — those values were the story of an execution that is not happening — but the run they were recorded in is now written on the row (`run_node.cached_from`), and the metrics endpoints read the series back from there. So a reused run answers `run.metrics("train.loss")` with the same points the run that trained did, rather than looking like a run that produced no numbers at all. Pointed at rather than copied: a sweep of 500 reusing one frozen node would otherwise duplicate its curve 500 times. That needed the cross-flow restore fixed first. The cache key has no flow in it while the stored outputs are named for the flow that produced them, so `quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s state and the next node was called without its argument. One rule now covers both halves: `requalify` reads a name owned by one flow as the same name in another, applied to the restored outputs, to the node id behind the pointer, and to the series names on the way out. Reuse across flows is kept. Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input, not only artifacts. Chaining a run's json config into the next one from a shell meant pasting the whole object inline, and the CLI could not even send the spelling — `_coerce` died in `json.loads` before the engine saw it. Both spellings are reserved on every input now, `str` included, and `_from_run` returns whatever the run's result holds rather than only a reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dp9L6gakMVro1K2C5zdtBE	2026-08-25 15:12:28 +02:00
stroblmeandClaude Opus 5	93374a310e	Refuse what a node cannot publish, and stop timing out work that is fine ... Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69	2026-08-25 07:30:14 +02:00
stroblmeandClaude Opus 5	400d7d9c5c	Stage caching for batch runs, and an engine that lives in the command ... A code node in a batch run is now fingerprinted by its source, its raw settings and the values it reads — an artifact input counting as its digest, which is what the content addressing was always for. A run that finds the key restores what the earlier one returned and skips the node, recorded as `cached`. The run history is the cache: `run_node.outputs` beside the `cache_key` the schema already had, no second store. On for code nodes, never for the built-in and connector types that have side effects; off per node with `@node(cache=False)` and per run with `--no-cache`. Emissions are not replayed on a hit, so a cached training node returns its result without redrawing its curve. Recorded in NOTEPAD.md with the two other deliberate limits. `fluksio run --local` boots the real app in the command's own process and drives it through its ASGI interface behind the ordinary client, so a run no longer needs a `serve` terminal beside it — same data directory, same history, and the cache carries between the two. It always waits, because the engine it starts lives exactly as long as the command. Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists, `run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited run rather than abandoning it, coloured statuses on a terminal, and `name` made optional on the metrics endpoint so a follower can ask for every series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-24 20:31:31 +02:00
stroblmeandClaude Opus 5	d4a9406c51	Fix the CI gates: Python 3.13, concurrency groups, hook violations ... The gates have never gone green on the new runners. Three separate reasons: - backend/Dockerfile shipped Python 3.10 while the code imports typing.Self and datetime.UTC, so the container exited on import and the suite could not even load its conftest. The image moves to 3.13 and the packages declare >=3.12, which is the floor the tests actually pass on; ruff's target follows and rewrites timezone.utc and asyncio.TimeoutError accordingly. Relocking drops the 3.10 branch, which bumps FastAPI and so regenerates the SDK. - frontend/README.md had no trailing newline and two dashboard widgets used arbitrary text-[…] sizes. Both are em-relative on purpose, so they move to the inline style the neighbouring ramp already uses. - Every commit left its own run queued: without a concurrency group a runner that was offline for a while works through a backlog nobody reads. A stack that fails to come up now prints its logs before the teardown removes it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-24 14:55:59 +02:00
stroblmeandClaude Fable 5	a38e2745eb	Add a Python SDK: flows declared in your own repository ... A data scientist keeps their code where it is and decorates it: `@node` declares a function's ports beside the function, `Flow(name, nodes=[...])` says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds one for a single flow. `fluksio sync` uploads the document plus a generated import shim per node, so the store still holds a complete, runnable, git-versioned definition while the code it imports stays theirs. `fluksio login|run|runs` and `flow.submit().wait()` are the client half, over the run endpoints that already existed. Runs record the user repository's commit beside the store's, so "what code produced this number" is answerable on the side that now holds the code. - `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim generator, an HTTP client and sync. Standard library only at import, so `from fluksio import node` in a training script pulls in no engine. - `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the repository's commit; `POST /modules/refresh` retires the workers without an install, which every sync calls — a worker holds the imported package in memory, so an edit to it is invisible until the process goes. - The canvas shows a generated body read-only and names the repository to edit instead; a body edited there stops the next sync rather than being discarded. - The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since the shim imports a module whose first line declares them. - `examples/myresearch` is the worked example, `make sync-example` uploads it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU	2026-08-23 20:16:08 +02:00
stroblmeandClaude Opus 5	961a8f881d	Keep the engine's state in SQLite, not Postgres ... One process owns this database — the image has run a single uvicorn worker for that reason since the four-engines bug — so a file beside the flows is the honest shape for it, and it is what lets `fluksio serve` need no infrastructure at all. Live values, node execution and the work queue never came here anyway; what does is a rollup a minute at a time, a row per cascade and the run history, and WAL keeps the readers going while that one writer works. DATA_DIR is now the one setting that moves everything an installation keeps; the rest derive from it and the images still spell theirs out. The schema is prepared in-process at startup, so the prestart service is gone, and the ten Postgres-only revisions collapse into one portable baseline. Three things only worked because psycopg was casting for us: a token's subject arriving as a string where the column is a UUID, `greatest`, and `date_bin`. The timestamps needed a column type of their own — SQLite stores no offset, and a naive datetime read back either raises against an aware `now` or serialises as local time. Postgres stays in the stack only for Umami, behind the analytics profile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 22:19:45 +02:00
stroblmeandClaude Opus 5	60d7ec81c0	Rename the import package app to fluksio ... A wheel whose top-level module is `app` collides with anything else in a user's venv, so the package that is about to be published takes the name it is published under. Only the Python package moves; the repo, the Docker WORKDIR and the compose project keep theirs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 21:48:05 +02:00