Commit Graph
141 Commits
Author SHA1 Message Date
stroblme a95be36dba Un-key stage-cache rows written before the current key shape
CACHE_SHAPE_SINCE marks the last change that moved every cache key. The
run service's periodic sweep clears cache_key on rows of runs created
before it, which nothing can match any more. Their outputs stay.
2026-10-15 10:54:22 +02:00
stroblme 7ea53ccf17 DELETE /artifacts/{digest}: superuser only, 409 while referenced
The keep-set the hourly sweep uses is now referenced_artifacts, shared by
the sweep and the route. The volatile ring is not touched.
2026-10-15 10:48:59 +02:00
stroblme 599e01b0bf Run child tables cascade from run.id
Migration ccb21eacf5d5 deletes rows whose run is gone, then adds foreign keys
with ON DELETE CASCADE from run_node, run_metric and run_artifact to run.id.
Deleting runs is now the one statement. A node or metric written after its
run was deleted is dropped with an info line. Migrations run with foreign keys
off, since a batch rebuild of run would otherwise cascade into every child row.
2026-10-15 10:32:33 +02:00
stroblme caa2cff6b1 A run keeps the inputs its flow declared at submit
Run.declared (migration 2e7ca7d10044) snapshots the declarations on submit
and import. The run detail's Inputs panel reads them, and a run from before
falls back to the flow's current declarations.
2026-10-15 10:20:29 +02:00
stroblme eca3fa0049 A panel writes only what its dashboards' inputs target
It still reads every message its dashboards show. A message a tile only
draws, or a setting is bound to, is no longer writable from the panel.
2026-10-15 10:10:44 +02:00
stroblme e85b18092f Flow inputs carry an origin hint, run or panel
The Inputs editor and the canvas label show it, and the run forms leave
panel inputs out.
2026-10-15 10:08:46 +02:00
stroblme 43498037be PUT /panels/ carries a version and refuses a stale one with 409
The Panels dialog re-reads the list on 409. A panels.json without a
version reads as 0.
2026-10-15 10:07:50 +02:00
stroblme 0562753d9c http sender drops reply keys no output port declares
A sender without outputs no longer fails on a {"status": "ok"} reply.
2026-10-15 09:57:51 +02:00
stroblme e6afc23952 sweep_artifacts: narrow the optional flow store for mypy 2026-10-15 09:55:41 +02:00
stroblme 5e5f1836cb wren runs its model on one BLAS thread
threadpoolctl scopes the limit to wren.act, so other numpy calls in the
engine keep their threads.
2026-10-15 09:55:02 +02:00
stroblme f2a823218a GPUs follow the engine's own CUDA_VISIBLE_DEVICES
Its entries are the cards present, and a local child's allocation is
named through them, so under =2,3 card 0 is device 2. Remote workers
keep mapping against their own mask.
2026-10-15 09:53:38 +02:00
stroblme 290b9fa705 Superuser route to flush the metrics collector now
POST /observability/flush handles what the collector has queued and writes
it, so the soak harness no longer sleeps FLUSH_INTERVAL_S before clearing
its own rows.
2026-10-15 09:53:19 +02:00
stroblme ae81d95fd7 Warn at startup when worker or cascade limits are far past the cores
More than four times os.cpu_count() logs one warning; the value is kept.
2026-10-15 09:51:38 +02:00
stroblme 9a029a85f8 Engine timings move to FLOW_* settings
NODE_STOP_TIMEOUT, REBUILD_WAIT, FLOW_REBUILD_WAIT, BACKOFF and
CANCEL_GRACE keep their defaults and are read from Settings.
2026-10-15 09:50:18 +02:00
stroblme 0004e4b9bd Retire a shared node in use with ?detach=true
DELETE /flows/library/{name} still answers 409 while nodes use the source.
With detach=true each of them keeps a copy of the code, in the published
flow and the draft, the source is deleted and the running users rebuild.
2026-10-15 09:48:47 +02:00
stroblme 96f4fa5a38 Warn when a node that runs no code names a device
A device on a node without source was accepted and silently ignored. The
build now reports it as ignored_device, an advisory issue.
2026-10-15 09:45:24 +02:00
stroblme c9fda50601 Connection nodes speak of listener and sender, and of send_queue_size
mqtt and http take websocket's vocabulary: their Mode is LISTENER/SENDER,
default names read mqtt_listener_… and http_listener_…, and mqtt's
publish_queue_size is send_queue_size. No alias stays behind, so the flow
store rewrites the old setting in every published and draft mqtt node when it
opens, in the same migration as trigger → debounce. Every mqtt and http
setting carries a one-line description for the editor, and the node-types
reference uses one wording for direction and says that an http sender
publishes a JSON object response on its outputs.
2026-10-15 09:44:22 +02:00
stroblme c032898135 A supervised loop that crashes reports its node down
A push session that failed before it listened left only a task_crashed
event. The wrapper in _run_supervised now reports down with the exception,
unless the loop already said why it is down.
2026-10-15 09:42:01 +02:00
stroblme 565d492f19 Give a rate-limit flush its own run, linked to the run that held the value
A flush ran its cascade under no id, so its executions and failures belonged
to no run. It now records a run with source "flush", and the holding run's id
travels on the queue item into a new `flow_run.parent` column, also on the
`/observability/runs` rows.
2026-10-15 09:37:30 +02:00
stroblme cfb9e53d68 Rename the trigger node type to debounce, migrating stored flows
The store rewrites `"type": "trigger"` to `debounce` in every published and
draft flow when it opens, bumps the version and commits once. No alias stays
behind, so `trigger` is now an unknown type.
2026-10-15 09:34:40 +02:00
stroblme 4a776e52c4 Delay node: hold the latest value inside the interval and send it at the window's end
The interval now rides the output ports, so the pipeline's own hold and flush
keep what arrives inside the window instead of the node dropping it.
2026-10-15 09:32:35 +02:00
stroblme 4be6a5b303 Test hygiene: forget the refused study, quiet _Tee on a closed stream, wait for connector recovery 2026-10-14 23:41:31 +02:00
stroblme b345587ab9 Shared node changes rebuild the flows whose published copy uses them, not their drafts 2026-10-14 23:34:47 +02:00
stroblme 245e68449f Memory-queue ids carry a per-instance boot token, so a restart without Redis no longer overwrites earlier run rows 2026-10-14 23:33:25 +02:00
stroblme 2ac997ce37 ValueSource kind includes panel; Panels dialog no longer suggests the screen port openpanel does not serve 2026-10-14 23:32:32 +02:00
stroblme 30fa2b07db Remote exception names are cut to 128 identifier characters before a class is built for them 2026-10-14 23:31:47 +02:00
stroblme df78cdac72 Artifact sweep keeps what a flow input's initial references, published or draft 2026-10-14 23:30:46 +02:00
stroblme 8c50c22444 Add websocket node: client or served socket, listener or sender
ws:// and wss:// URLs dial out and hold a supervised connection; a path is
served under /hooks/<flow>/ with an optional secret. Outputs only listen,
inputs send. Frames are JSON keyed by port, with json_key and on_connect.
2026-10-14 18:42:19 +02:00
Melvin Strobl 2008150692 Round dashboards: one face at a time, a face per widget type, paged by swipe, dots and keys; slider reading; face row in the editor 2026-10-14 15:24:25 +02:00
Melvin Strobl be3446cb16 Round dashboards: shape and ring, the ring drawn and turned in every look, Knob preset and face guide in the editor 2026-10-14 14:23:55 +02:00
Melvin Strobl 659799d1e5 Panel I/O: PanelDef.io ports, scoped publish and upload, I/O section in the Panels dialog 2026-10-13 21:10:54 +02:00
stroblme d482b32970 Do not check a pinned certificate's dates
A pin trusts one exact certificate, so an expired one or an engine clock
that is off no longer locks a device out (X509_V_FLAG_NO_CHECK_TIME).
2026-10-07 10:11:08 +02:00
stroblme cd6f8fec55 Lift certificate pinning into the connector contract
fluksio.flow.connector_tls: trust the certificate a device presented on
first contact, kept in ConnectorNode.store, and name the new fingerprint
when another one is refused. Additive, so no contract version bump; hue
and wfrac use it.
2026-10-07 09:48:24 +02:00
stroblme ff3a31e515 Hand a provisioned worker its token in the environment, not argv
The generated batch script exported nothing and passed `--token`, which
any user on a shared compute node reads from `ps`. It now exports
FLUKSIO_WORKER_TOKEN after `prerun`, which the worker already reads.
2026-10-07 09:32:01 +02:00
stroblme 953a983919 Bind a worker's name and labels into its token; list and revoke tokens by jti 2026-10-07 08:42:14 +02:00
stroblme 243d39b591 Give connectors a store that survives restarts, and a way to clear it
`ConnectorNode.store` keeps short strings per connector package in the
engine's database (`connector_state`, migration f8c3e5a91d27), so the nodes of
one package share them and `fluksio serve` needs nothing new: get, set and
delete, at most 64 kB a value and 256 keys a package. A superuser lists the
keys (never the values) and clears them through `/connectors/{node_type}/state`,
or under Stored state in a connector node's panel. Documented in the connector
contract.
2026-10-06 17:19:48 +02:00
stroblme a168bcbdee Refuse names that are not names at the store boundary
DELETE /flows/%2e%2e reached FlowStore.delete_flow(".."), which removed
the data directory, and /flows/_lib removed every shared node. Every
path the flow and dashboard stores build from a flow, node, library or
dashboard name now goes through checked(), which raises InvalidName
(400) for anything _validate_name refuses. _validate_name uses
fullmatch, so a trailing newline is refused too.
2026-10-06 11:59:25 +02:00
stroblme 6802cd2203 Make changing engine-run code superuser-only
Writing node source, installing modules and any flow save that adds,
retypes or repoints a code node (Function, Command), edits a Command
node's settings or changes a node's resources.env now answer 403 for
everyone but a superuser. MCP tools and the assistant reach the same
routes, so they inherit the gate. Node types report runs_code.

The editor shows code read-only with a note, disables a Command node's
settings, leaves code nodes and reusable nodes out of the palette and
paste, and the Modules page offers no install.
2026-10-06 11:36:30 +02:00
stroblme 14d3208a5d Add the wren node: a typed command into values for the devices wired to it 2026-10-05 20:58:59 +02:00
stroblme 929933d7c3 Record which inputs a run's caller passed, and mark defaults from it
`Run.passed` (migration d5a3c8e1f246) keeps the names the caller chose, so
the run detail and the comparison table can mark a value as the flow's own
again; rows from before it fall back to the names their params lack. Retries
hand on their parent's list, imports record their entry's.

Also: a cancel of a queued run no longer leaves its id in `_cancelled`, a
linked comparison no longer crashes before the list loads, and the pre-commit
failures are fixed (stale client, `text-[10px]`, worker formatting).
2026-10-05 15:01:33 +02:00
stroblme bca28dbf75 Keep a run's lease while its pipeline builds; allow cancelling it then
A claimed run only joined the leased set once its pipeline was built, so a
build waiting for a busy worker pool got the run swept as abandoned. Cancel
now also stops such a run, and a terminal status being overwritten is logged.
2026-10-05 11:23:54 +02:00
stroblme 0649062327 Bound warm workers' host memory and account local RAM
A worker over FLOW_WORKER_MAX_RSS_MB (--worker-max-rss) when its node
returns is retired instead of kept warm, in the shared pool and in every
declared one; 0, the default, keeps them all. The engine now accounts its
own memory (FLOW_RAM / --ram, else the box or the cgroup limit less 1 GB),
so a node's `ram` declaration holds locally. A declared environment the
shared pool already has is that pool rather than a duplicate set of warm
workers.
2026-09-27 16:11:13 +02:00
stroblme 7240b7b823 Add the assistant: a chat that drives the MCP tools in-process
A provider set under Settings (OpenAI-compatible; key encrypted in
flow-data/ai.json) turns on an Assistant panel in the sidebar. Each turn runs
the FastMCP tools in-process as the person chatting, streams the rounds over
SSE and keeps one transcript per user. The panel floats on any screen and
shares the editor's lane: a settings panel keeps the top half, the chat the
bottom. The MCP client is now built once, for the endpoint and the chat.
2026-09-16 23:28:23 +02:00
stroblme af18d134a4 Add POST /runs/import for runs executed elsewhere
A history recorded before the move (MLflow experiments) had nowhere to
go, since runs were only made by running a flow. Each entry is stored as
it ended: parameters completed and checked like a submit, cause
"import", empty commit and code digest, metrics and artifacts written
as given, nothing queued or announced. Run.external_id (new migration)
makes a second import a no-op and is an export column so a row joins
back. The request is all or nothing and capped at 200 entries;
Client.import_runs pages a longer history.
2026-09-16 17:01:07 +02:00
stroblme 831bda9c46 Sync only the declaring module before a run; skip git-ignored paths
`fluksio run` walked the working directory and imported every module in
it, which ran a research repository's analysis scripts before the run was
submitted. A flow's origin now records the file and module name that
declared it, and run/sweep import just that, under the same name so the
generated node bodies do not change. A flow with no record still walks,
now skipping what .gitignore covers.

Every code-defined flow reports `flow` changed once on its next sync,
since the origin gained two fields.
2026-09-16 16:55:02 +02:00
stroblme 0f040b6e1d Merge branch 'main' of git.stroblme.de:Fluksio/app
# Conflicts:
#	frontend/src/client/sdk.gen.ts
2026-09-07 08:41:45 +02:00
stroblme 4e8d04d5d7 Hold a node down for a minute before calling it down
A connector that misses one poll turned the health badge amber and the node
on the canvas red until it recovered. `LoadedNode.health_since` marks when the
status last actually changed — a rotating error detail under an unchanged
status does not move it, or a connector retrying with a different errno each
poll would never debounce — and `unhealthy_nodes` applies a 60 s floor to it.

The floor is on the surfacing, not on the transition: the stored health is the
truth the moment a node reports it, so a flow reading node health is never
told a stale story. Only going down is held back; recovery clears at once.
2026-09-07 07:54:10 +02:00
stroblme e7bf767d40 Bound the work-queue durability claim to the appendfsync window
Redis runs with appendfsync everysec, so an unclean stop loses up to a
second of journaled entries. Name that where the docstrings promised a
crash costs nothing, and state the setting and its trade-off on the
configuration page.
2026-09-06 17:16:01 +02:00
stroblme fa5ce4f694 Give both summaries a modified time, and the dashboard list a footprint
Home's "recently modified" order was a proxy — drafts first, then the
version counter, then the name. Both summaries now carry `updated_at`,
read as the mtime of the working copy: every write in the store commits
immediately, so a file's mtime is its commit time, and `git log -1 --
<path>` costs ~990ms across this instance's 15 documents (it walks the
history back to the last commit touching each one, so it is slowest for
the stalest) against ~1.8ms for the stats. No cache needed.

`DashboardSummary` also carries `footprint`: each widget as its type plus
the placement a panel resolves, which is the whole of what the mosaic
draws. That drops the document fetch per tile and with it the cap of
eight, past which tiles showed a name and nothing else. Verified against
this instance's 8 dashboards: the blocks are identical to what the old
client-side derivation produced.
2026-09-06 15:26:48 +02:00
stroblme 8c097d9222 Make revoking an agent and locking a dashboard actually revoke and lock
An MCP access token is a stateless JWT good until it expires, so deleting
the client row revoked nothing already handed out — on the MCP endpoint or
on the REST API, which takes the same token directly. Both doors now look
the client up by the `client_id` the token has always carried, so tokens
already in circulation are held to it too.

A dashboard's `locked` setting stopped the client drawing a control and
nothing else; the server took a publish from a panel showing it anyway. It
now bounds the panel's write scope, resolved live where a flow drives the
flag, exactly as the client resolves it. Reads are untouched — read-only is
not blind — and so is a querying chart's request, which is how that tile
reads rather than something anyone touched.
2026-09-06 15:24:53 +02:00