346 Commits

Include renames

Author	SHA1	Message	Date
stroblme	40eaf026c0	Merge remote-tracking branch 'origin/main'	2026-11-06 14:39:38 +01:00
stroblme	5eec267a7e	A code-declared node's shim names only its triggering ports, so a non-triggering input with no value yet reaches the function as its own default instead of failing the call	2026-11-06 14:35:12 +01:00
stroblme	40e60b422b	A querying chart offers its own windows: an optional ranges list of {label, range_s, interval_s}, up to 8 and checked by the server, where range_s 0 asks for the whole history; the editor lists and picks them, and the refresh is capped at what a timer can hold	2026-11-06 14:33:25 +01:00
Melvin Strobl	261cf1ec2a	Assistant answers general questions and stays silent for overheard talk ("Ignored." is not said); stt prompt setting sent as the transcription prompt	2026-11-06 07:27:44 +01:00
stroblme	4c15f4abe1	Test hygiene: empty the flow store between modules, unbind demand on shutdown, poll the hook state in test_flow_ids, count only the watcher under test in the revoked-link test, delete runs.spec's runs	2026-11-05 08:03:56 +01:00
stroblme	c6ac7f80a9	A batch run fails a node with no value on a triggering input instead of calling it, naming the input and its producer, and skips what waits on it	2026-11-05 07:35:02 +01:00
stroblme	78b2c1438c	pause_intake answers False when claimed work outlasts its timeout, and reload then resumes intake and raises RebuildBusy before tearing anything down	2026-11-05 06:43:33 +01:00
stroblme	1a622edb1f	RedisState's compare-and-swap watches through its pipeline, so a write between the compare and EXEC fails the swap; pause_intake waits out a claim already under way, so nothing dispatches after it returns	2026-11-04 21:53:41 +01:00
stroblme	d028bf326a	A batch run skips the nodes waiting on a failed one and ends error; a run whose nodes cannot load fails with the load error before anything executes; cancelling a run stops its pipeline without announcing a pause of the live flow; batch_issues refuses a delay node's interval; the client accepts an engine URL with or without /api/v1; RedisState's compare-and-swap records the names it writes	2026-11-04 21:38:21 +01:00
stroblme	eb17528ce6	FlowNotFound says whether a published flow or any flow is missing; read_all stamps its cache with the revision from before its scan; dashboard discard and delete commits name the key; a RedisState ttl expires counters and the names set too; bench_startup publishes the version after its saves, and only when there is a draft	2026-11-04 21:20:29 +01:00
stroblme	b7f1d21712	Docs for key renames: ids and keys in the flow concepts, the rename routes, events and 503s in the API reference, hooks during a rename, sync and run metrics after one; a pinned restore refused for its recorded key names that key in the 409	2026-11-03 08:09:35 +01:00
stroblme	f9d5f510c2	Startup deletes the work queue's keys from before it moved from pipeline: to queue: (stream, timers, dead letters, parked lists, done markers), logged with what they held; an old stream still holding undelivered work stays, is logged as an error, and counts as queued work for the keys it names	2026-11-03 07:55:14 +01:00
stroblme	ba4aae390d	Rename orchestration, splice and routes: a Renamer runs one rename at a time, closing the gate, taking the controller lock and holding the engine, each waiting without a limit and saying what it waits for (summary rename status, rename_waiting); DELETE on the pending request or a closed connection calls it off until the journal begins (409 aborted); then journal, steps and audit, retried in the background on failure (503 completing); splice_renamed rebuilds the affected flows under the held lock keeping stopped, paused and parked work, history limits and device wiring refreshed before activation; AlertManager.rename and EventBus.rename/settle follow the names; flow and dashboard rename routes answer by code and repeat stored results; renamed, dashboard_changed and pipeline_rebuilt after release; fluksio serve reads flow files without building a FlowStore	2026-11-02 22:14:44 +01:00
stroblme	2d65118d82	Rename journal, steps and recovery: rename.json (RENAME_JOURNAL_FILE) records each step, documents are written whole and committed alone (FlowStore.replace_documents, identity.write_documents) and rolled back before the journal goes forward; history per run scoped by flow id, alert ledger, alerts.json, state and queue steps idempotent and checkpointed; audit row once per op; recover_documents before assign_ids and recover_runtime before the engine is built, retried until done; repeated requests get the stored result, stale ones 409; plan refuses rename_in_progress; alerts.json written atomically; completeness scanner in the test utilities	2026-11-02 21:30:01 +01:00
stroblme	fa2a213631	Rename planner and document rewrite: rewrite_documents moves qualified names of the renamed flow in every flow (ports, inputs, outputs) and dashboard binding (one key table, WIDGET_BINDINGS, shared with the WidgetDef accessors, legacy pages included), published and draft, as plain JSON with both variants above either version; plan_rename reports documents, counts, fingerprint, warnings and refusals; superuser plan routes for flows and dashboards; key residue classifies state as the mapper does; duplicate retargets inputs and outputs too; queue reports its Redis version and other consumers	2026-11-02 21:06:39 +01:00
stroblme	81bd7c2724	Admission gate for a rename: RenameGate in flow/rename.py, installed by main.AdmissionGate inside CORS, turns mutating API requests away with 503 renaming and Retry-After while closed and drains those admitted without a limit, abortable; affected flows' hooks get 503 or socket close 1013, reads, sign-in, assistant, workers, artifacts and rename routes pass; the run cache misses while the gate is closed; trigger_node lets EngineHeld through instead of reporting a node failure	2026-11-02 20:53:08 +01:00
stroblme	cb709c19ac	Merge branch 'rename-s2'	2026-11-02 20:32:36 +01:00
stroblme	badcf94491	Engine holds for a rename: the pipeline refuses publish and inline cascades with EngineHeld once those running have finished, the execution service stops claiming and its timers once claimed cascades have finished; both wait without a limit, report what is busy and can be aborted. REST answers 503 with Retry-After, a device frame busy. Redis adds no longer wait for each other, only a queue hold waits for them	2026-11-02 20:21:17 +01:00
stroblme	be18a94089	Merge branch 'rename-s2'	2026-11-02 19:58:45 +01:00
stroblme	46b57de4ec	Rename migrations for live state and the work queue: state keys move with RENAMENX in batches (type, TTL and history capacity kept), the queue rebuilds its streams with the same ids, group and pending entries, rewrites timers, parked and held items and done markers, and can hold new work in a list released in order; nothing calls them yet	2026-11-02 19:50:54 +01:00
stroblme	53916d869b	Clients take a flow's id or key: one resolver in the SDK, id first, explicit id:/key:, a warning for reads and a refusal for changes when a name means two flows, deleted flows found by their last key; sync creates by POST, retries a create race through the drift checks and reports the id; MCP gets create_flow and resolves every flow argument with the caller's token; displays show flow_name; an engine from before ids is named as such	2026-11-02 19:39:14 +01:00
stroblme	ebcc480a8c	Rename mapper: flow/namespace.py maps state keys, work items and alert dedup keys from an old flow key to a new one; the state key families and WorkItem fields are registered there, the key builders take their prefixes from it, and done markers read back with queue.done_marker	2026-11-02 19:28:25 +01:00
stroblme	fd637e0f1e	Tests clear flow tombstones between modules with the other engine rows, and the tombstone test commits its prune before checking the count, so a wrong count no longer holds the write lock over the tests after it	2026-11-02 19:14:59 +01:00
stroblme	d4a1b1db47	History is kept by flow id: runs, cascades, rollups and events file the id in their flow column and nodes as <flow id>.<node>, the API answers flow plus flow_name and shows and takes nodes as <key>.<node>; events carry flow_id from the moment they are published; run queue items, the stage cache and cached curves keep ids and keys apart; endpoints, the brain graph, device bindings and search name the id to open	2026-11-02 19:01:19 +01:00
stroblme	e6027af8fe	Prove the rename's stream rebuild on a real Redis: same ids, group, lag and pending entries after RENAME, ENTRIESREAD set after XSETID because Redis caps it; backend CI gets a Redis 8.2 service	2026-11-02 18:51:14 +01:00
stroblme	08a6376836	Flows and dashboards are created by POST under an issued or superuser-pinned id; PUT only updates; deleted flows leave a tombstone with their key; the controller stores by id and runs by key, hooks live under /hooks/<flow id>; a key with state or queued work behind it is refused until cleared	2026-11-02 17:41:23 +01:00
stroblme	fae85f5ecb	Flows and dashboards get ids: the store is addressed by id with a key index, the first start gives every document its key as id, fluksio store migrate --dry-run and ls; the rename routes are gone	2026-11-02 16:54:11 +01:00
stroblme	64ef7c84d6	Portal members: a request that loses the adoption race reads back the winner's user instead of being refused	2026-11-01 11:21:42 +01:00
stroblme	d8eb6968f0	test_wren: load numpy itself, so the BLAS test passes alone	2026-11-01 10:59:45 +01:00
stroblme	7a7315043d	Alerts: a delivery no channel took leaves the cooldown unspent	2026-11-01 10:59:28 +01:00
stroblme	48d4be6093	Starting a running flow restarts its nodes, so interval and cron Injects keep firing; a params-dict node is told settings arrive by name	2026-11-01 10:59:14 +01:00
Melvin Strobl	e0f5ddc3a4	Devices may publish to their own out ports' messages (a level set on the device itself)	2026-10-29 21:58:38 +01:00
stroblme	75b3a93094	Fix work recovery and reduce engine and chart overhead	2026-10-29 14:24:17 +01:00
stroblme	5d0ea480bf	Alerts: a Telegram channel, one chat or group each, connected by a one-time link ... The channel sends through the Bot API with httpx only: HTML with escaped text, cut to fit a message, silent for a warning, a short 429 waited out once, and errors that say what to do about a blocked bot, a lost chat or a bad token and never quote the request URL. The bot token is a stored secret: saving a channel refuses a literal one, the card's "Store token" writes the secret and a reference, and no endpoint returns it. A chat id is never typed. Connect starts a ten minute code, and the chat that sends `/start <code>` back through the bot is the one connected, so it is a chat somebody opened a link for. The engine fetches updates only while a code is waiting (no standing poll, no webhook, no inbound address) and ignores everything else without a reply. Connect needs a saved form, and the form takes the connected chat in without touching what was typed since. The token is part of every Bot API URL, so the httpx logger gets the same filter the request lines have, which now also masks an httpx.URL argument.	2026-10-28 13:06:58 +01:00
stroblme	bfaaa7cd03	Alerts: cooldowns survive a restart, and a connection must stay down a minute before it alerts ... The alert manager kept its cooldowns and its ceiling per hour in memory, so a node that stayed down was announced again by every restart. Each delivered alert is now a row in the new alert_sent table (additive migration), loaded for the last seven days when the manager starts and pruned as new ones come in. A ledger that fails never stops an alert. The manager's clock is wall time for the same reason. A node_health "down" alert also waits 60 s (HEALTH_GRACE_S) in a pending task per node; a later event that is not "down" cancels it, as does stopping the manager, so a broker restarting with the stack alerts nobody. The cooldown used to be stamped before the ceiling per hour was checked, so an alert the ceiling held back still spent its cooldown. It is stamped only once the alert is sent now.	2026-10-28 12:46:42 +01:00
stroblme	00842fa175	Health summary: a node is failing after two failures in a row; one is a warning, a good run ends both ... The summary read degraded for 15 minutes after any single node_error, even once the node had run well again and including one bad input from a visitor. The controller now counts a node's failures since it last ran well and notes when it recovered; both carry across rebuilds and an acknowledgement resets the count. Two in a row inside the 15 minute window is "failing" (a problem), one is listed under warnings, and nodes.failing keeps meaning "failing now".	2026-10-28 12:42:35 +01:00
stroblme	2dc20911a3	Alerts: configuration, test and search entries are a superuser's; subscribing a browser stays open ... Any signed-in account could save an ntfy channel with its own `server` and a `{"$secret": ...}` token and press Test, which sent the stored secret to that host. The config GET/PUT, the test route and the global search's alert entries now answer to superusers, like /secrets. The webpush key and subscribe routes stay open to every account, and the Alerts page shows a non-superuser only the button for its own browser. The events table in the docs gains run_finished.	2026-10-28 12:40:44 +01:00
stroblme	29f4d93c2a	Search: node settings are indexed, with credentials masked on the server ... /search gains a `setting` category, `<node>.<setting>` under its flow with the node type and the value in a new `detail` field, so Ctrl-K finds which node uses a broker or a timeout and opens the node. `setting_text` is the one place that decides what may be said: a reference to a stored secret shows its name (to superusers only, as the secrets list does), a plain value in an `x-secret` field or under a key like password, token or api_key is masked and never indexed, nested credentials in headers or query are masked, long values are clipped. The global search ranks what contains the query as typed ahead of loose spellings before it caps a group, which long setting values would otherwise have crowded out.	2026-10-28 11:07:27 +01:00
stroblme	c04e4883c2	MCP: mcp 2.3 (MCPServer, transport arguments on streamable_http_app, bearer read in middleware); tests for the 2026-07-28 revision and the resource check	2026-10-28 09:21:01 +01:00
stroblme	c04d6b83c6	MCP tests: the SDK's own client connects over streamable HTTP with a bearer, and older protocol revisions still answer	2026-10-28 09:11:58 +01:00
stroblme	b5e1a4c5b2	Request log: a hook's secret is masked in uvicorn's lines and the tunnel connector's failure line	2026-10-28 08:41:52 +01:00
stroblme	5c7a8773ad	SDK: sync and run import the declaring module in a child process ... The command line no longer imports a research module, so import-time GPU work ends with a short-lived child instead of holding a CUDA context for the length of a followed run. fluksio.sdk.declare starts the child, which writes each flow's document and shims as JSON; Declared stands in for Flow in sync(). Errors keep their type and message, a child's traceback reaches the user.	2026-10-28 08:11:14 +01:00
stroblme	34243e6902	SDK: artifacts work outside a node, run --follow prints status and queue waits, fluksio cancel ... save_artifact/load_artifact outside a node use the worker's own store code in $FLUKSIO_ARTIFACT_DIR or <tmp>/fluksio-artifacts, so a node can be called by hand. --follow prints status changes and each node's resource wait; the placer's waiting snapshot now carries the run. fluksio cancel <run>...	2026-10-28 08:07:35 +01:00
stroblme	f52d9c3077	Service node ends its call when the loop cancels every task; test order and timing leaks ... _serve_once treated a cancelled call future as finished, so an event loop cancelling all tasks at once (shutdown) never sent the worker a cancel and the call's thread kept asyncio.run waiting for SILENCE_S. The cameras audio test now removes its voice flow and device, the unpaired-frame test waits for its first frame before the unpair, and test_connector no longer inherits the controller an app test bound to demand.	2026-10-27 21:23:40 +01:00
Melvin Strobl	00cda9d2f1	connector: wanted(port) says whether anyone watches or reads a port	2026-10-27 15:20:24 +01:00
Melvin Strobl	7b453af3fe	wren node: calibration setting, passed to wren's training as the installation's calibration records	2026-10-26 22:17:48 +01:00
Melvin Strobl	82c64a10b6	Make documentation standalone and generalize examples	2026-10-25 14:43:20 +01:00
Melvin Strobl	96ba512d40	wren node: untrained targets left out of the prompt, a read beside a held-back set goes on whole	2026-10-25 12:09:37 +01:00
stroblme	e9941b770d	wren node: Train in the panel, as a run of wren's training flow ... POST/GET /flows/{name}/nodes/{id}/train submit wren_train on the node's targets, examples and tests and read its latest run (run.for_node, cause "train"); "Use this model" writes `model` into the draft.	2026-10-25 01:18:54 +02:00
stroblme	07fa5a2dc2	wren node: targets from the wiring, on wren 2 ... Every output is a target and every input besides prompt a reading. A targets setting gives descriptions, values, ranges and per-target guards, model takes a trained pack's artifact digest, min_confidence defaults to the model's threshold. A command is applied whole or not at all, and targets the model was not trained on read degraded. The home path (area/kind ports, groups, brightness, temperature answers) is gone.	2026-10-24 21:28:42 +02:00