77dd87f1b9675929f83e67dd145341f51b107b7b
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  stroblmeandClaude Opus 5
|
1069247085 |
Coalesce the event bus, and fix the socket that ended on a client frame
A three-node cascade publishes 13-16 events and each one crossed to the
event loop on its own. They are one `call_soon_threadsafe` now — whatever
was published between two turns of the loop goes over together — and every
subscriber still receives every event, oldest still dropped first when one
falls behind.
The socket end of the same path:
- **any frame from the client ended its stream.** `receive_text` was
awaited once, outside the loop, so a keepalive — or anything else a
client decided to say — satisfied it and was read as the client going
away. It is recreated per iteration; only a disconnect ends the stream.
- events go out in one frame per wave (`{"type": "batch", "events": [...]}`,
capped at 64), serialised once with orjson rather than per client with
the stdlib's `json.dumps` through `send_json`. The client unpacks a batch
and still understands single frames, so an older engine behind a newer
bundle keeps working.
- authenticating and building the snapshot happen on a thread. Both were on
the event loop: one is a database round trip, the other reads the whole
of state, per connect and again per `dashboard_changed` per panel.
`Pipeline.values()` — what that snapshot is — no longer SCANs the whole
Redis namespace. It scanned five bookkeeping keys for every message to find
the messages; `RedisState` keeps a set of the names beside them and answers
from it. Maintained wherever a message is written, so a seeded value or a
deleted flow keeps it exact.
On the client, while in the same file:
- a `node_health` event invalidates the flow's detail. The canvas draws
health from the server-derived `issues`, so a node going down or
recovering only showed on mount, navigation or a rebuild. The store had
a health map of its own that nothing ever read; it and `useNodeHealth`
are gone rather than wired up, since the server's view is the one the
canvas already uses.
- a reconnect invalidates the five key families this socket feeds instead
of the entire cache, and the backoff is jittered. The usual reason a
socket dropped is the engine restarting, so every tab and every wall
panel refetched everything, together, at the moment it was least able to
answer.
- a frame that will not parse costs the frame, not the connection. It was
the one unguarded `JSON.parse` in the app; an exception there escaped to
`window.onerror` and left whatever it had already applied behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
|
||
|
stroblmeandClaude Opus 5
|
da528340a9 |
Cut the round trips a message costs the engine
Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4 messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and 34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is what a pip install runs on, 262 -> 626. The two that bought most of it: - `StateBackend.record` puts a published value, its timestamp, its series and its version counter in one round trip. They were four calls building four pipelines, and a value crossing an edge pays them twice. A released rate-limit hold rides along instead of a DEL per port. - the readiness check reads a node's inputs and hands them to the node, rather than reading the triggering ones to count them and having the node read the same keys again a moment later. `apply_outputs` was a second copy of `_record_outputs` and is now the same code plus the event that distinguishes it. The rest, each small: - `_derive` builds a node-by-id map and a `consumes` index, so dispatching an item and publishing a value stop scanning every node in the installation. - `read_all` is memoised against the store revision — it sits on the publish path, so a dashboard slider was reading and validating every flow file per value. Same mechanism `_wiring` already uses. - the `message_value` source block is built once per node instead of per emission. - both timer threads ask the queue to promote only when something is actually due, which takes an idle engine from ~4 Redis round trips a second to one. - the shared httpx client is bounded (32 connections, one retry); its default pool is 100 with no per-host cap, so one slow endpoint could take it and every other sender node with it. - the MQTT and delay nodes no longer log a line per message at INFO. Robustness, in the same pass: - `MemoryWorkQueue._done` was a set nothing ever removed from — one entry per non-idempotent node per item, for the life of the process, in the default configuration. Capped, the way the Redis side expires its markers. - a saturated engine can claim from the due lane past the cascade limit. The capacity gate sits in front of the claim, so the due lane's priority — decided inside it — did not apply while every slot was held: a motor's stop was not behind the long nodes, it was unread. Only after a slot has genuinely failed to free for half a second, and briefly, so the backlog is not starved in turn. - `reclaim_stale` dispatches through that same gate. It could return sixty entries and push in-flight far past the limit the gate exists to hold. - a flow's nodes are stopped together rather than one after another. Each gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took five seconds per node — long enough to outlast `REBUILD_WAIT` and 503 the deploy. - the worker pool and the HTTP client are closed on a thread, not on the event loop, and a run closes the state backend it built (on Redis, a client and a connection pool per run). - the five background tasks say something when they die. Each catches exceptions inside its loop, so one raised anywhere else left the engine serving with no metrics, no alerts or no artifact sweep, silently. `tests/flow/test_round_trips.py` counts the state operations one message costs — four, where it was about eleven — because none of the above would fail a behavioural test if it were undone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T |
||
|
stroblmeandClaude Opus 5
|
c3675688c8 |
Wait for a deadline instead of polling for one
Docs / docs (push) Successful in 25s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m23s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 4m31s
Test Backend / test-backend (push) Successful in 2m55s
Compose Smoke Test / test-compose (push) Successful in 35s
Playwright Tests / merge-reports (push) Successful in 1m11s
The timer thread promoted due work on a fixed one-second tick, so every delayed item was 0-1000ms late whatever the load — measured on the house at 705ms mean on a rollershutter stop, which is 2-4% of a 26-second travel and accumulates in the position the motor node believes it is at. It now sleeps to the soonest deadline and is woken when a nearer one is scheduled, which measures 0.9ms end to end through Redis. A promoted timer also went to the back of the queue. It goes into a due lane of its own that `claim` reads first, so work that has waited out a deadline is not held up by work that is merely queued. Beside it, in the same code: seeding a message now bumps its version, so a re-put flow's synchronous nodes no longer wait forever on a value that is sitting in state; the consumer group drops the consumers of engines that are gone (138 had accumulated on this installation); and the cast that closes the long-standing `xclaim` mypy error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
stroblmeandClaude Opus 5
|
0ffcabfdb9 |
Media dtypes: image, audio and video as narrowed artifact references
Docs / docs (push) Successful in 30s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m7s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m54s
pre-commit / pre-commit (push) Failing after 4m24s
Test Backend / test-backend (push) Successful in 3m8s
Compose Smoke Test / test-compose (push) Successful in 40s
Playwright Tests / merge-reports (push) Successful in 1m33s
A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.
Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.
Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.
Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.
What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
stroblmeandClaude Opus 5
|
180da3d640 |
Stop paying five Redis round trips and a global lock per message
The engine was I/O-bound on its own state backend. `RedisState.lock()` is one key — `pipeline:_lock` — for the whole process, taken five times a message at two round trips each, and every cascade and every node read queued behind it. Inside it, reading a node's inputs was three round trips per input (an EXISTS for `in`, then EXISTS and GET for the value), writing was two updates that a single transaction already gives, and the version counters went one INCR at a time. Replaced with the atomic command that was always available: `get_present` is one MGET and tells a missing key from one holding null, so the lock it used to be read under bought nothing; value and timestamp land in one `update`, which is a MULTI/EXEC; `increment_multi` pipelines the counters. `values()` — what every websocket snapshot calls — is two reads whatever the message count instead of two per message. Beside that: every webhook did its blocking XADD on the asyncio event loop (MQTT already used `to_thread`); the per-execution `NodeOutcome` was built and validated even with no run watching; `_minute` built a tz-aware datetime per event on the loop thread to key a dict, and now keys on an int; `move_due` promoted delayed items one round trip each, every second; `FLOW_MAX_CASCADES` makes the in-flight ceiling a setting rather than a constant. `orjson` replaces stdlib json where a message pays for it — state, the journal, the engine side of the worker pipe. `fluksio-worker` stays dependency-free, and the run-cache digest stays on stdlib so no stored key is invalidated. A non-finite number now stores as `null` rather than the bare `NaN` that was never JSON. Measured with `scripts/bench_engine.py` against a real Redis, 200 messages: a five-node chain went from 43.9 to 103.1 msg/s with p50 latency 2110ms → 782ms and p95 3913ms → 1439ms; one source into twenty consumers went from 5.4 to 33.7 msg/s. In memory, twenty consumers went from 187 to 448 msg/s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf |
||
|
stroblmeandClaude Opus 5
|
5726c80948 |
Say how much work is waiting, not just how much is running
`RedisWorkQueue.stats` read XPENDING, which counts entries delivered to a consumer and not yet acknowledged — work in progress. Entries sitting in the stream undelivered were counted nowhere, so an engine hours behind reported itself idle: on the house, `pending: 4` while the group's lag was 1554. The group's own `lag` is the missing number. `backlog` now carries it on both queues (`len(_items)` in memory), leads the health tile, and a sustained one publishes `engine_degraded` from the timer thread — named with the flow most of the waiting work belongs to, sampled from the undelivered tail, since that is the actionable half. It is a summary problem rather than a /utils/health 503: a backlog should not restart the container. Also drops the keyspace `scan_iter` `stats()` did per poll to count parked items — it walked every state and idempotency key twice per ten seconds — for a set the park/unpark path maintains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf |
||
|
stroblmeandClaude Opus 5
|
df22475f54 |
Claim only what the cascade pool can run
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UytviPMJbXzD8P84nLvXcq |
||
|
stroblmeandClaude Opus 5
|
4941c2787c |
Add fluksio status, and ask for a run's parameters at a terminal
Two halves of the same gap: the CLI could start work but not show you any. `fluksio status` draws the home screen's top half in a terminal — health and what is wrong with it, every flow with its state and node count, and the recent runs and failures under them. `--watch` keeps it there. Rich does the drawing; it was already installed under fastapi's own CLI, and is named now because a command depends on it. `fluksio run` with no parameters at a terminal asks for them, one line per declared input with its declared value in brackets — so Enter through the lot is what running the defaults looks like, and an artifact input takes the `@run:` spelling the engine now resolves. A scripted run is untouched: passing any parameter, or piping the command, skips the questions, as does --defaults. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69 |
||
|
stroblmeandClaude Opus 5
|
93374a310e |
Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69 |
||
|
stroblmeandClaude Opus 5
|
60d7ec81c0 |
Rename the import package app to fluksio
A wheel whose top-level module is `app` collides with anything else in a user's venv, so the package that is about to be published takes the name it is published under. Only the Python package moves; the repo, the Docker WORKDIR and the compose project keep theirs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
stroblmeandClaude Opus 5
|
a8065ad91c |
Give the work queue its own Redis root, and pin what the suite runs as
The queue shared the `pipeline:` prefix with flow state, so `RedisState.clear()` could DEL the queue stream and `keys()` enumerated queue entries — only callers filtering `__`-prefixed names kept it safe. It moves to `queue:` without a migration: whatever is in flight at the upgrade is dropped once, documented in DEPLOY.md rather than papered over. Alongside it: `pool_pre_ping`, so a connection idle across a Postgres restart costs a round trip instead of a failed request; the test suite pins ENVIRONMENT=local and DOMAIN=localhost itself rather than inheriting a deployment's .env; and `depth` leaves the queue stats, where it reported the capped journal length as if it were a backlog. ALERTS_FILE and PANELS_FILE now point at /data. They defaulted to a path on no volume, so alert routing and every wall-panel pairing were living in the container's writable layer and vanishing on each rebuild. Carrying the existing files across is a manual step; DEPLOY.md has it. development.md was still the upstream template — compose.override.yml, localhost.tiangolo.com, `docker compose watch` as the dev flow — and said nothing about the Playwright suite. Rewritten against what the Makefiles actually do. deployment.md was template text too, duplicating the root DEPLOY.md, and is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs |
||
|
stroblmeandClaude Opus 5
|
2554488a73 |
Backend: real alert test results, state cleanup on delete/rename, node trigger errors, queue and collector fixes
`AlertManager.send` swallowed every delivery failure, so the alerts screen's Test button answered 200 whatever happened — the one thing it exists for. It takes `raise_on_error` now, which only the test route passes; the per-channel loop keeps the swallow, because one dead channel must not stop the others hearing about the same fault. A refused delivery answers 502 with whatever the sender said. Renaming a flow left its values under the old name for good: the delete path already swept them, the rename path never did. It calls the same `forget_flow`, which covers the messages and the `__ts__`/`__version__`/`__history__` bookkeeping keyed by message name. Cleanup, not migration — they repopulate under the new name on the next run. Triggering a node by hand ran `Node.__call__` with nothing catching it, so a node that raised produced a 500 and a stack trace in the server log, and nothing at all on the canvas. `Pipeline.publish_error` is the reporting half of `_execute_node` lifted out; both paths go through it, so a manual failure now reads the same on the canvas and in the metrics as a queued one. The route answers 400 with the node's error. `MemoryWorkQueue.stats()` counts claimed-but-unacknowledged work rather than reporting zero, so the health tile means something without Redis. The metrics collector's held tracebacks are capped at `DETAIL_CAP` and swept on the same `RUN_STALE_S` cutoff the open runs use, instead of one untruncated traceback per node kept for the life of the process — a traceback still survives the flush between the log and the failure it belongs to. `GET /observability/events` takes `since`/`until`, the window `/runs` already took, so a failures list can cover the span the charts beside it are drawn from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XC2jX6Hdj7pxGGKzBTrbqB |
||
|
stroblmeandClaude Fable 5
|
93b4a9a0b1 |
Step a paused flow, and deliver what a rate limit held back
Three things a pause and an interval were quietly losing:
- A rebuild builds a fresh pipeline, so nothing is paused any more and no
resume ever comes for what the old one parked. Release it on rebuild.
- POST /flows/{name}/step takes the oldest parked item and runs that one wave
while the flow stays paused, so a held-back cascade can be walked through.
Nothing parked answers plainly rather than failing.
- A per-port interval was leading-edge only: a producer going quiet inside the
window left the consumer on the value before it. The held value is kept and
a flush item scheduled on the queue's existing timer, so the window ends with
a delivery. One timer in flight per node, and none without a queue to run it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7LwYgJfpkbLCTeiAf8U4A
|
||
|
rootandClaude Fable 5
|
04329149b3 |
Journal work before running it, so a crash stops losing messages
Execution was fire-and-forget: an MQTT message or webhook ran a cascade on a ThreadPoolExecutor built for that one wave, and an engine that died halfway through simply lost whatever was in flight. Concurrent triggers each built their own pool, so load meant unbounded threads. Every external trigger is now journaled to a Redis Streams queue before anything runs, and acknowledged only once its cascade finishes. A consumer thread drives cascades on one long-lived pool while node bodies run on another, so a cascade cannot starve the nodes it is waiting for. A reaper reclaims what a dead consumer never acknowledged — verified end to end: work journaled while the engine was stopped runs on restart, and work abandoned mid-cascade comes back as a second delivery. At-least-once needs a guard, so nodes that reach outside are marked non-idempotent and skipped on a redelivery they already completed. Without Redis the queue degrades to an in-memory one that does not pretend to be durable, and interactive callers still run inline. Also fixes two things this turned up: a delay node was sleeping on a worker thread, where a handful of them could occupy the whole pool, and webhooks 404'd whenever MCP was enabled because the app mounted at / answered first for every path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY |