stroblmeandClaude Opus 5
180da3d640
The engine was I/O-bound on its own state backend. `RedisState.lock()` is one key — `pipeline:_lock` — for the whole process, taken five times a message at two round trips each, and every cascade and every node read queued behind it. Inside it, reading a node's inputs was three round trips per input (an EXISTS for `in`, then EXISTS and GET for the value), writing was two updates that a single transaction already gives, and the version counters went one INCR at a time. Replaced with the atomic command that was always available: `get_present` is one MGET and tells a missing key from one holding null, so the lock it used to be read under bought nothing; value and timestamp land in one `update`, which is a MULTI/EXEC; `increment_multi` pipelines the counters. `values()` — what every websocket snapshot calls — is two reads whatever the message count instead of two per message. Beside that: every webhook did its blocking XADD on the asyncio event loop (MQTT already used `to_thread`); the per-execution `NodeOutcome` was built and validated even with no run watching; `_minute` built a tz-aware datetime per event on the loop thread to key a dict, and now keys on an int; `move_due` promoted delayed items one round trip each, every second; `FLOW_MAX_CASCADES` makes the in-flight ceiling a setting rather than a constant. `orjson` replaces stdlib json where a message pays for it — state, the journal, the engine side of the worker pipe. `fluksio-worker` stays dependency-free, and the run-cache digest stays on stdlib so no stored key is invalidated. A non-finite number now stores as `null` rather than the bare `NaN` that was never JSON. Measured with `scripts/bench_engine.py` against a real Redis, 200 messages: a five-node chain went from 43.9 to 103.1 msg/s with p50 latency 2110ms → 782ms and p95 3913ms → 1439ms; one source into twenty consumers went from 5.4 to 33.7 msg/s. In memory, twenty consumers went from 187 to 448 msg/s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
Fluksio
Fluksio is a node-based automation software that brings trust and reliability to your flow. It just works and looks good. Get started by running
pip install fluksio
fluksio serve
and you're ready to go.
For data science
You can turn your existing data science project into a flow by decorating your functions with @node ...
# myresearch/train.py
import fluksio
from fluksio import Port, node
@node(
requires=["dataset", Port("lr", "float")],
provides=[Port("loss", "float", stream=True), Port("weights", "artifact")],
device="gpu", device_policy="prefer",
)
def fit(dataset, lr, epochs=25):
for epoch in range(epochs):
loss = step(...)
yield {"loss": loss} # published as it happens, kept as a series
return {"weights": fluksio.save_artifact("weights.pt")}
... and passing them to a Flow:
# myresearch/pipeline.py
from fluksio import Flow, Port
from myresearch.data import prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit
train = Flow("train", nodes=[prepare, fit, evaluate],
inputs=[Port("lr", "float", initial=0.01)], outputs=["score"])
Fluksio will automatically infer the order of nodes based on the inputs and outputs you defined. When everything is set, you can launch your first run as follows:
fluksio run train --lr 0.05 --wait
Checkout our documentation for more infos.
Some other features
- Flows: typed messages between nodes, wired by name, edited on a canvas or declared in code. Every change is a commit in a git repository you own.
- Runs: an experiment and a CI-style job are the same entity. Parameters, seed, result, per-node timings, artifacts and the commit it ran at.
- Dashboards: charts and controls bound to the same messages the flows carry, with no separate metrics pipeline.
- Remote workers:
pip install fluksio-workeron the GPU box; it dials out over one websocket, so nothing there has to be reachable.
Fluksio can also be used for facility automation. Visit us on fluksio.com or go straight to our documentation.
License
Copyright (C) 2026 Melvin Strobl - GNU Affero General Public License v3.0 or later. Running a modified version over a network obliges you to offer its users the corresponding source (AGPL §13).