Let a sweep run more than four at a time, and name the run a failure was in
Concurrent runs sat at 4 whatever FLOW_MAX_CASCADES said: that setting bounds cascades, and the run drivers read a hardcoded MAX_PARALLEL nobody could reach. FLOW_MAX_RUNS is the knob they read now, --max-runs/--max-cascades/--max-workers are the same three as flags on serve, and the engine says which numbers it started with — which is the only way to tell that a settings file was read. Events keep the run they happened in. The payload always carried it and the persist path dropped it, so reading one run's failures meant filtering the engine-wide list; a batch run's id reaches those events now too, since a run has no journaled item to name itself by. Also: a provisioner's 0 means "no deadline" rather than "cancel on the next reconcile", and a command that reaches no engine says how to start one.
This commit is contained in:
@@ -125,6 +125,7 @@ warning into a refusal to start.
|
||||
|---|---|---|
|
||||
| `FLOW_MAX_WORKERS` | `4` | node-code subprocesses run in parallel |
|
||||
| `FLOW_MAX_CASCADES` | `4` | cascades in flight at once; throughput is this over the mean cascade time, so raise it where nodes wait on a network rather than a CPU |
|
||||
| `FLOW_MAX_RUNS` | `4` | batch runs driven at once. A different limit from the one above: a run drives a whole graph, and its nodes are bounded by `FLOW_MAX_WORKERS`. This is what a sweep queues behind |
|
||||
| `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit |
|
||||
| `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy |
|
||||
| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. Not detected — say how many there are |
|
||||
@@ -132,6 +133,10 @@ warning into a refusal to start.
|
||||
| `ARTIFACT_GC_INTERVAL_S` | `3600` | how often artifact bytes nothing refers to are swept away; 0 never sweeps |
|
||||
| `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it |
|
||||
|
||||
The three concurrency limits are also flags on `fluksio serve` — `--max-workers`,
|
||||
`--max-cascades`, `--max-runs` — which outrank the file, and the engine says
|
||||
which numbers it started with in its first lines.
|
||||
|
||||
An artifact is referred to by a run that recorded it or by a message currently
|
||||
holding it; anything else is what a camera published four hours ago, and the
|
||||
sweep is what keeps a flow streaming media from filling the disk. It stands
|
||||
|
||||
Reference in New Issue
Block a user