Schedule a node across every machine, not just this one

The engine answered "where does this node run" twice, in two ways that could not see each other: a device sent it to a worker carrying that label, and resources were counted against the engine's own cores. Declaring both meant the second answer won and nothing was counted at all — which the data-science getting-started page and the worked example both do. One question now, in flow/placement.py: of every machine attached, which could grant what this node asked for, and which of those has it free. The books move onto each machine — one accountant per worker, built from the inventory it reported — and the waiting moves above them, where one condition variable can be woken by a release anywhere or by a worker attaching. Locks go one way: placer, then a machine's books, never back. So a node asking for a card now finds the box that has one, rather than being clamped down to none and run here. When nothing can grant the ask at all it is still cut down and run — a flow written on a cluster has to work on a laptop — but the ceiling is one real machine now, since taking the largest of each dimension separately can describe a machine nobody has. Two things fixed on the way. A device on a connector node held every batch run of its flow forever, waiting for a worker that could never run an entry point. And `prefer` falling back to the engine skipped the books, so the fallback held nothing. The bench flow's node has taken a `params` argument that with_settings has not forwarded for some time, so the benchmark could not run at all: 62 ms median submit-to-result with this, against the 61 ms on record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
2026-08-27 08:49:36 +02:00
parent 1a9753fa9d
commit 6ff56533f5
14 changed files with 1214 additions and 326 deletions
@@ -0,0 +1,31 @@
+"""run.needs
+
+A run recorded the worker *labels* its nodes asked for, which answered whether
+anything was attached to run them but nothing about whether that machine was
+big enough. The size goes beside the labels, so a queued run can tell waiting
+for a machine from having nowhere to run at all.
+
+Revision ID: b3f1a7c50d92
+Revises: e5b8c2f4a913
+Create Date: 2026-08-27
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "b3f1a7c50d92"
+down_revision = "e5b8c2f4a913"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # Nullable: a run submitted before this declared nothing, and null is that
+    # rather than a run that needs nothing in particular.
+    op.add_column("run", sa.Column("needs", sa.JSON(), nullable=True))
+
+
+def downgrade():
+    op.drop_column("run", "needs")