Coalesce the event bus, and fix the socket that ended on a client frame

A three-node cascade publishes 13-16 events and each one crossed to the
event loop on its own. They are one `call_soon_threadsafe` now — whatever
was published between two turns of the loop goes over together — and every
subscriber still receives every event, oldest still dropped first when one
falls behind.

The socket end of the same path:

- **any frame from the client ended its stream.** `receive_text` was
  awaited once, outside the loop, so a keepalive — or anything else a
  client decided to say — satisfied it and was read as the client going
  away. It is recreated per iteration; only a disconnect ends the stream.
- events go out in one frame per wave (`{"type": "batch", "events": [...]}`,
  capped at 64), serialised once with orjson rather than per client with
  the stdlib's `json.dumps` through `send_json`. The client unpacks a batch
  and still understands single frames, so an older engine behind a newer
  bundle keeps working.
- authenticating and building the snapshot happen on a thread. Both were on
  the event loop: one is a database round trip, the other reads the whole
  of state, per connect and again per `dashboard_changed` per panel.

`Pipeline.values()` — what that snapshot is — no longer SCANs the whole
Redis namespace. It scanned five bookkeeping keys for every message to find
the messages; `RedisState` keeps a set of the names beside them and answers
from it. Maintained wherever a message is written, so a seeded value or a
deleted flow keeps it exact.

On the client, while in the same file:

- a `node_health` event invalidates the flow's detail. The canvas draws
  health from the server-derived `issues`, so a node going down or
  recovering only showed on mount, navigation or a rebuild. The store had
  a health map of its own that nothing ever read; it and `useNodeHealth`
  are gone rather than wired up, since the server's view is the one the
  canvas already uses.
- a reconnect invalidates the five key families this socket feeds instead
  of the entire cache, and the backoff is jittered. The usual reason a
  socket dropped is the engine restarting, so every tab and every wall
  panel refetched everything, together, at the moment it was least able to
  answer.
- a frame that will not parse costs the frame, not the connection. It was
  the one unguarded `JSON.parse` in the app; an exception there escaped to
  `window.onerror` and left whatever it had already applied behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T

backend/fluksio/flow/events.py

@@ -9,6 +9,7 @@ from __future__ import annotations
 
 import asyncio
 import logging
+import threading
 from collections import deque
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
@@ -26,6 +27,13 @@ class EventBus:
     def __init__(self) -> None:
         self._loop: asyncio.AbstractEventLoop | None = None
         self._subscribers: set[asyncio.Queue[dict[str, Any]]] = set()
+        # Events published since the last flush was scheduled. A cascade
+        # publishes 13-16 of them and each used to cost its own
+        # `call_soon_threadsafe`; they now cost one between two turns of the
+        # loop, which is what a wave is.
+        self._pending: list[dict[str, Any]] = []
+        self._pending_lock = threading.Lock()
+        self._flush_scheduled = False
         # Kept whether or not anyone is listening, so opening the log panel
         # shows what just happened rather than an empty box.
         self.recent_logs: deque[dict[str, Any]] = deque(maxlen=LOG_HISTORY)
@@ -54,21 +62,38 @@ class EventBus:
         loop = self._loop
         if loop is None or not self._subscribers:
             return
+        with self._pending_lock:
+            self._pending.append(event)
+            if self._flush_scheduled:
+                # A flush is already on its way and has not run yet, so it
+                # will find this event too.
+                return
+            self._flush_scheduled = True
         try:
-            loop.call_soon_threadsafe(self._dispatch, event)
+            loop.call_soon_threadsafe(self._dispatch)
         except RuntimeError:
             # Loop already closed — shutting down.
-            pass
+            with self._pending_lock:
+                self._pending.clear()
+                self._flush_scheduled = False
 
-    def _dispatch(self, event: dict[str, Any]) -> None:
+    def _dispatch(self) -> None:
+        """Hand everything published since the last turn to every subscriber."""
+        with self._pending_lock:
+            batch = self._pending
+            self._pending = []
+            self._flush_scheduled = False
+        if not batch:
+            return
         for queue in self._subscribers:
-            if queue.full():
-                # Drop the oldest so a slow client never blocks the engine.
-                try:
-                    queue.get_nowait()
-                except asyncio.QueueEmpty:
-                    pass
-            queue.put_nowait(event)
+            for event in batch:
+                if queue.full():
+                    # Drop the oldest so a slow client never blocks the engine.
+                    try:
+                        queue.get_nowait()
+                    except asyncio.QueueEmpty:
+                        pass
+                queue.put_nowait(event)
 
     @asynccontextmanager
     async def subscribe(self) -> AsyncIterator[asyncio.Queue[dict[str, Any]]]:

backend/fluksio/flow/nodes/delay.py

@@ -162,9 +162,7 @@ class DelayNode(Node):
             if self._pipeline is not None and self._pipeline.defer(
                 self, self._to_messages(output) or {}, self.delay
             ):
-                logger.debug(
-                    "[%s] Sending %s in %ss", self.name, output, self.delay
-                )
+                logger.debug("[%s] Sending %s in %ss", self.name, output, self.delay)
                 return None
             time.sleep(self.delay)
 

backend/fluksio/flow/pipeline.py

@@ -1725,9 +1725,7 @@ class Pipeline:
         per timestamp, one at a time under the global state lock.
         """
         keys = [
-            k
-            for k in self._state.keys()
-            if not k.startswith("__") and (not flow or flow_of(k) == flow)
+            k for k in self._state.message_names() if not flow or flow_of(k) == flow
         ]
         if not keys:
             return {}

backend/fluksio/flow/state.py

@@ -9,7 +9,7 @@ from __future__ import annotations
 
 from abc import ABC, abstractmethod
 from collections import deque
-from collections.abc import Iterator
+from collections.abc import Iterable, Iterator
 from contextlib import contextmanager
 from threading import RLock
 from typing import Any, cast
@@ -98,6 +98,15 @@ class StateBackend(ABC):
         """Clear all keys in the state."""
         ...
 
+    def message_names(self) -> list[str]:
+        """The message names in state, without the bookkeeping keys.
+
+        Its own method because the Redis backend can answer it from a set it
+        maintains, rather than by scanning a namespace that holds five
+        bookkeeping keys for every message.
+        """
+        return [k for k in self.keys() if not k.startswith("__")]
+
     def close(self) -> None:  # noqa: B027
         """Release whatever the backend holds outside this process.
 
@@ -510,18 +519,44 @@ class RedisState(StateBackend):
         data = cast(bytes | None, self._client.get(self._key(key)))
         return self._deserialize(data) if data is not None else default
 
+    def _names_key(self) -> str:
+        return self._key("__names__")
+
+    def message_names(self) -> list[str]:
+        """The message names, from the set kept beside them.
+
+        `keys()` is a SCAN of the whole namespace, which holds a version, a
+        timestamp, a history list and a last-seen marker for every message —
+        so reading the message names cost several times as many round trips
+        as there are messages. Every websocket connect asks for this.
+        """
+        members = cast(set[bytes], self._client.smembers(self._names_key()))
+        return [m.decode("utf-8") for m in members]
+
+    def _note(self, pipe: Any, keys: Iterable[str]) -> None:
+        """Record the message names among `keys`, in the caller's pipeline."""
+        names = [k for k in keys if not k.startswith("__")]
+        if names:
+            pipe.sadd(self._names_key(), *names)
+
     def set(self, key: str, value: Any) -> None:
         data = self._serialize(value)
+        pipe = self._client.pipeline()
         if self._ttl:
-            self._client.setex(self._key(key), self._ttl, data)
+            pipe.setex(self._key(key), self._ttl, data)
         else:
-            self._client.set(self._key(key), data)
+            pipe.set(self._key(key), data)
+        self._note(pipe, [key])
+        pipe.execute()
 
     def exists(self, key: str) -> bool:
         return bool(self._client.exists(self._key(key)))
 
     def delete(self, key: str) -> None:
-        self._client.delete(self._key(key), self._history_key(key))
+        pipe = self._client.pipeline()
+        pipe.delete(self._key(key), self._history_key(key))
+        pipe.srem(self._names_key(), key)
+        pipe.execute()
 
     def clear(self) -> None:
         """Clear all keys in the namespace."""
@@ -568,6 +603,7 @@ class RedisState(StateBackend):
                 pipe.setex(self._key(key), self._ttl, data)
             else:
                 pipe.set(self._key(key), data)
+        self._note(pipe, mapping)
         pipe.execute()
 
     @contextmanager
@@ -780,6 +816,7 @@ class RedisState(StateBackend):
                 pipe.expire(history_key, self._ttl)
         for key in counters:
             pipe.incr(self._key(key))
+        self._note(pipe, values)
         pipe.execute()
 
     def history(self, key: str) -> list[tuple[float, float]]: