Files
stroblmeandClaude Opus 5 d471614e6a Push a frame instead of storing and fetching it
The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.

`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.

The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.

Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
2026-09-02 10:15:14 +02:00

330 lines
14 KiB
YAML

# Production stack. Layered by the Makefile:
# compose.yml → production (Traefik + TLS, restart always)
# + compose.dev.yml → local dev (published ports, hot reload)
# + compose.local.yml → integrated stack on the shared `proxy` net
# Paths are relative to this directory, which compose uses as the project
# directory (build contexts therefore point at `..`, the repo root).
services:
# Umami's database, and nothing else's. The engine keeps its own state in
# SQLite on the flow-data volume, which is why this is behind a profile now.
db:
image: postgres:18
container_name: fluksio-db
restart: always
profiles: ["analytics"]
security_opt:
- no-new-privileges:true
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-postgres} -d ${POSTGRES_DB:-app}"]
interval: 10s
retries: 5
start_period: 30s
timeout: 10s
volumes:
- app-db-data:/var/lib/postgresql/data/pgdata
env_file:
- ../.env
environment:
- PGDATA=/var/lib/postgresql/data/pgdata
# Not required of the stack at large any more: only the analytics
# profile starts this, and the engine has no Postgres to configure.
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres}
- POSTGRES_USER=${POSTGRES_USER:-postgres}
- POSTGRES_DB=${POSTGRES_DB:-app}
redis:
image: redis:8.2-alpine
container_name: fluksio-redis
restart: always
security_opt:
- no-new-privileges:true
# Flow state, not a cache: append-only persistence keeps the last value of
# every message across a restart.
command: redis-server --appendonly yes
# Deliberately not on `proxy` and not published: only the backend talks to it.
networks:
- default
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
retries: 5
start_period: 10s
timeout: 5s
volumes:
- app-redis-data:/data
adminer:
image: adminer
container_name: fluksio-adminer
restart: always
profiles: ["analytics"]
security_opt:
- no-new-privileges:true
# Deliberately not on `proxy`: the Postgres UI is reachable from the host
# in dev only (see compose.dev.yml), never routed from the internet.
networks:
- default
depends_on:
- db
environment:
- ADMINER_DESIGN=pepa-linha-dark
backend:
image: '${DOCKER_IMAGE_BACKEND?Variable not set}:${TAG-latest}'
container_name: fluksio-api
restart: always
# PID 1 that reaps orphans. A python node's worker processes outlive the
# process holding their handle whenever that one goes without stopping the
# pool -- which `--reload` does on every source edit -- and reparent onto
# PID 1, which is the app itself and waits for nobody else's children. The
# container filled up with zombie `python`. Declared here rather than in
# compose.dev.yml because an init closes the whole class, not just reload.
init: true
security_opt:
- no-new-privileges:true
networks:
- proxy
- default
depends_on:
redis:
condition: service_healthy
restart: true
env_file:
- ../.env
environment:
- DOMAIN=${DOMAIN}
- FRONTEND_HOST=${FRONTEND_HOST?Variable not set}
- ENVIRONMENT=${ENVIRONMENT}
- BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS}
- SECRET_KEY=${SECRET_KEY?Variable not set}
- FIRST_SUPERUSER=${FIRST_SUPERUSER?Variable not set}
- FIRST_SUPERUSER_PASSWORD=${FIRST_SUPERUSER_PASSWORD?Variable not set}
- SMTP_HOST=${SMTP_HOST}
- SMTP_USER=${SMTP_USER}
- SMTP_PASSWORD=${SMTP_PASSWORD}
- EMAILS_FROM_EMAIL=${EMAILS_FROM_EMAIL}
- SENTRY_DSN=${SENTRY_DSN}
# Flow state survives a restart in Redis; without a host it stays in memory.
- REDIS_HOST=redis
# Everything this instance keeps — the database, the flows, secrets,
# artifacts and the user venv — on one volume. The paths below derive
# from it; they are spelled out because each is load bearing.
- DATA_DIR=/data
# Flows are files in a git repository; secrets sit encrypted beside it.
- FLOWS_DIR=/data/flows
- SECRETS_FILE=/data/secrets.enc
# The MCP endpoint for agents, and the key its tokens are signed with.
# The key lives on the same volume as the flows, so it survives a rebuild
# and every issued token with it.
- MCP_ENABLED=${MCP_ENABLED-false}
- OAUTH_PRIVATE_KEY_FILE=/data/oauth-key.pem
# The portal enrolment, for the same reason: it holds this instance's
# credential and the key it pinned, so losing it on a rebuild would mean
# enrolling again by hand. Absent until someone connects a portal.
- CLOUD_CONFIG_FILE=/data/cloud.json
# Alert routing and the wall-panel pairings. Both are written at runtime,
# so off the volume they would land in the container's writable layer and
# be lost on the next rebuild — un-pairing every screen.
- ALERTS_FILE=/data/alerts.json
- PANELS_FILE=/data/panels.json
# The push keypair and the browsers subscribed to it. Off the volume, a
# rebuild would silently stop every phone being notified: the key they
# subscribed against would be gone.
- WEBPUSH_FILE=/data/webpush.json
# Schedules are written in local time: "off at 02:00" means the house's
# two in the morning, not the container's. Unset, an image is UTC, and a
# cron would be right twice a year.
- TZ=${TZ:-UTC}
volumes:
- app-flow-data:/data
# `/dev/shm`, where frames a flow only shows live are held. Docker's
# default is 64 MB, which the ring's own bound sits just under; raise both
# together (ARTIFACT_VOLATILE_BYTES) for more cameras or bigger frames.
shm_size: 128mb
# Deep health: fails when the event loop is wedged or Redis is gone, not
# just when the process is dead. Autoheal restarts on unhealthy.
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health/"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
build:
context: ..
dockerfile: backend/Dockerfile
labels:
- autoheal=true
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.constraint-label=proxy
- traefik.http.services.${STACK_NAME?Variable not set}-backend.loadbalancer.server.port=8000
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.rule=Host(`api.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.entrypoints=http
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.rule=Host(`api.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.entrypoints=https
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls=true
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls.certresolver=le
- traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.middlewares=https-redirect
frontend:
image: '${DOCKER_IMAGE_FRONTEND?Variable not set}:${TAG-latest}'
container_name: fluksio-app
restart: always
security_opt:
- no-new-privileges:true
networks:
- proxy
- default
build:
context: ..
dockerfile: frontend/Dockerfile
args:
# The address baked into the bundle. `scripts/setup.sh` already derives
# it from ENVIRONMENT and writes it to .env — http for a local target,
# because nothing terminates TLS there — so take it from the environment
# and keep the https form only as the default for a checkout that
# predates that key. Hardcoding https here made every `up --build` that
# did not also layer compose.local.yml ship a bundle calling
# https://api.localhost, which fails the certificate check and with it
# the whole login.
- VITE_API_URL=${VITE_API_URL:-https://api.${DOMAIN?Variable not set}}
- NODE_ENV=production
labels:
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.constraint-label=proxy
- traefik.http.services.${STACK_NAME?Variable not set}-frontend.loadbalancer.server.port=80
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.rule=Host(`app.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.entrypoints=http
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.rule=Host(`app.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.entrypoints=https
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls=true
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls.certresolver=le
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.middlewares=https-redirect
# Static documentation site (zensical), served at docs.${DOMAIN}. Content and
# config live in ../docs/ and ../zensical.toml; `make docs-serve` previews it.
docs:
container_name: fluksio-docs
restart: always
security_opt:
- no-new-privileges:true
build:
context: ..
dockerfile: docker/Dockerfile.docs
expose:
- "80"
networks:
- proxy
labels:
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.constraint-label=proxy
- traefik.http.services.${STACK_NAME?Variable not set}-docs.loadbalancer.server.port=80
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-http.rule=Host(`docs.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-http.entrypoints=http
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-http.middlewares=https-redirect
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-https.rule=Host(`docs.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-https.entrypoints=https
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-https.tls=true
- traefik.http.routers.${STACK_NAME?Variable not set}-docs-https.tls.certresolver=le
# Docker never restarts a merely *unhealthy* container on its own; autoheal
# closes that gap for the services labeled autoheal=true.
#
# Behind a profile because it needs the Docker socket, which is host-wide
# authority: on a machine that runs anything besides this stack, that is a
# deliberate operator decision. `make up` opts in; the dev stacks do not.
autoheal:
image: willfarrell/autoheal:latest
container_name: fluksio-autoheal
profiles: ["autoheal"]
restart: always
security_opt:
- no-new-privileges:true
networks:
- default
environment:
# Scoped by label, so it only ever restarts this stack's backend.
- AUTOHEAL_CONTAINER_LABEL=autoheal
- AUTOHEAL_INTERVAL=15
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
# ── Site analytics (optional) ──────────────────────────────────────
# Self-hosted, cookieless Umami on analytics.${DOMAIN}. Gated behind the
# `analytics` profile, so a plain `make up` never starts it; bring it up with
# `make umami`, which also provisions its database.
umami:
container_name: fluksio-umami
# Pinned by digest so a bump is a deliberate step: this is an internet-facing
# third-party dashboard, not a library.
image: ghcr.io/umami-software/umami:postgresql-latest@sha256:87312d334d009ee67ee0d2fba8fed01435547cc468e452243aef5133a9984d48
restart: always
security_opt:
- no-new-privileges:true
profiles: ["analytics"]
networks:
- proxy
- default
depends_on:
db:
condition: service_healthy
# Deliberately no `env_file: ../.env` -- this third-party image receives the
# three variables it needs and none of the app's secrets.
environment:
- DATABASE_TYPE=postgresql
# Least-privilege role owning only the umami database, never the shared
# Postgres superuser (provisioned by `make umami`).
- DATABASE_URL=postgresql://${UMAMI_DB_USER:-umami}:${UMAMI_DB_PASSWORD}@db:5432/${UMAMI_DB:-umami}
- APP_SECRET=${UMAMI_APP_SECRET:-}
expose:
- "3000"
# No healthcheck on purpose: the upstream image ships neither curl nor wget,
# so a probe would leave the container permanently `starting` and Traefik
# would never route it (the same trap the backend's python healthcheck
# documents).
labels:
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.constraint-label=proxy
- traefik.http.services.${STACK_NAME?Variable not set}-umami.loadbalancer.server.port=3000
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-http.rule=Host(`analytics.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-http.entrypoints=http
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-http.middlewares=https-redirect
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-https.rule=Host(`analytics.${DOMAIN?Variable not set}`)
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-https.entrypoints=https
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-https.tls=true
- traefik.http.routers.${STACK_NAME?Variable not set}-umami-https.tls.certresolver=le
volumes:
app-db-data:
app-redis-data:
app-flow-data:
networks:
# Shared with the website stack and whatever reverse proxy fronts them.
# Created once by the workspace root's scripts/setup.sh.
proxy:
external: true