Add the fluksio CLI: serve, enroll, worker
`pip install fluksio && fluksio serve` on a machine with no Docker, no database and no configuration — which is the case this is for: a node on a cluster where ports cannot be opened. It makes its data directory, its key and an admin account, prints the password once, and serves. Pairing is `fluksio enroll <code> --portal …`, doing what the Settings screen does through the same function, before the engine starts and without one running — a machine nobody can route to has no browser pointed at it either. The portal serves the dashboard, so nothing is served here. Two things had to give way. `fastapi[standard]` pulls a cloud CLI that wants sentry-sdk 2.x while we pinned below it — no pip resolution existed, so the pin is lifted, which the comment beside it had been waiting for and which also lets the Python cap go. And `uv` is now a dependency rather than something to find on PATH: the Modules screen is how a data scientist installs torch, and it was quietly falling back to the engine's own interpreter. The CLI imports nothing from the engine before it has set DATA_DIR — the settings are built on the first import of core.config, and reaching it early put the database in the working directory. There is a test for that now, because the failure is silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Publish both distributions to PyPI on a version tag.
|
||||
#
|
||||
# Tags are the trigger rather than pushes to main: a release is a decision, and
|
||||
# a version already on PyPI cannot be replaced.
|
||||
name: Publish
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
- name: Set up uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
- name: The tag is what the packages say they are
|
||||
# Publishing 0.2.0 from a v0.3.0 tag is the kind of thing nobody
|
||||
# notices until an install pulls the wrong one.
|
||||
run: |
|
||||
version="${GITHUB_REF_NAME#v}"
|
||||
for project in backend worker; do
|
||||
declared=$(uv version --short --directory "$project")
|
||||
[ "$declared" = "$version" ] || {
|
||||
echo "$project declares $declared, tag says $version" >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
- run: uv build --all-packages --out-dir dist
|
||||
- name: Both wheels install and run
|
||||
run: |
|
||||
uv run --no-project --with dist/fluksio_worker-*.whl fluksio-worker --help
|
||||
uv run --no-project --with dist/fluksio-*.whl --with dist/fluksio_worker-*.whl \
|
||||
fluksio --version
|
||||
- name: Publish
|
||||
env:
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}
|
||||
run: uv publish dist/*
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
.PHONY: dev-utils dev dev-local up down update install dev-backend dev-frontend \
|
||||
generate-client seed-example seed-demo seed-house seed-aircon seed-hosted-demo test test-backend test-frontend soak bench-startup lint lint-backend \
|
||||
lint-frontend format-frontend umami clean help
|
||||
lint-frontend format-frontend umami build clean help
|
||||
|
||||
COMPOSE_ROOT := $(CURDIR)
|
||||
# Explicit project name keeps this stack isolated from the sibling website
|
||||
@@ -161,6 +161,9 @@ bench-startup: ## Time submitting a run (BENCH_ARGS="--kedro ../some/kedro/proj
|
||||
|
||||
# ── Linting ───────────────────────────────────────────────────────
|
||||
|
||||
build: ## Build the fluksio and fluksio-worker wheels into dist/
|
||||
uv build --all-packages --out-dir dist
|
||||
|
||||
lint: lint-backend lint-frontend ## Run all linters
|
||||
|
||||
lint-backend: ## Lint backend with ruff + mypy
|
||||
|
||||
@@ -10,13 +10,35 @@ repo holds the FastAPI backend, the flow engine, and the dashboard SPA. It is se
|
||||
## Layout
|
||||
|
||||
```text
|
||||
backend/ FastAPI + SQLModel + Alembic + SQLite
|
||||
backend/ FastAPI + SQLModel + Alembic + SQLite — the `fluksio` distribution
|
||||
fluksio/flow/ the flow engine (nodes, pipeline, state backends, controller)
|
||||
fluksio/cli.py `fluksio serve` / `enroll` / `worker`
|
||||
worker/ the `fluksio-worker` distribution: the agent and the node runner
|
||||
frontend/ React 19 + TanStack Router + Tailwind 4 + shadcn/ui
|
||||
docker/ compose.yml → compose.dev.yml → compose.local.yml (+ compose.traefik.yml)
|
||||
scripts/ generate-client.sh, test.sh
|
||||
```
|
||||
|
||||
## Install without Docker
|
||||
|
||||
```sh
|
||||
pip install fluksio
|
||||
fluksio serve # ~/.fluksio, SQLite, prints an admin password once
|
||||
fluksio enroll <code> --portal https://hub.fluksio.com # watch it from the portal
|
||||
```
|
||||
|
||||
Nothing else has to be running. `--data-dir` puts the installation somewhere else —
|
||||
worth it on a cluster, where `$HOME` is often a network filesystem SQLite cannot use.
|
||||
The dashboard is served by the portal, so a machine with no inbound route is reached
|
||||
without opening a port: it dials out.
|
||||
|
||||
A machine that should only *run nodes* for an engine elsewhere installs less:
|
||||
|
||||
```sh
|
||||
pip install fluksio-worker
|
||||
fluksio-worker --url wss://api.example.com/api/v1/workers/attach --token "$TOKEN" --labels gpu
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
Normally driven from the workspace root (`make init` once, then `make dev`). Standalone:
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""The Fluksio engine.
|
||||
|
||||
Inside a node, ``import fluksio`` is not this package: the worker installs a
|
||||
reporter of its own under that name before any node code runs, so what a node
|
||||
gets is :mod:`fluksio_worker.worker_main`'s ``emit`` and ``save_artifact``.
|
||||
The stubs below stand in the same place everywhere else, and say so rather
|
||||
than failing as a missing attribute.
|
||||
|
||||
Nothing is imported from the rest of the package here. Every ``from fluksio.x
|
||||
import y`` in the engine passes through this module, and an import cycle or a
|
||||
second of start-up cost would both begin here.
|
||||
"""
|
||||
|
||||
from importlib.metadata import PackageNotFoundError
|
||||
from importlib.metadata import version as _version
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
__version__ = _version("fluksio")
|
||||
except PackageNotFoundError: # pragma: no cover - a checkout that was never installed
|
||||
__version__ = "0.0.0+unknown"
|
||||
|
||||
_OUTSIDE = (
|
||||
"fluksio.{name}() only works inside a node: the worker running it installs "
|
||||
"the real one. There is nothing to {verb} out here."
|
||||
)
|
||||
|
||||
|
||||
def emit(**ports: Any) -> None:
|
||||
"""Publish values on a node's declared output ports, mid-run."""
|
||||
raise RuntimeError(_OUTSIDE.format(name="emit", verb="emit to"))
|
||||
|
||||
|
||||
def save_artifact(
|
||||
source: Any, name: str = "", media_type: str = "application/octet-stream"
|
||||
) -> dict[str, Any]:
|
||||
"""Put bytes in the artifact store and return a reference to them."""
|
||||
raise RuntimeError(_OUTSIDE.format(name="save_artifact", verb="save to"))
|
||||
|
||||
|
||||
def load_artifact(ref: dict[str, Any]) -> bytes:
|
||||
"""Read back what an artifact reference points at."""
|
||||
raise RuntimeError(_OUTSIDE.format(name="load_artifact", verb="load from"))
|
||||
|
||||
@@ -15,11 +15,11 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlmodel import select
|
||||
|
||||
@@ -31,6 +31,7 @@ from fluksio.api.deps import (
|
||||
get_current_user,
|
||||
)
|
||||
from fluksio.cloud import config as cloud_config
|
||||
from fluksio.cloud import enroll as enroll_mod
|
||||
from fluksio.core.security import get_password_hash
|
||||
from fluksio.models import Message, User, UserPublic
|
||||
|
||||
@@ -83,72 +84,16 @@ async def enroll(
|
||||
installation, so the owner's portal sessions arrive here as them. Widening
|
||||
that to anyone else is a local decision made one person at a time, below —
|
||||
never something the portal can do from its side.
|
||||
|
||||
The same work as `fluksio enroll` on the command line, which is how a
|
||||
machine with no browser pointed at it does this.
|
||||
"""
|
||||
if cloud_config.exists():
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="This installation is already connected to a portal",
|
||||
)
|
||||
|
||||
base = body.portal_url.rstrip("/")
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15.0) as client:
|
||||
response = await client.post(
|
||||
f"{base}/api/v1/enroll/",
|
||||
json={"claim_code": body.claim_code, "app_version": "0.1.0"},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502, detail=f"Could not reach the portal: {exc}"
|
||||
) from exc
|
||||
if response.status_code == 404:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="That claim code is unknown or has expired"
|
||||
await run_in_threadpool(
|
||||
enroll_mod.enroll, session, current_user, body.portal_url, body.claim_code
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"The portal refused the claim ({response.status_code})",
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
owner_id = data.get("owner_id")
|
||||
if not owner_id:
|
||||
# A portal older than remote users does not say who owns the
|
||||
# installation, and without that the enrolling account cannot be mapped
|
||||
# to anyone — which would leave the portal connected but refused here.
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail="That portal is too old for this installation: it did not "
|
||||
"say which account owns the installation",
|
||||
)
|
||||
config = cloud_config.CloudConfig(
|
||||
portal_url=base,
|
||||
ws_url=data["ws_url"],
|
||||
installation_id=data["installation_id"],
|
||||
token=data["installation_token"],
|
||||
issuer=data["issuer"],
|
||||
# Pinned here, at the one moment the claim code proves who we are
|
||||
# talking to. Nothing refreshes this.
|
||||
jwks=data["jwks"],
|
||||
local_user_id=str(current_user.id),
|
||||
enrolled_at=datetime.now(timezone.utc).isoformat(),
|
||||
portal_account=current_user.email,
|
||||
)
|
||||
cloud_config.save(config)
|
||||
|
||||
owner_id = str(owner_id)
|
||||
# Re-enrolling from a different local account moves the mapping rather than
|
||||
# leaving two accounts claiming the same portal identity, which the unique
|
||||
# index would refuse and the lookup could not choose between anyway.
|
||||
for other in session.exec(
|
||||
select(User).where(User.portal_sub == owner_id, User.id != current_user.id)
|
||||
):
|
||||
other.portal_sub = None
|
||||
session.add(other)
|
||||
current_user.portal_sub = owner_id
|
||||
session.add(current_user)
|
||||
session.commit()
|
||||
except enroll_mod.EnrollError as exc:
|
||||
raise HTTPException(status_code=exc.status, detail=exc.detail) from exc
|
||||
|
||||
_start_connector(request.app)
|
||||
return Message(message="Connected to the portal")
|
||||
|
||||
@@ -0,0 +1,290 @@
|
||||
"""`fluksio serve`, `fluksio enroll`, `fluksio worker`.
|
||||
|
||||
The point of this module is a machine nobody can route to: a node on a cluster
|
||||
where ports cannot be opened, or a laptop with no Docker. `fluksio serve`
|
||||
starts the engine with no infrastructure and no configuration; `fluksio enroll`
|
||||
hands it a claim code, and it dials the portal itself. What a browser would
|
||||
have done locally is then done through the portal, which serves the dashboard
|
||||
from its own side.
|
||||
|
||||
Nothing from the engine is imported at module level. `fluksio.core.config`
|
||||
builds its settings when it is first imported, and the database engine and the
|
||||
user venv's path are computed from those — so the environment has to be right
|
||||
before any of that happens, which is what `_configure_environment` is for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import secrets
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import fluksio
|
||||
|
||||
#: Where an installation keeps everything, unless it is told otherwise.
|
||||
DEFAULT_HOME = Path("~/.fluksio")
|
||||
|
||||
#: WAL — what lets readers work while the engine writes — is not supported on
|
||||
#: these. The database would be corrupt or locked, so it is worth saying.
|
||||
NETWORK_FILESYSTEMS = ("nfs", "nfs4", "cifs", "smb", "smb3", "lustre", "fuse.sshfs")
|
||||
|
||||
|
||||
def _say(message: str = "") -> None:
|
||||
"""Print, and mean it.
|
||||
|
||||
Redirected to a file or a journal, stdout is block-buffered, and the admin
|
||||
password below is shown exactly once — sitting in a buffer until the
|
||||
process exits is the same as never printing it.
|
||||
"""
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
def _data_dir(raw: str | None) -> Path:
|
||||
path = Path(raw).expanduser() if raw else DEFAULT_HOME.expanduser()
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path.resolve()
|
||||
|
||||
|
||||
def _warn_if_networked(path: Path) -> None:
|
||||
"""A cluster's $HOME is often NFS, and SQLite's WAL does not work there."""
|
||||
try:
|
||||
mounts = Path("/proc/mounts").read_text().splitlines()
|
||||
except OSError: # pragma: no cover - not Linux
|
||||
return
|
||||
best, kind = "", ""
|
||||
for line in mounts:
|
||||
parts = line.split()
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
point, fstype = parts[1], parts[2]
|
||||
if (path == Path(point) or point in map(str, path.parents)) and len(
|
||||
point
|
||||
) > len(best):
|
||||
best, kind = point, fstype
|
||||
if kind in NETWORK_FILESYSTEMS:
|
||||
print(
|
||||
f"warning: {path} is on {kind}, where SQLite's write-ahead log does "
|
||||
"not work. Point --data-dir at local disk.",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
def load_or_create_secret_key(path: Path) -> str:
|
||||
"""The key that signs sessions and encrypts the secrets store, kept once.
|
||||
|
||||
Regenerating it per process would sign out every session on restart and,
|
||||
worse, leave `secrets.enc` unreadable.
|
||||
"""
|
||||
if path.exists():
|
||||
return path.read_text().strip()
|
||||
key = secrets.token_urlsafe(32)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(key)
|
||||
path.chmod(0o600)
|
||||
return key
|
||||
|
||||
|
||||
def _configure_environment(data_dir: Path) -> None:
|
||||
"""Everything the settings need, before anything reads them.
|
||||
|
||||
Imports nothing from the engine, deliberately: the first import of
|
||||
`fluksio.core.config` builds the settings, and any engine module reaches it
|
||||
within an import or two. Doing that here would fix `DATA_DIR` at whatever
|
||||
the current directory happened to be.
|
||||
|
||||
`setdefault` throughout: an operator who exported one of these means it.
|
||||
"""
|
||||
os.environ.setdefault("DATA_DIR", str(data_dir))
|
||||
# Without this the settings would read whatever `../.env` resolves to from
|
||||
# the current directory — a checkout's development configuration, if the
|
||||
# command happened to be run from inside one.
|
||||
os.environ.setdefault("FLUKSIO_ENV_FILE", str(data_dir / "env"))
|
||||
os.environ.setdefault(
|
||||
"SECRET_KEY", load_or_create_secret_key(data_dir / "secret_key")
|
||||
)
|
||||
|
||||
|
||||
def _prepare(data_dir: Path) -> None:
|
||||
_warn_if_networked(data_dir)
|
||||
_configure_environment(data_dir)
|
||||
from fluksio.core.db import engine, prepare
|
||||
|
||||
prepare(engine)
|
||||
|
||||
|
||||
def _enroll(portal: str, code: str, as_email: str | None) -> int:
|
||||
from sqlmodel import Session
|
||||
|
||||
from fluksio.cloud import enroll as enroll_mod
|
||||
from fluksio.core.bootstrap import ensure_superuser, pick_superuser
|
||||
from fluksio.core.config import settings
|
||||
from fluksio.core.db import engine
|
||||
|
||||
with Session(engine) as session:
|
||||
_, generated = ensure_superuser(session, email=as_email)
|
||||
if generated:
|
||||
_print_new_admin(session, generated)
|
||||
try:
|
||||
user = pick_superuser(session, as_email)
|
||||
except LookupError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr, flush=True)
|
||||
return 1
|
||||
try:
|
||||
config = enroll_mod.enroll(session, user, portal, code)
|
||||
except enroll_mod.AlreadyEnrolled as exc:
|
||||
print(
|
||||
f"error: {exc.detail} (see {settings.CLOUD_CONFIG_FILE}).",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
return 1
|
||||
except enroll_mod.EnrollError as exc:
|
||||
print(f"error: {exc.detail}", file=sys.stderr, flush=True)
|
||||
return 1
|
||||
|
||||
_say(
|
||||
f"Connected to {config.portal_url} as {user.email} "
|
||||
f"(installation {config.installation_id})."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _print_new_admin(session: object, password: str) -> None:
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from fluksio.models import User
|
||||
|
||||
assert isinstance(session, Session)
|
||||
user = session.exec(
|
||||
select(User).where(User.is_superuser == True) # noqa: E712
|
||||
).first()
|
||||
_say(f"Created the admin account {user.email if user else ''}")
|
||||
_say(f" password: {password}")
|
||||
_say(" Shown once. Change it from the dashboard.")
|
||||
|
||||
|
||||
def cmd_serve(args: argparse.Namespace) -> int:
|
||||
data_dir = _data_dir(args.data_dir)
|
||||
_prepare(data_dir)
|
||||
|
||||
from sqlmodel import Session
|
||||
|
||||
from fluksio.cloud import config as cloud_config
|
||||
from fluksio.core.bootstrap import ensure_superuser
|
||||
from fluksio.core.db import engine
|
||||
|
||||
with Session(engine) as session:
|
||||
_, generated = ensure_superuser(
|
||||
session, email=args.admin_email, password=args.admin_password
|
||||
)
|
||||
if generated:
|
||||
_print_new_admin(session, generated)
|
||||
|
||||
if args.enroll:
|
||||
if not args.portal:
|
||||
print("error: --enroll needs --portal", file=sys.stderr, flush=True)
|
||||
return 1
|
||||
if not cloud_config.exists():
|
||||
# Before the engine starts, so the connector finds the config and
|
||||
# dials out as part of coming up rather than needing a restart.
|
||||
failed = _enroll(args.portal, args.enroll, args.admin_email)
|
||||
if failed:
|
||||
return failed
|
||||
|
||||
import uvicorn
|
||||
|
||||
from fluksio.main import app
|
||||
|
||||
config = cloud_config.load()
|
||||
_say(f"Fluksio {fluksio.__version__} — data in {data_dir}")
|
||||
_say(f" API http://{args.host}:{args.port}{'/api/v1'}")
|
||||
if config is not None:
|
||||
_say(f" Portal {config.portal_url}, installation {config.installation_id}")
|
||||
_say(" The dashboard is served by the portal; nothing is served here.")
|
||||
else:
|
||||
_say(" No portal. Pair this installation with:")
|
||||
_say(" fluksio enroll <code> --portal https://hub.example.com")
|
||||
|
||||
# One process: it holds the flow engine, and a second worker would be a
|
||||
# second engine — duplicated subscriptions, cron ticks and webhooks.
|
||||
uvicorn.run(app, host=args.host, port=args.port, log_level=args.log_level)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_enroll(args: argparse.Namespace) -> int:
|
||||
data_dir = _data_dir(args.data_dir)
|
||||
_prepare(data_dir)
|
||||
result = _enroll(args.portal, args.code, args.as_email)
|
||||
if result == 0:
|
||||
_say("Start it with `fluksio serve`; it dials the portal as it comes up.")
|
||||
return result
|
||||
|
||||
|
||||
def cmd_worker(rest: list[str]) -> int:
|
||||
from fluksio_worker.agent import main as worker_main
|
||||
|
||||
return worker_main(rest)
|
||||
|
||||
|
||||
def _parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="fluksio", description="Node-based automation: flows, dashboards, runs."
|
||||
)
|
||||
parser.add_argument("--version", action="version", version=fluksio.__version__)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
def with_data_dir(sub: argparse.ArgumentParser) -> None:
|
||||
sub.add_argument(
|
||||
"--data-dir",
|
||||
default=os.environ.get("FLUKSIO_HOME"),
|
||||
help=f"where this installation keeps everything (default {DEFAULT_HOME})",
|
||||
)
|
||||
|
||||
serve = subparsers.add_parser("serve", help="run the engine")
|
||||
with_data_dir(serve)
|
||||
serve.add_argument("--host", default="127.0.0.1")
|
||||
serve.add_argument("--port", type=int, default=8000)
|
||||
serve.add_argument("--log-level", default="info")
|
||||
serve.add_argument("--admin-email", default=None)
|
||||
serve.add_argument("--admin-password", default=None)
|
||||
serve.add_argument("--enroll", metavar="CODE", help="claim code, if not yet paired")
|
||||
serve.add_argument("--portal", metavar="URL", help="the portal --enroll redeems at")
|
||||
serve.set_defaults(func=cmd_serve)
|
||||
|
||||
enroll = subparsers.add_parser(
|
||||
"enroll", help="pair this installation with a portal"
|
||||
)
|
||||
enroll.add_argument("code", help="the claim code minted on the portal")
|
||||
enroll.add_argument("--portal", required=True, metavar="URL")
|
||||
enroll.add_argument(
|
||||
"--as",
|
||||
dest="as_email",
|
||||
default=None,
|
||||
help="the local account a portal session arrives as",
|
||||
)
|
||||
with_data_dir(enroll)
|
||||
enroll.set_defaults(func=cmd_enroll)
|
||||
|
||||
subparsers.add_parser(
|
||||
"worker",
|
||||
help="run nodes for an engine elsewhere (fluksio-worker)",
|
||||
add_help=False,
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
argv = list(sys.argv[1:] if argv is None else argv)
|
||||
# Everything after `worker` belongs to the agent's own parser.
|
||||
if argv and argv[0] == "worker":
|
||||
return cmd_worker(argv[1:])
|
||||
args = _parser().parse_args(argv)
|
||||
result: int = args.func(args)
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -27,8 +27,8 @@ from typing import Any
|
||||
import httpx
|
||||
from fastapi import FastAPI
|
||||
|
||||
import fluksio
|
||||
from fluksio.cloud import config as cloud_config
|
||||
from fluksio.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -443,4 +443,4 @@ def _query_value(query: str, key: str) -> str:
|
||||
|
||||
|
||||
def _version() -> str:
|
||||
return settings.VERSION if hasattr(settings, "VERSION") else "0.1.0"
|
||||
return fluksio.__version__
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Redeeming a claim code, from the dashboard or from the command line.
|
||||
|
||||
The work is the same either way — ask the portal, keep what it answers, and
|
||||
map the account that asked to the portal identity that owns the installation —
|
||||
so it lives here rather than in the route. The command line matters because a
|
||||
machine on a cluster has no browser pointed at it: `fluksio enroll` does this
|
||||
before the engine has started, holding nothing but the database.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlmodel import Session, select
|
||||
|
||||
import fluksio
|
||||
from fluksio.cloud import config as cloud_config
|
||||
from fluksio.models import User
|
||||
|
||||
|
||||
class EnrollError(Exception):
|
||||
"""A failure with the status the API should answer with."""
|
||||
|
||||
def __init__(self, status: int, detail: str) -> None:
|
||||
super().__init__(detail)
|
||||
self.status = status
|
||||
self.detail = detail
|
||||
|
||||
|
||||
class AlreadyEnrolled(EnrollError):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(409, "This installation is already connected to a portal")
|
||||
|
||||
|
||||
def redeem_claim(
|
||||
portal_url: str, claim_code: str, *, timeout: float = 15.0
|
||||
) -> dict[str, Any]:
|
||||
"""Trade a claim code for this installation's credential and the portal's keys."""
|
||||
base = portal_url.rstrip("/")
|
||||
try:
|
||||
response = httpx.post(
|
||||
f"{base}/api/v1/enroll/",
|
||||
json={"claim_code": claim_code, "app_version": fluksio.__version__},
|
||||
timeout=timeout,
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise EnrollError(502, f"Could not reach the portal: {exc}") from exc
|
||||
if response.status_code == 404:
|
||||
raise EnrollError(400, "That claim code is unknown or has expired")
|
||||
if response.status_code != 200:
|
||||
raise EnrollError(502, f"The portal refused the claim ({response.status_code})")
|
||||
|
||||
data: dict[str, Any] = response.json()
|
||||
if not data.get("owner_id"):
|
||||
# A portal older than remote users does not say who owns the
|
||||
# installation, and without that the enrolling account cannot be mapped
|
||||
# to anyone — which would leave the portal connected but refused here.
|
||||
raise EnrollError(
|
||||
502,
|
||||
"That portal is too old for this installation: it did not say "
|
||||
"which account owns the installation",
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
def enroll(
|
||||
session: Session, user: User, portal_url: str, claim_code: str
|
||||
) -> cloud_config.CloudConfig:
|
||||
"""Redeem the code and write the config the connector dials with."""
|
||||
if cloud_config.exists():
|
||||
raise AlreadyEnrolled()
|
||||
|
||||
data = redeem_claim(portal_url, claim_code)
|
||||
config = cloud_config.CloudConfig(
|
||||
portal_url=portal_url.rstrip("/"),
|
||||
ws_url=data["ws_url"],
|
||||
installation_id=data["installation_id"],
|
||||
token=data["installation_token"],
|
||||
issuer=data["issuer"],
|
||||
# Pinned here, at the one moment the claim code proves who we are
|
||||
# talking to. Nothing refreshes this.
|
||||
jwks=data["jwks"],
|
||||
local_user_id=str(user.id),
|
||||
enrolled_at=datetime.now(timezone.utc).isoformat(),
|
||||
portal_account=user.email,
|
||||
)
|
||||
cloud_config.save(config)
|
||||
|
||||
owner_id = str(data["owner_id"])
|
||||
# Re-enrolling from a different local account moves the mapping rather than
|
||||
# leaving two accounts claiming the same portal identity, which the unique
|
||||
# index would refuse and the lookup could not choose between anyway.
|
||||
for other in session.exec(
|
||||
select(User).where(User.portal_sub == owner_id, User.id != user.id)
|
||||
):
|
||||
other.portal_sub = None
|
||||
session.add(other)
|
||||
user.portal_sub = owner_id
|
||||
session.add(user)
|
||||
session.commit()
|
||||
return config
|
||||
@@ -0,0 +1,64 @@
|
||||
"""First run: an account to sign in with.
|
||||
|
||||
An installation started from the command line is given no environment, so the
|
||||
superuser a deployment sets in `.env` has to come from somewhere. The session
|
||||
key is the CLI's own business — it has to be set before this module can be
|
||||
imported at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from fluksio import crud
|
||||
from fluksio.models import User, UserCreate
|
||||
|
||||
#: A generated admin's address. `example.com` is reserved for exactly this
|
||||
#: (RFC 2606) — `@localhost` and `.local` are too, and `EmailStr` refuses those.
|
||||
#: It is a name to sign in with, not somewhere mail is sent.
|
||||
DEFAULT_ADMIN = "admin@example.com"
|
||||
|
||||
|
||||
def ensure_superuser(
|
||||
session: Session, *, email: str | None = None, password: str | None = None
|
||||
) -> tuple[User, str | None]:
|
||||
"""The account to sign in with, made on first run.
|
||||
|
||||
Returns the user and, when it was just created, the password in clear —
|
||||
the caller prints it once. An installation that already has a superuser is
|
||||
left alone: this is a first run, not a password reset.
|
||||
"""
|
||||
existing = session.exec(
|
||||
select(User).where(User.is_superuser == True) # noqa: E712
|
||||
).first()
|
||||
if existing is not None:
|
||||
return existing, None
|
||||
|
||||
generated = password or secrets.token_urlsafe(12)
|
||||
user = crud.create_user(
|
||||
session=session,
|
||||
user_create=UserCreate(
|
||||
email=email or DEFAULT_ADMIN, password=generated, is_superuser=True
|
||||
),
|
||||
)
|
||||
return user, generated
|
||||
|
||||
|
||||
def pick_superuser(session: Session, email: str | None = None) -> User:
|
||||
"""The account an enrolment acts as; a portal session arrives as this one."""
|
||||
if email:
|
||||
user = session.exec(select(User).where(User.email == email)).first()
|
||||
if user is None:
|
||||
raise LookupError(f"No account here with the address {email}")
|
||||
return user
|
||||
users = session.exec(
|
||||
select(User).where(User.is_superuser == True) # noqa: E712
|
||||
).all()
|
||||
if not users:
|
||||
raise LookupError("This installation has no superuser to enrol as")
|
||||
if len(users) > 1:
|
||||
addresses = ", ".join(sorted(u.email for u in users))
|
||||
raise LookupError(f"Several superusers here — name one with --as: {addresses}")
|
||||
return users[0]
|
||||
@@ -16,6 +16,7 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import logging
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -37,11 +38,25 @@ VENV_DIR = settings.FLOWS_DIR.parent / "user-venv"
|
||||
SYNC_TIMEOUT = 300
|
||||
|
||||
|
||||
def uv_bin() -> str:
|
||||
"""Where ``uv`` is, without depending on what PATH happens to hold.
|
||||
|
||||
It is a dependency, so it is installed beside the interpreter running this
|
||||
— which is what a systemd unit naming an absolute ExecStart, or a container
|
||||
entrypoint, would otherwise miss.
|
||||
"""
|
||||
beside = Path(sys.executable).with_name("uv")
|
||||
if beside.exists():
|
||||
return str(beside)
|
||||
return shutil.which("uv") or "uv"
|
||||
|
||||
|
||||
def venv_python() -> str:
|
||||
"""The interpreter node code runs on.
|
||||
|
||||
Falls back to the engine's own when there is no venv — a deployment without
|
||||
``uv`` still runs python nodes, it just cannot add packages to them.
|
||||
Falls back to the engine's own when there is no venv — an installation that
|
||||
could not build one still runs python nodes, it just cannot add packages
|
||||
to them.
|
||||
"""
|
||||
path = VENV_DIR / "bin" / "python"
|
||||
return str(path) if path.exists() else sys.executable
|
||||
@@ -65,7 +80,7 @@ def ensure_venv() -> None:
|
||||
# inside another is how a user pin ends up resolving against app packages.
|
||||
subprocess.run(
|
||||
[
|
||||
"uv",
|
||||
uv_bin(),
|
||||
"venv",
|
||||
"--python",
|
||||
str(Path(sys.base_prefix, "bin", "python3")),
|
||||
@@ -95,7 +110,7 @@ def sync(requirements: str) -> tuple[bool, str]:
|
||||
# Empty means empty: without the flag uv refuses to clear a venv,
|
||||
# so deleting the last line would leave the package installed.
|
||||
[
|
||||
"uv",
|
||||
uv_bin(),
|
||||
"pip",
|
||||
"sync",
|
||||
"--allow-empty-requirements",
|
||||
|
||||
@@ -44,7 +44,8 @@ def custom_generate_unique_id(route: APIRoute) -> str:
|
||||
|
||||
|
||||
if settings.SENTRY_DSN and settings.ENVIRONMENT != "local":
|
||||
sentry_sdk.init(dsn=str(settings.SENTRY_DSN), enable_tracing=True)
|
||||
# `enable_tracing` was removed in sentry-sdk 2.x; this is what it meant.
|
||||
sentry_sdk.init(dsn=str(settings.SENTRY_DSN), traces_sample_rate=1.0)
|
||||
|
||||
|
||||
def _state_backend() -> StateBackend:
|
||||
|
||||
+14
-4
@@ -2,9 +2,7 @@
|
||||
name = "fluksio"
|
||||
version = "0.1.0"
|
||||
description = "Node-based automation engine: flows, dashboards, batch runs"
|
||||
# Capped below 3.14: the MCP SDK wants a newer starlette there than the
|
||||
# pinned sentry-sdk allows. Lift it when sentry-sdk moves to 2.x.
|
||||
requires-python = ">=3.10,<3.14"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"fastapi[standard]<1.0.0,>=0.114.2",
|
||||
"python-multipart<1.0.0,>=0.0.7",
|
||||
@@ -17,7 +15,7 @@ dependencies = [
|
||||
"httpx<1.0.0,>=0.25.1",
|
||||
"sqlmodel<1.0.0,>=0.0.21",
|
||||
"pydantic-settings<3.0.0,>=2.2.1",
|
||||
"sentry-sdk[fastapi]<2.0.0,>=1.40.6",
|
||||
"sentry-sdk[fastapi]>=2.20.0",
|
||||
"pyjwt<3.0.0,>=2.8.0",
|
||||
"pwdlib[argon2,bcrypt]>=0.3.0",
|
||||
"numpy>=2.2.6",
|
||||
@@ -28,8 +26,15 @@ dependencies = [
|
||||
"croniter>=1.3.0",
|
||||
"mcp>=1.29,<2",
|
||||
"fluksio-worker>=0.1,<0.2",
|
||||
# The Modules screen installs node code's packages with it, into a venv of
|
||||
# the user's own. Present in the image; a pip install would otherwise have
|
||||
# to find one on PATH, and quietly fall back to the engine's interpreter.
|
||||
"uv>=0.5",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
fluksio = "fluksio.cli:main"
|
||||
|
||||
[tool.uv.sources]
|
||||
fluksio-worker = { workspace = true }
|
||||
|
||||
@@ -93,6 +98,11 @@ ignore = [
|
||||
]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
# The node API's stubs carry the real signatures and raise; unused arguments
|
||||
# are what a stub is.
|
||||
"fluksio/__init__.py" = ["ARG001"]
|
||||
# It talks to whoever ran it; that is what a command line is.
|
||||
"fluksio/cli.py" = ["T201"]
|
||||
# Node functions take `params` whether or not they use it — that is the
|
||||
# contract the engine calls them with.
|
||||
"fluksio/flow/nodes.py" = ["ARG001", "ARG002"]
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""The command line, and the one ordering it depends on."""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from fluksio.cli import load_or_create_secret_key
|
||||
|
||||
|
||||
def test_importing_the_cli_does_not_build_the_settings() -> None:
|
||||
"""`_configure_environment` has to run before anything reads a setting.
|
||||
|
||||
The settings are built on the first import of `fluksio.core.config`, and
|
||||
every engine module reaches it within an import or two. If importing the
|
||||
CLI pulled it in, `DATA_DIR` would be fixed at whatever directory the
|
||||
command was run from — which is how the database ends up in the cwd.
|
||||
"""
|
||||
leaked = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"import fluksio.cli, sys; print('fluksio.core.config' in sys.modules)",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
assert leaked.stdout.strip() == "False", leaked.stdout
|
||||
|
||||
|
||||
def test_the_secret_key_is_kept_rather_than_regenerated(tmp_path: Path) -> None:
|
||||
"""A new key each start would sign out every session and orphan secrets.enc."""
|
||||
path = tmp_path / "secret_key"
|
||||
first = load_or_create_secret_key(path)
|
||||
assert load_or_create_secret_key(path) == first
|
||||
assert path.stat().st_mode & 0o777 == 0o600
|
||||
@@ -10,7 +10,7 @@ nothing, which is what makes deleting that local account a revocation.
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from unittest.mock import AsyncMock, Mock, patch
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import jwt
|
||||
import pytest
|
||||
@@ -206,9 +206,7 @@ def test_adding_a_remote_user_maps_and_revokes(
|
||||
return_value={"user_id": "portal-user-9", "email": "remote@example.com"}
|
||||
),
|
||||
)
|
||||
with patch(
|
||||
"fluksio.api.routes.cloud.httpx.post", return_value=portal_reply
|
||||
) as post:
|
||||
with patch("fluksio.cloud.enroll.httpx.post", return_value=portal_reply) as post:
|
||||
added = client.post(
|
||||
f"{settings.API_V1_STR}/cloud/users",
|
||||
headers=superuser_token_headers,
|
||||
@@ -226,7 +224,7 @@ def test_adding_a_remote_user_maps_and_revokes(
|
||||
resolved = user_from_token(db, theirs)
|
||||
assert resolved is not None and resolved.email == "remote@example.com"
|
||||
|
||||
with patch("fluksio.api.routes.cloud.httpx.post", return_value=portal_reply):
|
||||
with patch("fluksio.cloud.enroll.httpx.post", return_value=portal_reply):
|
||||
again = client.post(
|
||||
f"{settings.API_V1_STR}/cloud/users",
|
||||
headers=superuser_token_headers,
|
||||
@@ -281,10 +279,7 @@ def test_enrolling_against_a_portal_without_an_owner_is_refused(
|
||||
),
|
||||
)
|
||||
try:
|
||||
with patch("fluksio.api.routes.cloud.httpx.AsyncClient") as client_cls:
|
||||
client_cls.return_value.__aenter__.return_value.post = AsyncMock(
|
||||
return_value=reply
|
||||
)
|
||||
with patch("fluksio.cloud.enroll.httpx.post", return_value=reply):
|
||||
response = client.post(
|
||||
f"{settings.API_V1_STR}/cloud/enroll",
|
||||
headers=superuser_token_headers,
|
||||
|
||||
Reference in New Issue
Block a user