One installation per project, and no login to reach it
Two things a local install should not have asked for. `fluksio serve` now signs you in. Logging in to your own machine was a formality — the password was printed by the same process that would have checked it, and the database it authenticates against sits in the directory the token goes into — so `serve` mints the token itself and says where it put it. `fluksio login` is left for an engine somewhere else. And an installation is `.fluksio` beside the code, found the way `.git` is, rather than one `~/.fluksio` for the machine. A repository with its own venv was already getting its own engine; it now gets its own flows, run history and token too, instead of three repositories sharing one database and fighting over one port. `--global` asks for the shared one, `--data-dir` still names any directory, and when both exist the banner says which you are looking at and how to reach the other. The directory ignores itself from within — a `.gitignore` of `*`, the way uv writes one into `.venv` — because it holds a credential and a database, and neither belongs in anybody's history. The token is written mode 600. A login an older version wrote to ~/.config/fluksio is still read, so nothing that worked stops working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
This commit is contained in:
+11
-3
@@ -9,8 +9,17 @@ pip install fluksio
|
|||||||
fluksio serve
|
fluksio serve
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Then login once (the token is stored on your device) with the credentials shown after the previous command
|
||||||
|
|
||||||
|
```sh
|
||||||
|
fluksio login --url http://127.0.0.1:8000
|
||||||
|
```
|
||||||
|
|
||||||
and you're ready to go!
|
and you're ready to go!
|
||||||
Fluksio keeps a SQLite database, a git repository of your flows and an artifact store under `~/.fluksio`, and prints an admin password once upon start.
|
|
||||||
|
Fluksio keeps a SQLite database, a git repository of your flows and an artifact
|
||||||
|
store in a `.fluksio` beside your code — one installation per project, found
|
||||||
|
the way `.git` is. It prints an admin password once, and signs you in itself.
|
||||||
|
|
||||||
## For data science
|
## For data science
|
||||||
|
|
||||||
@@ -45,10 +54,9 @@ train = Flow("train", nodes=[prepare, fit, evaluate],
|
|||||||
```
|
```
|
||||||
|
|
||||||
Fluksio will automatically infer the order of nodes based on the inputs and outputs you defined.
|
Fluksio will automatically infer the order of nodes based on the inputs and outputs you defined.
|
||||||
When everything is set, you can access a dashboard as follows:
|
When everything is set, you can launch your first run as follows:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
fluksio login --url http://127.0.0.1:8000
|
|
||||||
fluksio sync myresearch
|
fluksio sync myresearch
|
||||||
fluksio run train --lr 0.05 --wait
|
fluksio run train --lr 0.05 --wait
|
||||||
```
|
```
|
||||||
|
|||||||
+76
-7
@@ -20,6 +20,7 @@ import os
|
|||||||
import secrets
|
import secrets
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
import fluksio
|
import fluksio
|
||||||
|
|
||||||
@@ -41,12 +42,37 @@ def _say(message: str = "") -> None:
|
|||||||
print(message, flush=True)
|
print(message, flush=True)
|
||||||
|
|
||||||
|
|
||||||
def _data_dir(raw: str | None) -> Path:
|
def _data_dir(raw: str | None, shared: bool = False) -> Path:
|
||||||
path = Path(raw).expanduser() if raw else DEFAULT_HOME.expanduser()
|
"""Which installation this command is for.
|
||||||
|
|
||||||
|
A repository with its own venv wants its own engine too — its own flows,
|
||||||
|
its own run history, its own token — so the default is a `.fluksio` beside
|
||||||
|
the code, found the way `.git` is. `--global` asks for the shared one
|
||||||
|
instead, and `--data-dir` names any directory outright.
|
||||||
|
"""
|
||||||
|
from fluksio.sdk.client import find_data_dir, ignore_self
|
||||||
|
|
||||||
|
if raw:
|
||||||
|
path = Path(raw).expanduser()
|
||||||
|
elif shared:
|
||||||
|
path = DEFAULT_HOME.expanduser()
|
||||||
|
else:
|
||||||
|
found = find_data_dir()
|
||||||
|
path = found if found is not None else Path.cwd() / ".fluksio"
|
||||||
path.mkdir(parents=True, exist_ok=True)
|
path.mkdir(parents=True, exist_ok=True)
|
||||||
|
ignore_self(path)
|
||||||
return path.resolve()
|
return path.resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def _mention_other_installation(data_dir: Path) -> None:
|
||||||
|
"""Say when there is a second engine, so neither goes looking lost."""
|
||||||
|
shared = DEFAULT_HOME.expanduser().resolve()
|
||||||
|
if data_dir == shared or not (shared / "fluksio.db").exists():
|
||||||
|
return
|
||||||
|
_say(f" Note {shared} holds another installation; this one is separate.")
|
||||||
|
_say(" `fluksio serve --global` runs that one instead.")
|
||||||
|
|
||||||
|
|
||||||
def _warn_if_networked(path: Path) -> None:
|
def _warn_if_networked(path: Path) -> None:
|
||||||
"""A cluster's $HOME is often NFS, and SQLite's WAL does not work there."""
|
"""A cluster's $HOME is often NFS, and SQLite's WAL does not work there."""
|
||||||
try:
|
try:
|
||||||
@@ -161,8 +187,35 @@ def _print_new_admin(email: str, password: str) -> None:
|
|||||||
_say(" Shown once. Change it from the dashboard.")
|
_say(" Shown once. Change it from the dashboard.")
|
||||||
|
|
||||||
|
|
||||||
|
#: How long the token `serve` writes for its own machine is good for. Long,
|
||||||
|
#: because the thing it saves you from is logging in again, and it is rewritten
|
||||||
|
#: on every start anyway — an engine left running for a year is the only case
|
||||||
|
#: this has to cover.
|
||||||
|
LOCAL_TOKEN_DAYS = 365
|
||||||
|
|
||||||
|
|
||||||
|
def _sign_in(admin_id: Any, url: str, data_dir: Path) -> Path:
|
||||||
|
"""Write the credential for the engine this command is about to start.
|
||||||
|
|
||||||
|
Logging in to your own machine is a formality: the password was printed by
|
||||||
|
this same process, and the database it authenticates against is in the
|
||||||
|
directory the token goes into. So it costs nothing to be honest about it
|
||||||
|
and hand the client a token, rather than asking somebody to type back
|
||||||
|
something we just told them.
|
||||||
|
"""
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from fluksio.core import security
|
||||||
|
from fluksio.sdk.client import write_config
|
||||||
|
|
||||||
|
token = security.create_access_token(
|
||||||
|
admin_id, expires_delta=timedelta(days=LOCAL_TOKEN_DAYS)
|
||||||
|
)
|
||||||
|
return write_config(url, token, data_dir)
|
||||||
|
|
||||||
|
|
||||||
def cmd_serve(args: argparse.Namespace) -> int:
|
def cmd_serve(args: argparse.Namespace) -> int:
|
||||||
data_dir = _data_dir(args.data_dir)
|
data_dir = _data_dir(args.data_dir, args.shared)
|
||||||
_prepare(data_dir)
|
_prepare(data_dir)
|
||||||
|
|
||||||
from sqlmodel import Session
|
from sqlmodel import Session
|
||||||
@@ -175,8 +228,9 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
|||||||
admin, generated = ensure_superuser(
|
admin, generated = ensure_superuser(
|
||||||
session, email=args.admin_email, password=args.admin_password
|
session, email=args.admin_email, password=args.admin_password
|
||||||
)
|
)
|
||||||
|
admin_id, admin_email = admin.id, admin.email
|
||||||
if generated:
|
if generated:
|
||||||
_print_new_admin(admin.email, generated)
|
_print_new_admin(admin_email, generated)
|
||||||
|
|
||||||
if args.enroll:
|
if args.enroll:
|
||||||
if not args.portal:
|
if not args.portal:
|
||||||
@@ -194,9 +248,15 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
|||||||
from fluksio.flow import modules
|
from fluksio.flow import modules
|
||||||
from fluksio.main import app
|
from fluksio.main import app
|
||||||
|
|
||||||
|
# The client talks to this engine, and 0.0.0.0 is not an address to talk
|
||||||
|
# to — it is a statement about which interfaces to listen on.
|
||||||
|
reachable = "127.0.0.1" if args.host in ("0.0.0.0", "::", "") else args.host
|
||||||
|
url = f"http://{reachable}:{args.port}"
|
||||||
|
token_path = _sign_in(admin_id, url, data_dir)
|
||||||
|
|
||||||
config = cloud_config.load()
|
config = cloud_config.load()
|
||||||
_say(f"Fluksio {fluksio.__version__} — data in {data_dir}")
|
_say(f"Fluksio {fluksio.__version__} — data in {data_dir}")
|
||||||
_say(f" API http://{args.host}:{args.port}{'/api/v1'}")
|
_say(f" API {url}/api/v1")
|
||||||
# Which environment node code imports from is the thing a data scientist
|
# Which environment node code imports from is the thing a data scientist
|
||||||
# most needs to know at this moment, and the answer differs depending on
|
# most needs to know at this moment, and the answer differs depending on
|
||||||
# how Fluksio was installed. Saying it costs one line.
|
# how Fluksio was installed. Saying it costs one line.
|
||||||
@@ -215,6 +275,9 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
|||||||
else:
|
else:
|
||||||
_say(" No portal. Pair this installation with:")
|
_say(" No portal. Pair this installation with:")
|
||||||
_say(" fluksio enroll <code> --portal https://hub.example.com")
|
_say(" fluksio enroll <code> --portal https://hub.example.com")
|
||||||
|
_say(f" Signed in as {admin_email}")
|
||||||
|
_say(f" token in {token_path}")
|
||||||
|
_mention_other_installation(data_dir)
|
||||||
|
|
||||||
# One process: it holds the flow engine, and a second worker would be a
|
# One process: it holds the flow engine, and a second worker would be a
|
||||||
# second engine — duplicated subscriptions, cron ticks and webhooks.
|
# second engine — duplicated subscriptions, cron ticks and webhooks.
|
||||||
@@ -223,7 +286,7 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_enroll(args: argparse.Namespace) -> int:
|
def cmd_enroll(args: argparse.Namespace) -> int:
|
||||||
data_dir = _data_dir(args.data_dir)
|
data_dir = _data_dir(args.data_dir, args.shared)
|
||||||
_prepare(data_dir)
|
_prepare(data_dir)
|
||||||
result = _enroll(args.portal, args.code, args.as_email)
|
result = _enroll(args.portal, args.code, args.as_email)
|
||||||
if result == 0:
|
if result == 0:
|
||||||
@@ -248,7 +311,13 @@ def _parser() -> argparse.ArgumentParser:
|
|||||||
sub.add_argument(
|
sub.add_argument(
|
||||||
"--data-dir",
|
"--data-dir",
|
||||||
default=os.environ.get("FLUKSIO_HOME"),
|
default=os.environ.get("FLUKSIO_HOME"),
|
||||||
help=f"where this installation keeps everything (default {DEFAULT_HOME})",
|
help="where this installation keeps everything (default: ./.fluksio)",
|
||||||
|
)
|
||||||
|
sub.add_argument(
|
||||||
|
"--global",
|
||||||
|
dest="shared",
|
||||||
|
action="store_true",
|
||||||
|
help=f"use the shared installation at {DEFAULT_HOME} instead",
|
||||||
)
|
)
|
||||||
|
|
||||||
serve = subparsers.add_parser("serve", help="run the engine")
|
serve = subparsers.add_parser("serve", help="run the engine")
|
||||||
|
|||||||
@@ -17,9 +17,10 @@ from typing import Any
|
|||||||
|
|
||||||
from fluksio.sdk import FLOWS, Flow, SyncError
|
from fluksio.sdk import FLOWS, Flow, SyncError
|
||||||
from fluksio.sdk.client import (
|
from fluksio.sdk.client import (
|
||||||
|
GLOBAL_DATA_DIR,
|
||||||
ApiError,
|
ApiError,
|
||||||
Client,
|
Client,
|
||||||
config_path,
|
data_dir,
|
||||||
login,
|
login,
|
||||||
origin_of,
|
origin_of,
|
||||||
repo_root,
|
repo_root,
|
||||||
@@ -104,13 +105,15 @@ def discover(targets: list[str]) -> list[Flow]:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_login(args: argparse.Namespace) -> int:
|
def cmd_login(args: argparse.Namespace) -> int:
|
||||||
|
"""Only needed for an engine somewhere else — `serve` signs you in here."""
|
||||||
email = args.email or input("Email: ")
|
email = args.email or input("Email: ")
|
||||||
password = args.password or getpass.getpass("Password: ")
|
password = args.password or getpass.getpass("Password: ")
|
||||||
|
directory = GLOBAL_DATA_DIR.expanduser() if args.shared else data_dir()
|
||||||
try:
|
try:
|
||||||
login(args.url, email, password)
|
path = login(args.url, email, password, directory=directory)
|
||||||
except ApiError as exc:
|
except ApiError as exc:
|
||||||
return _fail(f"could not log in: {exc.detail}")
|
return _fail(f"could not log in: {exc.detail}")
|
||||||
_say(f"Logged in to {args.url}; the token is in {config_path()}.")
|
_say(f"Logged in to {args.url}; the token is in {path}.")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
@@ -263,10 +266,18 @@ def add_parsers(subparsers: Any) -> None:
|
|||||||
)
|
)
|
||||||
sub.add_argument("--token", default="", help="override the stored token")
|
sub.add_argument("--token", default="", help="override the stored token")
|
||||||
|
|
||||||
parser = subparsers.add_parser("login", help="store a token for an engine")
|
parser = subparsers.add_parser(
|
||||||
|
"login", help="store a token for an engine elsewhere"
|
||||||
|
)
|
||||||
parser.add_argument("--url", default="http://localhost:8000")
|
parser.add_argument("--url", default="http://localhost:8000")
|
||||||
parser.add_argument("--email", default="")
|
parser.add_argument("--email", default="")
|
||||||
parser.add_argument("--password", default="")
|
parser.add_argument("--password", default="")
|
||||||
|
parser.add_argument(
|
||||||
|
"--global",
|
||||||
|
dest="shared",
|
||||||
|
action="store_true",
|
||||||
|
help="store it for the machine rather than this project",
|
||||||
|
)
|
||||||
parser.set_defaults(func=cmd_login)
|
parser.set_defaults(func=cmd_login)
|
||||||
|
|
||||||
parser = subparsers.add_parser(
|
parser = subparsers.add_parser(
|
||||||
|
|||||||
@@ -17,21 +17,72 @@ from typing import Any
|
|||||||
|
|
||||||
from fluksio.sdk import MARKER, Flow, SyncError
|
from fluksio.sdk import MARKER, Flow, SyncError
|
||||||
|
|
||||||
__all__ = ["Client", "RunHandle", "SyncReport", "config_path", "login", "sync"]
|
__all__ = [
|
||||||
|
"Client",
|
||||||
|
"RunHandle",
|
||||||
|
"SyncReport",
|
||||||
|
"config_path",
|
||||||
|
"data_dir",
|
||||||
|
"find_data_dir",
|
||||||
|
"ignore_self",
|
||||||
|
"login",
|
||||||
|
"sync",
|
||||||
|
"write_config",
|
||||||
|
]
|
||||||
|
|
||||||
API = "/api/v1"
|
API = "/api/v1"
|
||||||
#: A run is over when it reaches one of these.
|
#: A run is over when it reaches one of these.
|
||||||
DONE = frozenset({"ok", "error", "cancelled", "abandoned"})
|
DONE = frozenset({"ok", "error", "cancelled", "abandoned"})
|
||||||
|
|
||||||
|
|
||||||
def config_path() -> Path:
|
#: What a project-local installation is called, beside `.venv` and `.git`.
|
||||||
"""Where ``fluksio login`` leaves the engine it talked to."""
|
DATA_DIR_NAME = ".fluksio"
|
||||||
|
|
||||||
|
#: The shared one, for a machine that wants a single engine rather than one
|
||||||
|
#: per repository. `fluksio serve --global` is how you ask for it.
|
||||||
|
GLOBAL_DATA_DIR = Path("~/.fluksio")
|
||||||
|
|
||||||
|
|
||||||
|
def find_data_dir(start: Path | None = None) -> Path | None:
|
||||||
|
"""The nearest project-local installation, walking up from ``start``.
|
||||||
|
|
||||||
|
The same search `.git` and `.venv` get, and for the same reason: which
|
||||||
|
installation you mean is a fact about where you are standing, not about
|
||||||
|
which machine you are on. Several repositories on one device each keep
|
||||||
|
their own flows, runs and token this way rather than sharing one.
|
||||||
|
"""
|
||||||
|
directory = (start or Path.cwd()).resolve()
|
||||||
|
for candidate in (directory, *directory.parents):
|
||||||
|
local = candidate / DATA_DIR_NAME
|
||||||
|
if local.is_dir():
|
||||||
|
return local
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def data_dir(start: Path | None = None) -> Path:
|
||||||
|
"""The installation this working directory belongs to."""
|
||||||
|
found = find_data_dir(start)
|
||||||
|
return found if found is not None else GLOBAL_DATA_DIR.expanduser()
|
||||||
|
|
||||||
|
|
||||||
|
def config_path(directory: Path | None = None) -> Path:
|
||||||
|
"""Where the token for an installation lives — beside the data it opens.
|
||||||
|
|
||||||
|
Not a single file per machine: a token is for one engine, and with an
|
||||||
|
installation per repository there is more than one. Keeping it inside the
|
||||||
|
data directory means the client finds the credential for the engine whose
|
||||||
|
directory it is standing in, without either having to be told.
|
||||||
|
"""
|
||||||
|
return (directory or data_dir()) / "client.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _legacy_config_path() -> Path:
|
||||||
|
"""Where `fluksio login` used to write, before installations were local."""
|
||||||
base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
|
base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
|
||||||
return Path(base) / "fluksio" / "client.json"
|
return Path(base) / "fluksio" / "client.json"
|
||||||
|
|
||||||
|
|
||||||
def _stored() -> dict[str, str]:
|
def _read(path: Path) -> dict[str, str]:
|
||||||
path = config_path()
|
|
||||||
if not path.exists():
|
if not path.exists():
|
||||||
return {}
|
return {}
|
||||||
try:
|
try:
|
||||||
@@ -41,6 +92,15 @@ def _stored() -> dict[str, str]:
|
|||||||
return data if isinstance(data, dict) else {}
|
return data if isinstance(data, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _stored() -> dict[str, str]:
|
||||||
|
"""The nearest credential: this project's, then the machine's."""
|
||||||
|
for path in (config_path(), _legacy_config_path()):
|
||||||
|
found = _read(path)
|
||||||
|
if found:
|
||||||
|
return found
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
class ApiError(Exception):
|
class ApiError(Exception):
|
||||||
"""The engine refused, with what it said."""
|
"""The engine refused, with what it said."""
|
||||||
|
|
||||||
@@ -247,7 +307,37 @@ class RunHandle:
|
|||||||
return f"RunHandle({self.id!r}, status={self.status!r})"
|
return f"RunHandle({self.id!r}, status={self.status!r})"
|
||||||
|
|
||||||
|
|
||||||
def login(url: str, email: str, password: str, timeout: float = 30.0) -> str:
|
def ignore_self(directory: Path) -> None:
|
||||||
|
"""Keep an installation out of the repository it sits in.
|
||||||
|
|
||||||
|
It holds a token and a database, neither of which belongs in anybody's
|
||||||
|
history. A `.gitignore` of `*` inside the directory ignores it from within,
|
||||||
|
so nothing has to be added to the project's own — the same thing `uv` does
|
||||||
|
for the venv it builds.
|
||||||
|
"""
|
||||||
|
marker = directory / ".gitignore"
|
||||||
|
if not marker.exists():
|
||||||
|
directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
marker.write_text("# A Fluksio installation: a database, and a token.\n*\n")
|
||||||
|
|
||||||
|
|
||||||
|
def write_config(url: str, token: str, directory: Path | None = None) -> Path:
|
||||||
|
"""Store the credential for an engine, readable only by its owner."""
|
||||||
|
path = config_path(directory)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
ignore_self(path.parent)
|
||||||
|
path.write_text(json.dumps({"url": url.rstrip("/"), "token": token}, indent=2))
|
||||||
|
path.chmod(0o600)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def login(
|
||||||
|
url: str,
|
||||||
|
email: str,
|
||||||
|
password: str,
|
||||||
|
timeout: float = 30.0,
|
||||||
|
directory: Path | None = None,
|
||||||
|
) -> Path:
|
||||||
"""Exchange credentials for a token and remember the engine."""
|
"""Exchange credentials for a token and remember the engine."""
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
@@ -258,12 +348,7 @@ def login(url: str, email: str, password: str, timeout: float = 30.0) -> str:
|
|||||||
)
|
)
|
||||||
if response.status_code >= 400:
|
if response.status_code >= 400:
|
||||||
raise ApiError(response.status_code, _detail(response))
|
raise ApiError(response.status_code, _detail(response))
|
||||||
token = str(response.json()["access_token"])
|
return write_config(url, str(response.json()["access_token"]), directory)
|
||||||
path = config_path()
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_text(json.dumps({"url": url.rstrip("/"), "token": token}, indent=2))
|
|
||||||
path.chmod(0o600)
|
|
||||||
return token
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
"""Which installation a command is for, and where its token lives.
|
||||||
|
|
||||||
|
A repository with its own venv gets its own engine, so "which one" is a fact
|
||||||
|
about the working directory rather than about the machine.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from fluksio.sdk import client
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def elsewhere(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||||
|
"""A working directory with no installation above it."""
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg"))
|
||||||
|
monkeypatch.setattr(client, "GLOBAL_DATA_DIR", tmp_path / "home" / ".fluksio")
|
||||||
|
return tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_project_local_installation_is_found_from_below(elsewhere: Path):
|
||||||
|
(elsewhere / ".fluksio").mkdir()
|
||||||
|
deep = elsewhere / "src" / "pkg" / "sub"
|
||||||
|
deep.mkdir(parents=True)
|
||||||
|
|
||||||
|
assert client.find_data_dir(deep) == elsewhere / ".fluksio"
|
||||||
|
assert client.config_path(client.data_dir(deep)).parent.name == ".fluksio"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_nearest_one_wins(elsewhere: Path):
|
||||||
|
"""An installation inside another belongs to the directory it is in."""
|
||||||
|
(elsewhere / ".fluksio").mkdir()
|
||||||
|
inner = elsewhere / "inner"
|
||||||
|
(inner / ".fluksio").mkdir(parents=True)
|
||||||
|
|
||||||
|
assert client.find_data_dir(inner) == inner / ".fluksio"
|
||||||
|
|
||||||
|
|
||||||
|
def test_with_none_above_it_the_shared_one_answers(elsewhere: Path):
|
||||||
|
assert client.find_data_dir() is None
|
||||||
|
assert client.data_dir() == elsewhere / "home" / ".fluksio"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_token_is_written_only_for_its_owner(elsewhere: Path):
|
||||||
|
path = client.write_config("http://127.0.0.1:8000", "abc", elsewhere / ".fluksio")
|
||||||
|
|
||||||
|
assert json.loads(path.read_text()) == {
|
||||||
|
"url": "http://127.0.0.1:8000",
|
||||||
|
"token": "abc",
|
||||||
|
}
|
||||||
|
# It is a credential sitting in somebody's project directory.
|
||||||
|
assert path.stat().st_mode & 0o077 == 0
|
||||||
|
# And one nothing can commit by accident.
|
||||||
|
assert (path.parent / ".gitignore").read_text().endswith("*\n")
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_credential_beside_the_data_is_the_one_used(elsewhere: Path):
|
||||||
|
client.write_config("http://127.0.0.1:8131", "local", elsewhere / ".fluksio")
|
||||||
|
|
||||||
|
assert client._stored()["token"] == "local"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_login_from_before_installations_were_local_still_works(elsewhere: Path):
|
||||||
|
"""`fluksio login` wrote to XDG once; that must not stop answering."""
|
||||||
|
legacy = client._legacy_config_path()
|
||||||
|
legacy.parent.mkdir(parents=True)
|
||||||
|
legacy.write_text(json.dumps({"url": "http://elsewhere:8000", "token": "old"}))
|
||||||
|
|
||||||
|
assert client._stored()["token"] == "old"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_local_credential_wins_over_the_legacy_one(elsewhere: Path):
|
||||||
|
legacy = client._legacy_config_path()
|
||||||
|
legacy.parent.mkdir(parents=True)
|
||||||
|
legacy.write_text(json.dumps({"url": "http://elsewhere:8000", "token": "old"}))
|
||||||
|
client.write_config("http://127.0.0.1:8131", "local", elsewhere / ".fluksio")
|
||||||
|
|
||||||
|
assert client._stored()["token"] == "local"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignoring_itself_leaves_an_existing_gitignore_alone(elsewhere: Path):
|
||||||
|
directory = elsewhere / ".fluksio"
|
||||||
|
directory.mkdir()
|
||||||
|
(directory / ".gitignore").write_text("mine\n")
|
||||||
|
|
||||||
|
client.ignore_self(directory)
|
||||||
|
|
||||||
|
assert (directory / ".gitignore").read_text() == "mine\n"
|
||||||
@@ -86,3 +86,33 @@ def test_run_arguments_are_typed_by_the_flow_they_are_for() -> None:
|
|||||||
|
|
||||||
with pytest.raises(SyncError, match="not an input of this flow"):
|
with pytest.raises(SyncError, match="not an input of this flow"):
|
||||||
_params(definition, ["--nonesuch", "1"])
|
_params(definition, ["--nonesuch", "1"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_uses_the_installation_the_directory_belongs_to(
|
||||||
|
tmp_path: Path, monkeypatch
|
||||||
|
) -> None:
|
||||||
|
"""A repository with its own venv gets its own engine, not the machine's."""
|
||||||
|
from fluksio import cli
|
||||||
|
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
monkeypatch.setattr(cli, "DEFAULT_HOME", tmp_path / "home" / ".fluksio")
|
||||||
|
|
||||||
|
# Nothing above it yet: one is made here rather than in the home directory.
|
||||||
|
made = cli._data_dir(None)
|
||||||
|
assert made == (tmp_path / ".fluksio").resolve()
|
||||||
|
assert (made / ".gitignore").exists()
|
||||||
|
|
||||||
|
# And is then found again from anywhere below it.
|
||||||
|
deep = tmp_path / "src" / "deep"
|
||||||
|
deep.mkdir(parents=True)
|
||||||
|
monkeypatch.chdir(deep)
|
||||||
|
assert cli._data_dir(None) == made
|
||||||
|
|
||||||
|
# `--global` asks for the shared one even so.
|
||||||
|
assert (
|
||||||
|
cli._data_dir(None, shared=True) == (tmp_path / "home" / ".fluksio").resolve()
|
||||||
|
)
|
||||||
|
|
||||||
|
# And `--data-dir` still names any directory outright.
|
||||||
|
named = cli._data_dir(str(tmp_path / "named"))
|
||||||
|
assert named == (tmp_path / "named").resolve()
|
||||||
|
|||||||
+26
-8
@@ -15,6 +15,18 @@ The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
|||||||
installation, while `login`, `sync`, `run` and `runs` talk to one that may be
|
installation, while `login`, `sync`, `run` and `runs` talk to one that may be
|
||||||
anywhere.
|
anywhere.
|
||||||
|
|
||||||
|
## Where an installation lives
|
||||||
|
|
||||||
|
`.fluksio` beside your code, found the way `.git` is: from the working
|
||||||
|
directory, or any directory above it. Two repositories on one machine are
|
||||||
|
therefore two engines, with their own flows, runs and token. `fluksio serve`
|
||||||
|
makes one where there is none, and it ignores itself from within — a
|
||||||
|
`.gitignore` of `*`, so a database and a credential cannot be committed by
|
||||||
|
accident.
|
||||||
|
|
||||||
|
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
|
||||||
|
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
|
||||||
|
|
||||||
## `fluksio serve`
|
## `fluksio serve`
|
||||||
|
|
||||||
Runs the engine.
|
Runs the engine.
|
||||||
@@ -29,7 +41,7 @@ Docker.
|
|||||||
|
|
||||||
| Option | Default | What it does |
|
| Option | Default | What it does |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--data-dir PATH` | `~/.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
|
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
|
||||||
| `--host HOST` | `127.0.0.1` | what to bind |
|
| `--host HOST` | `127.0.0.1` | what to bind |
|
||||||
| `--port PORT` | `8000` | what to listen on |
|
| `--port PORT` | `8000` | what to listen on |
|
||||||
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
||||||
@@ -112,13 +124,17 @@ address an engine over its API rather than being one.
|
|||||||
### `fluksio login`
|
### `fluksio login`
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
fluksio login --url http://127.0.0.1:8000
|
fluksio login --url https://api.example.com
|
||||||
```
|
```
|
||||||
|
|
||||||
Asks for an email and password, and keeps the token it gets in
|
For an engine somewhere *else*. One you started yourself needs no login:
|
||||||
`~/.config/fluksio/client.json` (`$XDG_CONFIG_HOME` is honoured). Everything
|
`fluksio serve` writes the token as it comes up and says where it put it.
|
||||||
below reads it from there, or from `FLUKSIO_URL` and `FLUKSIO_TOKEN`, or from
|
|
||||||
its own `--url` and `--token`.
|
The token goes in this project's `.fluksio/client.json`, or with `--global` in
|
||||||
|
`~/.fluksio/client.json`. Every command below reads it from there — nearest
|
||||||
|
first, walking up from the working directory — or from `FLUKSIO_URL` and
|
||||||
|
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
|
||||||
|
version wrote to `~/.config/fluksio/client.json` is still read.
|
||||||
|
|
||||||
### `fluksio sync`
|
### `fluksio sync`
|
||||||
|
|
||||||
@@ -164,7 +180,9 @@ commit of the repository it came from, and its parameters.
|
|||||||
## What lives in the data directory
|
## What lives in the data directory
|
||||||
|
|
||||||
```text
|
```text
|
||||||
~/.fluksio/
|
.fluksio/ (or ~/.fluksio, with `--global`)
|
||||||
|
├── client.json the token `serve` wrote, mode 600
|
||||||
|
├── .gitignore `*` — a database and a credential, ignored from within
|
||||||
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
||||||
├── flows/ a git repository — one directory per flow
|
├── flows/ a git repository — one directory per flow
|
||||||
│ ├── house/
|
│ ├── house/
|
||||||
@@ -203,7 +221,7 @@ directory. The ones you are most likely to touch:
|
|||||||
|
|
||||||
| Variable | Default | What it does |
|
| Variable | Default | What it does |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this |
|
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
|
||||||
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
||||||
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
||||||
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
||||||
|
|||||||
@@ -6,36 +6,74 @@ answer to "what was the learning rate on the run that got 94%?", and the
|
|||||||
|
|
||||||
This page adds Fluksio to what you already have. It takes about five minutes,
|
This page adds Fluksio to what you already have. It takes about five minutes,
|
||||||
installs one Python package, and does not ask you to restructure anything.
|
installs one Python package, and does not ask you to restructure anything.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
cd my-research
|
||||||
pip install fluksio
|
pip install fluksio
|
||||||
fluksio serve
|
fluksio serve
|
||||||
```
|
```
|
||||||
|
|
||||||
That is the whole installation. No Docker, no database server, no ports to
|
That is the whole installation. No Docker, no database server, no ports to
|
||||||
open. The first run prints something like:
|
open, and no login. The first run prints something like:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Created the admin account admin@example.com
|
Created the admin account admin@example.com
|
||||||
password: k3Qm-8vTpLdX
|
password: k3Qm-8vTpLdX
|
||||||
Shown once. Change it from the dashboard.
|
Shown once. Change it from the dashboard.
|
||||||
Fluksio 0.1.0 — data in /home/you/.fluksio
|
Fluksio 0.1.0 — data in /home/you/my-research/.fluksio
|
||||||
API http://127.0.0.1:8000/api/v1
|
API http://127.0.0.1:8000/api/v1
|
||||||
|
Nodes /home/you/my-research/.venv/bin/python
|
||||||
|
your environment, adopted. Add packages with pip.
|
||||||
No portal. Pair this installation with:
|
No portal. Pair this installation with:
|
||||||
fluksio enroll <code> --portal https://hub.example.com
|
fluksio enroll <code> --portal https://hub.example.com
|
||||||
|
Signed in as admin@example.com
|
||||||
|
token in /home/you/my-research/.fluksio/client.json
|
||||||
```
|
```
|
||||||
|
|
||||||
**Write that password down.** It is shown once and it is how you authenticate
|
Read the last two lines: you are already signed in. Signing in to your own
|
||||||
from here on.
|
machine is a formality — the password was printed by the same process that
|
||||||
|
would have checked it — so `serve` writes the token itself and every command
|
||||||
|
below just works. `fluksio login` is for an engine somewhere *else*.
|
||||||
|
|
||||||
Everything the installation owns lives in `~/.fluksio`: a SQLite database, a
|
**Write that password down** anyway. It is shown once, and it is what the
|
||||||
git repository holding your flows, the artifact store, and a virtual
|
dashboard asks for.
|
||||||
environment your node code runs in. Move it with `--data-dir`, which is worth
|
|
||||||
doing on a cluster where `$HOME` is a network filesystem — SQLite's
|
### One installation per project
|
||||||
write-ahead log does not work on NFS, and `fluksio serve` warns you when it
|
|
||||||
notices.
|
`.fluksio` sits beside your code, and is found the way `.git` is — from the
|
||||||
|
directory you are standing in, or any directory above it. So two repositories
|
||||||
|
on one machine are two engines: separate flows, separate run history,
|
||||||
|
separate token, and no chance of one experiment's graph turning up in the
|
||||||
|
other's.
|
||||||
|
|
||||||
|
```text
|
||||||
|
~/research/protein-fold/
|
||||||
|
.venv/ torch, fluksio
|
||||||
|
.fluksio/ its own database, flows, artifacts, token
|
||||||
|
myresearch/
|
||||||
|
|
||||||
|
~/research/climate-sim/
|
||||||
|
.venv/ jax, fluksio
|
||||||
|
.fluksio/ its own everything
|
||||||
|
climate/
|
||||||
|
```
|
||||||
|
|
||||||
|
It holds a database and a credential, so it ignores itself from within — a
|
||||||
|
`.gitignore` of `*`, the same thing `uv` writes into `.venv`. Nothing to add
|
||||||
|
to your project's own.
|
||||||
|
|
||||||
|
Give them different ports (`--port`) if you want two running at once.
|
||||||
|
|
||||||
|
!!! tip "One engine for the machine instead"
|
||||||
|
|
||||||
|
`fluksio serve --global` uses `~/.fluksio` — shared by every directory,
|
||||||
|
which is what you want for a personal server rather than a project. When
|
||||||
|
both exist, the banner says which one you are looking at and how to reach
|
||||||
|
the other. `--data-dir` still names any directory outright, which is worth
|
||||||
|
doing on a cluster where `$HOME` is a network filesystem: SQLite's
|
||||||
|
write-ahead log does not work on NFS, and `fluksio serve` warns you when
|
||||||
|
it notices.
|
||||||
|
|
||||||
!!! tip "Keep it running"
|
!!! tip "Keep it running"
|
||||||
|
|
||||||
@@ -44,25 +82,23 @@ notices.
|
|||||||
orchestrator spends before it does anything. Leave it in a `tmux` window,
|
orchestrator spends before it does anything. Leave it in a `tmux` window,
|
||||||
or write a small `systemd --user` unit for it.
|
or write a small `systemd --user` unit for it.
|
||||||
|
|
||||||
## Log in
|
## Talking to it over HTTP
|
||||||
|
|
||||||
```sh
|
|
||||||
fluksio login --url http://127.0.0.1:8000
|
|
||||||
```
|
|
||||||
|
|
||||||
It asks for the email and password printed above and keeps the token in
|
|
||||||
`~/.config/fluksio/client.json`, so nothing below needs credentials again.
|
|
||||||
|
|
||||||
Everything the commands do is the HTTP API, and some of this page shows it
|
Everything the commands do is the HTTP API, and some of this page shows it
|
||||||
directly. For that, grab the same token as a shell variable:
|
directly. For that, take the token `serve` already wrote:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
export FLUKSIO=http://127.0.0.1:8000/api/v1
|
export FLUKSIO=http://127.0.0.1:8000/api/v1
|
||||||
export TOKEN=$(jq -r .token ~/.config/fluksio/client.json)
|
export TOKEN=$(jq -r .token .fluksio/client.json)
|
||||||
```
|
```
|
||||||
|
|
||||||
While you are experimenting, the interactive schema at
|
While you are experimenting, the interactive schema at
|
||||||
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
|
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
|
||||||
|
|
||||||
|
For an engine on another machine, `fluksio login --url https://…` asks for a
|
||||||
|
password and stores the token the same way — in this project's `.fluksio`, or
|
||||||
|
with `--global` in `~/.fluksio`.
|
||||||
|
|
||||||
## Your packages are already there
|
## Your packages are already there
|
||||||
|
|
||||||
If you installed Fluksio into the environment you work in — the venv that
|
If you installed Fluksio into the environment you work in — the venv that
|
||||||
@@ -492,7 +528,7 @@ every version of this thing I keep tweaking", note that **your flows are
|
|||||||
already a git repository**:
|
already a git repository**:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cd ~/.fluksio/flows
|
cd .fluksio/flows
|
||||||
git log --oneline
|
git log --oneline
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Anything already exported wins over the file.
|
|||||||
|
|
||||||
| Variable | Default | Notes |
|
| Variable | Default | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `DATA_DIR` | `flow-data` (`~/.fluksio` via the CLI) | everything below derives from this |
|
| `DATA_DIR` | `flow-data` (`./.fluksio` via the CLI; `~/.fluksio` with `--global`) | everything below derives from this |
|
||||||
| `DATABASE_URL` | SQLite in `DATA_DIR` | any SQLAlchemy URL |
|
| `DATABASE_URL` | SQLite in `DATA_DIR` | any SQLAlchemy URL |
|
||||||
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
|
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
|
||||||
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
|
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
|
||||||
|
|||||||
Reference in New Issue
Block a user