stroblme
bfaaa7cd03
The alert manager kept its cooldowns and its ceiling per hour in memory, so a node that stayed down was announced again by every restart. Each delivered alert is now a row in the new alert_sent table (additive migration), loaded for the last seven days when the manager starts and pruned as new ones come in. A ledger that fails never stops an alert. The manager's clock is wall time for the same reason. A node_health "down" alert also waits 60 s (HEALTH_GRACE_S) in a pending task per node; a later event that is not "down" cancels it, as does stopping the manager, so a broker restarting with the stack alerts nobody. The cooldown used to be stamped before the ceiling per hour was checked, so an alert the ceiling held back still spent its cooldown. It is stamped only once the alert is sent now.
115 lines
4.6 KiB
Markdown
115 lines
4.6 KiB
Markdown
# Secrets, modules and alerts
|
|
|
|
Store credentials, manage node dependencies and choose where failures go.
|
|
These screens help keep an instance humming.
|
|
|
|
## Secrets
|
|
|
|
Open **Secrets** as a superuser, then add a name and value. The list shows
|
|
names only; saved values are not displayed again.
|
|
|
|
Reference a secret from a node setting:
|
|
|
|
```json
|
|
{"$secret": "influx-token"}
|
|
```
|
|
|
|
Fluksio resolves it when building the node. Credential fields in the editor
|
|
offer a secret picker. Missing names produce an error on the affected node.
|
|
|
|
Secrets are encrypted with a key derived from `SECRET_KEY` and stored outside
|
|
the flow repository. Using references keeps credentials out of saved flows.
|
|
|
|
!!! warning "Keep your encryption key"
|
|
|
|
Changing `SECRET_KEY` makes existing secrets unreadable. Keep the old key
|
|
for re-encryption, or re-enter the secrets. Source installations include
|
|
`backend/scripts/rotate_secret_key.py` for rotating the secrets store.
|
|
|
|
## Modules
|
|
|
|
**Modules** shows the Python environment used by node code, its installed
|
|
packages and Python version. There are two setups:
|
|
|
|
- **Managed environment:** Docker uses a separate node environment. A
|
|
superuser edits **Requirements** and presses **Apply**. The list is resolved
|
|
with `uv pip compile` and synced with `uv pip sync`, so each package brings
|
|
its dependencies and deleting a requirement removes it. The manifest is
|
|
versioned with the flows, and successful installs refresh the workers.
|
|
- **Adopted environment:** a fresh pip installation inside your own virtual
|
|
environment can use that environment for nodes too. Modules is read-only.
|
|
Install dependencies there with pip or uv, then sync or restart Fluksio.
|
|
|
|
Set `NODE_VENV=managed` to request a separate environment. Installation
|
|
failures show the package manager's output on the page.
|
|
|
|
For an existing scientific environment on another machine, attach a worker
|
|
with `fluksio-worker --python /path/to/venv/bin/python`. See
|
|
[Remote workers](../code/workers.md).
|
|
|
|
## Alerts
|
|
|
|
Create a channel, choose events in a rule, and use **Test** to check delivery.
|
|
Enable alerts for the rules to send. Channels and rules belong to superusers,
|
|
whose Alerts page holds them. Any other account sees only the button that
|
|
subscribes its own browser to a **webpush** channel.
|
|
|
|
### Channels
|
|
|
|
| Kind | Destination |
|
|
|---|---|
|
|
| **ntfy** | server and topic; optional token |
|
|
| **SMTP** | recipient address, using the instance's mail settings |
|
|
| **Webhook** | URL that receives a POST |
|
|
| **Dashboard** | declared message read by a Notification widget |
|
|
| **webpush** | browsers subscribed from the Alerts page |
|
|
|
|
Channel settings may use `{"$secret": "name"}` references. Browser push also
|
|
needs notification permission and a secure browser context.
|
|
|
|
### Rules
|
|
|
|
A rule selects events and destination channels. Leaving all events unchecked
|
|
matches every supported alert event, including future additions.
|
|
|
|
| Event | Trigger |
|
|
|---|---|
|
|
| `node_error` | a node fails |
|
|
| `node_health` | a connection goes down |
|
|
| `flow_quarantined` | repeated crashes stop a flow's background work |
|
|
| `task_crashed` | a background task crashes |
|
|
| `engine_degraded` | engine health degrades |
|
|
| `cascade_dropped` | the engine drops work |
|
|
| `queue_unavailable` | the work queue becomes unreachable |
|
|
| `run_finished` | a batch run ends in error |
|
|
|
|
Repeated faults are deduplicated with a 15-minute default cooldown. The
|
|
engine remembers what it sent in its database, so a restart does not start
|
|
every cooldown over, and the ten-per-hour cap holds across one too. Cooldowns
|
|
longer than seven days are not kept. A `node_health` alert waits until the
|
|
connection has been down for a minute, so a broker that restarts with the
|
|
stack or a device that reconnects alerts nobody. Flapping connections are
|
|
suppressed. A quiet channel may be cooling down; **Test** helps distinguish
|
|
that from a broken configuration.
|
|
|
|
## Admin: users and clients
|
|
|
|
Superusers use **Admin** to add, edit and remove accounts, and manage OAuth
|
|
clients for [agents](../code/agents.md).
|
|
|
|
Accounts admitted from a portal are marked *Portal*. They sign in through the
|
|
portal, have no local password and are not superusers. To block local access,
|
|
edit the account and clear **Is active?**. See [Accounts and the portal](portal.md).
|
|
|
|
## Settings
|
|
|
|
Change your name, email, password and appearance under **Settings**.
|
|
|
|
**Remote access** manages enrollment and admitted users; see
|
|
[Accounts and the portal](portal.md).
|
|
|
|
A superuser can configure **Assistant**: choose a provider, store its key,
|
|
fetch models and select one. The encrypted key is shared for assistant access
|
|
on this instance. The chat appears after configuration; see
|
|
[The assistant](../code/agents.md#the-assistant).
|