Hand a provisioned worker its token in the environment, not argv
The generated batch script exported nothing and passed `--token`, which any user on a shared compute node reads from `ps`. It now exports FLUKSIO_WORKER_TOKEN after `prerun`, which the worker already reads.
This commit is contained in:
@@ -231,7 +231,6 @@ class SlurmProvisioner:
|
||||
worker = [
|
||||
"exec fluksio-worker",
|
||||
f"--url {self.engine_url}",
|
||||
f"--token {token}",
|
||||
f"--cpus {profile.cpus}",
|
||||
f"--gpus {profile.gpus}",
|
||||
f"--max-idle {self.max_idle_s:.0f}",
|
||||
@@ -248,6 +247,10 @@ class SlurmProvisioner:
|
||||
*[f"#SBATCH {option}" for option in profile.sbatch],
|
||||
"",
|
||||
*profile.prerun,
|
||||
# Not `--token`: argv is in `ps` for every user of a shared node,
|
||||
# the environment only for the job's owner. Exported after
|
||||
# `prerun`, so nothing it starts inherits the token.
|
||||
f"export FLUKSIO_WORKER_TOKEN={token}",
|
||||
" ".join(worker),
|
||||
"",
|
||||
]
|
||||
|
||||
@@ -87,6 +87,9 @@ def test_the_script_starts_a_worker_that_dials_back_and_stops_when_done(monkeypa
|
||||
assert "--labels gpu" in script
|
||||
# Without this the job holds its allocation until the walltime.
|
||||
assert "--max-idle 120" in script
|
||||
# The token travels in the environment: argv is in `ps` for anyone.
|
||||
assert "--token" not in script
|
||||
assert "\nexport FLUKSIO_WORKER_TOKEN=ey" in script
|
||||
|
||||
|
||||
def test_one_job_per_profile_however_often_it_is_asked_for(monkeypatch):
|
||||
|
||||
@@ -240,6 +240,9 @@ When a node needs a machine nothing attached can give, and a profile fits, the
|
||||
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
|
||||
so. `prerun` owns the environment: a `module load`, or a venv with
|
||||
`fluksio-worker` already in it. The generated script runs no `pip install`.
|
||||
It hands the worker a token of its own, good for 24 hours, as
|
||||
`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared
|
||||
compute node cannot read it from `ps`.
|
||||
|
||||
One outstanding request per profile, however often it is asked for. A job that
|
||||
never attaches within `provision_timeout_s` is `scancel`led, as is anything
|
||||
|
||||
Reference in New Issue
Block a user