Hand a provisioned worker its token in the environment, not argv
The generated batch script exported nothing and passed `--token`, which any user on a shared compute node reads from `ps`. It now exports FLUKSIO_WORKER_TOKEN after `prerun`, which the worker already reads.
This commit is contained in:
@@ -231,7 +231,6 @@ class SlurmProvisioner:
|
|||||||
worker = [
|
worker = [
|
||||||
"exec fluksio-worker",
|
"exec fluksio-worker",
|
||||||
f"--url {self.engine_url}",
|
f"--url {self.engine_url}",
|
||||||
f"--token {token}",
|
|
||||||
f"--cpus {profile.cpus}",
|
f"--cpus {profile.cpus}",
|
||||||
f"--gpus {profile.gpus}",
|
f"--gpus {profile.gpus}",
|
||||||
f"--max-idle {self.max_idle_s:.0f}",
|
f"--max-idle {self.max_idle_s:.0f}",
|
||||||
@@ -248,6 +247,10 @@ class SlurmProvisioner:
|
|||||||
*[f"#SBATCH {option}" for option in profile.sbatch],
|
*[f"#SBATCH {option}" for option in profile.sbatch],
|
||||||
"",
|
"",
|
||||||
*profile.prerun,
|
*profile.prerun,
|
||||||
|
# Not `--token`: argv is in `ps` for every user of a shared node,
|
||||||
|
# the environment only for the job's owner. Exported after
|
||||||
|
# `prerun`, so nothing it starts inherits the token.
|
||||||
|
f"export FLUKSIO_WORKER_TOKEN={token}",
|
||||||
" ".join(worker),
|
" ".join(worker),
|
||||||
"",
|
"",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -87,6 +87,9 @@ def test_the_script_starts_a_worker_that_dials_back_and_stops_when_done(monkeypa
|
|||||||
assert "--labels gpu" in script
|
assert "--labels gpu" in script
|
||||||
# Without this the job holds its allocation until the walltime.
|
# Without this the job holds its allocation until the walltime.
|
||||||
assert "--max-idle 120" in script
|
assert "--max-idle 120" in script
|
||||||
|
# The token travels in the environment: argv is in `ps` for anyone.
|
||||||
|
assert "--token" not in script
|
||||||
|
assert "\nexport FLUKSIO_WORKER_TOKEN=ey" in script
|
||||||
|
|
||||||
|
|
||||||
def test_one_job_per_profile_however_often_it_is_asked_for(monkeypatch):
|
def test_one_job_per_profile_however_often_it_is_asked_for(monkeypatch):
|
||||||
|
|||||||
@@ -240,6 +240,9 @@ When a node needs a machine nothing attached can give, and a profile fits, the
|
|||||||
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
|
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
|
||||||
so. `prerun` owns the environment: a `module load`, or a venv with
|
so. `prerun` owns the environment: a `module load`, or a venv with
|
||||||
`fluksio-worker` already in it. The generated script runs no `pip install`.
|
`fluksio-worker` already in it. The generated script runs no `pip install`.
|
||||||
|
It hands the worker a token of its own, good for 24 hours, as
|
||||||
|
`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared
|
||||||
|
compute node cannot read it from `ps`.
|
||||||
|
|
||||||
One outstanding request per profile, however often it is asked for. A job that
|
One outstanding request per profile, however often it is asked for. A job that
|
||||||
never attaches within `provision_timeout_s` is `scancel`led, as is anything
|
never attaches within `provision_timeout_s` is `scancel`led, as is anything
|
||||||
|
|||||||
Reference in New Issue
Block a user