[Crunch2] Prevent dispatch races
See Container dispatch for background.
Containers can go from "Locked" to "Queued" state if something fails before the container started actually running, or if the dispatcher loses track of the container and believes it is no longer running.
A Queued container can then be re-locked by the same dispatcher, which would initiate a second crunch-run process, which could race the first crunch-run process and lead to confusing results as both crunch-run processes have the ability to update the container record.
The proposed solution is to introduce an additional API token which will be issued when the container is Locked. This will be the API token that the crunch-run process will use to update the record. If the container is unlocked for any reason, the API token will be revoked, and as a result the crunch-run process will be unable to modify the container record and fail; the new crunch-run process will be able to take ownership of the container record safely with a new API token.
- Add an
run_auth_uuidfield to "containers" table on the API server
- When the container is Locked, set
run_auth_uuidto a system user with a read/write scope of just the container record
- When the dispatcher queues or executes the container using
crunch-run, set ARVADOS_API_TOKEN to the
- If the container returns to "Queued" the
run_auth_uuidtoken is revoked/deleted and the field is cleared.
If crunch-run is started multiple times, the old crunch-run will be unable to update the container record because its token is revoked. Only the new crunch-run with the new ARVADOS_API_TOKEN will be able to update the container record.
#5 Updated by Tom Clegg over 3 years ago
These races can be prevented without API changes, using the existing (since #8128) auth_uuid field.Implementation would look like:
- when updating state to Locked, crunch-dispatch notes the auth_uuid field in the response.
- when updating the container later, crunch-dispatch includes the known auth_uuid in the update fields. If the container has been unlocked and locked by a different dispatcher, even with the same dispatch token, auth_uuid will have changed, and the update will fail.
- when invoking crunch-run, crunch-dispatch passes the known auth_uuid as a command line flag or environment variable.
- crunch-run includes the known auth_uuid in every arvados.v1.containers.update call. If the container has been unlocked/relocked, updates will fail.