[Crunch2] [API] Add ephemeral "run token" for running containers
Currently it is possible for a container to be locked by two dispatch processes at once (see #9898).
Also, we want to avoid passing a long-lived auth token to compute nodes.
Resolution¶Add a "run_auth_uuid" column to the containers table.
- When state changes to Locked (i.e., during the "lock" action) generate a new token and store its UUID here.
- When state changes to Queued, Complete, or Cancelled, invalidate the token and set this attribute to nil.
- Include the token in the response to the "lock" API.
- Do not include the token in any other API response.
The new token should have its scope restricted to /arvados/v1/containers/ and should be permitted to update the container (currently, only the locked_by_uuid token can update a locked container).
The token indicated by locked_by_uuid should still be permitted to unlock the container, but not perform other updates.
Update crunch-dispatch-* to pass the new token (instead of its own dispatch token) to crunch-run.