Arvados

Update: current theory is that this is a race condition between Container.handle_completed and container reuse.

1. (A) Container completion starts
2. (A) Container gets list of container requests to finalize
3. (B) New container request is submitted
4. (B) New container request finds container to reuse
5. (B) Container appears to still be Running (because update to Complete happened in the other transaction, which hasn't committed yet) so it doesn't finalize
6. (A) Container completion finalizes container requests found in step 2, which does not include new container request

Result: new container request never gets finalized.

Proposed solution:

• Container.handle_completed gets a row level lock on the container, and then on each container request (only when creating a new retry container)
• ContainerRequest.finalize_if_needed gets a row level lock on the container, then the container request, only then checks state and determines whether to finalize.

Another variation on this race is that retries may be mishandled, a new container request could finalize on a container that failed, and not get a retry container.

15164-cr-finalize-lock @ 24301058687be0d42883871d168c15dac98668c2

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1222/

Addresses race condition between container completion and container
reuse. Without this locking, a container request can resolve and
attempt to reuse a container which is concurrently being completed,
resulting in a race condition that results in the container request
never being finalized.

This LGTM, thanks!