Project

General

Profile

Actions

Bug #13102

open

containers are not reused unless runtime constraints (including RAM) match exactly

Added by Joshua Randall about 6 years ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

The logic for deciding when to reuse an existing container appears to include the full set of runtime_constraints. I had expected more aggressive container reuse that applies appropriate reuse rules constraints given the semantics of the specific constraint (in particular, CWL constraints that are only hints should never be part of reuse logic). Alternatively, perhaps reuse should not consider runtime constraints for container reuse at all (or an option such as `--ignore-runtime-constraints-for-reuse` could allow for that behaviour).

If I have a CWL workflow that specifies `minRam: 1000` as a hint and runs a number of containers successfully (but perhaps a few fail because they have exceeded the memory limit), I would expect that if I then raise the hinted constraint to `minRam: 2000` and run the workflow again, that all of the existing containers that had completed successfully with the lower limit would be reused (or at least that I could configure the system to do that). Actually, I would also have hoped that the system would notice that those few containers died because they exceeded the RAM limit and automatically run them again with a higher limit on the second attempt (instead of just attempting three times to run them with exactly the same constraints, running into the memory limit each time).

In most cases, I would expect that changing runtime constraints should not change the output of a job, except from changing it from failing to succeeding or vice-versa. If the successful output for a step is not independent of a runtime constraint (for example, because it checks the memory limit from inside the container, and chooses a different code path), then the constraint could be separately listed as an input to document that fact.


Related issues

Related to Arvados - Idea #15502: endpoint to show why container requests didn't reuseNewActions
Actions

Also available in: Atom PDF