containers are not reused unless runtime constraints (including RAM) match exactly
The logic for deciding when to reuse an existing container appears to include the full set of runtime_constraints. I had expected more aggressive container reuse that applies appropriate reuse rules constraints given the semantics of the specific constraint (in particular, CWL constraints that are only hints should never be part of reuse logic). Alternatively, perhaps reuse should not consider runtime constraints for container reuse at all (or an option such as `--ignore-runtime-constraints-for-reuse` could allow for that behaviour).
If I have a CWL workflow that specifies `minRam: 1000` as a hint and runs a number of containers successfully (but perhaps a few fail because they have exceeded the memory limit), I would expect that if I then raise the hinted constraint to `minRam: 2000` and run the workflow again, that all of the existing containers that had completed successfully with the lower limit would be reused (or at least that I could configure the system to do that). Actually, I would also have hoped that the system would notice that those few containers died because they exceeded the RAM limit and automatically run them again with a higher limit on the second attempt (instead of just attempting three times to run them with exactly the same constraints, running into the memory limit each time).
In most cases, I would expect that changing runtime constraints should not change the output of a job, except from changing it from failing to succeeding or vice-versa. If the successful output for a step is not independent of a runtime constraint (for example, because it checks the memory limit from inside the container, and chooses a different code path), then the constraint could be separately listed as an input to document that fact.
#2 Updated by Tom Clegg about 1 year ago
Currently, a request for ram=[A,B] is satisfied by a container that ran with ram=M only if M=A. According to the Containers API design, the correct condition is A≤M ∧ B≥M.
Currently, when a request is satisfied by running a new container, the new container runs with M=A. Using M=B would also be correct.If we make both of those changes, the "ran out of memory → retry the failed jobs with more memory" workflow would be much easier to accomplish:
- run with ram=[1G,1G]
- if some jobs fail, run again with ram=[1G,4G]
- if some jobs fail, run again with ram=[1G,16G]
Even better, given a range [1G,16G] on a container request with max_retries=3, Arvados could try first with 1G, then 4G, then 16G if the container gets killed for reaching max memory. This might be hard to recognize reliably, but there is probably a strategy that works in most cases, like increasing max memory on attempt N+1 if crunchstat reports that attempt N used more than 90% of its allotted memory.