Arvados

19975-oom-resubmit @ 9e76a12ff0b25322f86caf6d5ea70c09cbfd8829

developer-run-tests: #3536

Re-run failing test developer-run-tests-apps-workbench-integration: #3803

Summary:

• When enabled, always retry on exit code 137 (I originally thought we should also check for a low memory condition but a bit of experimentation showed it often times will die without any warning)
• It only retries once, with the supplied multiplier. I had originally thought of allowing some arbitrary number of retries but that seemed overcomplicated.
• Is able to find retries containers and reuse them provided the RAM request / multiplier haven't changed.
• Also scans for several different substrings to infer an out of memory condition occurred
• Option to provide your own regex if your tool produces some other error message
• Integration test fakes these conditions because reliably triggering them for real is tricky

Some comments:

• File arvcointainer.py
  • Line 370: Just a code style suggestion: keep the var naming snake_cased like the rest?
  • Line 489: The log warning message could be updated to offer this new feature to the user
• Do you think it would be useful to notify when a retry has been made because of an OOM situation? I suspect the workflow developer could be particularly interested in these kind of events for optimization/cost estimations purposes.
• I think this needs to be added to the doc page user/cwl/cwl-extensions.html

Other than that, it LGTM.