Arvados

If the toplevel arvados-cwl-runner process is reused, then it doesn't execute and thus doesn't have a chance to decide whether the child steps should be re-run.

Either the arvados-cwl-runner client should be more clever and traverse the workflow to look for steps that may need to be re-run (with WorkReuse: false) or arvados-cwl-runner client should never enable reuse for the top level workflow runner process (child steps can still be reused).

I've been doing some tests on arvbox to fully understand what the issue is about.

I've attached my own test files and a drawing with how it's all assembled: there's a super-wf workflow that just calls another hash-output-wf that in turn calls a workflow and a couple of command line tools. One of those tools has the reuse=false hint to simulate an external resource that changes every time is read.

When super-wf is submitted with reuse=true, the whole job tree gets reused as this bug report description. Following Peter's comments on submitting the root wf with reuse=false, I did just that on super-wf and the result was that the entire tree of jobs was not reused, even the count-grepped-lines-wf and its children steps.

This is happening whether or not I run arvados-cwl-runner with --submit.

For testing purposes, you can reproduce what I did by running:

$ arvados-cwl-runner super-wf.cwl args.yml

It seems that a root step with reuse=false avoids its children being reused, is this behavior expected?

Lucas Di Pentima wrote:

I've been doing some tests on arvbox to fully understand what the issue is about.

I've attached my own test files and a drawing with how it's all assembled: there's a super-wf workflow that just calls another hash-output-wf that in turn calls a workflow and a couple of command line tools. One of those tools has the reuse=false hint to simulate an external resource that changes every time is read.

When super-wf is submitted with reuse=true, the whole job tree gets reused as this bug report description. Following Peter's comments on submitting the root wf with reuse=false, I did just that on super-wf and the result was that the entire tree of jobs was not reused, even the count-grepped-lines-wf and its children steps.

This is happening whether or not I run arvados-cwl-runner with --submit.

For testing purposes, you can reproduce what I did by running:

[...]

It seems that a root step with reuse=false avoids its children being reused, is this behavior expected?

Yes that's expected because workflow level requirements are inherited by steps of the workflow, so it would only reuse steps which were then explicitly marked as with reuse=true.

• https://ci.curoverse.com/job/developer-run-tests/1612/
• https://ci.curoverse.com/job/arvados-cwl-conformance-tests/272/
• WB integration retry: https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1707/

Never reuse runner containers so that individual reuse=false hints can be evaluated on the workflow.

Lucas Di Pentima wrote:

Updates at 36994ebb9 - branch 15497-reuse-fix
Test runs:

• https://ci.curoverse.com/job/developer-run-tests/1612/
• https://ci.curoverse.com/job/arvados-cwl-conformance-tests/272/
• WB integration retry: https://ci.curoverse.com/job/developer-run-tests-apps-workbench-integration/1707/

Never reuse runner containers so that individual reuse=false hints can be evaluated on the workflow.

So setting arv:ReuseRequirement to false in a workflow step never worked? Is there a way to write some integration tests for this? It also might be worth mentioning the bug number in the comment in arvcontainer.py.

Otherwise, this LGTM.

The issue is when the user submits the wf with --submit, what happens is that a-c-r creates a "runner container" that will in turn run a-c-r inside de cluster. That runner container by default is set with reuse=true, so it will get reused after the 1st run and the included workflow doesn't get evaluated no matter which reuse behavior.

• Added the issue # to the comment (was relying on git blame for that).
• I think the current test suite is enough because the expected containers still have the --enable-reuse in their command key, but better to ask Peter for confirmation.

I think this is fine. The more complicated solution would be to peer into the workflow and determine if it has non-reusable steps, but in my experience re-running the workflow runner is almost always what you want.