Arvados

Updated test package, adds --defer-dowload and --varying-url-params

Example command line:

arvados-cwl-runner --defer-download --varying-url-params=AWSAccessKeyId,Signature,Expires workflow.cwl params.yml

• --defer-download will perform the download after the workflow is submitted (when the runner process on the compute node actually starts)
• --varying-url-params tells it to ignore these URL query parameters from any HTTP URLs when checking to see if a URL has already been downloaded to Keep.

19699-cwl-http-dl @ 420b3b25875fd56814d1ff9027b9283ff4446571

• See comment 8 for list of new options
• This branch is on #19688 so you should review that one first

developer-run-tests: #3364

19699-cwl-http-dl @ c31c6528cac695bc86d4244516e07ea316cac979

Rebased to get test fixes

developer-run-tests: #3367

Just a couple comments:

• The a-c-r options page (user/cwl/cwl-run-options.html) needs to be updated with these new flags. I think this feature may also deserve a proper doc section, but maybe should not block this story.
• IIRC, compute nodes don't have internet access by default. If this is the case, do you think it would be convenient to remind this potential issue when documenting --defer-downloads?

Lucas Di Pentima wrote in #note-14:

Just a couple comments:

• The a-c-r options page (user/cwl/cwl-run-options.html) needs to be updated with these new flags. I think this feature may also deserve a proper doc section, but maybe should not block this story.

You're right, I forgot about docs. Let's keep the issue open and I'll follow up.

• IIRC, compute nodes don't have internet access by default. If this is the case, do you think it would be convenient to remind this potential issue when documenting --defer-downloads?

arvados-cwl-runner always has network access to the API enabled. Compute nodes can be firewalled off from the general Internet but that's something you need to configure at the gateway level which isn't part of our standard configuration.

19699-cwl-http-dl LGTM

19699-cwl-dl-docs @ 3cdc1e47bf435c364644ce8ef792cb42e95ac183

• Update table of options
• Add section about downloading from HTTP

• At cwl-style.html.textile file:
  • Line 175: The first sentence is the same as the section's title.
  • Line 211: $(runtime.outdir) formatting is missing.
  • Lines 255, 256, 257: Ignoring formatting of flags with ==, I think it would look nicer if they're formatted in monospaced font like other variables, commands, etc.
  • Line 263: I think the example command would be better formatted inside a codeblock.
• Even though the feature is fully described on the guide, I think we could clarify a bit more about its utility, for example: time savings, reduced traffic costs, enhanced automation, wdyt?

The rest LGTM.

Lucas Di Pentima wrote in #note-19:

• At cwl-style.html.textile file:
  • Line 175: The first sentence is the same as the section's title.
  • Line 211: $(runtime.outdir) formatting is missing.
  • Lines 255, 256, 257: Ignoring formatting of flags with ==, I think it would look nicer if they're formatted in monospaced font like other variables, commands, etc.
  • Line 263: I think the example command would be better formatted inside a codeblock.
• Even though the feature is fully described on the guide, I think we could clarify a bit more about its utility, for example: time savings, reduced traffic costs, enhanced automation, wdyt?

The rest LGTM.

Addressed above comments

19699-cwl-dl-docs @ da952d583d65e9c6c7ff24ae40c4e0d0a21efd22

This LGTM, thanks!