Arvados

19688-cwl-fast-path @ 9f39acdc1d65d8b2714ffccd51a0311e6df9ec4a

Adds a new "fast path" for running workflows that have been registered on the API server.

This leverages the fact that all the dependency rewriting and workflow packing has already taken place. Previously, it was re-doing this work. For some large customer workflows, this is extremely expensive (taking multiple minutes).

Key points:

• Enables fast path when the workflow is being submitted and is provided as a workflow UUID
• It minimizes overhead by only loading the top level workflow and suppressing the load of the workflow steps
• When submitting the workflow, it avoid going through the "packing" step and simply copies the already-packed workflow from the workflow record into the container request
• To further reduce overhead in submitting, the registered workflow is now a wrapper workflow which has the same input/output signature as the original workflow, but only a single step, which invokes the packed workflow. Combined with the first bullet (not loading workflow steps), this avoids virtually all of the parsing overhead during submit.
  • Puts the packed workflow into a collection, this has the added benefit of partially addressing a couple of other long-standing issues
  • Workflow definitions are now visible in workbench by visiting the collection.
  • No longer storing potentially huge workflow definitions in the "mounts" field of container requests, which led to slow load times of project views.
• Fixed tests

developer-run-tests: #3358

19688-cwl-fast-path @ 0b39c68ee38afbfec9f7d6d082a52cc2681edbea

See note 6 for details. This is a follow up fix so that git info is copied into the wrapper.

developer-run-tests: #3363

19688-cwl-fast-path @ 12aba9fd35f8813ff30cd6e892574505a24fcea3

• Fix test

developer-run-tests: #3366

This LGTM, thanks!