Project

General

Profile

Actions

Bug #19688

closed

Launch registered workflows faster

Added by Peter Amstutz 29 days ago. Updated 9 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Target version:
Start date:
11/14/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Launching very large registered workflows is slow and this is a big problem. Related, registered workflows can be huge records that make project list responses slow.

Fast launch

Assume that the workflow has been fully processed. Make a fast path which takes the workflow as-is and constructs a container request.

Lightweight records

Copy the input and output sections into a wrapper workflow. The wrapper workflow has one step that invokes the real workflow, which is stored in a collection.


Files


Subtasks 1 (0 open1 closed)

Task #19719: Review 19688-cwl-fast-pathResolvedLucas Di Pentima11/14/2022

Actions

Related issues

Related to Arvados Epics - Story #17848: Improve a-c-r usabilityIn Progress07/01/202111/30/2022

Actions
Related to Arvados - Bug #19699: HTTP download creates collections with too-long names, needs flag to run in runner process after submissionResolvedPeter Amstutz11/14/2022

Actions
Actions #1

Updated by Peter Amstutz 29 days ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 29 days ago

Actions #4

Updated by Peter Amstutz 27 days ago

Dev package for testing. Includes code from this branch as well as #19699

Actions #5

Updated by Peter Amstutz 22 days ago

  • Status changed from New to In Progress
Actions #6

Updated by Peter Amstutz 22 days ago

19688-cwl-fast-path @ 9f39acdc1d65d8b2714ffccd51a0311e6df9ec4a

Adds a new "fast path" for running workflows that have been registered on the API server.

This leverages the fact that all the dependency rewriting and workflow packing has already taken place. Previously, it was re-doing this work. For some large customer workflows, this is extremely expensive (taking multiple minutes).

Key points:

  • Enables fast path when the workflow is being submitted and is provided as a workflow UUID
  • It minimizes overhead by only loading the top level workflow and suppressing the load of the workflow steps
  • When submitting the workflow, it avoid going through the "packing" step and simply copies the already-packed workflow from the workflow record into the container request
  • To further reduce overhead in submitting, the registered workflow is now a wrapper workflow which has the same input/output signature as the original workflow, but only a single step, which invokes the packed workflow. Combined with the first bullet (not loading workflow steps), this avoids virtually all of the parsing overhead during submit.
    • Puts the packed workflow into a collection, this has the added benefit of partially addressing a couple of other long-standing issues
    • Workflow definitions are now visible in workbench by visiting the collection.
    • No longer storing potentially huge workflow definitions in the "mounts" field of container requests, which led to slow load times of project views.
  • Fixed tests

developer-run-tests: #3358

Actions #7

Updated by Peter Amstutz 21 days ago

  • Target version changed from 2022-11-09 sprint to 2022-11-23 sprint
Actions #8

Updated by Peter Amstutz 20 days ago

  • Related to Bug #19699: HTTP download creates collections with too-long names, needs flag to run in runner process after submission added
Actions #9

Updated by Peter Amstutz 17 days ago

When registering a single command line tool, it will run with the same of the original workflow including git description.

On second thought, this isn't what we want. Another package coming up.

Actions #10

Updated by Peter Amstutz 17 days ago

Make sure the wrapper info has git info. Which is what we really want. Also name the step for the cwl file.

Actions #11

Updated by Peter Amstutz 17 days ago

19688-cwl-fast-path @ 0b39c68ee38afbfec9f7d6d082a52cc2681edbea

See note 6 for details. This is a follow up fix so that git info is copied into the wrapper.

developer-run-tests: #3363

Actions #13

Updated by Lucas Di Pentima 16 days ago

This LGTM, thanks!

Actions #14

Updated by Peter Amstutz 9 days ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF