Project

General

Profile

Actions

Idea #19132

open

Registered workflow improvements

Added by Peter Amstutz about 2 years ago. Updated 28 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
09/01/2023
Due date:
09/30/2024 (Due in about 4 months)
Story points:
-
Release:
Release relationship:
Auto

Description

Improve UX around registered workflows.

Better picking

  • Metadata about workflow purpose and function
  • Faceted search

Better launching

  • Change flow so that the user picks a project then a "draft" state container request is created, where the inputs and arvados-cwl-runner options are editable.

Support versioning for registered workflows in Workbench.

Possible features:

  • When launching a workflow, user can choose among multiple versions
  • Can easily determine which version of the workflow was used to run a particular pipeline.
  • Text of workflow that is used for execution is more like the text that the user started from (instead of workflow packing).
  • Text of workflow as submitted (i.e. original) is also saved so that users can compare the "original" they submitted against their VCS later (good for auditability).
  • Requirement from customer: when making a copy, record the upstream collection it was copied from, indicate in the user interface that the upstream version has changed, offer to bump up to current version.

Possible design:

  • Store CWL in a collection with "type: workflow" property
  • Use collection versioning
  • Store workflow metadata in properties
    • entry point
    • input spec
    • git commit/branch/repo? if uploaded a git directory
    • docker jobs image to use
  • Deprecate "workflow" records
  • Identify the workflow version used for a given pipeline by matching the PDH

Implementation:

  • arvados-cwl-runner support for creating/updating new-style registered workflows
  • Workbench 2 support (this is probably the majority of the work)
    • picking
    • viewing
    • launching

Proposed new behavior for a-c-r: always create a workflow collection when running a workflow, so we have accessible record of what was run.


Related issues

Related to Arvados - Idea #19464: record git information when registering a workflowResolvedPeter Amstutz09/08/2022Actions
Related to Arvados - Idea #19636: Editing inputs on "draft" tasksNewActions
Related to Arvados - Idea #20981: Visual design for projects, search, and workflow pickingResolvedPeter Amstutz10/25/2023Actions
Related to Arvados - Idea #20924: Search supports workflowsNewActions
Related to Arvados - Feature #21074: Migrate "workflow" table to be backed by collections but maintain APINewActions
Related to Arvados - Feature #21292: New workflow picker panelNewActions
Related to Arvados - Feature #20880: Can link to wb2 URL to prompt user to launch workflow, specify project uuid in queryNewStephen SmithActions
Related to Arvados - Bug #15814: Running a workflow from WB2 exposes secret inputsNewActions
Related to Arvados - Bug #20929: Launching workflow creates a draft process for editingNewActions
Related to Arvados - Idea #3094: No UI to edit runtime constraints for draft container requestsNewActions
Related to Arvados - Idea #21709: CWL hints for ordering and grouping inputs which affects WB2 displayNewActions
Related to Arvados - Feature #21710: Render HTML in CWL 'label' and 'doc' fieldsNewActions
Related to Arvados - Feature #21711: CWL File picker option to directly upload files, create and select new collectionNewActions
Related to Arvados - Feature #14168: [Running a workflow] Input presetsNewActions
Related to Arvados - Feature #19086: Can provide input JSON/YAML when launching a Workflow as an alternative to the input formNewPeter AmstutzActions
Actions #1

Updated by Peter Amstutz about 2 years ago

  • Start date set to 05/15/2022
  • Due date set to 08/15/2022
Actions #2

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz about 2 years ago

  • Start date changed from 05/15/2022 to 05/23/2022
Actions #4

Updated by Ward Vandewege about 2 years ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz almost 2 years ago

  • Release set to 28
Actions #7

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz almost 2 years ago

  • Due date changed from 08/15/2022 to 09/14/2022
Actions #9

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 05/23/2022 to 07/01/2022
  • Due date changed from 09/14/2022 to 09/30/2022
Actions #10

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 07/01/2022 to 08/31/2022
  • Due date changed from 09/30/2022 to 10/31/2022
Actions #11

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 08/31/2022 to 09/01/2022
Actions #12

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 10/31/2022 to 12/31/2022
Actions #13

Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
Actions #14

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 12/31/2022 to 03/31/2023
Actions #15

Updated by Peter Amstutz over 1 year ago

  • Related to Idea #19464: record git information when registering a workflow added
Actions #16

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 03/31/2023 to 06/30/2023
Actions #17

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 09/01/2022 to 03/01/2023
  • Due date changed from 06/30/2023 to 09/30/2023
Actions #18

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 03/01/2023 to 05/01/2023
  • Due date changed from 09/30/2023 to 11/30/2023
Actions #19

Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)
  • Subject changed from Versioning registered workflows to Registered workflow improvements
Actions #20

Updated by Peter Amstutz about 1 year ago

  • Related to Idea #19636: Editing inputs on "draft" tasks added
Actions #21

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 05/01/2023 to 09/01/2023
  • Due date changed from 11/30/2023 to 12/31/2023
Actions #22

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 09/01/2023 to 10/01/2023
Actions #23

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 10/01/2023 to 09/01/2023
Actions #24

Updated by Peter Amstutz 8 months ago

  • Related to Idea #20981: Visual design for projects, search, and workflow picking added
Actions #25

Updated by Brett Smith 8 months ago

  • Related to Idea #20924: Search supports workflows added
Actions #26

Updated by Peter Amstutz 7 months ago

  • Related to Feature #21074: Migrate "workflow" table to be backed by collections but maintain API added
Actions #27

Updated by Peter Amstutz 5 months ago

  • Due date changed from 12/31/2023 to 03/31/2024
Actions #28

Updated by Peter Amstutz 28 days ago

  • Due date changed from 03/31/2024 to 06/30/2024
Actions #29

Updated by Peter Amstutz 28 days ago

  • Due date changed from 06/30/2024 to 09/30/2024
Actions #30

Updated by Peter Amstutz 28 days ago

Actions #31

Updated by Peter Amstutz 28 days ago

  • Related to Feature #20880: Can link to wb2 URL to prompt user to launch workflow, specify project uuid in query added
Actions #32

Updated by Peter Amstutz 28 days ago

  • Related to Bug #15814: Running a workflow from WB2 exposes secret inputs added
Actions #33

Updated by Peter Amstutz 28 days ago

  • Related to Bug #20929: Launching workflow creates a draft process for editing added
Actions #35

Updated by Peter Amstutz 28 days ago

  • Related to Idea #3094: No UI to edit runtime constraints for draft container requests added
Actions #37

Updated by Peter Amstutz 28 days ago

  • Related to Idea #21709: CWL hints for ordering and grouping inputs which affects WB2 display added
Actions #38

Updated by Peter Amstutz 28 days ago

  • Related to Feature #21710: Render HTML in CWL 'label' and 'doc' fields added
Actions #39

Updated by Peter Amstutz 28 days ago

  • Related to Feature #21711: CWL File picker option to directly upload files, create and select new collection added
Actions #40

Updated by Peter Amstutz 28 days ago

Actions #41

Updated by Peter Amstutz 28 days ago

  • Related to Feature #19086: Can provide input JSON/YAML when launching a Workflow as an alternative to the input form added
Actions

Also available in: Atom PDF