Project

General

Profile

Actions

Feature #21074

open

Migrate "workflow" table to be backed by collections but maintain API

Added by Peter Amstutz 7 months ago. Updated 18 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
API
Story points:
-

Description

Idea: the "workflow" table is an odd duck. It stores a single data string in the "definition" field, but doesn't support properties, versioning, trashing, etc. We want these things for workflows but we don't want to duplicate all the logic. It would be better if we could just store workflows in collections.

However, eliminating the "workflows" API endpoint would be disruptive, as Workbench and arvados-cwl-runner both rely on it. (We can synchronize workbench updates but people frequently use older versions of arvados-cwl-runner with newer API servers).

To migrate workflow records to collections, I propose the following:

  1. Workflow records are migrated over to collections. The "name" and "description" fields are straightforward. The contents of the "definition" field would be put in Keep as "workflow.yml". The collection record would have metadata "type: cwl-workflow"
  2. The Workflow endpoint is migrated to controller
  3. On controller, GET/PUT/POST operations are translated to apply to only collections with "type: cwl-workflow". The contents of "definition" would be read from / written to Keep
    1. another option would be to generate definition the on the fly from metadata
  4. when going through the workflows endpoint, collection UUIDs would be mapped to workflow UUIDs with the same cluster and random part just with 7fd4e substituting for 4zz18
  5. Going forward, we can choose to either expose additional fields and capabilities through the workflows endpoint (properties, versioning), or phase out the workflows endpoint by updating client code that uses workflows to instead use collections of "type: cwl-workflow"

This is probably also an opportunity to extract other metadata from the CWL document and put them in collection properties so that Workbench has it on hand without having to parse the CWL document as it currently does.


Related issues

Related to Arvados Epics - Idea #19132: Registered workflow improvementsIn Progress09/01/202309/30/2024Actions
Actions #1

Updated by Peter Amstutz 7 months ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 6 months ago

  • Related to Idea #19132: Registered workflow improvements added
Actions #3

Updated by Peter Amstutz 5 months ago

  • Target version changed from Future to Development 2024-01-17 sprint
Actions #4

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-01-31 sprint
Actions #5

Updated by Peter Amstutz 5 months ago

  • Category set to API
Actions #6

Updated by Peter Amstutz 5 months ago

  • Description updated (diff)
Actions #7

Updated by Peter Amstutz 5 months ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz 5 months ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-01-31 sprint to Development 2024-02-14 sprint
Actions #10

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Actions #11

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-13 sprint
Actions #12

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-03-13 sprint to Development 2024-03-27 sprint
Actions #13

Updated by Peter Amstutz about 2 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #14

Updated by Peter Amstutz about 2 months ago

  • Tracker changed from Idea to Feature
Actions #15

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Actions #16

Updated by Peter Amstutz 18 days ago

  • Target version changed from Development 2024-04-24 sprint to Development 2024-05-08 sprint
Actions #17

Updated by Peter Amstutz 18 days ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-06-05 sprint
Actions

Also available in: Atom PDF