Story #2992

[Workbench] Support "display_order" (integer) key for pipeline template components. Obey when rendering pipeline summary/details in Workbench.

Added by Tom Clegg over 4 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Workbench
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

background

Pipeline components are stored in a hash. In Ruby, and in the YAML representation of components that is stored in the database, the ordering of hash keys is consistent. However, when the components are handled by other languages such as Python (for example, by arv-copy) the order of components is likely to get permuted because key order is generally not predictable in hash/dictionary implementations.

Pipeline template authors should be able to control the display order (without relying on ordered dictionary implementations) by supplying a display_order key (with a numeric value) in each component.

solution / implementation

Workbench should sort components by display_order when rendering pipeline templates and instances.
  • If no display_order is given, don't sort: assume the hash implementation provides the correct order (as it does now).
API server, before saving a pipeline template, should:
  • Renumber the given display_order figures so they start at 1 and increase by 1.
  • Set max = number of components that had a display_order provided.
  • Iterate over all components (in whatever order is provided by the hash implementation, i.e., the order they appeared in the JSON document on the wire). Assign ++max to display_order for any component where it is missing.
  • Leave the components hash alone if it doesn't have the expected structure (e.g., it looks like {"foo":[1,2,3]} and Workbench wouldn't be able to render it anyway).

Older templates (if they are never updated) will still have undefined order if they end up getting copied/consumed by a Python program like arv-copy, etc. -- this is OK.

Update fixtures and examples in docs to include display_order.


Subtasks

Task #5855: Add Workbench supportNewTom Clegg

Task #5854: Add default display_order on server sideNewTom Clegg


Related issues

Related to Arvados - Story #3490: [Workbench] Create and edit pipeline templatesNew

Related to Arvados - Bug #6095: [SDKs] Make arv-copy preserve pipeline template component orderResolved2015-05-20

Has duplicate Arvados - Feature #4051: [Workbench] Pipeline component display order should be controlled by sort_order attributes if present.Rejected

Has duplicate Arvados - Bug #4432: [SDKs] When copying pipeline instances using arv-copy, the component jobs get re-arrangedClosed2014-11-05

History

#1 Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)

#2 Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz about 4 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)

#5 Updated by Brett Smith about 4 years ago

I'm not sure lexical ordering is what our users want. Pipeline templates often have their components ordered by the logical order that the jobs run in. Arbitrarily reordering them by default might be surprising. That's why the current issue with arv-copy is so surprising: the copied pipeline, when viewed in Workbench, "looks" wrong.

What if the proposed display_order field was, by default, set to match the ordering in the components as submitted? That would go farther to preserve the current behavior, and still provide the flexibility proposed in this ticket. I think it should be possible to refresh it correctly on update by checking if components.changed? and not display_order.changed?.

#6 Updated by Peter Amstutz about 4 years ago

  • Target version changed from Arvados Future Sprints to 2015-01-07 sprint

#7 Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2015-01-07 sprint to Arvados Future Sprints

#8 Updated by Tom Clegg over 3 years ago

  • Target version changed from Arvados Future Sprints to 2015-05-20 sprint

#9 Updated by Tom Clegg over 3 years ago

  • Subject changed from Support "display_order" (integer) key for pipeline template components. Obey when rendering pipeline summary/details in Workbench. to [Workbench] Support "display_order" (integer) key for pipeline template components. Obey when rendering pipeline summary/details in Workbench.
  • Description updated (diff)
  • Category set to Workbench

#10 Updated by Tom Clegg over 3 years ago

  • Assigned To set to Tom Clegg

#11 Updated by Brett Smith over 3 years ago

  • Target version changed from 2015-05-20 sprint to Arvados Future Sprints

Also available in: Atom PDF