Project

General

Profile

Actions

Feature #17004

closed

Can set output_properties on output collection of a container request

Added by Peter Amstutz over 2 years ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
05/13/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

I would like for the container, which can specify its own output collection PDH, to also be able to specify properties on the output collection.

Because the output is a PDH, it can't just put the properties on the collection, since it isn't guaranteed to find the correct collection.

Proposal:

  1. The container request gets "output_properties" which is can be set when the container request is created / committed
  2. The container gets an "output_properties" field which can be set by the running process
  3. When finalizing the container request, merge the output properties of the container with the output properties of the container request (the container request takes precedence) and set those on the output collection.

Motivation:

Want arvados-cwl-runner to be able to mark intermediate collection results as "intermediate" so they can be filtered from workbench view.

Want arvados-cwl-runner to be able to set properties on its own final collection output based on dynamic expressions based on the workflow inputs or results.


Subtasks 1 (0 open1 closed)

Task #19067: Review 17004-properties-on-outputResolvedPeter Amstutz05/13/2022

Actions

Related issues

Related to Arvados - Story #11942: [CWL] arvados-cwl-runner should support tagging output collection using propertiesClosed07/05/2017

Actions
Related to Arvados Epics - Story #17848: CWL runner improvementsIn Progress07/01/202106/30/2023

Actions
Related to Arvados Epics - Story #16945: WB2 Workflows / containers feature parityIn Progress08/01/202103/31/2023

Actions
Blocks Arvados - Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.NewPeter Amstutz05/19/2022

Actions
Blocks Arvados - Feature #17981: Hint to set properties on output collection based on workflow input or output parameter valuesResolved

Actions
Actions #1

Updated by Peter Amstutz over 1 year ago

  • Related to Story #11942: [CWL] arvados-cwl-runner should support tagging output collection using properties added
Actions #2

Updated by Peter Amstutz over 1 year ago

Actions #3

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
  • Subject changed from Can set output_properties on container request to Can set output_properties on output collection of a container request
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Related to deleted (Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.)
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Blocks Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Blocks Feature #17981: Hint to set properties on output collection based on workflow input or output parameter values added
Actions #8

Updated by Peter Amstutz 11 months ago

  • Related to Story #16945: WB2 Workflows / containers feature parity added
Actions #9

Updated by Peter Amstutz 10 months ago

  • Target version set to 2022-04-27 Sprint
Actions #10

Updated by Peter Amstutz 10 months ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz 10 months ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #12

Updated by Peter Amstutz 10 months ago

  • Assigned To set to Peter Amstutz
Actions #13

Updated by Peter Amstutz 9 months ago

  • Status changed from New to In Progress
Actions #15

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #16

Updated by Peter Amstutz 9 months ago

17004-properties-on-output @ 1e7856bffea0d0ecfcf940de90243dea0fbd3c2f

  • Add 'output_properties' to 'container' and 'container' request
    • merge 'output_properties' to get the properties to set on the output collection
    • priority is container_request > container > defaults
    • if requesting_container_uuid is not nil, assign a 'type' of 'intermediate' instead of 'output'
    • add tests
  • Add "arv:OutputCollectionProperties"
    • This sets output_properties on workflow step intermediate collections by setting container_request.output_properties
    • This sets output_properties on the final workflow output collection by setting container.output_properties of the workflow runner container.
    • Add tests
  • Related to cleaner workflow execution output, also changed the default collection output name "Output from workflow XYZ".
    • Previously this wasn't set, so the output collection would be called "Container output for request XYZ" which is pretty meaningless to humans.
    • Fix tests

developer-run-tests: #3129

Actions #18

Updated by Peter Amstutz 9 months ago

Failing tests were a couple of commits behind, this is the latest version.

developer-run-tests: #3131

Actions #19

Updated by Peter Amstutz 9 months ago

Hopefully fixes the unit test error.

17004-properties-on-output @ d49403ac749fffe24ac6357597e6014bb65efa30

developer-run-tests: #3132

Workbench re-run:

developer-run-tests-apps-workbench-integration: #3366

Actions #20

Updated by Tom Clegg 9 months ago

in doc/api/methods/container_requests.html.textile.liquid
  • the output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}
in doc/user/cwl/cwl-extensions.html.textile.liquid
  • duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
  • markdown `$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)
in services/api/app/models/container.rb
  • in validate_change, I think it would be more consistent to add :output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).
Actions #21

Updated by Peter Amstutz 9 months ago

Tom Clegg wrote:

in doc/api/methods/container_requests.html.textile.liquid
  • the output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}

Fixed.

in doc/user/cwl/cwl-extensions.html.textile.liquid
  • duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
  • markdown `$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)

Fixed.

in services/api/app/models/container.rb
  • in validate_change, I think it would be more consistent to add :output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).

Fixed.

17004-properties-on-output @ f35aae3c6732a7e08253367517c61ab3071086d1

developer-run-tests: #3149

Actions #22

Updated by Tom Clegg 9 months ago

LGTM, thanks!

Actions #23

Updated by Peter Amstutz 9 months ago

  • Status changed from In Progress to Resolved
Actions #24

Updated by Peter Amstutz 9 months ago

  • Release set to 51
Actions

Also available in: Atom PDF