Feature #17004

Can set output_properties on output collection of a container request

Added by Peter Amstutz over 1 year ago. Updated 5 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
05/13/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

I would like for the container, which can specify its own output collection PDH, to also be able to specify properties on the output collection.

Because the output is a PDH, it can't just put the properties on the collection, since it isn't guaranteed to find the correct collection.

Proposal:

  1. The container request gets "output_properties" which is can be set when the container request is created / committed
  2. The container gets an "output_properties" field which can be set by the running process
  3. When finalizing the container request, merge the output properties of the container with the output properties of the container request (the container request takes precedence) and set those on the output collection.

Motivation:

Want arvados-cwl-runner to be able to mark intermediate collection results as "intermediate" so they can be filtered from workbench view.

Want arvados-cwl-runner to be able to set properties on its own final collection output based on dynamic expressions based on the workflow inputs or results.


Subtasks

Task #19067: Review 17004-properties-on-outputResolvedPeter Amstutz


Related issues

Related to Arvados - Story #11942: [CWL] arvados-cwl-runner should support tagging output collection using propertiesClosed07/05/2017

Related to Arvados Epics - Story #17848: Improve a-c-r usabilityIn Progress07/01/202105/31/2022

Related to Arvados Epics - Story #16945: WB2 Workflows / containers feature parityIn Progress08/01/202105/31/2022

Blocks Arvados - Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.In Progress05/19/2022

Blocks Arvados - Feature #17981: Hint to set properties on output collection based on workflow input or output parameter valuesNew

History

#1 Updated by Peter Amstutz 10 months ago

  • Related to Story #11942: [CWL] arvados-cwl-runner should support tagging output collection using properties added

#2 Updated by Peter Amstutz 10 months ago

#3 Updated by Peter Amstutz 10 months ago

  • Description updated (diff)
  • Subject changed from Can set output_properties on container request to Can set output_properties on output collection of a container request

#4 Updated by Peter Amstutz 10 months ago

  • Related to Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added

#5 Updated by Peter Amstutz 10 months ago

  • Related to deleted (Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.)

#6 Updated by Peter Amstutz 10 months ago

  • Blocks Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added

#7 Updated by Peter Amstutz 10 months ago

  • Blocks Feature #17981: Hint to set properties on output collection based on workflow input or output parameter values added

#8 Updated by Peter Amstutz 2 months ago

  • Related to Story #16945: WB2 Workflows / containers feature parity added

#9 Updated by Peter Amstutz about 1 month ago

  • Target version set to 2022-04-27 Sprint

#10 Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)

#11 Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint

#12 Updated by Peter Amstutz 28 days ago

  • Assigned To set to Peter Amstutz

#13 Updated by Peter Amstutz 19 days ago

  • Status changed from New to In Progress

#15 Updated by Peter Amstutz 14 days ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint

#16 Updated by Peter Amstutz 12 days ago

17004-properties-on-output @ 1e7856bffea0d0ecfcf940de90243dea0fbd3c2f

  • Add 'output_properties' to 'container' and 'container' request
    • merge 'output_properties' to get the properties to set on the output collection
    • priority is container_request > container > defaults
    • if requesting_container_uuid is not nil, assign a 'type' of 'intermediate' instead of 'output'
    • add tests
  • Add "arv:OutputCollectionProperties"
    • This sets output_properties on workflow step intermediate collections by setting container_request.output_properties
    • This sets output_properties on the final workflow output collection by setting container.output_properties of the workflow runner container.
    • Add tests
  • Related to cleaner workflow execution output, also changed the default collection output name "Output from workflow XYZ".
    • Previously this wasn't set, so the output collection would be called "Container output for request XYZ" which is pretty meaningless to humans.
    • Fix tests

developer-run-tests: #3129

#18 Updated by Peter Amstutz 9 days ago

Failing tests were a couple of commits behind, this is the latest version.

developer-run-tests: #3131

#19 Updated by Peter Amstutz 9 days ago

Hopefully fixes the unit test error.

17004-properties-on-output @ d49403ac749fffe24ac6357597e6014bb65efa30

developer-run-tests: #3132

Workbench re-run:

developer-run-tests-apps-workbench-integration: #3366

#20 Updated by Tom Clegg 7 days ago

in doc/api/methods/container_requests.html.textile.liquid
  • the output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}
in doc/user/cwl/cwl-extensions.html.textile.liquid
  • duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
  • markdown `$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)
in services/api/app/models/container.rb
  • in validate_change, I think it would be more consistent to add :output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).

#21 Updated by Peter Amstutz 7 days ago

Tom Clegg wrote:

in doc/api/methods/container_requests.html.textile.liquid
  • the output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}

Fixed.

in doc/user/cwl/cwl-extensions.html.textile.liquid
  • duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
  • markdown `$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)

Fixed.

in services/api/app/models/container.rb
  • in validate_change, I think it would be more consistent to add :output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).

Fixed.

17004-properties-on-output @ f35aae3c6732a7e08253367517c61ab3071086d1

developer-run-tests: #3149

#22 Updated by Tom Clegg 7 days ago

LGTM, thanks!

#23 Updated by Peter Amstutz 6 days ago

  • Status changed from In Progress to Resolved

#24 Updated by Peter Amstutz 5 days ago

  • Release set to 51

Also available in: Atom PDF