Feature #17004
closed
Can set output_properties on output collection of a container request
Added by Peter Amstutz over 3 years ago.
Updated almost 2 years ago.
Release relationship:
Auto
Description
I would like for the container, which can specify its own output collection PDH, to also be able to specify properties on the output collection.
Because the output is a PDH, it can't just put the properties on the collection, since it isn't guaranteed to find the correct collection.
Proposal:
- The container request gets "output_properties" which is can be set when the container request is created / committed
- The container gets an "output_properties" field which can be set by the running process
- When finalizing the container request, merge the output properties of the container with the output properties of the container request (the container request takes precedence) and set those on the output collection.
Motivation:
Want arvados-cwl-runner to be able to mark intermediate collection results as "intermediate" so they can be filtered from workbench view.
Want arvados-cwl-runner to be able to set properties on its own final collection output based on dynamic expressions based on the workflow inputs or results.
- Related to Idea #11942: [CWL] arvados-cwl-runner should support tagging output collection using properties added
- Description updated (diff)
- Subject changed from Can set output_properties on container request to Can set output_properties on output collection of a container request
- Related to Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
- Related to deleted (Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.)
- Blocks Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
- Blocks Feature #17981: Hint to set properties on output collection based on workflow input or output parameter values added
- Related to Idea #16945: WB2 Workflows / containers feature parity added
- Target version set to 2022-04-27 Sprint
- Description updated (diff)
- Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
- Assigned To set to Peter Amstutz
- Status changed from New to In Progress
- Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
17004-properties-on-output @ 1e7856bffea0d0ecfcf940de90243dea0fbd3c2f
- Add 'output_properties' to 'container' and 'container' request
- merge 'output_properties' to get the properties to set on the output collection
- priority is container_request > container > defaults
- if requesting_container_uuid is not nil, assign a 'type' of 'intermediate' instead of 'output'
- add tests
- Add "arv:OutputCollectionProperties"
- This sets output_properties on workflow step intermediate collections by setting container_request.output_properties
- This sets output_properties on the final workflow output collection by setting container.output_properties of the workflow runner container.
- Add tests
- Related to cleaner workflow execution output, also changed the default collection output name "Output from workflow XYZ".
- Previously this wasn't set, so the output collection would be called "Container output for request XYZ" which is pretty meaningless to humans.
- Fix tests
developer-run-tests: #3129 
in doc/api/methods/container_requests.html.textile.liquid
- the
output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}
in doc/user/cwl/cwl-extensions.html.textile.liquid
- duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
- markdown
`$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)
in services/api/app/models/container.rb
- in validate_change, I think it would be more consistent to add
:output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).
Tom Clegg wrote:
in doc/api/methods/container_requests.html.textile.liquid
- the
output_properties row might be a good place to mention there are also some automatic/default properties {"type":"intermediate","container_request":"..."}
Fixed.
in doc/user/cwl/cwl-extensions.html.textile.liquid
- duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
- markdown
`$(inputs)` looks like it was meant to be textile @$(inputs)@ (existing error copied over from ProcessProperties?)
Fixed.
in services/api/app/models/container.rb
- in validate_change, I think it would be more consistent to add
:output_properties to progress_attrs instead of just the "when Running" case, so it can be updated at all the other times when output can be updated (e.g., while changing state from Running to Complete).
Fixed.
17004-properties-on-output @ f35aae3c6732a7e08253367517c61ab3071086d1
developer-run-tests: #3149 
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by