Actions
Feature #17004
closed
Can set output_properties on output collection of a container request
Story points:
-
Release:
Release relationship:
Auto
Description
I would like for the container, which can specify its own output collection PDH, to also be able to specify properties on the output collection.
Because the output is a PDH, it can't just put the properties on the collection, since it isn't guaranteed to find the correct collection.
Proposal:
- The container request gets "output_properties" which is can be set when the container request is created / committed
- The container gets an "output_properties" field which can be set by the running process
- When finalizing the container request, merge the output properties of the container with the output properties of the container request (the container request takes precedence) and set those on the output collection.
Motivation:
Want arvados-cwl-runner to be able to mark intermediate collection results as "intermediate" so they can be filtered from workbench view.
Want arvados-cwl-runner to be able to set properties on its own final collection output based on dynamic expressions based on the workflow inputs or results.
Related issues
Updated by Peter Amstutz over 2 years ago
- Related to Idea #11942: [CWL] arvados-cwl-runner should support tagging output collection using properties added
Updated by Peter Amstutz over 2 years ago
- Related to Idea #17848: CWL runner improvements added
Updated by Peter Amstutz over 2 years ago
- Description updated (diff)
- Subject changed from Can set output_properties on container request to Can set output_properties on output collection of a container request
Updated by Peter Amstutz over 2 years ago
- Related to Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
Updated by Peter Amstutz over 2 years ago
- Related to deleted (Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.)
Updated by Peter Amstutz over 2 years ago
- Blocks Feature #16583: Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs. added
Updated by Peter Amstutz over 2 years ago
- Blocks Feature #17981: Hint to set properties on output collection based on workflow input or output parameter values added
Updated by Peter Amstutz about 2 years ago
- Related to Idea #16945: WB2 Workflows / containers feature parity added
Updated by Peter Amstutz about 2 years ago
- Target version set to 2022-04-27 Sprint
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Updated by Peter Amstutz almost 2 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz almost 2 years ago
- Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Updated by Peter Amstutz almost 2 years ago
17004-properties-on-output @ 1e7856bffea0d0ecfcf940de90243dea0fbd3c2f
- Add 'output_properties' to 'container' and 'container' request
- merge 'output_properties' to get the properties to set on the output collection
- priority is container_request > container > defaults
- if requesting_container_uuid is not nil, assign a 'type' of 'intermediate' instead of 'output'
- add tests
- Add "arv:OutputCollectionProperties"
- This sets output_properties on workflow step intermediate collections by setting container_request.output_properties
- This sets output_properties on the final workflow output collection by setting container.output_properties of the workflow runner container.
- Add tests
- Related to cleaner workflow execution output, also changed the default collection output name "Output from workflow XYZ".
- Previously this wasn't set, so the output collection would be called "Container output for request XYZ" which is pretty meaningless to humans.
- Fix tests
Updated by Peter Amstutz almost 2 years ago
Failing tests were a couple of commits behind, this is the latest version.
Updated by Peter Amstutz almost 2 years ago
Hopefully fixes the unit test error.
17004-properties-on-output @ d49403ac749fffe24ac6357597e6014bb65efa30
Workbench re-run:
Updated by Tom Clegg almost 2 years ago
in doc/api/methods/container_requests.html.textile.liquid
- the
output_propertiesrow might be a good place to mention there are also some automatic/default properties{"type":"intermediate","container_request":"..."}
- duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
- markdown
`$(inputs)`looks like it was meant to be textile@$(inputs)@(existing error copied over from ProcessProperties?)
- in validate_change, I think it would be more consistent to add
:output_propertiestoprogress_attrsinstead of just the "when Running" case, so it can be updated at all the other times whenoutputcan be updated (e.g., while changing state from Running to Complete).
Updated by Peter Amstutz almost 2 years ago
Tom Clegg wrote:
in doc/api/methods/container_requests.html.textile.liquid
- the
output_propertiesrow might be a good place to mention there are also some automatic/default properties{"type":"intermediate","container_request":"..."}
Fixed.
in doc/user/cwl/cwl-extensions.html.textile.liquid
- duplicate anchor id #ProcessProperties should probably be #OutputCollectionProperties
- markdown
`$(inputs)`looks like it was meant to be textile@$(inputs)@(existing error copied over from ProcessProperties?)
Fixed.
in services/api/app/models/container.rb
- in validate_change, I think it would be more consistent to add
:output_propertiestoprogress_attrsinstead of just the "when Running" case, so it can be updated at all the other times whenoutputcan be updated (e.g., while changing state from Running to Complete).
Fixed.
17004-properties-on-output @ f35aae3c6732a7e08253367517c61ab3071086d1
Updated by Peter Amstutz almost 2 years ago
- Status changed from In Progress to Resolved
Actions