Project

General

Profile

Actions

Feature #16583

closed

Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.

Added by Peter Amstutz almost 4 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Workbench2
Target version:
-
Story points:
-
Release relationship:
Auto

Description

Output collections have properties "container_request: <uuid>" and "type: output". This allows workbench2 to offer a filter on outputs, logs, and general.

Similar to how I can filter on just top level containers, I'd like to be able to filter on top level outputs. We also want to be able to do scripted cleaning up of intermediate outputs while leaving the final output alone.

Top level containers have "requested_by_container_uuid: null". Technically we could look up the container request for each collection to determine if it is top level or not, but that can't be written as a single query filter.

Proposed approach:

  1. Add ability to set properties of container outputs: #17004
  2. Arvados-cwl-runner starts marking intermediate and final collections, using output_properties to set it
  3. Add a rake task that the admin can optionally use to infer the intermediate and final collection output property from requested_by_container_uuid.
  4. Add to Workbench2 the ability to filter out intermediate collections from project display.
  5. Document the rake task in the upgrade notes

Subtasks 3 (0 open3 closed)

Task #19118: Review 16583-intermediate-collections on Workbench 2ResolvedPeter Amstutz05/19/2022Actions
Task #19163: Write rake taskClosedPeter Amstutz05/19/2022Actions
Task #19173: Review rake task branchClosed05/19/2022Actions

Related issues

Related to Arvados Epics - Idea #17848: CWL runner improvementsResolved07/01/202103/29/2023Actions
Blocked by Arvados - Feature #17004: Can set output_properties on output collection of a container requestResolvedPeter Amstutz05/13/2022Actions
Actions

Also available in: Atom PDF