Project

General

Profile

Actions

Feature #9964

closed

arvados-cwl-runner limits output data to keep using output_glob

Added by Bryan Cosca about 8 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Story points:
-
Release:
Release relationship:
Auto

Description

Once we have a feature to tell crunch-run to limit which files to be uploaded, have arvados-cwl-runner start using it.


Subtasks 2 (0 open2 closed)

Task #21684: Review 9964-output-glob-acr ResolvedPeter Amstutz06/11/2024Actions
Task #21892: Fix broken integration testsResolvedPeter Amstutz06/13/2024Actions

Related issues

Related to Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationResolvedPeter Amstutz05/30/2023Actions
Blocked by Arvados - Feature #12430: Crunch2 limit output collection to glob patternsResolvedTom CleggActions
Actions #1

Updated by Tom Morris almost 8 years ago

  • Assigned To set to Tom Morris
  • Target version set to Arvados Future Sprints
Actions #2

Updated by Ward Vandewege over 3 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #3

Updated by Peter Amstutz almost 2 years ago

  • Release set to 60
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Release deleted (60)
  • Target version set to Future
  • Assigned To deleted (Tom Morris)
  • Category set to CWL
  • Subject changed from [CWL][Crunch2][Crunch] crunchrunner should use CWL globs to output data to keep to arvados-cwl-runner limits output data to keep using output_glob
  • Tracker changed from Bug to Feature
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Related to deleted (Feature #12430: Crunch2 limit output collection to glob patterns)
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Blocked by Feature #12430: Crunch2 limit output collection to glob patterns added
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz over 1 year ago

  • Related to Bug #20561: Log when files from input are being propagated to output in crunch-run finalization added
Actions #9

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Future to Development 2024-01-03 sprint
Actions #10

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Actions #11

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-01-31 sprint
Actions #12

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-01-31 sprint to Development 2024-02-14 sprint
Actions #13

Updated by Peter Amstutz 10 months ago

  • Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Actions #14

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-27 sprint
Actions #15

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #16

Updated by Peter Amstutz 8 months ago

  • Release set to 70
Actions #17

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Actions #18

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-04-24 sprint to Development 2024-05-08 sprint
Actions #19

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-04-24 sprint
Actions #20

Updated by Peter Amstutz 7 months ago

  • Assigned To set to Peter Amstutz
Actions #21

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-04-24 sprint to Development 2024-05-08 sprint
Actions #22

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-06-05 sprint
Actions #23

Updated by Peter Amstutz 6 months ago

  • Status changed from New to In Progress
Actions #24

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-06-05 sprint to Development 2024-06-19 sprint
Actions #25

Updated by Peter Amstutz 6 months ago

9964-output-glob-acr @ 59ac77450a978b8bcfcbfdcc01b191ea91a2bc86

developer-run-tests: #4274

  • All agreed upon points are implemented / addressed.
    • Translates CWL outputBinding->glob to output_glob
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • One corner case came up in development. If both outputBinding glob and cwl.output.json are used in the same CommandLineTool and cwl.output.json refers to files that don't match any globs, those files won't be uploaded. This situation cannot be detected automatically, but also may not be something that ever appears in practice. I discuss this in a comment.
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • added a unit test and ran the CWL 1.2 test suite
  • Documentation has been updated.
    • improves the implementation of existing CWL functionality, doesn't require any documentation changes.
  • Behaves appropriately at the intended scale (describe intended scale).
    • will improve performance in situations where there are a lot of files in the working directory and only a small subset are considered output by the workflow, by excluding those extra files from the output collection
  • Considered backwards and forwards compatibility issues between client and server.
    • Updated the discovery doc revision to 20240502 (this is the date the output_glob feature was merged, it really should have been updated then) and made the feature conditional on the API server advertising the minimum version
  • Follows our coding standards and GUI style guidelines.
    • yes
Actions #26

Updated by Peter Amstutz 6 months ago

Not sure why this is failing, re-run:

developer-run-tests: #4276

Actions #28

Updated by Peter Amstutz 6 months ago

Actions #29

Updated by Tom Clegg 5 months ago

This LGTM, thanks.

Actions #30

Updated by Peter Amstutz 5 months ago

  • Status changed from In Progress to Resolved
Actions #31

Updated by Peter Amstutz 5 months ago

  • Status changed from Resolved to Feedback
Actions #32

Updated by Peter Amstutz 5 months ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF