Story #10401

[CWL] Limit expansion of Directory inputs

Added by Peter Amstutz 5 months ago. Updated about 17 hours ago.

Status:In ProgressStart date:03/08/2017
Priority:NormalDue date:
Assignee:Peter Amstutz% Done:

33%

Category:-
Target version:2017-04-12 sprint
Story points0.5Remaining (hours)0.00 hour
Velocity based estimate-

Description

Currently, the default behavior of cwltool and arvados-cwl-runner is to recursively expand directory listings to enumerate all files. For directory trees with hundreds of thousands of files, this is very expensive in terms of both time and memory consumption.

Rework cwltool behavior to accommodate directories which are not expanded by default. Allow the user to explicitly request by a hint whether or not to expand directory listings.

Submit PR to CWL v1.1 spec to standardize feature enabling user to specify whether and how to expand directory listings.


Subtasks

Task #11194: ReviewNewTom Clegg

Task #11225: Update cwltool In ProgressPeter Amstutz

Task #11340: Review 10401-run-upload-dirResolvedTom Clegg

Associated revisions

Revision ca19a29f
Added by Peter Amstutz 3 days ago

Merge branch '10401-run-upload-dir' refs #10401

Revision 922e79be
Added by Peter Amstutz 3 days ago

Update cwl test_submit refs #10401

History

#1 Updated by Peter Amstutz 5 months ago

  • Subject changed from [CWL] Limit expansion of Directory objects to [CWL] Limit expansion of Directory inputs

#2 Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)
  • Story points set to 2.0

#3 Updated by Peter Amstutz 29 days ago

  • Target version set to Arvados Future Sprints

#4 Updated by Peter Amstutz 29 days ago

  • Status changed from New to In Progress

#5 Updated by Tom Morris 29 days ago

  • Target version changed from Arvados Future Sprints to 2017-03-15 sprint

#6 Updated by Peter Amstutz 29 days ago

  • Assignee set to Peter Amstutz

#7 Updated by Peter Amstutz 15 days ago

  • Target version changed from 2017-03-15 sprint to 2017-03-29 sprint

#8 Updated by Peter Amstutz 3 days ago

10401-run-upload-dir

Add support for uploading Directories in the "uploadfiles" method of arv-run (used by a-c-r)

#9 Updated by Tom Clegg 3 days ago

10401-run-upload-dir @ 7e9156a LGTM

#10 Updated by Peter Amstutz 2 days ago

10401-limit-dir-expansion

  • Bump cwltool version, add support for cwltool:LoadListing hint which controls expansion of directory listings (no_listing, shallow_listing, deep_listing), still defaults to deep_listing behavio for compatibility
  • Because this changes how directories are handled (no longer assumes that a directory object has a fully enumerated list of files) this required updates to how directory uploads & path mappings are handled
  • Also explicitly checks that the things it tries to upload are file URIs, fixes #11257
  • Rework how Arvados-specific CWL extensions are declared, so that document validation is properly aware of the Arvados extensions (this fixes the phone-home bug #11333)

#11 Updated by Peter Amstutz about 18 hours ago

  • Target version changed from 2017-03-29 sprint to 2017-04-12 sprint

#12 Updated by Peter Amstutz about 18 hours ago

  • Story points changed from 2.0 to 0.5

Also available in: Atom PDF