Feature #9308

[CWL] Crunchrunner supports using writable keep mount for output

Added by Peter Amstutz almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/06/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

Crunchrunner used by crunch v1 for CWL jobs does not support using the writable keep mount as the output directory.

  1. Define a job runtime constraint that requests use of writable keep mount.
  2. Update crunchrunner to use writable keep mount as the output directory if the runtime constraint is set
  3. Update arvados-cwl-runner to set the job runtime constraint for writable keep when there is an appropriate hint in the CWL document (depends on #9307)

Subtasks

Task #10193: Review 9308-crunchrunner-writable-keepResolvedPeter Amstutz


Related issues

Blocks Arvados - Feature #9307: [CWL] Arvados specific hint to use writable keep mountResolved10/05/2016

Associated revisions

Revision 74ec5b86
Added by Peter Amstutz over 4 years ago

Merge branch '9307-cwl-use-tmp-output' closes #9307 closes #9308

History

#1 Updated by Peter Amstutz almost 5 years ago

  • Description updated (diff)

#2 Updated by Peter Amstutz over 4 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2016-10-12 sprint

#3 Updated by Lucas Di Pentima over 4 years ago

Some observations:

  • File sdk/go/crunchrunner/crunchrunner.go
    • func setupCommand: Is it possible that the file copy return some error? Maybe it’s convenient to check for that and return it if exist.
    • func getKeepTmp: Can be the case of a corrupted json data being read from ‘buf’ making json.Unmarshall return an error?
  • File sdk/cwl/arvados_cwl/arv-cwl-schema.yml
    • There’s a repeated “this” word on the OutputDirType documentation (line 25)

#4 Updated by Peter Amstutz over 4 years ago

  • Status changed from New to Resolved

Applied in changeset arvados|commit:74ec5b86db46257f75ea1eb94c136ce18e65c906.

#5 Updated by Tom Morris over 4 years ago

  • Status changed from Resolved to In Progress

Running bcl2fastq on e51c5 after this was deployed causes the job to deadlock/hang with 4 cores spinning solidly burning sys time with no user time or I/O

#6 Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF