Bug #12601

[CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations

Added by Peter Amstutz almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
11/15/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Container request mounts:

    "/keep/257d929aeba48f3a34ee3ad6cd6e1912+2101/55001703461433.chr1.bam": {
      "portable_data_hash": "257d929aeba48f3a34ee3ad6cd6e1912+2101",
      "kind": "collection",
      "path": "55001703461433.chr1.bam" 
    },
    "/keep/257d929aeba48f3a34ee3ad6cd6e1912+2101/55001703461433.chr1.bam.bai": {
      "portable_data_hash": "257d929aeba48f3a34ee3ad6cd6e1912+2101",
      "kind": "collection",
      "path": "55001703461433.chr1.bam.bai" 
    },

cwl.input.json:

  "samtools-index-bam": {
    "basename": "55001703461433.chr1.bam",
    "checksum": "sha1$d14040d6f85f4cb56ed31b5f964db085feca67fa",
    "class": "File",
    "location": "/keep/8079bc6e40be8ebbcd2f263966996725+2265/55001703461433.chr1.bam",
    "secondaryFiles": [
      {
        "basename": "55001703461433.chr1.bam.bai",
        "checksum": "sha1$0f47819ed3a744d52845ae2a306ce967c89fa90c",
        "class": "File",
        "location": "/keep/8079bc6e40be8ebbcd2f263966996725+2265/55001703461433.chr1.bam.bai",
        "size": 692656
      }
    ],
    "size": 3213913729
  }

Pathmapping copies the files to a new collection to improve container reuse, however when using RunInSingleContainer it happens after the cwl.input.json has been generated. Need to apply pathmapping to ensure that cwl.input.json and the actual input record are consistent.


Subtasks

Task #12602: Review 12601-pathmap-subwfResolvedPeter Amstutz

Associated revisions

Revision 6be9a2c8
Added by Peter Amstutz almost 2 years ago

Merge branch '12601-pathmap-subwf' closes #12601

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)

#2 Updated by Peter Amstutz almost 2 years ago

  • Status changed from New to In Progress
  • Assigned To set to Peter Amstutz

#3 Updated by Peter Amstutz almost 2 years ago

  • Subject changed from [CWL] RunInSingleContainer job order needs to be consistent with pathmapper manipulations to [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations

#4 Updated by Peter Amstutz almost 2 years ago

12601-pathmap-subwf @ 2b713e548510a99803c41ceb13d47fa2f38b51bf

I'm working on a test case.

#5 Updated by Lucas Di Pentima almost 2 years ago

Running sdk/cwl tests with master's version of sdk/cwl/arvados_cwl/arvworkflow.py pass ok, is that a false positive on the test?

#6 Updated by Peter Amstutz almost 2 years ago

Lucas Di Pentima wrote:

Running sdk/cwl tests with master's version of sdk/cwl/arvados_cwl/arvworkflow.py pass ok, is that a false positive on the test?

What are you running it exactly? You should be running this:

arvados-tests.sh -n12

#7 Updated by Lucas Di Pentima almost 2 years ago

I'm having lots of issues trying to run the tests successfully on my arvbox instance (required to install a newer docker, re-create the image, etc and still fails), and also I've somehow broken my user on 4xphq when doing tests for my remote-groups story, so I cannot test it there, don't want to make you wait any longer.

The code updates LGTM.

#8 Updated by Anonymous almost 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:6be9a2c8066a633327f797e537c228fb9b9d91c5.

Also available in: Atom PDF