Project

General

Profile

Actions
Copy link

Bug #12601

closed

[CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations

Added by Peter Amstutz over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Peter Amstutz
Category:
-
Target version:
2017-11-22 Sprint
Story points:
-

Description

Container request mounts:

    "/keep/257d929aeba48f3a34ee3ad6cd6e1912+2101/55001703461433.chr1.bam": {
      "portable_data_hash": "257d929aeba48f3a34ee3ad6cd6e1912+2101",
      "kind": "collection",
      "path": "55001703461433.chr1.bam" 
    },
    "/keep/257d929aeba48f3a34ee3ad6cd6e1912+2101/55001703461433.chr1.bam.bai": {
      "portable_data_hash": "257d929aeba48f3a34ee3ad6cd6e1912+2101",
      "kind": "collection",
      "path": "55001703461433.chr1.bam.bai" 
    },

cwl.input.json:

  "samtools-index-bam": {
    "basename": "55001703461433.chr1.bam",
    "checksum": "sha1$d14040d6f85f4cb56ed31b5f964db085feca67fa",
    "class": "File",
    "location": "/keep/8079bc6e40be8ebbcd2f263966996725+2265/55001703461433.chr1.bam",
    "secondaryFiles": [
      {
        "basename": "55001703461433.chr1.bam.bai",
        "checksum": "sha1$0f47819ed3a744d52845ae2a306ce967c89fa90c",
        "class": "File",
        "location": "/keep/8079bc6e40be8ebbcd2f263966996725+2265/55001703461433.chr1.bam.bai",
        "size": 692656
      }
    ],
    "size": 3213913729
  }

Pathmapping copies the files to a new collection to improve container reuse, however when using RunInSingleContainer it happens after the cwl.input.json has been generated. Need to apply pathmapping to ensure that cwl.input.json and the actual input record are consistent.

Subtasks 1 (0 open — 1 closed)

Task #12602: Review 12601-pathmap-subwfResolvedPeter Amstutz11/15/2017Actions
Actions
Copy link
 #1

Updated by Peter Amstutz over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Peter Amstutz over 6 years ago

  • Status changed from New to In Progress
  • Assigned To set to Peter Amstutz
Actions
Copy link
 #3

Updated by Peter Amstutz over 6 years ago

  • Subject changed from [CWL] RunInSingleContainer job order needs to be consistent with pathmapper manipulations to [CWL] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations
Actions
Copy link
 #4

Updated by Peter Amstutz over 6 years ago

12601-pathmap-subwf @ 2b713e548510a99803c41ceb13d47fa2f38b51bf

I'm working on a test case.

Actions
Copy link
 #5

Updated by Lucas Di Pentima over 6 years ago

Running sdk/cwl tests with master's version of sdk/cwl/arvados_cwl/arvworkflow.py pass ok, is that a false positive on the test?

Actions
Copy link
 #6

Updated by Peter Amstutz over 6 years ago

Lucas Di Pentima wrote:

Running sdk/cwl tests with master's version of sdk/cwl/arvados_cwl/arvworkflow.py pass ok, is that a false positive on the test?

What are you running it exactly? You should be running this:

arvados-tests.sh -n12

Actions
Copy link
 #7

Updated by Lucas Di Pentima over 6 years ago

I'm having lots of issues trying to run the tests successfully on my arvbox instance (required to install a newer docker, re-create the image, etc and still fails), and also I've somehow broken my user on 4xphq when doing tests for my remote-groups story, so I cannot test it there, don't want to make you wait any longer.

The code updates LGTM.

Actions
Copy link
 #8

Updated by Anonymous over 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:6be9a2c8066a633327f797e537c228fb9b9d91c5.

Actions
Copy link

Also available in: Atom PDF