Bug #13849

[CWL] secondaryFiles checks failed in single container

Added by Jiayong Li almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

single container
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-1vziek0s8udmajx

non-single container
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-g0u996tfq07d4mp

Note that the non-single container run failed, with the secondaryFiles checks prompted errors, as they should.

cwltool ERROR: Cannot make scatter job: Missing required secondary file 'chr1.1kg.phase3.v5a.vcf.gz.tbi' from file object: {
    "basename": "chr1.1kg.phase3.v5a.vcf.gz", 
    "nameroot": "chr1.1kg.phase3.v5a.vcf", 
    "nameext": ".gz", 
    "location": "keep:ba7cd392bc4aa229c3c771b496e79628+9990/chr1.1kg.phase3.v5a.vcf.gz", 
    "secondaryFiles": [], 
    "class": "File" 
}

This means the secondaryFiles checks failed in the single container version. See the attached tar ball for cwl. Run with

arvados-cwl-runner phasing-wf.cwl phasing-NA12878.yml

Phasing.tar.gz (1.49 KB) Phasing.tar.gz Jiayong Li, 07/18/2018 05:52 PM

History

#1 Updated by Jiayong Li almost 3 years ago

  • Status changed from New to In Progress

#2 Updated by Jiayong Li almost 3 years ago

#3 Updated by Tom Morris almost 3 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2018-08-15 Sprint

#4 Updated by Peter Amstutz almost 3 years ago

The expectation is that it would fail when run with RunInSingleContainer, but it didn't fail.

The diagnosis is that RunInSingleContainer uses cwltool inside the container. If the version of cwltool is older, in isn't enforcing those checks.

The solution is to rebuild the container image with a newer arvados-cwl-runner / cwltool.

#5 Updated by Peter Amstutz almost 3 years ago

  • Status changed from New to Resolved

#6 Updated by Tom Morris over 2 years ago

  • Release set to 13

Also available in: Atom PDF