Bug #13849

[CWL] secondaryFiles checks failed in single container

Added by Jiayong Li over 3 years ago. Updated almost 3 years ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:
Release relationship:


single container

non-single container

Note that the non-single container run failed, with the secondaryFiles checks prompted errors, as they should.

cwltool ERROR: Cannot make scatter job: Missing required secondary file 'chr1.1kg.phase3.v5a.vcf.gz.tbi' from file object: {
    "basename": "chr1.1kg.phase3.v5a.vcf.gz", 
    "nameroot": "chr1.1kg.phase3.v5a.vcf", 
    "nameext": ".gz", 
    "location": "keep:ba7cd392bc4aa229c3c771b496e79628+9990/chr1.1kg.phase3.v5a.vcf.gz", 
    "secondaryFiles": [], 
    "class": "File" 

This means the secondaryFiles checks failed in the single container version. See the attached tar ball for cwl. Run with

arvados-cwl-runner phasing-wf.cwl phasing-NA12878.yml

Phasing.tar.gz (1.49 KB) Phasing.tar.gz Jiayong Li, 07/18/2018 05:52 PM


#1 Updated by Jiayong Li over 3 years ago

  • Status changed from New to In Progress

#2 Updated by Jiayong Li over 3 years ago

#3 Updated by Tom Morris about 3 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2018-08-15 Sprint

#4 Updated by Peter Amstutz about 3 years ago

The expectation is that it would fail when run with RunInSingleContainer, but it didn't fail.

The diagnosis is that RunInSingleContainer uses cwltool inside the container. If the version of cwltool is older, in isn't enforcing those checks.

The solution is to rebuild the container image with a newer arvados-cwl-runner / cwltool.

#5 Updated by Peter Amstutz about 3 years ago

  • Status changed from New to Resolved

#6 Updated by Tom Morris almost 3 years ago

  • Release set to 13

Also available in: Atom PDF