Bug #11182

[CWL] mismatch between local run and cwl run using pipes

Added by Bryan Cosca 29 days ago. Updated 29 days ago.

Status:ResolvedStart date:02/28/2017
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Story points-
Velocity based estimate-

Description

whats going on in cwl land with the error |.gz: No such file or directory?

(acr82-2-14)bcosc@bcosc.qr1hi:~/veritasgenetics/cwl/myGenomePipe$ arvados-cwl-runner --submit vcf-sort.cwl vcf-sort.yml 
2017-02-28 20:26:27 cwltool INFO: /home/bcosc/acr82-2-14/bin/arvados-cwl-runner 5763c67176e8e34656cd96881074777b14b2dc4a 1.0.20170213211952, arvados-python-client 0.1.20170210210022, cwltool 1.0.20170213175853
2017-02-28 20:26:27 cwltool INFO: Resolved 'vcf-sort.cwl' to 'file:///data-sdc/home/bcosc/veritasgenetics/cwl/myGenomePipe/vcf-sort.cwl'
2017-02-28 20:26:28 arvados.cwl-runner INFO: Uploaded to e51c5-4zz18-xkxj3ya9thwpvco
2017-02-28 20:26:33 arvados.cwl-runner INFO: Created pipeline e51c5-d1hrv-alw96lfb6hi446c
2017-02-28 20:26:42 arvados.cwl-runner INFO: [job vcf-sort.cwl] e51c5-8i9sb-w8hdagsbyklljhp is Running
2017-02-28 20:30:43 arvados.cwl-runner INFO: [job vcf-sort.cwl] e51c5-8i9sb-w8hdagsbyklljhp is Failed
2017-02-28 20:30:43 arvados.cwl-runner INFO: [job vcf-sort.cwl] error log:
2017-02-28 20:30:43 arvados.cwl-runner INFO: 
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr arvados.cwl-runner INFO: [job workflow.cwl#main] error log:
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr arvados.cwl-runner INFO:
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:49 e51c5-8i9sb-mlvqz433xjydozb 17208  status: 0 done, 1 running, 0 todo
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:50 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:50 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr Using Arvados SDK version 0.1.20161216224559
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:51 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:51 crunchrunner: $(task.tmpdir)=/tmp/crunch-job-task-work/compute0.1/tmpdir
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:51 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:51 crunchrunner: $(task.outdir)=/tmp/crunch-job-task-work/compute0.1/outdir
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:51 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:51 crunchrunner: $(task.keep)=/keep
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:51 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:51 Running [gunzip -c /keep/19b8ca7f57e05d1258e1021f2322c86f+72/foo.fixed.vcf.gz | vcf-sort -c | gzip] > /tmp/crunch-job-task-work/compute0.1/outdir/foo.fixed.vcf.sorted.vcf.gz
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr gzip: |.gz: No such file or directory
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr gzip: vcf-sort.gz: No such file or directory
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr gzip: |.gz: No such file or directory
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr gzip: gzip.gz: No such file or directory
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:55 Completed with exit code 1
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:29:55 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr 2017/02/28 20:29:55 Uploading ./foo.fixed.vcf.sorted.vcf.gz (219795563 bytes)
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208 0 stderr srun: error: compute0: task 0: Exited with exit code 1
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208 0 child 17938 on compute0.1 exit 1 success=false
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208 0 failure (#1, permanent) after 34 seconds
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208 0 task output (414 bytes): . 9b1e0503e8f710a2bdb80fc9b0cd080a+67108864+A3694ddcdf9a30e9240e5ef2f0715437bcadf0b7f@58c852c5 5039400454094765ce9cd1d4fa66de58+67108864+A0be4abee5c3b586558b41b5308c8c8dd8a031ce1@58c852c7 72ca22683bb4a6297d398279a21a9c39+67108864+A395ceab7d191754e75da4ce892710b3497c41248@58c852ca 27914322206482732de21fdeabe7af33+18468971+A7484e151ef8fa388b56947cac75cb1de2fa4e1e4@58c852cb 0:219795563:foo.fixed.vcf.sorted.vcf.gz
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  status: 0 done, 0 running, 1 todo
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  wait for last 0 children to finish
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  release job allocation
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  Freeze not implemented
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  collate
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  collated output manifest text to send to API server is 414 bytes with access tokens
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  job output 99af2c3ce1e890b72ec8c87de3bcbdac+210
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr   2017-02-28_20:30:23 e51c5-8i9sb-mlvqz433xjydozb 17208  finish
  2017-02-28_20:30:26 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr arvados.cwl-runner WARNING: Overall process status is permanentFail
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 stderr arvados.cwl-runner INFO: Final output collection 80f47175e19ed15cffacfb024e745f9f+271 "Output of main (2017-02-28T20:30:26.967Z)" (e51c5-4zz18-q8jx7kklzwx3zwv)
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 child 15874 on compute2.1 exit 0 success=false
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 failure (#1, permanent) after 229 seconds
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780 0 task output (36 bytes): 80f47175e19ed15cffacfb024e745f9f+271
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  status: 0 done, 0 running, 1 todo
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  wait for last 0 children to finish
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  release job allocation
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  Freeze not implemented
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  collate
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  collated output manifest text to send to API server is 526 bytes with access tokens
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  job output 80f47175e19ed15cffacfb024e745f9f+271
  2017-02-28_20:30:27 e51c5-8i9sb-w8hdagsbyklljhp 15780  finish
2017-02-28 20:30:43 arvados.cwl-runner WARNING: Overall process status is permanentFail
2017-02-28 20:30:43 arvados.cwl-runner INFO: Final output collection 80f47175e19ed15cffacfb024e745f9f+271
{
    "vcf": {
        "size": 219795563, 
        "class": "File", 
        "location": "keep:80f47175e19ed15cffacfb024e745f9f+271/foo.fixed.vcf.sorted.vcf.gz" 
    }
}
2017-02-28 20:30:43 cwltool WARNING: Final process status is permanentFail

(acr82-2-14)bcosc@bcosc.qr1hi:~/veritasgenetics/cwl/myGenomePipe$ gunzip -c ~/keep-e51c5/19b8ca7f57e05d1258e1021f2322c86f+72/foo.fixed.vcf.gz | vcf-sort -c | gzip > ~/temp/foo.sorted.vcf.gz
(acr82-2-14)bcosc@bcosc.qr1hi:~/veritasgenetics/cwl/myGenomePipe$ ls -lah ~/temp/foo.sorted.vcf.gz 
-rw-rw-r-- 1 bcosc bcosc 42M Feb 28 20:32 /home/bcosc/temp/foo.sorted.vcf.gz

History

#1 Updated by Bryan Cosca 29 days ago

  • Description updated (diff)

#2 Updated by Bryan Cosca 29 days ago

  • Status changed from New to Resolved

silly me forgot to put ShellCommandRequirement

Also available in: Atom PDF