Bug #20484
closed
Check reuse behavior in WGS workflow on pirca
Added by Tom Clegg over 1 year ago. Updated over 1 year ago.
Description
https://workbench2.pirca.arvadosapi.com/processes/pirca-xvhdp-vjbg3wuxiz5oipo reused the fastqc steps, but did not reuse bwamem-samtools-view steps.
Updated by Tom Clegg over 1 year ago
- Description updated (diff)
This CR on 2023-05-03 did not reuse an existing container:
https://workbench2.pirca.arvadosapi.com/processes/pirca-xvhdp-qupraygeq1fid2m
Reuse did work for a CR submitted by a different user 2023-05-04.
https://workbench2.pirca.arvadosapi.com/processes/pirca-xvhdp-2zz4ezjgigqkytu
The first difference I see is that the command is quoted differently.
Not reused:
/bin/sh -c '/bwa-0.7.17/bwa mem -M -t 8 /keep/18657d75efb4afd31a14bb204d073239+13611/GRCh38_no_alt_plus_hs38d1_analysis_set.fna -R '\''@RG\tID:sample\tSM:sample\tLB:sample\tPL:ILLUMINA\tPU:sample1'\'' -c 250 /keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_1.fastq.gz /keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_2.fastq.gz | samtools view -@ 8 -b -S -'
Reused:
/bin/sh -c \''/bwa-0.7.17/bwa'\'' '\''mem'\'' '\''-M'\'' '\''-t'\'' '\''16'\'' '\''/keep/18657d75efb4afd31a14bb204d073239+13611/GRCh38_no_alt_plus_hs38d1_analysis_set.fna'\'' '\''-R'\'' '\''@RG\tID:sample\tSM:sample\tLB:sample\tPL:ILLUMINA\tPU:sample1'\'' '\''-c'\'' '\''250'\'' '\''/keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_1.fastq.gz'\'' '\''/keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_2.fastq.gz'\'' | '\''samtools'\'' '\''view'\'' '\''-@'\'' '\''16'\'' '\''-b'\'' '\''-S'\'' -'
Updated by Tom Clegg over 1 year ago
Here they are without the extra quoting added by workbench2.
Not reused:
"command": [ "/bin/sh", "-c", "/bwa-0.7.17/bwa mem -M -t 8 /keep/18657d75efb4afd31a14bb204d073239+13611/GRCh38_no_alt_plus_hs38d1_analysis_set.fna -R '@RG\\tID:sample\\tSM:sample\\tLB:sample\\tPL:ILLUMINA\\tPU:sample1' -c 250 /keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_1.fastq.gz /keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_2.fastq.gz | samtools view -@ 8 -b -S -" ],
Reused:
"command": [ "/bin/sh", "-c", "'/bwa-0.7.17/bwa' 'mem' '-M' '-t' '16' '/keep/18657d75efb4afd31a14bb204d073239+13611/GRCh38_no_alt_plus_hs38d1_analysis_set.fna' '-R' '@RG\\tID:sample\\tSM:sample\\tLB:sample\\tPL:ILLUMINA\\tPU:sample1' '-c' '250' '/keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_1.fastq.gz' '/keep/a146a06222f9a66b7d141e078fc67660+376237/ERR2122553_2.fastq.gz' | 'samtools' 'view' '-@' '16' '-b' '-S' -" ],
Updated by Tom Clegg over 1 year ago
$ pip3 install shellescape==3.4.1
$ python -c 'import shellescape; print(shellescape.quote("ls"))'
'ls'
$ pip3 install shellescape==3.8.1
$ python -c 'import shellescape; print(shellescape.quote("ls"))'
ls
$ grep shellescape ~/src/cwltool/setup.py
"shellescape >= 3.4.1, < 3.9",
Updated by Tom Clegg over 1 year ago
The arvados-cwl-runner doing the shell quoting is running in a container. The one that reused existing work was running the arvados/jobs:2.1.0 docker image; the one that ran a new container was running arvados/jobs:2.6.1.
Updated by Tom Clegg over 1 year ago
- Status changed from In Progress to Feedback
Updated by Peter Amstutz over 1 year ago
- Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint
Updated by Peter Amstutz over 1 year ago
- Status changed from Feedback to Resolved