Bug #9361
closed
[CWL] arvados-cwl-runner unknown issue of job reuse
Description
I'm running snap_freebayes_hu34D5B9.
I cancelled "process_alignment_3" due to resource constraint in the run https://workbench.f48sn.arvadosapi.com/pipeline_instances/f48sn-d1hrv-su9zu4wms03rg7p
Then I used arvados-cwl-runner --enable-reuse and ran the same pipeline https://workbench.f48sn.arvadosapi.com/pipeline_instances/f48sn-d1hrv-vi7dgz6phq60fm8
The output of "process_alignment", "process_alignment_2", and "process_alignment_3" are exactly the same. Something went wrong here.
The command I'm using
arvados-cwl-runner --debug --enable-reuse --local --wait --project-uuid f48sn-j7d0g-fnuojiyi5vnwigu main-snap_freebayes_hu34D5B9.cwl main-snap_freebayes_hu34D5B9-samples.json
Files
Related issues
Updated by Jiayong Li almost 8 years ago
- Subject changed from [CWL] arvados-cwl-runner reuse unknown issue to [CWL] arvados-cwl-runner unknown issue of job reuse
Updated by Jiayong Li almost 8 years ago
Updated by Jiayong Li almost 8 years ago
Updated by Brad Chapman almost 8 years ago
- Status changed from New to Resolved
Jiayong;
Thanks for the report. This was a bug from me when we moved to CWL definitions that enable re-use and require specification of all inputs. I was no longer passing the split region (align_split) to the alignment runs so all of the sections that were meant to get run in sections were unfortunately re-running the full alignment. Arvados re-used them since they had the same inputs (and were the same). I pushed a fix to bcbio to avoid this, so you need to:
- bcbio_vm.py upgrade --wrapper
- Regenerate the CWL so it includes the align_split
You don't need a new Docker container. Sorry about the problem and thanks again for catching it.