Bug #9362
Updated by Jiayong Li almost 8 years ago
Right now, arvados-cwl-runner --enable-reuse reuses the oldest version of the run, I think it should reuse the most recent run instead. To be more precise, consider the following scenario, where the tools are all deterministic. In run 1, job A went through, but upon examining the output I concluded that the compute nodes made an error. In run 2, I used --disable-reuse and had a reasonable result of job A. Now in all of the future pipelines that have job A, I would like to reuse the result of job A of run 2, but the default behavior of arvados-cwl-runner reuses that of run 1 instead. I understand the functionality of specifying which run to reuse would be quite complicated, but I think the simpler solution is to reuse the most recent run instead of the oldest one. The rationale here is that if the very first run was successful, one wouldn't want to run it again. Here's a concrete example of the above behavior. I ran snap_freebayes_hu34D5B9. The pipeline https://workbench.f48sn.arvadosapi.com/pipeline_instances/f48sn-d1hrv-vi7dgz6phq60fm8 had a bad run (explained in #9361). So I reran the pipeline with --disable-reuse and had a successful run https://workbench.f48sn.arvadosapi.com/pipeline_instances/f48sn-d1hrv-of9pj0lw5h830m5 Now I'm running snap_gatk_hu34D5B9, naturally I turned on --enable-reuse to reuse the successful alignments in f48sn-d1hrv-of9pj0lw5h830m5, but it reused the bad alignments in f48sn-d1hrv-vi7dgz6phq60fm8 instead. https://workbench.f48sn.arvadosapi.com/pipeline_instances/f48sn-d1hrv-soqcqpjh26gntc4 I think the arvados-cwl-runner should reuse the most recent run instead of an earlier one. Or maybe there should be a way to specify which run to reuse. The command I was I'm using is <pre> arvados-cwl-runner --debug --enable-reuse --local --wait --project-uuid f48sn-j7d0g-fnuojiyi5vnwigu main-snap_freebayes_hu34D5B9.cwl main-snap_freebayes_hu34D5B9-samples.json </pre> and <pre> arvados-cwl-runner --debug --enable-reuse --local --wait --project-uuid f48sn-j7d0g-fnuojiyi5vnwigu main-snap_gatk_hu34D5B9.cwl main-snap_gatk_hu34D5B9-samples.json </pre>