Bug #10292

[CWL] Sometimes deadlocks on completion of subworkflow

Added by Brad Chapman almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/18/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

Peter;
This is a ticket to follow up on our discussion about deadlocks during bcbio CWL runs. These are intermittent and thus hard to provide a test case for but the behavior is that the run will report on a deadlock with all successfully completed tasks. Restarting a workflow with reuse enabled picks up where it left off and will keep running to completion. The deadlock occurs in multiple places although is most often seen during alignment and variant calling in steps with multiple points of parallelization:

[step vc_output_record] completion status is success
[workflow variantcall] outdir is /tmp/user/1001/tmpBDstpi
[step variantcall] completion status is success
2016-10-18 07:35:42 arvados.cwl-runner[14721] ERROR: Workflow is deadlocked, no runnable jobs and not waiting on any pending jobs.
Workflow error, try again with --debug for more information:
  Workflow failed.

It would be useful if we had better reporting so we could provide useful feedback during failures that would help with isolating the problem.


Subtasks

Task #10919: Wait and see if problem reoccursResolved

Associated revisions

Revision df39fa64
Added by Peter Amstutz over 4 years ago

Merge branch '10292-cwl-deadlock' refs #10292

History

#1 Updated by Tom Morris over 4 years ago

  • Target version set to 2017-02-01 sprint

#2 Updated by Tom Morris over 4 years ago

  • Story points set to 0.5

#3 Updated by Peter Amstutz over 4 years ago

  • Subject changed from [CWL] Improved debugging for deadlocks with bcbio runs to [CWL] Sometimes deadlocks on completion of subworkflow
  • Status changed from New to In Progress

#5 Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF