Project

General

Profile

Actions

Bug #10292

closed

[CWL] Sometimes deadlocks on completion of subworkflow

Added by Brad Chapman over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5

Description

Peter;
This is a ticket to follow up on our discussion about deadlocks during bcbio CWL runs. These are intermittent and thus hard to provide a test case for but the behavior is that the run will report on a deadlock with all successfully completed tasks. Restarting a workflow with reuse enabled picks up where it left off and will keep running to completion. The deadlock occurs in multiple places although is most often seen during alignment and variant calling in steps with multiple points of parallelization:

[step vc_output_record] completion status is success
[workflow variantcall] outdir is /tmp/user/1001/tmpBDstpi
[step variantcall] completion status is success
2016-10-18 07:35:42 arvados.cwl-runner[14721] ERROR: Workflow is deadlocked, no runnable jobs and not waiting on any pending jobs.
Workflow error, try again with --debug for more information:
  Workflow failed.

It would be useful if we had better reporting so we could provide useful feedback during failures that would help with isolating the problem.


Subtasks 1 (0 open1 closed)

Task #10919: Wait and see if problem reoccursResolved10/18/2016Actions
Actions #1

Updated by Tom Morris over 7 years ago

  • Target version set to 2017-02-01 sprint
Actions #2

Updated by Tom Morris over 7 years ago

  • Story points set to 0.5
Actions #3

Updated by Peter Amstutz over 7 years ago

  • Subject changed from [CWL] Improved debugging for deadlocks with bcbio runs to [CWL] Sometimes deadlocks on completion of subworkflow
  • Status changed from New to In Progress
Actions #5

Updated by Peter Amstutz over 7 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF