Arvados

Peter;
Thanks for the debugging session yesterday. I re-ran am also getting further than before. Awesome. I fixed the issue with using a directory and also got past the `batch_for_variantcall` stage where you were stuck. Here is an updated pipeline (sorry, not with Arvados tracking yet -- need to get an automated way to do that -- but it does have a VERSION file with the date):

https://s3.amazonaws.com/bcbio/cwl/test_bcbio_cwl.tar.gz

The run I'm on is here:

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-wvhnu50v15p6lw6

and I get stuck on the next step using the results from `batch_for_variantcall`. batch_for_variantcall returns records which is the major change here. Either I'm not returning them right, or they are not handled correctly at some point. I get this error:

Exception on step 'get_parallel_regions'
Exception on step 'variantcall'
2016-03-08 09:13:38 arvados.cwl-runner[3444] ERROR: Caught unhandled exception, marking pipeline as failed
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/arvados_cwl/__init__.py", line 384, in arvExecutor
    for runnable in jobiter:
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 370, in job
    for w in wj.job(builder.job, basedir, output_callback, **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 287, in job
    for newjob in step.iterable:
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 243, in try_make_job
    for j in jobs:
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 528, in dotproduct_scatter
    for j in process.job(jo, basedir, functools.partial(rc.receive_scatter_output, n), **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 145, in job
    for j in self.step.job(joborder, basedir, output_callback, **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 472, in job
    **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 370, in job
    for w in wj.job(builder.job, basedir, output_callback, **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 287, in job
    for newjob in step.iterable:
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 243, in try_make_job
    for j in jobs:
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 145, in job
    for j in self.step.job(joborder, basedir, output_callback, **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/workflow.py", line 472, in job
    **kwargs):
  File "build/bdist.linux-x86_64/egg/cwltool/draft2tool.py", line 166, in job
    builder.pathmapper = self.makePathMapper(reffiles, input_basedir, **kwargs)
  File "build/bdist.linux-x86_64/egg/arvados_cwl/__init__.py", line 285, in makePathMapper
    return ArvPathMapper(self.arvrunner, reffiles, input_basedir, **kwargs)
  File "build/bdist.linux-x86_64/egg/arvados_cwl/__init__.py", line 252, in __init__
    raise cwltool.workflow.WorkflowException("Input file path '%s' is invalid" % st)
WorkflowException: Input file path '/home/chapmanb/bio/bcbio-nextgen/tests/test_bcbio_cwl/run_info-cwl-workflow/e' is invalid

If I add some debugging to print out `builder.files` and reffiles in cwltool/draft2tool (https://github.com/common-workflow-language/cwltool/blob/893de464b3dd6f359e39ee00d8d4f69cfdb1e1bc/cwltool/draft2tool.py#L161) I see a lot of e's. The file structure looks to have remained in place but all the paths got converted over to e:

set([u'e'])
[{u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'},
                      {u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'},
                      {u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File',
  u'path': u'e',
  u'secondaryFiles': [{u'class': u'File', u'path': u'e'}]},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'},
 {u'class': u'File', u'path': u'e'}]

Peter;
We have a successful run. Awesome work:

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-3q50swaspsu3mn8

The two issues I'm still running into:

- I got the ERROR: Workflow cannot make any more progress failure two additional times when running the eventually successful pipeline. It was the same CWL, same docker image, same cwl-runner/cwltool, just literally restarting again right after. For the successful run it failed twice and then worked fine the third time. So there is some kind of issue still lurking but I don't know how to reproduce other than running a few times. Unfortunately the third time when I ran with --debug to get more information it finished successfully so I don't have additional details.

- Re-use of previously run pipelines works intermittently as well. For the two re-runs above some tasks re-used (immediately returned Completed on the command line) which others did not. I didn't notice an obvious pattern, but maybe there is an easier way that interpreting the logs to check what re-used and what didn't. These were again runs with the same CWL and docker image, and the docker image includes the sort_keys=True fixes for JSON dumping from bcbio.

Thanks for all the support on this. Excited to have a working run.

Peter;
For help with debugging here are two pipelines run with the same CWL, bcbio image and cwl-runner. The first failed with the `ERROR: Workflow cannot make any progress` issue:

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-y1z1nxb2enndigu#

The second, run immediately after, failed with the same error and also only re-used some components:

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-qmxnsnfa6l5fuu9#

8 of the 10 steps were re-run. It looks like prep_align_inputs and prep_align_inputs_2 were reused but not the others.