Arvados

$ crunchstat-summary --format html --job 962eh-8i9sb-vrfiobkau7bilws > blah.html
Traceback (most recent call last):
  File "/home/peter/work/scripts/venv/bin/crunchstat-summary", line 4, in <module>
    __import__('pkg_resources').run_script('crunchstat-summary==0.1.20170105025304', 'crunchstat-summary')
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/home/peter/work/scripts/venv/lib/python2.7/site-packages/crunchstat_summary-0.1.20170105025304-py2.7.egg/EGG-INFO/scripts/crunchstat-summary", line 15, in <module>
    for r in cmd.report():
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20170105025304-py2.7.egg/crunchstat_summary/command.py", line 65, in report
    yield self.summer.html_header()
AttributeError: 'JobSummarizer' object has no attribute 'html_header'

$ crunchstat-summary --format text --job 962eh-8i9sb-vrfiobkau7bilws > blah.html
Traceback (most recent call last):
  File "/home/peter/work/scripts/venv/bin/crunchstat-summary", line 4, in <module>
    __import__('pkg_resources').run_script('crunchstat-summary==0.1.20170105025304', 'crunchstat-summary')
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/home/peter/work/scripts/venv/lib/python2.7/site-packages/crunchstat_summary-0.1.20170105025304-py2.7.egg/EGG-INFO/scripts/crunchstat-summary", line 15, in <module>
    for r in cmd.report():
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20170105025304-py2.7.egg/crunchstat_summary/command.py", line 60, in report
    yield self.summer.text_header()
AttributeError: 'JobSummarizer' object has no attribute 'text_header'

This fits the story description so long as we define a "reasonable number of threads" as N=1. Parallel processing with a thread pool would be better, since the reason for having threads in the first place is that going through 100s of jobs serially means that (at ~5 seconds per job) it will take crunchstat-summary 10 minutes or more to analyze a large workflow.

An easy solution might be something like:

1. Take the next N jobs
2. Spin them out to N threads, wait for all of them to complete (basically the existing logic)
3. yield N results
4. repeat until everything is processed

Thanks for the quick review. I'll look at the job failure, but the cluster you used isn't familiar and doesn't seem to be resolvable via *.arvadosapi.com Where is it? I was mostly focused on pipeline instances, so it wouldn't surprise me if there were issues specific to jobs (although any bugs are likely to be in the other branch that this one depends on).

As for performance, reports for a pipeline with 370 jobs that runs 3 days and uses thousands of core hours take 11.5 minutes for text and 13.8 minutes for html, which is acceptable to me.

I have a branch with a capped number of threads, but decided the complexity wasn't warranted.