Arvados

 cwltool INFO: /home/tfmorris/venv/bin/arvados-cwl-runner 863280201573355f380c59226015373885b50617 1.0.20170112194703, arvados-python-client 0.1.20170112173420, cwltool 1.0.20170112185927

2017-01-13_01:58:09 e51c5-8i9sb-q0w9end7q7mszgc 18554  docker image hash is 9974f6d72cf4a57923b0c4c3ecbaf58c9af43c35b2003e3ad765de69fa314f7c

2017-01-13_01:58:42 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr Using Arvados SDK version 0.1.20170112173420
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr Traceback (most recent call last):
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/tmp/crunch-job/src/crunch_scripts/cwl-runner", line 13, in <module>
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     arvados_cwl.crunch_script.run()
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 35, in run
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     logger.info(arvados_cwl.versionstring())
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 495, in versionstring
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     arvcwlpkg = pkg_resources.require("arvados-cwl-runner")
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 745, in require
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     needed = self.resolve(parse_requirements(requirements))
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 639, in resolve
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     raise DistributionNotFound(req)
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr pkg_resources.DistributionNotFound: futures>=3.0.5
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr srun: error: compute2: task 0: Exited with exit code 1
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 child 18967 on compute2.1 exit 1 success=

New dependency on python-futures, apparently. So where is it coming from?

(venv)...$ pipdeptree -r -p futures
Warning!! Cyclic dependencies found:
* SPARQLWrapper => rdflib => SPARQLWrapper
* rdflib => SPARQLWrapper => rdflib
------------------------------------------------------------------------
futures==3.0.5
  - cwltest==1.0.20161227194859 [requires: futures>=3.0.5]
    - cwltool==1.0.20170112185927 [requires: cwltest>=1.0.20161227194859]
      - arvados-cwl-runner==1.0.20170112194703 [requires: cwltool==1.0.20170112185927]

Huh, it's there in the python packages, no problem.

But we don't package futures in deb/rpm and the distribution packages are missing the dependency (here, for example, is the ubuntu1204 version of cwltest):

shell.e51c5:~# apt-cache show python-cwltest
Package: python-cwltest
Version: 1.0.20161227194859-2
License: Apache 2.0
Vendor: none
Architecture: all
Maintainer: Ward Vandewege <ward@curoverse.com>
Installed-Size: 23
Depends: python-schema-salad (>= 1.14), python-typing (>= 3.5.2), python-junit-xml (>= 1.7), python2.7
Section: default
Priority: extra
Homepage: https://github.com/common-workflow-language/cwltest
Description: Common workflow language testing framework
Description-md5: 4071a470dc6bcdbae7fb8dd3f5e6863b
Filename: pool/trusty/main/p/python-cwltest/python-cwltest_1.0.20161227194859-2_all.deb
MD5sum: 1defaa6918fe3dccb6c090233b6b840b
SHA1: 01b49291cb6ca07cfa526641bd3b52ecb92ae928
SHA256: e973b583333df03817b87d49cfb9011eda0c5a7427bc81a730d4a80066c9882b
Size: 10722
...

So... fpm bug?

Hrm, looks like https://pypi.python.org/pypi/cwltest/1.0.20161227194859 does something fancy in setup.py:

      extras_require={
          ':python_version == "2.7"': [
              'futures >= 3.0.5',
          ],
      },

So... extras_require is optional according to the docs at http://setuptools.readthedocs.io/en/latest/setuptools.html

extras_require
A dictionary mapping names of “extras” (optional features of your project) to strings or lists of strings specifying what other distributions must be installed to support those features. See the section below on Declaring Dependencies for details and examples of the format of this argument.

That explains why fpm isn't picking it up automatically as a dependency when we build the cwltest distro package.

If it isn't optional for a-c-r, we should list futures as an explicit dependency there, I think. Or is there a better way to fix this problem? In any case I have added python-futures version 3.0.5 to our list of backports in branch

10881-build-python-futures-package

which is ready for review.