Bug #10881

[cwl] arvados-cwl-runner submitted job fails due to missing dependency futures>=3.0.5 in docker/jobs image

Added by Tom Morris almost 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
01/13/2017
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

 cwltool INFO: /home/tfmorris/venv/bin/arvados-cwl-runner 863280201573355f380c59226015373885b50617 1.0.20170112194703, arvados-python-client 0.1.20170112173420, cwltool 1.0.20170112185927

2017-01-13_01:58:09 e51c5-8i9sb-q0w9end7q7mszgc 18554  docker image hash is 9974f6d72cf4a57923b0c4c3ecbaf58c9af43c35b2003e3ad765de69fa314f7c

2017-01-13_01:58:42 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr Using Arvados SDK version 0.1.20170112173420
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr Traceback (most recent call last):
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/tmp/crunch-job/src/crunch_scripts/cwl-runner", line 13, in <module>
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     arvados_cwl.crunch_script.run()
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 35, in run
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     logger.info(arvados_cwl.versionstring())
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 495, in versionstring
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     arvcwlpkg = pkg_resources.require("arvados-cwl-runner")
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 745, in require
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     needed = self.resolve(parse_requirements(requirements))
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr   File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 639, in resolve
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr     raise DistributionNotFound(req)
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr pkg_resources.DistributionNotFound: futures>=3.0.5
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 stderr srun: error: compute2: task 0: Exited with exit code 1
2017-01-13_01:58:43 e51c5-8i9sb-q0w9end7q7mszgc 18554 0 child 18967 on compute2.1 exit 1 success=

New dependency on python-futures, apparently. So where is it coming from?

(venv)...$ pipdeptree -r -p futures
Warning!! Cyclic dependencies found:
* SPARQLWrapper => rdflib => SPARQLWrapper
* rdflib => SPARQLWrapper => rdflib
------------------------------------------------------------------------
futures==3.0.5
  - cwltest==1.0.20161227194859 [requires: futures>=3.0.5]
    - cwltool==1.0.20170112185927 [requires: cwltest>=1.0.20161227194859]
      - arvados-cwl-runner==1.0.20170112194703 [requires: cwltool==1.0.20170112185927]

Huh, it's there in the python packages, no problem.

But we don't package futures in deb/rpm and the distribution packages are missing the dependency (here, for example, is the ubuntu1204 version of cwltest):

shell.e51c5:~# apt-cache show python-cwltest
Package: python-cwltest
Version: 1.0.20161227194859-2
License: Apache 2.0
Vendor: none
Architecture: all
Maintainer: Ward Vandewege <ward@curoverse.com>
Installed-Size: 23
Depends: python-schema-salad (>= 1.14), python-typing (>= 3.5.2), python-junit-xml (>= 1.7), python2.7
Section: default
Priority: extra
Homepage: https://github.com/common-workflow-language/cwltest
Description: Common workflow language testing framework
Description-md5: 4071a470dc6bcdbae7fb8dd3f5e6863b
Filename: pool/trusty/main/p/python-cwltest/python-cwltest_1.0.20161227194859-2_all.deb
MD5sum: 1defaa6918fe3dccb6c090233b6b840b
SHA1: 01b49291cb6ca07cfa526641bd3b52ecb92ae928
SHA256: e973b583333df03817b87d49cfb9011eda0c5a7427bc81a730d4a80066c9882b
Size: 10722
...

So... fpm bug?

Hrm, looks like https://pypi.python.org/pypi/cwltest/1.0.20161227194859 does something fancy in setup.py:

      extras_require={
          ':python_version == "2.7"': [
              'futures >= 3.0.5',
          ],
      },

So... extras_require is optional according to the docs at http://setuptools.readthedocs.io/en/latest/setuptools.html

extras_require
A dictionary mapping names of “extras” (optional features of your project) to strings or lists of strings specifying what other distributions must be installed to support those features. See the section below on Declaring Dependencies for details and examples of the format of this argument.

That explains why fpm isn't picking it up automatically as a dependency when we build the cwltest distro package.

If it isn't optional for a-c-r, we should list futures as an explicit dependency there, I think. Or is there a better way to fix this problem? In any case I have added python-futures version 3.0.5 to our list of backports in branch

10881-build-python-futures-package

which is ready for review.


Related issues

Related to Arvados - Bug #14326: Our custom-compiled `python-future` and `python3-future` packages can't be installed together and have precedenceNew

Associated revisions

Revision eddb8f68 (diff)
Added by Ward Vandewege almost 2 years ago

Build the python-futures package, version 3.0.5, for our supported
distributions.

refs #10881

Revision 7f105aaf
Added by Peter Amstutz almost 2 years ago

Merge branch '10881-build-python-futures-package' refs #10881

History

#1 Updated by Ward Vandewege almost 2 years ago

  • Description updated (diff)

#2 Updated by Ward Vandewege almost 2 years ago

  • Description updated (diff)

#3 Updated by Ward Vandewege almost 2 years ago

  • Description updated (diff)

#4 Updated by Ward Vandewege almost 2 years ago

  • Description updated (diff)

#5 Updated by Peter Amstutz almost 2 years ago

Let do the obvious fix of building the backport package. If fpm isn't picking up the dependency, we need to add an explicit dependency.

I'll look at the setup.py in cwltest and see if there's anything that should be done differently.

#6 Updated by Peter Amstutz almost 2 years ago

I'm peeved because I did run package testing locally and had found and fixed other dependency issues. This slipped through because extra_requires was missed by fpm. I think you are right that using extra_requires is just wrong in this case. I think it should have been declared this way:

"install_requires:" [
  "futures > 3.0.5 ; python_version == '2.7'" 
]

I'm going to fix the cwltest upstream and see if that enables fpm to detect the correct dependencies.

#7 Updated by Peter Amstutz almost 2 years ago

Unfortunately, using install_requires in that way only works with setuptools > 20. So I'm going with adding an explicit dependency with fpm.

#8 Updated by Tom Morris over 1 year ago

  • Target version set to Arvados Future Sprints

#9 Updated by Javier Bértoli 2 months ago

  • Related to Bug #14326: Our custom-compiled `python-future` and `python3-future` packages can't be installed together and have precedence added

Also available in: Atom PDF