python3-arvados-cwl-runner package broken on Debian 10 and Ubuntu 18.04
This is version 2.0.2 on a clean buster install:
21:27:37 + cwl-runner hasher-workflow.cwl hasher-workflow-job.yml 21:27:38 Traceback (most recent call last): 21:27:38 File "/usr/bin/cwl-runner", line 8, in <module> 21:27:38 from arvados_cwl import main 21:27:38 File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 58, in <module> 21:27:38 import cwltool.main 21:27:38 File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 35, in <module> 21:27:38 from . import command_line_tool, workflow 21:27:38 File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 38, in <module> 21:27:38 from .docker import DockerCommandLineJob 21:27:38 File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/docker.py", line 4, in <module> 21:27:38 from distutils import spawn 21:27:38 File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py", line 44, in <module> 21:27:38 from distutils import dist, sysconfig # isort:skip 21:27:38 ImportError: cannot import name 'dist' from 'distutils' (/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py) 21:27:38 Build step 'Execute shell' marked build as failure
The python2 version works. Our other distro packages may also be affected. We need to add a test for this in the packaging tests (just add a simple execution of cwl-runner with a fake API host and token).
#3 Updated by Ward Vandewege about 2 months ago
- Target version changed from To Be Groomed to 2020-05-20 Sprint
- Assigned To set to Ward Vandewege
- Status changed from New to In Progress
- this affects master as well as the 2.0 branch
- this affects Debian 10 (python 3.7) and Ubuntu 18.04 (python 3.6)
- CentOS 7 (python 3.6), Debian 9 (python 3.5) and Ubuntu 16.04 (python 3.5) are not affected
What's happening is that /usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py goes out of its way to load the system distutils to get to stdlib. On Debian 10, the (partial) system python3 distutils comes from the libpython3.7-stdlib package, which does not include dist.py. Boom.
It's worth noting that it works fine on Debian 9 and Ubuntu 1604, where the (partial) system python3 distutils comes from the libpython3.5-stdlib package, which includes dist.py.
The workaround is easy: install the python3-distutils systemwide to provide the necessary dist.py. I added it as a dependency on our python3-arvados-cwl-runner packages for Debian 10+ and Ubuntu 18.04+.