Bug #16434

python3-arvados-cwl-runner package broken on Debian 10 and Ubuntu 18.04

Added by Ward Vandewege about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-
Release relationship:
Auto

Description

This is version 2.0.2 on a clean buster install:

21:27:37 + cwl-runner hasher-workflow.cwl hasher-workflow-job.yml
21:27:38 Traceback (most recent call last):
21:27:38   File "/usr/bin/cwl-runner", line 8, in <module>
21:27:38     from arvados_cwl import main
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 58, in <module>
21:27:38     import cwltool.main
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 35, in <module>
21:27:38     from . import command_line_tool, workflow
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 38, in <module>
21:27:38     from .docker import DockerCommandLineJob
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/docker.py", line 4, in <module>
21:27:38     from distutils import spawn
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py", line 44, in <module>
21:27:38     from distutils import dist, sysconfig  # isort:skip
21:27:38 ImportError: cannot import name 'dist' from 'distutils' (/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py)
21:27:38 Build step 'Execute shell' marked build as failure

The python2 version works. Our other distro packages may also be affected. We need to add a test for this in the packaging tests (just add a simple execution of cwl-runner with a fake API host and token).

Associated revisions

Revision 595e6ca9
Added by Ward Vandewege about 2 months ago

Merge branch '16434-fix-python3-arvados-cwl-runner-packaging'

closes #16434

History

#1 Updated by Ward Vandewege about 2 months ago

  • Description updated (diff)

#2 Updated by Ward Vandewege about 2 months ago

  • Description updated (diff)

#3 Updated by Ward Vandewege about 2 months ago

  • Target version changed from To Be Groomed to 2020-05-20 Sprint
  • Assigned To set to Ward Vandewege
  • Status changed from New to In Progress

More information:

  • this affects master as well as the 2.0 branch
  • this affects Debian 10 (python 3.7) and Ubuntu 18.04 (python 3.6)
  • CentOS 7 (python 3.6), Debian 9 (python 3.5) and Ubuntu 16.04 (python 3.5) are not affected

What's happening is that /usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py goes out of its way to load the system distutils to get to stdlib. On Debian 10, the (partial) system python3 distutils comes from the libpython3.7-stdlib package, which does not include dist.py. Boom.

It's worth noting that it works fine on Debian 9 and Ubuntu 1604, where the (partial) system python3 distutils comes from the libpython3.5-stdlib package, which includes dist.py.

The workaround is easy: install the python3-distutils systemwide to provide the necessary dist.py. I added it as a dependency on our python3-arvados-cwl-runner packages for Debian 10+ and Ubuntu 18.04+.

#4 Updated by Anonymous about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#5 Updated by Ward Vandewege about 2 months ago

  • Subject changed from python3-arvados-cwl-runner package broken on buster to python3-arvados-cwl-runner package broken on Debian 10 and Ubuntu 18.04

#6 Updated by Ward Vandewege about 1 month ago

  • Release set to 33

Also available in: Atom PDF