Project

General

Profile

Actions

Bug #16434

closed

python3-arvados-cwl-runner package broken on Debian 10 and Ubuntu 18.04

Added by Ward Vandewege over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-
Release relationship:
Auto

Description

This is version 2.0.2 on a clean buster install:

21:27:37 + cwl-runner hasher-workflow.cwl hasher-workflow-job.yml
21:27:38 Traceback (most recent call last):
21:27:38   File "/usr/bin/cwl-runner", line 8, in <module>
21:27:38     from arvados_cwl import main
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 58, in <module>
21:27:38     import cwltool.main
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 35, in <module>
21:27:38     from . import command_line_tool, workflow
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 38, in <module>
21:27:38     from .docker import DockerCommandLineJob
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/docker.py", line 4, in <module>
21:27:38     from distutils import spawn
21:27:38   File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py", line 44, in <module>
21:27:38     from distutils import dist, sysconfig  # isort:skip
21:27:38 ImportError: cannot import name 'dist' from 'distutils' (/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py)
21:27:38 Build step 'Execute shell' marked build as failure

The python2 version works. Our other distro packages may also be affected. We need to add a test for this in the packaging tests (just add a simple execution of cwl-runner with a fake API host and token).


Related issues

Related to Arvados - Bug #16611: arvados-docker-cleaner package broken on Debian 10 and Ubuntu 18.04ResolvedWard Vandewege

Actions
Actions #1

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege over 2 years ago

  • Target version changed from To Be Groomed to 2020-05-20 Sprint
  • Assigned To set to Ward Vandewege
  • Status changed from New to In Progress

More information:

  • this affects master as well as the 2.0 branch
  • this affects Debian 10 (python 3.7) and Ubuntu 18.04 (python 3.6)
  • CentOS 7 (python 3.6), Debian 9 (python 3.5) and Ubuntu 16.04 (python 3.5) are not affected

What's happening is that /usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/distutils/__init__.py goes out of its way to load the system distutils to get to stdlib. On Debian 10, the (partial) system python3 distutils comes from the libpython3.7-stdlib package, which does not include dist.py. Boom.

It's worth noting that it works fine on Debian 9 and Ubuntu 1604, where the (partial) system python3 distutils comes from the libpython3.5-stdlib package, which includes dist.py.

The workaround is easy: install the python3-distutils systemwide to provide the necessary dist.py. I added it as a dependency on our python3-arvados-cwl-runner packages for Debian 10+ and Ubuntu 18.04+.

Actions #4

Updated by Anonymous over 2 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #5

Updated by Ward Vandewege over 2 years ago

  • Subject changed from python3-arvados-cwl-runner package broken on buster to python3-arvados-cwl-runner package broken on Debian 10 and Ubuntu 18.04
Actions #6

Updated by Ward Vandewege over 2 years ago

  • Release set to 33
Actions #7

Updated by Javier Bértoli over 2 years ago

  • Related to Bug #16611: arvados-docker-cleaner package broken on Debian 10 and Ubuntu 18.04 added
Actions #8

Updated by Ward Vandewege over 2 years ago

The fix on the 2.0-dev branch (in release 2.0.3 and 2.0.4) was botched, but the fix in master is correct. I pushed a fix in 9b6a214f4feb90845640bce53efa34d544c701e9 on the 2.0-dev branch, but we have no plans for a 2.0.5 at the moment. Release 2.1 will have the proper fix.

Actions

Also available in: Atom PDF