Bug #12215

arvados-docker-cleaner broken on ubuntu precise (12.04)

Added by Joshua Randall 3 months ago. Updated 3 months ago.

Status:NewStart date:09/04/2017
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Docker
Target version:-
Story points-
Velocity based estimate-

Description

arvados-docker-cleaner does not start on trusty (ubuntu 12.04).

this appears to be because one of its dependencies has a dependency that does not work with python 3.2 (which is the newest version of python3 on trusty).

docker-py seems to require idna, and the version we have is incompatible with the system python and results in a stack trace like this:

 python3 -m arvados_docker.cleaner --quota 250G
Traceback (most recent call last):
  File "/usr/lib/python3.2/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python3.2/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 17, in <module>
    import docker
  File "/usr/lib/python3.2/dist-packages/docker/__init__.py", line 20, in <module>
    from .client import Client, AutoVersionClient # flake8: noqa
  File "/usr/lib/python3.2/dist-packages/docker/client.py", line 19, in <module>
    import requests
  File "/usr/lib/python3.2/dist-packages/requests/__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/usr/lib/python3.2/dist-packages/requests/packages/__init__.py", line 39, in <module>
    from . import idna
  File "/usr/lib/python3.2/dist-packages/requests/packages/idna/__init__.py", line 1, in <module>
    from .core import *
  File "/usr/lib/python3.2/dist-packages/requests/packages/idna/core.py", line 10
    _unicode_dots_re = re.compile(u'[\u002e\u3002\uff0e\uff61]')
                                                              ^
SyntaxError: invalid syntax

History

#1 Updated by Joshua Randall 3 months ago

My guess at a fix would be to pin the version of python3-docker-py built on trusty so that it uses an older version of idna (maybe that means installing idna separately and pinning that?)

https://github.com/curoverse/arvados/blob/32f2a67ecfd7a5b4169d4e3fbc162306771f0d34/jenkins/run-build-packages.sh#L487

#2 Updated by Joshua Randall 3 months ago

Actually it looks like python 3.4 is available on trusty but it isn't required by docker-py (which apparently should require python3 > 3.2).

#3 Updated by Joshua Randall 3 months ago

haha, right - I meant to say precise (12.04) on which the newest python is indeed 3.2 - ARGH!

#4 Updated by Joshua Randall 3 months ago

  • Subject changed from arvados-docker-cleaner broken on trusty to arvados-docker-cleaner broken on ubuntu precise (12.04)

#5 Updated by Joshua Randall 3 months ago

I've installed python 3.6 from source on all humgen-0x-0x nodes and arvados-docker-cleaner is now running out of a virtualenv, so our ~18TB of docker crap is now slowly being cleaned up and this is no longer an emergency for us

Also available in: Atom PDF