Bug #20754: docker-cleaner possibly broken - Arvados

Actions

Copy link

Bug #20754

closed

docker-cleaner possibly broken

Added by Tom Clegg 9 months ago. Updated 8 months ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Brett Smith

Category:

Docker

Target version:

Development 2023-08-16

Story points:

2.0

Release:

Arvados 2.7.0

Release relationship:

Auto

Description

observed in systemd journal on a pirca compute instance while troubleshooting something else:

Stopped Arvados Docker Image Cleaner.
Started Arvados Docker Image Cleaner.
Traceback (most recent call last):
  File "/usr/bin/arvados-docker-cleaner", line 8, in <module>
    sys.exit(main())
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 365, in main
    run(config, docker.Client(version='1.14'))
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 345, in run
    images = DockerImages.from_daemon(config['Quota'], docker_client)
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 68, in from_daemon
    for image in docker_client.images():
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/api/image.py", line 39, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/utils/decorators.py", line 47, in inner
    return f(self, *args, **kwargs)
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/client.py", line 112, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'
arvados-docker-cleaner.service: Main process exited, code=exited, status=1/FAILURE
arvados-docker-cleaner.service: Failed with result 'exit-code'.

It seems to be in a crash-and-restart loop.

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Peter Amstutz 9 months ago

Story points set to 2.0
Target version changed from Future to Development 2023-08-16
Assigned To set to Brett Smith

Actions

Copy link

Updated by Peter Amstutz 9 months ago

Category changed from Docker to Docker
Project changed from 35 to Arvados

Actions

Copy link

Updated by Brett Smith 9 months ago

Assuming this is installed from the bullseye deb for arvados-docker-cleaner-2.6.3, versions are:

arvados_docker_cleaner-2.6.3
certifi-2023.5.7
charset_normalizer-3.1.0
docker_py-1.7.2
idna-3.4
pip-23.1.2
requests-2.31.0
setuptools-44.1.1
six-1.16.0
urllib3-2.0.3
websocket_client-1.5.2
wheel-0.40.0

Actions

Copy link

Updated by Brett Smith 9 months ago

The shortest fix is to "pin urllib3<2.0":https://github.com/docker/docker-py/issues/3113.

But holy crap y'all, we currently have the Docker library we used pinned to one specific version, and I committed that pin last time I worked here, in 95253d987794df9c27db93b1b9f8be2d32229309.

I would at least like to try an upgrade. The library has a changelog that conveniently labels Breaking changes.

They supported Python 3.6 until version 6.0, we can start there.
Their minimum Docker API version is 1.21, ours is 1.35.
From a quick skim, it looks like there will be some API renames to work through, but that's it. While there are other breaking changes documented like return type changes, I didn't notice any that pertain to the images API the image cleaner uses.

As long as I haven't missed anything major this should be doable with a story point-ish.

Actions

Copy link

Updated by Brett Smith 9 months ago

Brett Smith wrote in #note-4:

I would at least like to try an upgrade. The library has a changelog that conveniently labels Breaking changes.

Done in 20754-docker-py-upgrade @ c071d7aaed53241f68bad09d8dd8d20b21b8bb40 - developer-run-tests: #3761

The switch from docker-py to docker is expected, the module itself renamed. See their 2.0.0 changelog
The updates to the tests are mostly to get mocks to match API changes in the module.
Because there are no "real" tests, I did run the service manually on my system, and it seemed to work fine. It removed containers, it deleted images, it followed some events.

Actions

Copy link