Bug #20754
closeddocker-cleaner possibly broken
Description
observed in systemd journal on a pirca compute instance while troubleshooting something else:
Stopped Arvados Docker Image Cleaner. Started Arvados Docker Image Cleaner. Traceback (most recent call last): File "/usr/bin/arvados-docker-cleaner", line 8, in <module> sys.exit(main()) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 365, in main run(config, docker.Client(version='1.14')) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 345, in run images = DockerImages.from_daemon(config['Quota'], docker_client) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 68, in from_daemon for image in docker_client.images(): File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/api/image.py", line 39, in images res = self._result(self._get(self._url("/images/json"), params=params), File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/utils/decorators.py", line 47, in inner return f(self, *args, **kwargs) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/client.py", line 112, in _get return self.get(url, **self._set_request_timeout(kwargs)) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( TypeError: request() got an unexpected keyword argument 'chunked' arvados-docker-cleaner.service: Main process exited, code=exited, status=1/FAILURE arvados-docker-cleaner.service: Failed with result 'exit-code'.
It seems to be in a crash-and-restart loop.
Updated by Peter Amstutz 12 months ago
- Story points set to 2.0
- Target version changed from Future to Development 2023-08-16
- Assigned To set to Brett Smith
Updated by Peter Amstutz 12 months ago
- Category changed from Docker to Docker
- Project changed from 35 to Arvados
Updated by Brett Smith 12 months ago
Assuming this is installed from the bullseye deb for arvados-docker-cleaner-2.6.3, versions are:
arvados_docker_cleaner-2.6.3
certifi-2023.5.7
charset_normalizer-3.1.0
docker_py-1.7.2
idna-3.4
pip-23.1.2
requests-2.31.0
setuptools-44.1.1
six-1.16.0
urllib3-2.0.3
websocket_client-1.5.2
wheel-0.40.0
Updated by Brett Smith 12 months ago
The shortest fix is to "pin urllib3<2.0":https://github.com/docker/docker-py/issues/3113.
But holy crap y'all, we currently have the Docker library we used pinned to one specific version, and I committed that pin last time I worked here, in 95253d987794df9c27db93b1b9f8be2d32229309.
I would at least like to try an upgrade. The library has a changelog that conveniently labels Breaking changes.
- They supported Python 3.6 until version 6.0, we can start there.
- Their minimum Docker API version is 1.21, ours is 1.35.
- From a quick skim, it looks like there will be some API renames to work through, but that's it. While there are other breaking changes documented like return type changes, I didn't notice any that pertain to the images API the image cleaner uses.
As long as I haven't missed anything major this should be doable with a story point-ish.
Updated by Brett Smith 12 months ago
Brett Smith wrote in #note-4:
I would at least like to try an upgrade. The library has a changelog that conveniently labels Breaking changes.
Done in 20754-docker-py-upgrade @ c071d7aaed53241f68bad09d8dd8d20b21b8bb40 - developer-run-tests: #3761
- The switch from
docker-py
todocker
is expected, the module itself renamed. See their 2.0.0 changelog - The updates to the tests are mostly to get mocks to match API changes in the module.
- Because there are no "real" tests, I did run the service manually on my system, and it seemed to work fine. It removed containers, it deleted images, it followed some events.
Updated by Brett Smith 12 months ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|912fd82a23277fc4f0dbd2c1587c8dfd6af52aa5.