Idea #7927
closed
[Crunch] arvados-docker-cleaner specifies a Docker API version
Description
arvados-docker-cleaner currently instantiates docker.Client() without any arguments. In this case, the docker module has a hardcoded constant API version that it tries to use. If that version is newer than the server supports, the connection will fail.
Determine the lowest API version that arvados-docker-cleaner can use, and pass that string as a version kwarg to docker.Client(). If that's too difficult to do, or causes other problems, consider also specifying version='auto'—this will use the server's native API version.
Updated by Ward Vandewege almost 9 years ago
I'm also seeing this on a fresh Wheezy install with docker 1.8.2 and arvados-docker-cleaner 0.1.20151104051940.
2015-12-04_20:47:31.17790 Starting arvados-docker-cleaner from /etc/sv/arvados-docker-cleaner
2015-12-04_20:47:31.27240 Traceback (most recent call last):
2015-12-04_20:47:31.27241 File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 138, in _raise_for_status
2015-12-04_20:47:31.27241 response.raise_for_status()
2015-12-04_20:47:31.27241 File "/usr/local/lib/python3.2/dist-packages/requests/models.py", line 837, in raise_for_status
2015-12-04_20:47:31.27241 raise HTTPError(http_error_msg, response=self)
2015-12-04_20:47:31.27242 requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localunixsocket/v1.21/images/json?only_ids=0&all=0
2015-12-04_20:47:31.27242
2015-12-04_20:47:31.27242 During handling of the above exception, another exception occurred:
2015-12-04_20:47:31.27242
2015-12-04_20:47:31.27243 Traceback (most recent call last):
2015-12-04_20:47:31.27243 File "/usr/lib/python3.2/runpy.py", line 161, in _run_module_as_main
2015-12-04_20:47:31.27243 "__main__", fname, loader, pkg_name)
2015-12-04_20:47:31.27243 File "/usr/lib/python3.2/runpy.py", line 74, in _run_code
2015-12-04_20:47:31.27243 exec(code, run_globals)
2015-12-04_20:47:31.27244 File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 294, in <module>
2015-12-04_20:47:31.27244 main(sys.argv[1:])
2015-12-04_20:47:31.27244 File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 291, in main
2015-12-04_20:47:31.27244 run(args, docker.Client())
2015-12-04_20:47:31.27245 File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 274, in run
2015-12-04_20:47:31.27246 images = DockerImages.from_daemon(args.quota, docker_client)
2015-12-04_20:47:31.27246 File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 57, in from_daemon
2015-12-04_20:47:31.27246 for image in docker_client.images():
2015-12-04_20:47:31.27246 File "/usr/local/lib/python3.2/dist-packages/docker/api/image.py", line 40, in images
2015-12-04_20:47:31.27247 True)
2015-12-04_20:47:31.27247 File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 146, in _result
2015-12-04_20:47:31.27247 self._raise_for_status(response)
2015-12-04_20:47:31.27247 File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 142, in _raise_for_status
2015-12-04_20:47:31.27248 raise errors.APIError(e, response, explanation=explanation)
2015-12-04_20:47:31.27248 docker.errors.APIError: 400 Client Error: Bad Request ("b'client is newer than server (client API version: 1.21, server API version: 1.20)'")
2015-12-04_20:47:31.27988 Stopping arvados-docker-cleaner
Updated by Brett Smith almost 9 years ago
Declaring the version we know we want is ideal. See what the oldest version(s) supported by current Docker are. If they're old enough to be supported by much-older Docker (like, around 1.3), declare that. Otherwise, declare 'auto'.
Updated by Brett Smith almost 9 years ago
- Target version set to Arvados Future Sprints
Updated by Brett Smith almost 9 years ago
- Status changed from New to In Progress
- Assigned To set to Brett Smith
- Target version changed from Arvados Future Sprints to 2016-01-06 sprint
Updated by Nico César almost 9 years ago
mmmh test are failing-
https://ci.curoverse.com/job/developer-test-job/74/consoleFull
1) Failure: TestCrunchJob#test_fail_clean_tmp [/data/1/jenkins/workspace/developer-test-job/sdk/cli/test/test_crunch-job.rb:94]: Expected /Clean work dirs: exit 1\n$/ to match " 12841 Sanity check is `true`\n 12841 starting: ['true']\n 12841 Sanity check OK\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 running from /data/1/jenkins/workspace/developer-test-job/sdk/cli/bin/crunch-job with arvados-cli Gem version(s) 0.1.20151207150126\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 check slurm allocation\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 node localhost - 2 slots\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 start\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 Clean work dirs\nstarting: ['bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk \"(index(\\\\$3, \\\\\"$CRUNCH_TMP\\\\\") == 1){print \\\\$3}\" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']\nFailing mount stub was called\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841 Clean work dirs: exit 1\nclose failed in file object destructor:\nsys.excepthook is missing\nlost sys.stderr\n".
Updated by Brett Smith almost 9 years ago
Nico Cesar wrote:
mmmh test are failing-
https://ci.curoverse.com/job/developer-test-job/74/consoleFull
This is caused by a race condition in a Python SDK tool that sometimes causes it to write some extra errors as it exits. It's not related to this change: Docker cleaner doesn't use the SDK at all, and vice versa. I'm going to push accordingly. Thanks.
Updated by Brett Smith almost 9 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:ace5807988dd1db2e8bd63a788fb0f0d9da152d7.