Story #7927

[Crunch] arvados-docker-cleaner specifies a Docker API version

Added by Brett Smith almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
Crunch
Target version:
Start date:
12/16/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

arvados-docker-cleaner currently instantiates docker.Client() without any arguments. In this case, the docker module has a hardcoded constant API version that it tries to use. If that version is newer than the server supports, the connection will fail.

Determine the lowest API version that arvados-docker-cleaner can use, and pass that string as a version kwarg to docker.Client(). If that's too difficult to do, or causes other problems, consider also specifying version='auto'—this will use the server's native API version.


Subtasks

Task #8025: Review 7927-docker-cleaner-api-version-wipResolvedNico César

Associated revisions

Revision ace58079
Added by Brett Smith almost 5 years ago

Merge branch '7927-docker-cleaner-api-version-wip'

Closes #7927, #8025.

History

#1 Updated by Ward Vandewege almost 5 years ago

I'm also seeing this on a fresh Wheezy install with docker 1.8.2 and arvados-docker-cleaner 0.1.20151104051940.

2015-12-04_20:47:31.17790 Starting arvados-docker-cleaner from /etc/sv/arvados-docker-cleaner
2015-12-04_20:47:31.27240 Traceback (most recent call last):
2015-12-04_20:47:31.27241   File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 138, in _raise_for_status
2015-12-04_20:47:31.27241     response.raise_for_status()
2015-12-04_20:47:31.27241   File "/usr/local/lib/python3.2/dist-packages/requests/models.py", line 837, in raise_for_status
2015-12-04_20:47:31.27241     raise HTTPError(http_error_msg, response=self)
2015-12-04_20:47:31.27242 requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localunixsocket/v1.21/images/json?only_ids=0&all=0
2015-12-04_20:47:31.27242 
2015-12-04_20:47:31.27242 During handling of the above exception, another exception occurred:
2015-12-04_20:47:31.27242 
2015-12-04_20:47:31.27243 Traceback (most recent call last):
2015-12-04_20:47:31.27243   File "/usr/lib/python3.2/runpy.py", line 161, in _run_module_as_main
2015-12-04_20:47:31.27243     "__main__", fname, loader, pkg_name)
2015-12-04_20:47:31.27243   File "/usr/lib/python3.2/runpy.py", line 74, in _run_code
2015-12-04_20:47:31.27243     exec(code, run_globals)
2015-12-04_20:47:31.27244   File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 294, in <module>
2015-12-04_20:47:31.27244     main(sys.argv[1:])
2015-12-04_20:47:31.27244   File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 291, in main
2015-12-04_20:47:31.27244     run(args, docker.Client())
2015-12-04_20:47:31.27245   File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 274, in run
2015-12-04_20:47:31.27246     images = DockerImages.from_daemon(args.quota, docker_client)
2015-12-04_20:47:31.27246   File "/usr/local/lib/python3.2/dist-packages/arvados_docker/cleaner.py", line 57, in from_daemon
2015-12-04_20:47:31.27246     for image in docker_client.images():
2015-12-04_20:47:31.27246   File "/usr/local/lib/python3.2/dist-packages/docker/api/image.py", line 40, in images
2015-12-04_20:47:31.27247     True)
2015-12-04_20:47:31.27247   File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 146, in _result
2015-12-04_20:47:31.27247     self._raise_for_status(response)
2015-12-04_20:47:31.27247   File "/usr/local/lib/python3.2/dist-packages/docker/client.py", line 142, in _raise_for_status
2015-12-04_20:47:31.27248     raise errors.APIError(e, response, explanation=explanation)
2015-12-04_20:47:31.27248 docker.errors.APIError: 400 Client Error: Bad Request ("b'client is newer than server (client API version: 1.21, server API version: 1.20)'")
2015-12-04_20:47:31.27988 Stopping arvados-docker-cleaner

#2 Updated by Brett Smith almost 5 years ago

Declaring the version we know we want is ideal. See what the oldest version(s) supported by current Docker are. If they're old enough to be supported by much-older Docker (like, around 1.3), declare that. Otherwise, declare 'auto'.

#3 Updated by Brett Smith almost 5 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Brett Smith almost 5 years ago

  • Story points set to 0.5

#5 Updated by Brett Smith almost 5 years ago

  • Status changed from New to In Progress
  • Assigned To set to Brett Smith
  • Target version changed from Arvados Future Sprints to 2016-01-06 sprint

#6 Updated by Nico César almost 5 years ago

mmmh test are failing-

https://ci.curoverse.com/job/developer-test-job/74/consoleFull

  1) Failure:
TestCrunchJob#test_fail_clean_tmp [/data/1/jenkins/workspace/developer-test-job/sdk/cli/test/test_crunch-job.rb:94]:
Expected /Clean work dirs: exit 1\n$/ to match " 12841  Sanity check is `true`\n 12841  starting: ['true']\n 12841  Sanity check OK\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  running from /data/1/jenkins/workspace/developer-test-job/sdk/cli/bin/crunch-job with arvados-cli Gem version(s) 0.1.20151207150126\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  check slurm allocation\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  node localhost - 2 slots\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  start\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  Clean work dirs\nstarting: ['bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk \"(index(\\\\$3, \\\\\"$CRUNCH_TMP\\\\\") == 1){print \\\\$3}\" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']\nFailing mount stub was called\nzzzzz-8i9sb-eyhwnlas72ojxs8 12841  Clean work dirs: exit 1\nclose failed in file object destructor:\nsys.excepthook is missing\nlost sys.stderr\n".

#7 Updated by Brett Smith almost 5 years ago

Nico Cesar wrote:

mmmh test are failing-

https://ci.curoverse.com/job/developer-test-job/74/consoleFull

This is caused by a race condition in a Python SDK tool that sometimes causes it to write some extra errors as it exits. It's not related to this change: Docker cleaner doesn't use the SDK at all, and vice versa. I'm going to push accordingly. Thanks.

#8 Updated by Brett Smith almost 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:ace5807988dd1db2e8bd63a788fb0f0d9da152d7.

Also available in: Atom PDF