Bug #5246
closed
[OPS] Write to keep.qr1hi.arvadosapi.com raising OpenSSL.SSL.WantWriteError when trying to put a block more than 108 KiB
Description
WTF?
$ dd if=/dev/zero of=zeros bs=1K count=109
109+0 records in
109+0 records out
111616 bytes (112 kB) copied, 0.00175415 s, 63.6 MB/s
peter@peter:[pts/7]:~
$ arv-put zeros
0M / 0M 0.0% Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 370, in run
self.run_with_limiter(limiter)
File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 382, in run_with_limiter
timeout=self.args.get('timeout', None)))
File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 338, in put
timeout=timeout)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 105, in put
return request('put', url, data=data, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in send
timeout=timeout
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
body=body, headers=headers)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 308, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1001, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 208, in sendall
return self.connection.sendall(data)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 977, in sendall
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 849, in _raise_ssl_error
raise WantWriteError()
WantWriteError
peter@peter:[pts/7]:~
$ dd if=/dev/zero of=zeros bs=1K count=108
108+0 records in
108+0 records out
110592 bytes (111 kB) copied, 0.000724505 s, 153 MB/s
peter@peter:[pts/7]:~
$ arv-put zeros
0M / 0M 100.0%
Collection saved as 'Saved at 2015-02-18 16:44:25 UTC by peter@peter'
qr1hi-4zz18-603am3us8zzwlrq
Updated by Ward Vandewege almost 10 years ago
- Target version changed from Bug Triage to 2015-03-11 sprint
Updated by Peter Amstutz almost 10 years ago
This appears to be a requests and/or urllib3 bug. The "OpenSSL.SSL.WantWriteError" exception is a non-fatal error that indicates "the socket buffer is full, try again later".
Works (installed from pip in a virtualenv):
- requests 2.5.1
- urllib3 1.10.1
Doesn't work (default Debian packages in Jesse):
- python-requests 2.4.3-4
- python-urllib3 1.9.1-3
Updated by Peter Amstutz almost 10 years ago
Tracked it down. The bug is in urllib3 and is fixed in 1.10.
Updated by Peter Amstutz almost 10 years ago
For reference, the github issue for this bug was:
Updated by Brett Smith over 9 years ago
Peter Amstutz wrote:
Doesn't work (default Debian packages in Jesse):
- python-requests 2.4.3-4
- python-urllib3 1.9.1-3
Can't reproduce on a squeeze system inside virtualenv:
(arv)locke % pip freeze | grep -e urllib -e requests requests==2.4.3 urllib3==1.9.1 (arv)locke % dd if=/dev/zero of=/tmp/zeroes bs=1K count=109 109+0 records in 109+0 records out 111616 bytes (112 kB) copied, 0.00330214 s, 33.8 MB/s (arv)locke % arv-put /tmp/zeroes 0M / 0M 100.0% Collection saved as 'Saved at 2015-02-23 03:53:49 UTC by brett@locke' qr1hi-4zz18-5dj10r61chli72r
I care because I don't want to require users to install a very modern library unless it's absolutely necessary. We've already had a user try to run with Python 2.6. How do you think a Python library less than a year old is going to fly?
The bug report you linked to is compelling, but it seems like this might also depend on the version of OpenSSL installed. Let's hash out how we want to tackle this.
Updated by Peter Amstutz over 9 years ago
If you don't have the OpenSSL wrapper that it wants to use, it will silently disable the feature. That might be what is happening in your test. This OpenSSL feature is optional in urllib3 but is enabled by default by requests on import. Possibly we could figure out some way to hack requests so that it doesn't use it.
Updated by Peter Amstutz over 9 years ago
From requests init.py
try:
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
except ImportError:
pass
There is pyopenssl.extract_from_urllib3() so possibly in keep.py we could version test urllib3 and remove openssl support for versions < 1.10.
Updated by Brett Smith over 9 years ago
Reviewing 0b4ad564. I'm fine with the fix in principle.
The version-parsing code isn't rich enough. As one example, it considers version 2.0 to be buggy. pkg_resources.parse_version parses PEP 440 version strings and gives you back comparable tuples. Suggest the following implementation:
try:
import urllib3
from pkg_resources import parse_version
if parse_version(urllib3.__version__) < parse_version('1.10'):
from urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()
except ImportError:
pass
In the comment:
- For future readers, let's be more specific: s/a certain major Linux distribution/Debian jessie/.
- "stablizing" is a typo (should be "stabilizing").
Thanks.
Updated by Peter Amstutz over 9 years ago
Brett Smith wrote:
Reviewing 0b4ad564. I'm fine with the fix in principle.
The version-parsing code isn't rich enough. As one example, it considers version 2.0 to be buggy. pkg_resources.parse_version parses PEP 440 version strings and gives you back comparable tuples. Suggest the following implementation:
Your solution is much nicer. Done.
In the comment:
- For future readers, let's be more specific: s/a certain major Linux distribution/Debian jessie/.
- "stablizing" is a typo (should be "stabilizing").
Fixed.
Updated by Peter Amstutz over 9 years ago
- Status changed from New to Resolved
- % Done changed from 50 to 100
Applied in changeset arvados|commit:3979c83819a07b544aa4a0510bbeb58d1c92905a.