Bug #5246

[OPS] Write to keep.qr1hi.arvadosapi.com raising OpenSSL.SSL.WantWriteError when trying to put a block more than 108 KiB

Added by Peter Amstutz almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/19/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

WTF?

$ dd if=/dev/zero of=zeros bs=1K count=109
109+0 records in
109+0 records out
111616 bytes (112 kB) copied, 0.00175415 s, 63.6 MB/s
peter@peter:[pts/7]:~
$ arv-put zeros 
0M / 0M 0.0% Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 370, in run
    self.run_with_limiter(limiter)
  File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 382, in run_with_limiter
    timeout=self.args.get('timeout', None)))
  File "/home/peter/work/arvados/sdk/python/arvados/keep.py", line 338, in put
    timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 105, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 308, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 208, in sendall
    return self.connection.sendall(data)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 977, in sendall
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 849, in _raise_ssl_error
    raise WantWriteError()
WantWriteError

peter@peter:[pts/7]:~
$ dd if=/dev/zero of=zeros bs=1K count=108
108+0 records in
108+0 records out
110592 bytes (111 kB) copied, 0.000724505 s, 153 MB/s
peter@peter:[pts/7]:~
$ arv-put zeros 
0M / 0M 100.0% 
Collection saved as 'Saved at 2015-02-18 16:44:25 UTC by peter@peter'
qr1hi-4zz18-603am3us8zzwlrq

Subtasks

Task #5367: Review 5246-urllib3-workaroundResolvedPeter Amstutz

Task #5323: Find another workaroundResolvedPeter Amstutz

Task #5268: Investigate furtherResolvedPeter Amstutz

Task #5280: Review 5246-update-urllib3ClosedPeter Amstutz

Associated revisions

Revision 71aa90b8 (diff)
Added by Peter Amstutz almost 6 years ago

5246: Add minimum version constraint on urllib3 because of bug #5246

Revision 3979c838
Added by Peter Amstutz over 5 years ago

Merge branch '5246-urllib3-workaround' closes #5246

History

#1 Updated by Peter Amstutz almost 6 years ago

  • Description updated (diff)

#2 Updated by Ward Vandewege almost 6 years ago

  • Target version changed from Bug Triage to 2015-03-11 sprint

#3 Updated by Ward Vandewege almost 6 years ago

  • Story points set to 0.5

#4 Updated by Peter Amstutz almost 6 years ago

  • Assigned To set to Peter Amstutz

#5 Updated by Peter Amstutz almost 6 years ago

This appears to be a requests and/or urllib3 bug. The "OpenSSL.SSL.WantWriteError" exception is a non-fatal error that indicates "the socket buffer is full, try again later".

Works (installed from pip in a virtualenv):

  • requests 2.5.1
  • urllib3 1.10.1

Doesn't work (default Debian packages in Jesse):

  • python-requests 2.4.3-4
  • python-urllib3 1.9.1-3

#6 Updated by Peter Amstutz almost 6 years ago

Tracked it down. The bug is in urllib3 and is fixed in 1.10.

#7 Updated by Peter Amstutz almost 6 years ago

For reference, the github issue for this bug was:

https://github.com/shazow/urllib3/issues/412

#8 Updated by Brett Smith almost 6 years ago

Peter Amstutz wrote:

Doesn't work (default Debian packages in Jesse):

  • python-requests 2.4.3-4
  • python-urllib3 1.9.1-3

Can't reproduce on a squeeze system inside virtualenv:

(arv)locke % pip freeze | grep -e urllib -e requests
requests==2.4.3
urllib3==1.9.1
(arv)locke % dd if=/dev/zero of=/tmp/zeroes bs=1K count=109
109+0 records in
109+0 records out
111616 bytes (112 kB) copied, 0.00330214 s, 33.8 MB/s
(arv)locke % arv-put /tmp/zeroes
0M / 0M 100.0%
Collection saved as 'Saved at 2015-02-23 03:53:49 UTC by brett@locke'
qr1hi-4zz18-5dj10r61chli72r

I care because I don't want to require users to install a very modern library unless it's absolutely necessary. We've already had a user try to run with Python 2.6. How do you think a Python library less than a year old is going to fly?

The bug report you linked to is compelling, but it seems like this might also depend on the version of OpenSSL installed. Let's hash out how we want to tackle this.

#9 Updated by Peter Amstutz almost 6 years ago

If you don't have the OpenSSL wrapper that it wants to use, it will silently disable the feature. That might be what is happening in your test. This OpenSSL feature is optional in urllib3 but is enabled by default by requests on import. Possibly we could figure out some way to hack requests so that it doesn't use it.

#10 Updated by Peter Amstutz almost 6 years ago

From requests init.py

try:
    from urllib3.contrib import pyopenssl
    pyopenssl.inject_into_urllib3()
except ImportError:
    pass

There is pyopenssl.extract_from_urllib3() so possibly in keep.py we could version test urllib3 and remove openssl support for versions < 1.10.

#11 Updated by Brett Smith over 5 years ago

Reviewing 0b4ad564. I'm fine with the fix in principle.

The version-parsing code isn't rich enough. As one example, it considers version 2.0 to be buggy. pkg_resources.parse_version parses PEP 440 version strings and gives you back comparable tuples. Suggest the following implementation:

try:
    import urllib3
    from pkg_resources import parse_version
    if parse_version(urllib3.__version__) < parse_version('1.10'):
        from urllib3.contrib import pyopenssl
        pyopenssl.extract_from_urllib3()
except ImportError:
    pass

In the comment:

  • For future readers, let's be more specific: s/a certain major Linux distribution/Debian jessie/.
  • "stablizing" is a typo (should be "stabilizing").

Thanks.

#12 Updated by Peter Amstutz over 5 years ago

Brett Smith wrote:

Reviewing 0b4ad564. I'm fine with the fix in principle.

The version-parsing code isn't rich enough. As one example, it considers version 2.0 to be buggy. pkg_resources.parse_version parses PEP 440 version strings and gives you back comparable tuples. Suggest the following implementation:

Your solution is much nicer. Done.

In the comment:

  • For future readers, let's be more specific: s/a certain major Linux distribution/Debian jessie/.
  • "stablizing" is a typo (should be "stabilizing").

Fixed.

#13 Updated by Brett Smith over 5 years ago

b434a3a8 is good to merge. Thanks.

#14 Updated by Peter Amstutz over 5 years ago

  • Status changed from New to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:3979c83819a07b544aa4a0510bbeb58d1c92905a.

Also available in: Atom PDF