Arvados

Testing Keepstore. Uploading with 'arv-put' midsize files like up to a few GB works.

First attempt to upload 60GB fails with 'arv-put' crashing at the 100% mark. Attempts to upload the same 60GB file also crash at the same point. Second and further attempts as expected show only read activity on the Keepstores: all blobs have been uploaded.

Context: recently installed, freshly updated setup. Not using SSO but direct token.

From API server log:

Started PUT "/arvados/v1/users/gcam1-tpzed-42l58gq9xqdzxkb" for 127.0.0.1 at 2015-10-13 14:52:32 +0000
Processing by Arvados::V1::UsersController#update as */*
  Parameters: {"api_token"=>"60l1om9jukg1y7qpu1a6uqeevd29zc9rruqe0yc3anbg3k6b7f", "reader_tokens"=>"[false]", "user"=>"{\"first_name\":\"Librarian\",\"prefs\":{\"getting_started_shown\":\"2015-09-09T12:05:04.170+00:00\"}}", "id"=>"gcam1-tpzed-42l58gq9xqdzxkb"}
WARNING: Can't verify CSRF token authenticity
  Rendered text template (0.0ms)
Completed 200 OK in 65.3ms (Views: 0.6ms | ActiveRecord: 9.3ms)

The crash report itself:

$ arv-put --replication 1 --no-resume --project-uuid gcam1-j7d0g-soxcyrmt3m87u2z --name dbNSFP2.5 dbNSFP2.5/
    60492M / 60492M 100.0% 
    Traceback (most recent call last):
      File "/usr/local/bin/arv-put", line 4, in <module> main()
      File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 517, in main
        ).execute(num_retries=args.retries)
      File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 142, in positional_wrapper
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 722, in execute
        body=self.body, headers=self.headers)
      File "/usr/local/lib/python2.7/dist-packages/arvados/api.py", line 54, in _intercept_http_request
        return self.orig_http_request(uri, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
        (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
        (response, content) = self._conn_request(conn, request_uri, method, body, headers)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1273, in _conn_request
        conn.request(method, request_uri, body, headers)
      File "/usr/lib/python2.7/httplib.py", line 979, in request
        self._send_request(method, url, body, headers)
      File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
        self.endheaders(body)
      File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
        self._send_output(message_body)
      File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
        self.send(msg)
      File "/usr/lib/python2.7/httplib.py", line 811, in send
        self.sock.sendall(data)
      File "/usr/lib/python2.7/ssl.py", line 329, in sendall
        v = self.send(data[count:])
      File "/usr/lib/python2.7/ssl.py", line 298, in send
        v = self._sslobj.write(data)
    socket.error: [Errno 32] Broken pipe

My current wild guess is some kind of timeout.