Arvados

Re-ran the job and got the same error: qr1hi-8i9sb-77epylk1ecakg3a

Was able to recreate the error. Notes:

• This is a severely degenerate manifest. There are many instances of sequential file segments that could be collapsed. There is also interleaving of writes to multiple files in the same block which greatly bloats the number of segments required to represent the file. The same information could probably be represented by a manifest 10% of the size (or even smaller.)
• The manifest text is successfully written to Keep. The failure is when trying to create the collection record.
• I can't find anything in the API server logs indicating that the request made it to the Rails code. Along with the 500 error code, this suggests it is getting rejected earlier by the nginx proxy (?) or in passenger.

Here's the log:

10.13.26.53 - - [23/Feb/2015:19:02:00 +0000] "POST /arvados/v1/collections?alt=json HTTP/1.1" 500 192 "-" "google-api-python-client/1.3.1 (gzip)" 

Whoops.

This manifest isn't 28 MiB. It's at least two orders of magnitude larger than that. The 28 MiB manifest is only the first of 60 tasks. The actual requests concatenates all those tasks and results in a manifest that is around 887 MiB.

Analysis:

There are two underlying causes here:

1. CollectionWriter was used in a way that created fragmented manifests which were orders of magnitude larger than necessary
2. Failure to have explicit limits on how large a single API request and/or manifest is allowed to be, so that instead of getting specific errors (413 This manifest is too large) we get generic errors (500 Internal Server Error) that are much harder to debug.

The test in sdk/python/tests/test_api.py would be a bit more stable if it looked up the request limit in the discovery doc, instead of assuming 128 MiB. (If it's an error when limit>128MiB then we could check for that on the API side in rake config:check... but I'm assuming it isn't.)

If I'm following correctly, this added line would make more sense in __init__() than in __authorize__():

     def authorize(self, http):
         http.arvados_api_token = self.api_token
         http.orig_http_request = http.request
         http.request = types.MethodType(self.http_request, http)
+        http.max_request_size = 0
         return http

The comment in services/api/config/application.default.yml would probably be a good place to add a note about enforcement. Perhaps:

"Note this limit only affects clients that pay attention to it. Therefore we also recommend configuring the web server, or an upstream proxy server, to enforce a request size limit."

Everything else LGTM.