Bug #5182

[Workbench] CORS error uploading via browser - better JS error handling needed

Added by Peter Amstutz almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Workbench
Target version:
Start date:
02/09/2015
Due date:
% Done:

67%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

Firefox 35.0.1 on OS X 10.6.8

Trying to upload via browser

Javascript console:

The connection to wss://ws.qr1hi.arvadosapi.com/websocket?api_token=xxx was interrupted while the page was loading. application-1de82a2a4d45ee318cb51cf20380c35d.js:59619
OPTIONS https://keep.qr1hi.arvadosapi.com/ [HTTP/1.1 503 Service Temporarily Unavailable 53ms]
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://keep.qr1hi.arvadosapi.com/. This can be fixed by moving the resource to the same domain or enabling CORS. keep.qr1hi.arvadosapi.com
"slice 0: error, retry 1" application-1de82a2a4d45ee318cb51cf20380c35d.js:63084
OPTIONS https://keep.qr1hi.arvadosapi.com/ [HTTP/1.1 503 Service Temporarily Unavailable 54ms]
"slice 0: error, retry 2" application-1de82a2a4d45ee318cb51cf20380c35d.js:63084

If file upload fails, the user should get an error.

5182-network-error.png (27.4 KB) 5182-network-error.png Tom Clegg, 03/10/2015 02:11 PM
5182-mixed-content-error.png (22.5 KB) 5182-mixed-content-error.png Tom Clegg, 03/10/2015 02:20 PM

Subtasks

Task #5272: Detect CORS failure and display error messageResolvedTom Clegg

Task #5419: Document how to set CORS headers on nginx proxyResolvedTom Clegg

Task #5427: Review 5182-cors-error-reportingResolvedRadhika Chippada


Related issues

Related to Arvados Workbench 2 - Bug #13564: [Workbench] Uploader should have a timeoutNew

Associated revisions

Revision 855a790d
Added by Tom Clegg over 5 years ago

Merge branch '5182-cors-error-reporting' closes #5182

History

#1 Updated by Peter Amstutz almost 6 years ago

Using arvados-python-client-0.1.20141016164709.8297039 ("latest" via pip at this time)

Possibly related?

$ arv-put stuff.rtf 
0M / 0M 0.0% 2015-02-09 13:07:44 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:44 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:44 arvados.keep[1866] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 4624592896)> proceeding a23356ce7051569a2427f55ed78cc176 https://[keep.qr1hi.arvadosapi.com]:443/
2015-02-09 13:07:44 arvados.keep[1866] DEBUG: Request: PUT https://[keep.qr1hi.arvadosapi.com]:443/a23356ce7051569a2427f55ed78cc176
2015-02-09 13:07:44 arvados.keep[1866] DEBUG: Request fail: PUT a23356ce7051569a2427f55ed78cc176 => 503 <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>

2015-02-09 13:07:48 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:48 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:48 arvados.keep[1866] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 4624592896)> proceeding a23356ce7051569a2427f55ed78cc176 https://[keep.qr1hi.arvadosapi.com]:443/
2015-02-09 13:07:48 arvados.keep[1866] DEBUG: Request: PUT https://[keep.qr1hi.arvadosapi.com]:443/a23356ce7051569a2427f55ed78cc176
2015-02-09 13:07:48 arvados.keep[1866] DEBUG: Request fail: PUT a23356ce7051569a2427f55ed78cc176 => 503 <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>

2015-02-09 13:07:56 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:56 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:07:56 arvados.keep[1866] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 4624592896)> proceeding a23356ce7051569a2427f55ed78cc176 https://[keep.qr1hi.arvadosapi.com]:443/
2015-02-09 13:07:56 arvados.keep[1866] DEBUG: Request: PUT https://[keep.qr1hi.arvadosapi.com]:443/a23356ce7051569a2427f55ed78cc176
2015-02-09 13:07:56 arvados.keep[1866] DEBUG: Request fail: PUT a23356ce7051569a2427f55ed78cc176 => 503 <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>

2015-02-09 13:08:12 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:08:12 arvados.keep[1866] DEBUG: ['https://[keep.qr1hi.arvadosapi.com]:443/']
2015-02-09 13:08:12 arvados.keep[1866] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 4624592896)> proceeding a23356ce7051569a2427f55ed78cc176 https://[keep.qr1hi.arvadosapi.com]:443/
2015-02-09 13:08:12 arvados.keep[1866] DEBUG: Request: PUT https://[keep.qr1hi.arvadosapi.com]:443/a23356ce7051569a2427f55ed78cc176
2015-02-09 13:08:12 arvados.keep[1866] DEBUG: Request fail: PUT a23356ce7051569a2427f55ed78cc176 => 503 <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>

Traceback (most recent call last):
  File "/usr/local/bin/arv-put", line 4, in <module>
    main()
  File "/Library/Python/2.7/site-packages/arvados/commands/put.py", line 443, in main
    writer.finish_current_stream()
  File "/Library/Python/2.7/site-packages/arvados/collection.py", line 401, in finish_current_stream
    self.flush_data()
  File "/Library/Python/2.7/site-packages/arvados/commands/put.py", line 291, in flush_data
    super(ArvPutCollectionWriter, self).flush_data()
  File "/Library/Python/2.7/site-packages/arvados/collection.py", line 352, in flush_data
    self._my_keep().put(data_buffer[0:self.KEEP_BLOCK_SIZE]))
  File "/Library/Python/2.7/site-packages/arvados/retry.py", line 157, in num_retries_setter
    return orig_func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/arvados/keep.py", line 708, in put
    (data_hash, copies, thread_limiter.done()))
arvados.errors.KeepWriteError: Write fail for a23356ce7051569a2427f55ed78cc176: wanted 2 but wrote 0

#2 Updated by Peter Amstutz almost 6 years ago

  • Description updated (diff)

#3 Updated by Ward Vandewege almost 6 years ago

We're hitting the keepproxy connection limit. We need better error reporting.

#4 Updated by Ward Vandewege almost 6 years ago

  • Subject changed from CORS error uploading via browser to CORS error uploading via browser - better JS error handling needed

#5 Updated by Ward Vandewege almost 6 years ago

  • Subject changed from CORS error uploading via browser - better JS error handling needed to [Workbench] CORS error uploading via browser - better JS error handling needed

#6 Updated by Ward Vandewege almost 6 years ago

  • Target version changed from Bug Triage to 2015-03-11 sprint

#7 Updated by Tom Clegg almost 6 years ago

  • Category set to Workbench
  • Assigned To set to Tom Clegg

#8 Updated by Ward Vandewege almost 6 years ago

  • Description updated (diff)

#9 Updated by Ward Vandewege almost 6 years ago

  • Story points set to 0.5

#11 Updated by Tom Clegg over 5 years ago

Three improvements in 5182-cors-error-reporting:

  • By design, browsers make the "CORS headers do not permit this request" error indistinguishable from "network error during cross-origin request": i.e., since no CORS headers explicitly permitted the request, the JS program is not allowed to know anything about what happened. We now have a generic "check your console" message for this case, instead of just "error:" like before.

  • If the keep proxy is http:// but Workbench is https://, Workbench JS won't be allowed to connect to the keep proxy. (In Chrome in my dev setup, neither done() nor fail() handler gets called for such a request, which makes it awkward to detect -- fortunately it's easy enough to predict based on the URIs.) We now catch this condition before even bothering to attempt a Keep request:

  • If the keep proxy is itself behind a web proxy like nginx, and the web proxy has trouble connecting to keep proxy (as is the case for this bug report), the web proxy will likely provide a useful error message -- but the browser will avoid passing it to Workbench JS if the web proxy doesn't send CORS headers. Recent versions of nginx have an "add_header ... always" feature that can add CORS headers to error responses. The keepproxy install page has been updated accordingly. 4d71b77

#12 Updated by Tom Clegg over 5 years ago

  • Status changed from New to In Progress

#13 Updated by Radhika Chippada over 5 years ago

Review comments:

Just a couple minor text suggestions.

  • This error message seems too technical: “A network error occurred, or there is a CORS configuration problem”. A typical user may not know what CORS configuration is. Can we say something like: “A network error occurred. The server may be unreachable or this could be a resource sharing issue due to CORS configuration on the server. Please check your browser debug console for a more specific error message (browser security features prevent us from showing the details here).” I think it does not hurt to make the word “CORS” to link to CORS wiki page?
  • Can we make this error message also a bit easier to understand? —— “server setup problem …” - Can we say something like “Mixed content error. Proxy http://… cannot be accessed from origin https://… (highlight http and https?)
  • Documentation “add_header 'Access-Control-Allow-Origin' '*' always” : Can you add an example url instead of “*” here.

#14 Updated by Tom Clegg over 5 years ago

Radhika Chippada wrote:

Review comments:

Just a couple minor text suggestions.

  • This error message seems too technical: “A network error occurred, or there is a CORS configuration problem”. A typical user may not know what CORS configuration is. Can we say something like: “A network error occurred. The server may be unreachable or this could be a resource sharing issue due to CORS configuration on the server. Please check your browser debug console for a more specific error message (browser security features prevent us from showing the details here).” I think it does not hurt to make the word “CORS” to link to CORS wiki page?

Agreed, the word CORS is probably useless to the user. It's a bit of a wild guess anyway: even in this bug report CORS made it harder to see what the real server problem was, but no amount of CORS perfection would have made the upload actually work. Changed to:

A network error occurred: either the server was unreachable, or there is a server configuration problem. Please check [...]

  • Can we make this error message also a bit easier to understand? —— “server setup problem …” - Can we say something like “Mixed content error. Proxy http://… cannot be accessed from origin https://… (highlight http and https?)

Putting html markup on http/https seems to creep quickly into explicit tracking of trusted/untrusted strings throughout all of our status reporting, which doesn't seem worth it. Instead I changed the message to:

There is a server configuration problem. Proxy http://foobar:12345/ cannot be used from origin https://foobar:3031 due to the browser's mixed-content (https/http) policy.

I suppose the most important idea to convey is that it's a server/sysadmin problem, and can't be fixed on the client side. Hopefully "mixed-content (https/http)" is a good enough clue for the sysadmin to check the schemes in the URIs...

  • Documentation “add_header 'Access-Control-Allow-Origin' '*' always” : Can you add an example url instead of “*” here.

The literal string * is what the header should say. I'm guessing you're guessing it's supposed to be replaced with something? ([How] should I make it clearer that it's supposed to be left just like that?)

Now at fc79e9d

#15 Updated by Radhika Chippada over 5 years ago

Tom,

  • The error message are SO MUCH better now. They are quite readable and helpful now. Thanks.
  • Regarding "add_header 'Access-Control-Allow-Origin' '*' always": On second thought, yes you are right; it should be "*" or whatever the site admin wants to limit it to.

Thanks.

LGTM

#16 Updated by Anonymous over 5 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:855a790d7be3ff345f04226bde3719fb90e6d9a5.

#17 Updated by Tom Morris over 2 years ago

  • Related to Bug #13564: [Workbench] Uploader should have a timeout added

Also available in: Atom PDF