Bug #6651

"This website is under heavy load" error causes a parsing error

Added by Nico César almost 5 years ago. Updated 9 months ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


/usr/local/arvados/install-arvados-tokens.rb production

Error: 757: unexpected token at '<h1>This website is under heavy load</h1><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>'
/usr/local/rvm/gems/ruby-2.1.4/gems/json-1.8.2/lib/json/common.rb:155:in `parse'
/usr/local/rvm/gems/ruby-2.1.4/gems/json-1.8.2/lib/json/common.rb:155:in `parse'
/usr/local/rvm/gems/ruby-2.1.4/gems/arvados-0.1.20150615153458/lib/arvados.rb:214:in `api_exec'
/usr/local/rvm/gems/ruby-2.1.4/gems/arvados-0.1.20150615153458/lib/arvados.rb:83:in `block (4 levels) in initialize'
/usr/local/arvados/install-arvados-tokens.rb:81:in `<main>'

passenger default of overloaded is in HTML not in json


#1 Updated by Joshua Randall over 4 years ago

I just observed this on our system as well - when the API server is overloaded, workbench ends up putting up a generic error about parsing rather than any message about the heavy load.

#2 Updated by Joshua Randall over 4 years ago

I have worked around this problem with the following nginx config:

in nginx.conf:

    passenger_request_queue_overflow_status_code 513;
    passenger_intercept_errors on;

in cond.d/arvados-api.conf, inside the "server { listen ... }" block:

  error_page 513 =503 /heavyload.json;

In /var/www/arvados-api/current/public/heavyload.json:

{"errors":["This website is under heavy load (queue full)"]}

I change the status code for the "heavy load" situation within passenger from 503 to 513 (since there are other situations that could also cause 503 errors). I then configure nginx to process the 513 status by returning the /heavyload.json file instead of the default handler, and also recodes the status back to 503.

#3 Updated by Tom Morris almost 3 years ago

  • Assigned To set to Nico César
  • Target version set to 2017-08-16 sprint

#4 Updated by Nico César almost 3 years ago

  • Target version changed from 2017-08-16 sprint to Arvados Future Sprints

this is a duplicate

#5 Updated by Nico César about 1 year ago

  • Status changed from New to Duplicate

#6 Updated by Tom Morris 9 months ago

  • Assigned To deleted (Nico César)
  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF