[Workbench] Times out before API server loading active pipeline instances
When loading a dashboard with lots of "active" pipeline_instances, the workbench server times out and hangs up on the API server before it can return results.
#3 Updated by Brett Smith about 3 years ago
- Subject changed from workbench times out before API server to [Workbench] Times out before API server loading active pipeline instances
I assume this happens while loading the Dashboard? We may also want to consider limiting the number of pipeline instances that loads. In general, we're getting more cautious about Workbench loading unbounded numbers of objects lately (e.g., #8286, #8441).
I also wonder if we need to make these timeouts configurable. Fundamentally, they're configurable in Nginx on the API server end, so these values may not be appropriate for all clusters.
#4 Updated by Joshua Randall about 3 years ago
Yes, I had this problem with the dashboard.
I'd be happy with a hard limit on the number of things returned (although in that case it might be nice if the section heading said something like "showing 20 out of 350 total").
One thing I tried to do to address this was to limit the "Active pipelines" section on the dashboard to only pipelines whose jobs had actually started running, as I am typically not all that interested in the pipelines that are "running" but which are really still queued because none of their jobs have started yet.
Unfortunately the information I need to do that did not appear to be readily available in the pipeline_instance object that the API server returns, so it can't sent as a filter parameter in the list request. Since the API server is embedding job information in the returned pipeline_instance object, it would be in a good position to augment that with some kind of 'job_status' read only field that could be filtered on.
#7 Updated by Brett Smith about 3 years ago
Discussed this a bit, and we agreed on a couple of things:
- There's really no reason for Workbench's API client to timeout at all, since it's effectively bridging the Web browser with the API server. It should only hang up when one of those ends does: either the user stops the browser request, or Nginx hangs up on the API server end. Anything that moves the current client closer to that goal is an improvement.
- Workbench shouldn't be displaying so many items in its dashboard, but that can be a separate story.
#10 Updated by Brett Smith about 3 years ago
So, I'm struggling a little bit to understand mechanically how this works. Here's my train of thought:
- We're using httpclient 18.104.22.168
- The way to set timeouts for it seems to be to set the right timeout attributes on the Session object, which are proxied through the HTTPClient object.
- There doesn't seem to be any special shortcut for setting timeouts through request methods. A hash passed in ultimately goes down to the create_request method where it sets HTTP request headers.
So... what do you know that I don't? :)