Bug #8504

[Workbench] Times out before API server loading active pipeline instances

Added by Joshua Randall over 3 years ago. Updated over 2 years ago.

Status:
Feedback
Priority:
Normal
Assigned To:
Category:
Workbench
Target version:
Start date:
03/16/2016
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

When loading a dashboard with lots of "active" pipeline_instances, the workbench server times out and hangs up on the API server before it can return results.


Subtasks

Task #8719: Review PR #40In ProgressJoshua Randall


Related issues

Related to Arvados - Story #8657: [Workbench] Hard limit the number of items displayed on the dashboardNew03/07/2016

History

#1 Updated by Joshua Randall over 3 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

#2 Updated by Joshua Randall about 3 years ago

  • Assigned To deleted (Joshua Randall)

#3 Updated by Brett Smith about 3 years ago

  • Subject changed from workbench times out before API server to [Workbench] Times out before API server loading active pipeline instances

Josh,

I assume this happens while loading the Dashboard? We may also want to consider limiting the number of pipeline instances that loads. In general, we're getting more cautious about Workbench loading unbounded numbers of objects lately (e.g., #8286, #8441).

I also wonder if we need to make these timeouts configurable. Fundamentally, they're configurable in Nginx on the API server end, so these values may not be appropriate for all clusters.

Thanks.

#4 Updated by Joshua Randall about 3 years ago

Yes, I had this problem with the dashboard.

I'd be happy with a hard limit on the number of things returned (although in that case it might be nice if the section heading said something like "showing 20 out of 350 total").

One thing I tried to do to address this was to limit the "Active pipelines" section on the dashboard to only pipelines whose jobs had actually started running, as I am typically not all that interested in the pipelines that are "running" but which are really still queued because none of their jobs have started yet.

Unfortunately the information I need to do that did not appear to be readily available in the pipeline_instance object that the API server returns, so it can't sent as a filter parameter in the list request. Since the API server is embedding job information in the returned pipeline_instance object, it would be in a good position to augment that with some kind of 'job_status' read only field that could be filtered on.

#5 Updated by Joshua Randall about 3 years ago

  • Assigned To set to Joshua Randall

#6 Updated by Brett Smith about 3 years ago

  • Target version set to Arvados Future Sprints

#7 Updated by Brett Smith about 3 years ago

Discussed this a bit, and we agreed on a couple of things:

  • There's really no reason for Workbench's API client to timeout at all, since it's effectively bridging the Web browser with the API server. It should only hang up when one of those ends does: either the user stops the browser request, or Nginx hangs up on the API server end. Anything that moves the current client closer to that goal is an improvement.
  • Workbench shouldn't be displaying so many items in its dashboard, but that can be a separate story.

#8 Updated by Brett Smith about 3 years ago

  • Target version changed from Arvados Future Sprints to 2016-03-30 sprint

#9 Updated by Brett Smith about 3 years ago

  • Story points set to 0.5

#10 Updated by Brett Smith about 3 years ago

So, I'm struggling a little bit to understand mechanically how this works. Here's my train of thought:

So... what do you know that I don't? :)

#11 Updated by Brett Smith about 3 years ago

  • Target version changed from 2016-03-30 sprint to Arvados Future Sprints

#12 Updated by Tom Clegg almost 3 years ago

Brett Smith wrote:

  • Workbench shouldn't be displaying so many items in its dashboard, but that can be a separate story.

Radhika reports this is being addressed in #9318 (limit=8)

#13 Updated by Radhika Chippada over 2 years ago

Several dashboard performance related improvements were made in #10078. Please recheck with latest. Thanks.

Also available in: Atom PDF