Feature #17074
closed
Use count=none when fetching pages
Description
Calculating "items_available" in list requests is expensive.
For example, on a user cluster, they have a project with over 2 million collections and it takes around 10 seconds to return 50 items.
On another user cluster, they have 600,000 projects and the "shared with me" groups list takes over 5 seconds to return 50 items.
I will have to make some manual API calls to double check but I believe if we use "count=none" then it will skip having to count all rows and results will be returned much faster (a few hundred ms).
Using "offset" for paging is also expensive, because it throws away results. Without knowing the total count, offset paging cannot provide a "navigate to last page" option, as it does not know how many pages there are. Offset paging also produces unexpected results if list order changes during navigation.
We want to make the following changes:
- Use count=none when getting populating project contents.
- Continue to use offset for paging
- We issue a separate query to get the total number of items, with
limit=0andcount=exact-- this way, populating the page isn't held up by waiting for the full count. We can also limit the full count to only on an initial load or refresh of the project panel.
NOTE¶
We have to continue using offset paging because the "contents" endpoint doesn't completely support keyset paging; it works by concatenating tables (manually in ruby code instead of a UNION query) and the order that tables are queried and returned isn't lexically ordered.
Old note about contents¶
klasses = [Group,
Job, PipelineInstance, PipelineTemplate, ContainerRequest, Workflow,
Collection,
Human, Specimen, Trait]
If we remove the deprecated stuff:
klasses = [Group,
ContainerRequest, Workflow,
Collection]
We get get object ids
- j7d0g
- xvhdp
- 7fd4e
- 4zz18
Which means Workflow and Collection are out of order. So getting contents ordered by uuid won't be totally ordered.
There's a workaround where the client passes back last_object_class, so we might be able to get away with that.
Updated by 
Updated by