Project

General

Profile

Actions

Bug #22141

open

Picking dialog collection search is very slow

Added by Peter Amstutz 12 days ago. Updated 2 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Workbench2
Story points:
-
Release:
Release relationship:
Auto

Description

Search in the picking dialog is incredibly slow.

Specifically, this is "Filter Collections list in Projects" with a project selected. So it is getting the listing for a specific project. The project itself has at least 16,000 collections in it.

Here's a couple of queries:

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22]]

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22]]

These both time out.

Here is the basic query

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22">https://xngs1.rdcloud.bms.com/arvados/v1/groups/xngs1-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22]]]

Improvements

The regular listing excludes logs and intermediates, but when filtering, it stops doing that, which means it is filtering on a much much larger set of collections. It should continue excluding logs and intermediates.

It should be using count=none. If exactly 200 items are returned, it should add the entry, "there may be more items, try filtering"

There should be some kind of indication that it is reloading the list, currently there's no feedback and so when the search takes too long, it feels like nothing is happening at all.


Subtasks 1 (1 open0 closed)

Task #22150: Review 22141-picking-dialog-searchIn ProgressStephen Smith10/04/2024Actions

Related issues

Related to Arvados - Feature #22046: Picker understands UUID and PDH searchIn ProgressPeter AmstutzActions
Related to Arvados - Feature #22045: Project/collection/file picker search shows parent projectsIn ProgressPeter AmstutzActions
Actions #1

Updated by Peter Amstutz 12 days ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz 12 days ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz 12 days ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz 12 days ago

  • Related to Feature #22046: Picker understands UUID and PDH search added
Actions #5

Updated by Peter Amstutz 12 days ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz 12 days ago

  • Release set to 70
Actions #7

Updated by Peter Amstutz 11 days ago

  • Assigned To set to Peter Amstutz
Actions #8

Updated by Peter Amstutz 3 days ago

  • Related to Feature #22045: Project/collection/file picker search shows parent projects added
Actions #9

Updated by Peter Amstutz 2 days ago

22141-picking-dialog-search @ e9d7b4ef3c2a02bb357f768cc8a87aae29395d91

developer-run-tests: #4484

This branch ended up including #22045 and #22046 as well, so the review should cover the requirements from all three tickets.

  • All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
    • Now uses count=none in queries so they don't take a long time on large databases
    • When using project or collection search, the parent projects are displayed in the list
    • When clicking on a project or collection, user gets details about the selected item
    • User can paste a UUID or PDH into the search boxes to get the exact item they want
    • Dialog box is set to a fixed height of 80vh so it doesn't change size as elements in the tree are added or removed
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • This does not try to solve the data loading race in #21838, this branch will need to be manually merged with that one
    • #22044 would also be a quality of life improvement for picking, but I didn't work on it
  • Code is tested and passing, both automated and manual, what manual testing was done is described.
    • Added e2e tests for the new searching and details panel behavior
    • It occurs to me now writing this that I didn't make an e2e test that includes the interaction of searching both projects and collection, but this behavior was tested manually.
  • New or changed UX/UX and has gotten feedback from stakeholders.
    • This went through a few rounds of review with the team to land on a UI design we all liked
  • Documentation has been updated.
    • Unfortunately our documentation doesn't describe how to use workbench, so there isn't anything to update. (I checked the tutorial, but it doesn't have any screenshots of this UI either).
  • Behaves appropriately at the intended scale (describe intended scale).
    • Improves scaling by using count=none for faster queries.
  • Considered backwards and forwards compatibility issues between client and server.
    • I ended up discovering and working around #22154 ('kind' is wrong for users in 'included') by checking the uuid type, so this branch doesn't assume that is fixed (which would be an API change).
  • Follows our coding standards and GUI style guidelines.
    • comments

This started out as a ticket to just revise the queries to be more efficient, but I ended up attacking usability of the picking dialog generally.

Entering something into the project or collection search individually gets those hits and displays them under their parent projects.

Now you can also click on items and get details about the item on the right side, including clickable links to the parent project.

As a result, showing the parent project in the tree is not strictly necessary (the information is available in details now) but I think it's still helpful to quickly identify the item you're looking when there are multiple hits, without a lot of clicking around.

If you enter something into both projects and collections search, it searches for projects, then filters the list of collections under those projects (this is more or less the previous search behavior of the picking dialog).

This branch also has a commit that moves around some declarations in order to break circular imports. This was because one of the component tests refused to load. The refactoring is pretty straightforward, it mostly involves moving base type and enum declarations located in 'views' into their own modules and/or into the related 'action' module. This allows other modules to import just the individual types they need without having to depend on a react component definition that requires on lots of other things.

Actions

Also available in: Atom PDF