Project

General

Profile

Actions

Bug #22141

closed

Picking dialog collection search is very slow

Added by Peter Amstutz about 1 month ago. Updated 24 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Workbench2
Story points:
-
Release:
Release relationship:
Auto

Description

Search in the picking dialog is incredibly slow.

Specifically, this is "Filter Collections list in Projects" with a project selected. So it is getting the listing for a specific project. The project itself has at least 16,000 collections in it.

Here's a couple of queries:

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22]]

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22]]

These both time out.

Here is the basic query

%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22">https://xngs1.rdcloud.bms.com/arvados/v1/groups/xngs1-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22]]]

Improvements

The regular listing excludes logs and intermediates, but when filtering, it stops doing that, which means it is filtering on a much much larger set of collections. It should continue excluding logs and intermediates.

It should be using count=none. If exactly 200 items are returned, it should add the entry, "there may be more items, try filtering"

There should be some kind of indication that it is reloading the list, currently there's no feedback and so when the search takes too long, it feels like nothing is happening at all.


Subtasks 1 (0 open1 closed)

Task #22150: Review 22141-picking-dialog-searchResolvedStephen Smith10/09/2024Actions

Related issues

Related to Arvados - Feature #22046: Picker understands UUID and PDH searchResolvedPeter AmstutzActions
Related to Arvados - Feature #22045: Project/collection/file picker search shows parent projectsResolvedPeter AmstutzActions
Actions #1

Updated by Peter Amstutz about 1 month ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz about 1 month ago

  • Related to Feature #22046: Picker understands UUID and PDH search added
Actions #5

Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz about 1 month ago

  • Release set to 70
Actions #7

Updated by Peter Amstutz about 1 month ago

  • Assigned To set to Peter Amstutz
Actions #8

Updated by Peter Amstutz about 1 month ago

  • Related to Feature #22045: Project/collection/file picker search shows parent projects added
Actions #9

Updated by Peter Amstutz 30 days ago

22141-picking-dialog-search @ e9d7b4ef3c2a02bb357f768cc8a87aae29395d91

developer-run-tests: #4484

This branch ended up including #22045 and #22046 as well, so the review should cover the requirements from all three tickets.

  • All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
    • Now uses count=none in queries so they don't take a long time on large databases
    • When using project or collection search, the parent projects are displayed in the list
    • When clicking on a project or collection, user gets details about the selected item
    • User can paste a UUID or PDH into the search boxes to get the exact item they want
    • Dialog box is set to a fixed height of 80vh so it doesn't change size as elements in the tree are added or removed
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • This does not try to solve the data loading race in #21838, this branch will need to be manually merged with that one
    • #22044 would also be a quality of life improvement for picking, but I didn't work on it
  • Code is tested and passing, both automated and manual, what manual testing was done is described.
    • Added e2e tests for the new searching and details panel behavior
    • It occurs to me now writing this that I didn't make an e2e test that includes the interaction of searching both projects and collection, but this behavior was tested manually.
  • New or changed UX/UX and has gotten feedback from stakeholders.
    • This went through a few rounds of review with the team to land on a UI design we all liked
  • Documentation has been updated.
    • Unfortunately our documentation doesn't describe how to use workbench, so there isn't anything to update. (I checked the tutorial, but it doesn't have any screenshots of this UI either).
  • Behaves appropriately at the intended scale (describe intended scale).
    • Improves scaling by using count=none for faster queries.
  • Considered backwards and forwards compatibility issues between client and server.
    • I ended up discovering and working around #22154 ('kind' is wrong for users in 'included') by checking the uuid type, so this branch doesn't assume that is fixed (which would be an API change).
  • Follows our coding standards and GUI style guidelines.
    • comments

This started out as a ticket to just revise the queries to be more efficient, but I ended up attacking usability of the picking dialog generally.

Entering something into the project or collection search individually gets those hits and displays them under their parent projects.

Now you can also click on items and get details about the item on the right side, including clickable links to the parent project.

As a result, showing the parent project in the tree is not strictly necessary (the information is available in details now) but I think it's still helpful to quickly identify the item you're looking when there are multiple hits, without a lot of clicking around.

If you enter something into both projects and collections search, it searches for projects, then filters the list of collections under those projects (this is more or less the previous search behavior of the picking dialog).

This branch also has a commit that moves around some declarations in order to break circular imports. This was because one of the component tests refused to load. The refactoring is pretty straightforward, it mostly involves moving base type and enum declarations located in 'views' into their own modules and/or into the related 'action' module. This allows other modules to import just the individual types they need without having to depend on a react component definition that requires on lots of other things.

Actions #10

Updated by Stephen Smith 26 days ago

Here's a few things I noticed:

  • When searching projects, selecting one of the parents of the results shows a "search-" prefix in the details panel uuid - this also seems to prevent starting a WF in one of the "parent" projects of a search result, if we stick to using the prefix it would be good to put that in a const so it's more easily traceable
  • There is a compile error in tree.tsx from the childItemNameDark and childItemNameLight style keys not present in the style object
  • Style-wise it seems the indentation got flattened for a large part of projects-tree-picker.tsx
Actions #11

Updated by Peter Amstutz 25 days ago

Stephen Smith wrote in #note-10:

Here's a few things I noticed:

  • When searching projects, selecting one of the parents of the results shows a "search-" prefix in the details panel uuid - this also seems to prevent starting a WF in one of the "parent" projects of a search result, if we stick to using the prefix it would be good to put that in a const so it's more easily traceable

There might be a better way to encode special behavior for certain tree nodes, but using special ids seems to work. I made it a const and found the place where the gets returned to redux-form and added a check to strip off the prefix.

  • There is a compile error in tree.tsx from the childItemNameDark and childItemNameLight style keys not present in the style object

Should be fixed.

  • Style-wise it seems the indentation got flattened for a large part of projects-tree-picker.tsx

Should be indented properly now.

22141-picking-dialog-search @ f8b2a1f5617511aa75a5aae636811798b1bc3311

developer-run-tests: #4496

Actions #12

Updated by Peter Amstutz 24 days ago

  • Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint
Actions #13

Updated by Peter Amstutz 24 days ago

22141-picking-dialog-search @ 3f86ac062745cd18c2b56862058977e58c744c23

developer-run-tests: #4499

Adjusted one workbench test and merged main.

Actions #14

Updated by Stephen Smith 24 days ago

Lgtm!

Actions #15

Updated by Peter Amstutz 24 days ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF