Bug #22141
closedPicking dialog collection search is very slow
Description
Search in the picking dialog is incredibly slow.
Specifically, this is "Filter Collections list in Projects" with a project selected. So it is getting the listing for a specific project. The project itself has at least 16,000 collections in it.
Here's a couple of queries:
%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25Output%25%22],[%22collections.any%22,%22ilike%22,%22%25from%25%22],[%22collections.any%22,%22ilike%22,%22%25WE%25%22]]
%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22">https://xxxxx.example.com/arvados/v1/groups/xxxxx-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.any%22,%22ilike%22,%22%25%25%22],[%22collections.any%22,%22ilike%22,%22%25Output+from+WES%25%22]]
These both time out.
Here is the basic query
%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22">https://xngs1.rdcloud.bms.com/arvados/v1/groups/xngs1-j7d0g-gp9u6hjrcgwy40d/contents?limit=200&filters=%22uuid%22,%22is_a%22,[%22arvados%23group%22,%22arvados%23collection%22,[%22collections.properties.type%22,%22not+in%22,[%22intermediate%22,%22log%22]]]
Improvements¶
The regular listing excludes logs and intermediates, but when filtering, it stops doing that, which means it is filtering on a much much larger set of collections. It should continue excluding logs and intermediates.
It should be using count=none. If exactly 200 items are returned, it should add the entry, "there may be more items, try filtering"
There should be some kind of indication that it is reloading the list, currently there's no feedback and so when the search takes too long, it feels like nothing is happening at all.
Related issues
Updated by Peter Amstutz about 2 months ago
- Status changed from New to In Progress
Updated by Peter Amstutz about 2 months ago
- Related to Feature #22046: Picker understands UUID and PDH search added
Updated by Peter Amstutz about 2 months ago
- Related to Feature #22045: Project/collection/file picker search shows parent projects added
Updated by Peter Amstutz about 2 months ago
22141-picking-dialog-search @ e9d7b4ef3c2a02bb357f768cc8a87aae29395d91
This branch ended up including #22045 and #22046 as well, so the review should cover the requirements from all three tickets.
- All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
- Now uses count=none in queries so they don't take a long time on large databases
- When using project or collection search, the parent projects are displayed in the list
- When clicking on a project or collection, user gets details about the selected item
- User can paste a UUID or PDH into the search boxes to get the exact item they want
- Dialog box is set to a fixed height of 80vh so it doesn't change size as elements in the tree are added or removed
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- Code is tested and passing, both automated and manual, what manual testing was done is described.
- Added e2e tests for the new searching and details panel behavior
- It occurs to me now writing this that I didn't make an e2e test that includes the interaction of searching both projects and collection, but this behavior was tested manually.
- New or changed UX/UX and has gotten feedback from stakeholders.
- This went through a few rounds of review with the team to land on a UI design we all liked
- Documentation has been updated.
- Unfortunately our documentation doesn't describe how to use workbench, so there isn't anything to update. (I checked the tutorial, but it doesn't have any screenshots of this UI either).
- Behaves appropriately at the intended scale (describe intended scale).
- Improves scaling by using count=none for faster queries.
- Considered backwards and forwards compatibility issues between client and server.
- I ended up discovering and working around #22154 ('kind' is wrong for users in 'included') by checking the uuid type, so this branch doesn't assume that is fixed (which would be an API change).
- Follows our coding standards and GUI style guidelines.
- comments
This started out as a ticket to just revise the queries to be more efficient, but I ended up attacking usability of the picking dialog generally.
Entering something into the project or collection search individually gets those hits and displays them under their parent projects.
Now you can also click on items and get details about the item on the right side, including clickable links to the parent project.
As a result, showing the parent project in the tree is not strictly necessary (the information is available in details now) but I think it's still helpful to quickly identify the item you're looking when there are multiple hits, without a lot of clicking around.
If you enter something into both projects and collections search, it searches for projects, then filters the list of collections under those projects (this is more or less the previous search behavior of the picking dialog).
This branch also has a commit that moves around some declarations in order to break circular imports. This was because one of the component tests refused to load. The refactoring is pretty straightforward, it mostly involves moving base type and enum declarations located in 'views' into their own modules and/or into the related 'action' module. This allows other modules to import just the individual types they need without having to depend on a react component definition that requires on lots of other things.
Updated by Stephen Smith about 1 month ago
Here's a few things I noticed:
- When searching projects, selecting one of the parents of the results shows a "search-" prefix in the details panel uuid - this also seems to prevent starting a WF in one of the "parent" projects of a search result, if we stick to using the prefix it would be good to put that in a const so it's more easily traceable
- There is a compile error in tree.tsx from the
childItemNameDark
andchildItemNameLight
style keys not present in the style object - Style-wise it seems the indentation got flattened for a large part of projects-tree-picker.tsx
Updated by Peter Amstutz about 1 month ago
Stephen Smith wrote in #note-10:
Here's a few things I noticed:
- When searching projects, selecting one of the parents of the results shows a "search-" prefix in the details panel uuid - this also seems to prevent starting a WF in one of the "parent" projects of a search result, if we stick to using the prefix it would be good to put that in a const so it's more easily traceable
There might be a better way to encode special behavior for certain tree nodes, but using special ids seems to work. I made it a const and found the place where the gets returned to redux-form and added a check to strip off the prefix.
- There is a compile error in tree.tsx from the
childItemNameDark
andchildItemNameLight
style keys not present in the style object
Should be fixed.
- Style-wise it seems the indentation got flattened for a large part of projects-tree-picker.tsx
Should be indented properly now.
22141-picking-dialog-search @ f8b2a1f5617511aa75a5aae636811798b1bc3311
Updated by Peter Amstutz about 1 month ago
- Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint
Updated by Peter Amstutz about 1 month ago
22141-picking-dialog-search @ 3f86ac062745cd18c2b56862058977e58c744c23
Adjusted one workbench test and merged main.
Updated by Peter Amstutz about 1 month ago
- Status changed from In Progress to Resolved