Arvados

Based on the extensive amount of profiling of collection performance, it appears that below are some of the big areas of potential performance improvement:

• signing locators and other pre-save operations in API server during CREATE / UPDATE / GET of collections
  • See https://arvados.org/issues/6087#note-10, https://arvados.org/issues/6087#note-11, and https://arvados.org/issues/6087#note-12 for full detail. Created #6203 for this.

• Performance improvements in collections/_show_files page: #6050

• Performance improvements in collection#show page source_summary section: #6042

• Fetching same collection repeatedly
  • #6041
  • In case of an combine collections, the collections being combined are again fetched in the combine_selected_files_into_collection action. After combining, new collection is returned by server; however, the redirect operation again fetches the newly created collection during show operation

• Combining collections
  • https://arvados.org/issues/6087#note-13

Radhika Chippada wrote:

Based on the extensive amount of profiling of collection performance, it appears that below are some of the big areas of potential performance improvement:

I would point out that we haven't done any performance testing on the Python SDK or FUSE yet, both of which are common ways for jobs to access and manipulate collections. I am reluctant to prioritize any changes to Workbench until we have some numbers for those. In general, I would rather prioritize changes that improve job performance and stability to those that don't.

• signing locators and other pre-save operations in API server during CREATE / UPDATE / GET of collections
  • See https://arvados.org/issues/6087#note-10, https://arvados.org/issues/6087#note-11, and https://arvados.org/issues/6087#note-12 for full detail. Created #6203 for this.

I agree that improving the performance of access token parsing and manipulation seems like a great place to start. It affects literally every operation on collections, and will improve performance for all components.

I will move #6203 to this sprint and assign to you, as the instantiation of this ticket. I will leave this ticket open for now. If you finish #6203 and your other tickets on this sprint promptly, let's revisit to see if there's another good change to make in the time remaining.

Thanks.