Arvados

What I believe is the primary source of lag is the way Data Explorer middlewares are initiated combined with the fact that all each new middleware triggers its own SET_COLUMNS call in each Data Explorer. There are 19 Data Explorers and 1 middleware for each, leading to an n^2 situation (19 * 19 = 361). Each Data Explorer has its own set of actions that are dispatched as well.

Here are all the actions dispatched when Workbench, viewing the Project Panel, is refreshed on the main branch (note that many things happen exactly 19 times, some happen 38 times (19 * 2), and SET_COLUMNS happens 380 times (19^2 + 19):

If we remove the calls to SET_COLUMNS, we see a marked improvement:

If we remove the middlwares that are not directly affecting the Project Panel:

In this state, workbench works 100% as expected as long as the only Data Explorer viewed is the Project Panel, including switching between projects. There is still some lag when switching between projects. This might be addressed by using web workers to offload some of the work to another JS thread and/or using scheduler.yield() where appropriate, which is where I will focus my efforts after this initial phase.

The long-term solution to the SET_COLUMNS problem is a substantial rewrite with the aim of lazily-loading each Data Explorer when needed, instead of loading all 19 on the initial page load/refresh.

I am still investigating alternatives to our current middleware implementation. These cannot be done lazily, but if we can offload their side effects until they are needed, we can still save a thousands of unnecessary actions.

I also investigated the "requestContainerStatusCount" issue, and implemented a wrapper that just yields control momentarily after each of the requests resolves. This doesn't speed up the page load time (it actually increases it by a tiny but nonzero amount), but it does break up the jank at several points, meaning UI input can be handled more often.

I have also applied some memoization to reduce some of the more problematic actions. The Data Table now renders twice instead of 16 times per page load, and a slight refactor means the side panel tree item renderer gets called 98 times vs 512 times. The reason for this is that it used a calculation that was located within the mapStateToDispatch for that component. This means it was re-calculated every time Redux state updated, whether or not the calculation was relevant to the component. Best prectice is that mapStateToProps should only select and map state properties directly, not compute derived data. Going through and removing any similar instances of this will go a long way to removing excess re-renders.

So far, my work on 22127-optimization @ 1b8d65a0bf940d5a7fe4a13bafc377ecdd3d0dad has yielded the following results:

The gross jank time, beginning to end, in the above picture is just under 3 seconds with the longest individual jank lasting 835ms. By comparison, Instagram, possibly the most optimized React application there is, averages 1.5-2 seconds of jank time on (logged-in) refresh.