Bug #13593
Updated by Peter Amstutz over 6 years ago
A user reported that creating many projects at once resulted in extreme slowdown and eventually crashing the API server unless the create requests were throttled. The hypthothesis is that each Each time a project (or group) is created, it triggers a refresh of the permissions graph, and this is creating load. Notes: Assuming groups are created sequentially by graph. If a single client, and it doesn't return until the graph is refreshed, it shouldn't have lots of parallel activity. Possibly, other requests are getting backed up behind the permission graph refreshes (however, the refresh happens in a transaction, other requests are supposed user needs to just get an old copy of create many projects at once, refreshing the permission graph until the new one for every project is ready). Based on what we know so far, slow group creation very expensive. Consider a batch-create API which makes sense, but having it overload the server while is is doing it does not make sense. Action items: # Collect more information / run experiments possible to see if we can reproduce locally. # Example options to reduce overhead from permission graph refreshes: ## Consider an flag on API call that requests asynchronous permission refresh, and coalesce updates ## Alternately, introduce a batch create API which creates multiple groups projects in one a single API call call, but only runs permission refresh once refreshes permissions once.