Project

General

Profile

Bug #13593

Updated by Tom Clegg almost 6 years ago

A user reported that creating many projects at once resulted in extreme slowdown and eventually crashing the API server unless the create requests were throttled. 

 Creating The hypthothesis is that each time a project (or any other group) also refreshes is created, it triggers a refresh of the permissions graph, and this is creating load. 

 Notes: 

 Assuming groups are created sequentially by a single client, and it doesn't return until the graph in the same transaction. is refreshed, it shouldn't have lots of parallel activity. 

 This makes the API call slow, but we intend/expect that Possibly, other API calls can proceed while requests are getting backed up behind the update is permission graph refreshes (however, the refresh happens in progress, so a transaction, other (non-permission-modifying) API calls aren't expected to be slowed down. 

 h2. Avoiding impact on other API requests 

 Problem TBD. Try are supposed to reproduce. 

 h2. Faster "create" calls 

 Possible approach, covering both serial and concurrent updates: 
 * Accept just get an "asynchronous update" flag with create/update requests, indicating old copy of the client doesn't need a guarantee that the permission graph until the new one is updated when ready). 

 Based on what we know so far, slow group creation makes sense, but having it receives its response. 
 * Either way, update overload the permission graph _after_ the create/update transaction server while is committed, and skip redundant updates. is doing it does not make sense. 

 Skipping redundant updates: Action items: 

 # Collect more information / run experiments to see if we can reproduce locally. 
 * Acquire lock # Example options to reduce overhead from permission graph refreshes: 
 * Check whether any ## Consider an flag on API call that requests asynchronous permission refresh, and coalesce updates have occurred since the lock was acquired for the existing version of the graph 
 * If so, update graph 
 * Release lock 
  
 ## Alternately, introduce a batch create API which creates multiple groups in one API call but only runs permission refresh once

Back