Project

General

Profile

Actions

Bug #20533

open

Better handling of request surges when canceling a large workflow

Added by Peter Amstutz 10 days ago. Updated 1 day ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Specific test case: running a workflow with 100s of containers and then canceling them all at once leads to a massive surge of requests to the API server as all the containers finalize all at once.

Want to test ways that we can mitigate this traffic surge so that:

  1. all the containers finalize without fatal 503 errors (#20540, #20541)
  2. the workbench remains responsive (at least for GET requests during this time)
    1. evaluate configuration changes
    2. load balancing #20539
    3. controller request prioritization #20602
    4. Send out cancellations at a slower rate than whatever it's doing right now

Subtasks 1 (1 open0 closed)

Task #20556: ReviewNewTom Clegg

Actions

Related issues

Related to Arvados Epics - Story #20599: Scaling to 1000s of concurrent containersNew06/01/202309/30/2023

Actions
Related to Arvados - Story #20602: Prioritize requests made by workbench 2New

Actions
Actions #1

Updated by Peter Amstutz 10 days ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz 9 days ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz 9 days ago

  • Target version changed from To be groomed to To be scheduled
Actions #5

Updated by Peter Amstutz 9 days ago

  • Target version changed from To be scheduled to Development 2023-06-07
Actions #6

Updated by Peter Amstutz 8 days ago

  • Description updated (diff)
  • Subject changed from Better handling of request surges to Better handling of request surges when canceling a large workflow
Actions #7

Updated by Peter Amstutz 8 days ago

  • Assigned To set to Peter Amstutz
Actions #8

Updated by Peter Amstutz 1 day ago

  • Related to Story #20599: Scaling to 1000s of concurrent containers added
Actions #10

Updated by Peter Amstutz 1 day ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz 1 day ago

  • Related to Story #20602: Prioritize requests made by workbench 2 added
Actions #12

Updated by Peter Amstutz 1 day ago

  • Description updated (diff)
Actions #13

Updated by Peter Amstutz 1 day ago

  • Target version changed from Development 2023-06-07 to To be groomed
Actions

Also available in: Atom PDF