Project

General

Profile

Actions
Copy link

Bug #20533

closed

Better handling of request surges when canceling a large workflow

Added by Peter Amstutz 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Peter Amstutz
Category:
API
Target version:
-
Story points:
-
Release:
Arvados 2.7.0
Release relationship:
Auto

Description

Specific test case: running a workflow with 100s of containers and then canceling them all at once leads to a massive surge of requests to the API server as all the containers finalize all at once.

Want to test ways that we can mitigate this traffic surge so that:

  1. all the containers finalize without fatal 503 errors (#20540, #20541)
  2. the workbench remains responsive (at least for GET requests during this time)
    1. evaluate configuration changes
    2. load balancing #20539
    3. controller request prioritization #20602
    4. Send out cancellations at a slower rate than whatever it's doing right now

Related issues

Related to Arvados Epics - Idea #20599: Scaling to 1000s of concurrent containersResolved06/01/202303/31/2024Actions
Related to Arvados - Idea #20602: Prioritize requests made by workbench 2ResolvedTom Clegg06/08/2023Actions
Actions
Copy link
 #1

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Peter Amstutz 11 months ago

  • Target version changed from Future to To be scheduled
Actions
Copy link
 #5

Updated by Peter Amstutz 11 months ago

  • Target version changed from To be scheduled to Development 2023-06-07
Actions
Copy link
 #6

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
  • Subject changed from Better handling of request surges to Better handling of request surges when canceling a large workflow
Actions
Copy link
 #7

Updated by Peter Amstutz 11 months ago

  • Assigned To set to Peter Amstutz
Actions
Copy link
 #8

Updated by Peter Amstutz 11 months ago

  • Related to Idea #20599: Scaling to 1000s of concurrent containers added
Actions
Copy link
 #10

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
Actions
Copy link
 #11

Updated by Peter Amstutz 11 months ago

  • Related to Idea #20602: Prioritize requests made by workbench 2 added
Actions
Copy link
 #12

Updated by Peter Amstutz 11 months ago

  • Description updated (diff)
Actions
Copy link
 #13

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2023-06-07 to Future
Actions
Copy link
 #14

Updated by Peter Amstutz 10 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #15

Updated by Peter Amstutz 10 months ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #16

Updated by Peter Amstutz 10 months ago

I ran a test with 250 containers and hitting cancel. The request queue does immediately fill up with requests as the containers try to finalize, but

  1. Workbench remains responsive (very important)
  2. I believe all the crunch-run processes retry and terminate gracefully (but a bug I thought I fixed might still be a think: https://dev.arvados.org/issues/20614#note-14)
Actions
Copy link
 #17

Updated by Peter Amstutz 10 months ago

  • Release set to 66
  • Target version deleted (Future)
  • Status changed from Feedback to Resolved
Actions
Copy link

Also available in: Atom PDF