Bug #20533
closed
Better handling of request surges when canceling a large workflow
Added by Peter Amstutz 11 months ago.
Updated 10 months ago.
Release relationship:
Auto
Description
Specific test case: running a workflow with 100s of containers and then canceling them all at once leads to a massive surge of requests to the API server as all the containers finalize all at once.
Want to test ways that we can mitigate this traffic surge so that:
- all the containers finalize without fatal 503 errors (#20540, #20541)
- the workbench remains responsive (at least for GET requests during this time)
- evaluate configuration changes
- load balancing #20539
- controller request prioritization #20602
- Send out cancellations at a slower rate than whatever it's doing right now
- Description updated (diff)
- Description updated (diff)
- Target version changed from Future to To be scheduled
- Target version changed from To be scheduled to Development 2023-06-07
- Description updated (diff)
- Subject changed from Better handling of request surges to Better handling of request surges when canceling a large workflow
- Assigned To set to Peter Amstutz
- Related to Idea #20599: Scaling to 1000s of concurrent containers added
- Description updated (diff)
- Related to Idea #20602: Prioritize requests made by workbench 2 added
- Description updated (diff)
- Target version changed from Development 2023-06-07 to Future
- Status changed from New to In Progress
- Status changed from In Progress to Feedback
I ran a test with 250 containers and hitting cancel. The request queue does immediately fill up with requests as the containers try to finalize, but
- Workbench remains responsive (very important)
- I believe all the crunch-run processes retry and terminate gracefully (but a bug I thought I fixed might still be a think: https://dev.arvados.org/issues/20614#note-14)
- Release set to 66
- Target version deleted (
Future)
- Status changed from Feedback to Resolved
Also available in: Atom
PDF
Updated by