Project

General

Profile

Actions

Bug #21904

closed

Remove feature that rejects POST arvados/v1/logs immediately when any requests are queued

Added by Tom Clegg 6 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Story points:
-
Release:
Release relationship:
Auto

Description

In #20602 we started rejecting "POST /arvados/v1/logs" requests instead of queueing them when all active request slots are full. The idea was to drop container-logging events outright during busy periods (i.e., when any requests are queued) -- otherwise, when lots of containers were running, these frequent requests would quickly fill up the queue and cause more important non-interactive requests, like saving container output collections, to fail.

This is no longer needed because crunch-run no longer sends POST /arvados/v1/logs requests at all.

Now, POST /arvados/v1/logs requests are more likely to come from the "log uploads and downloads" feature of keep-web, and the above rationale does not apply.

Instead, these should be queued the same way as other non-interactive requests.

(This has also caused an additional problem where keep-web's outgoing request throttle gets triggered by the premature 503s. That problem could be fixed on the client side, but a client-side fix wouldn't address the problem of upload/download logs being skipped whenever the controller is even moderately busy. So it makes more sense to just fix the server side.)


Subtasks 1 (0 open1 closed)

Task #22007: Review 21904-no-unqueueable-reqsResolvedTom Clegg07/30/2024Actions

Related issues 1 (0 open1 closed)

Related to Arvados - Bug #21748: awscli downloads from keep-web slowlyResolvedTom CleggActions
Actions #1

Updated by Tom Clegg 6 months ago

  • Related to Bug #21748: awscli downloads from keep-web slowly added
Actions #2

Updated by Peter Amstutz 6 months ago

  • Target version set to Development 2024-08-07 sprint
Actions #3

Updated by Tom Clegg 5 months ago

  • Status changed from New to In Progress

We can also revert #20200, an earlier attempt to address the "too many log requests" issue.

Actions #5

Updated by Tom Clegg 5 months ago

  • Assigned To set to Tom Clegg
Actions #6

Updated by Brett Smith 5 months ago

Tom Clegg wrote in #note-4:

21904-no-unqueueable-reqs @ df73561dc9cc547df259d5215a482cd7d787aed6 -- developer-run-tests: #4352

Looks good to me. grepped for a few different patterns to look for remnant code and didn't find anything relevant. Thanks.

Actions #7

Updated by Tom Clegg 5 months ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF