Bug #21904
closedRemove feature that rejects POST arvados/v1/logs immediately when any requests are queued
Description
In #20602 we started rejecting "POST /arvados/v1/logs" requests instead of queueing them when all active request slots are full. The idea was to drop container-logging events outright during busy periods (i.e., when any requests are queued) -- otherwise, when lots of containers were running, these frequent requests would quickly fill up the queue and cause more important non-interactive requests, like saving container output collections, to fail.
This is no longer needed because crunch-run no longer sends POST /arvados/v1/logs
requests at all.
Now, POST /arvados/v1/logs
requests are more likely to come from the "log uploads and downloads" feature of keep-web, and the above rationale does not apply.
Instead, these should be queued the same way as other non-interactive requests.
(This has also caused an additional problem where keep-web's outgoing request throttle gets triggered by the premature 503s. That problem could be fixed on the client side, but a client-side fix wouldn't address the problem of upload/download logs being skipped whenever the controller is even moderately busy. So it makes more sense to just fix the server side.)
Related issues