Arvados

I am once again running into problems with API server resource starvation due to long-running requests causing normal clients with reasonable timeouts and retry behaviour essentially denial-of-service attacking the server (with, in this case, large collection creation requests).

IMHO it is not possible to run a stable Arvados API server under heavy load without applying a patch like the one that was implemented at Sanger to introduce statement timeouts in the database to mitigate this issue (see: https://github.com/wtsi-hgi/hgi-systems/blob/master/ansible/roles/arvados-master/files/hgi-integration-1.1.4.20180723133344-8.patch). Apologies that the OP for this issue did not include the patch!

Definitions of "heavy load" may vary depending on the capabilities of the API server and database hosts (and passenger settings such as the number of workers), but based on prior experience, the sorts of things that may take a long time (longer than the timeout) and lead to client-retry DoS attacks are:
- attempts to create very large collections (I think the thing that takes a long time is validating manifest block signatures)
- attempts to read very large collections with signed manifests (unsigned manifests are fast)
- attempts to work with container requests with large inline input documents
- certain queries against the log table, when it has grown very large

Tom Clegg wrote:

13697-database-timeout @ fdd3dd304439a7912db1c6ef257cf512ec7b3b52 -- developer-run-tests: #2707

• Set the database statement and lock timeouts to RequestTimeout (similar to the contributed patch, but implemented in the "db connection checkout" hook instead of the ArvadosModel initializer which runs each time an object is created/loaded)
• In Go services that use lib/service (including controller and keepstore, but not yet keepproxy or keep-web), the request context is cancelled if the handler doesn't finish or hijack the connection before reaching RequestTimeout.
• When keep-web gets migrated to use lib/service, this won't cut off long/slow downloads (the webdav handler doesn't interrupt copy-bytes-to-ResponseWriter when context cancels) but we might (?) need some more special handling so large/slow uploads are allowed to take longer than RequestTimeout.

For this, would it be worth putting a note along these lines (with a reference to this ticket) in the webdav source for when we refactor it to use lib/service? That way it won't be overlooked.

• Remove httpserver.HandlerWithContext in favor of the BaseContext feature already provided by http.Server.

Nice!

I see a singularity related test failure at developer-run-tests-remainder: #2814 /consoleFull, is that unrelated?

LGTM otherwise!

Ward Vandewege wrote:

I see a singularity related test failure at developer-run-tests-remainder: #2814 /consoleFull, is that unrelated?

Hm, the same failure happened to an #8363 branch as well, and went away on the next attempt.

executor_test.go:166:
    c.Check(code, Equals, expectCode)
... obtained int = 151
... expected int = 0

151 is SIGURG. This seems to be a singularity bug, reported in 3.5.2 and fixed April 2020. The Go runtime (starting at/before 1.14) sends itself SIGURG to preempt goroutines, and these are forwarded to child processes by default.

• https://github.com/golang/go/issues/37942
• https://github.com/hpcng/singularity/issues/5231

Perhaps the fix is to update arvados-server install from singularity 3.5.2 to 3.7.4 (source:tools/compute-images/scripts/base.sh already uses 3.7.4).

Added #18184

Tom Clegg wrote:

Ward Vandewege wrote:

I see a singularity related test failure at developer-run-tests-remainder: #2814 /consoleFull, is that unrelated?

Hm, the same failure happened to an #8363 branch as well, and went away on the next attempt.

[...]

151 is SIGURG. This seems to be a singularity bug, reported in 3.5.2 and fixed April 2020. The Go runtime (starting at/before 1.14) sends itself SIGURG to preempt goroutines, and these are forwarded to child processes by default.

• https://github.com/golang/go/issues/37942
• https://github.com/hpcng/singularity/issues/5231

Perhaps the fix is to update arvados-server install from singularity 3.5.2 to 3.7.4 (source:tools/compute-images/scripts/base.sh already uses 3.7.4).

Good sleuthing! Yes please upgrade to 3.7.4, that is the minimum version we are intending to support.