Arvados

Typical system default is LimitNOFILE=1024:524288 (soft:hard).

1024 was too low, so in 73018eed5cd93c9db7467760f5f1484fa2750554 (Feb 2020) we added LimitNOFILE=65536 to our systemd service files.

1024 was too low for everybody else too, so starting with Go 1.19 (Aug 2022) any Go program that imports the os package automatically raises its own softlimit to hardlimit. (See Go release notes, Go commit, systemd advice "Typically applications should increase their soft limit to the hard limit on their own")

So now our LimitNOFILE=65536 solution is effectively reducing the limit from 524288 to 65536.

The default hard limit is 524288 on debian:11, debian:12, ubuntu:20.04, ubuntu:22.04.

So, I think the most obvious improvement is to remove LimitNOFILE from our service files, and enjoy the default limit of 524288.

21640-max-nofile @ e672f484160faae900fb7f7e281d06952fd35d28 -- developer-run-tests: #4245

Maybe a lesson for future work here is: if you want to increase a soft limit, make sure to write it as soft:hard unless you specifically want to lower the hard limit. It sounds like this issue probably wouldn't have even come up if we had originally written LimitNOFILE=65536:524288 in our service definitions.

...or if systemd had offered a way to write "raise soft limit, leave hard limit alone".

21640-max-nofile @ e672f484160faae900fb7f7e281d06952fd35d28

For posterity, yes I'm reviewing this without the checklist, because I confirm it is a single commit that only removes configuration from systemd service units. There's basically no point in running tests because nothing is actually going to test this, there's no style guide to follow, etc.

Also just for posterity, I think it's worth calling out:

diff --git a/services/crunch-dispatch-local/crunch-dispatch-local.service b/services/crunch-dispatch-local/crunch-dispatch-local.service
index b4fc10f83e..f40359de2e 100644
--- a/services/crunch-dispatch-local/crunch-dispatch-local.service
+++ b/services/crunch-dispatch-local/crunch-dispatch-local.service
@@ -11,11 +11,8 @@ StartLimitIntervalSec=0
 Restart=always
 RestartSec=1
-LimitNOFILE=1000000

 [Install]

This change actually does lower the limit for crunch-dispatch-local, because now we're going from 1 million to 512 kibi-file-handles. I'm okay with that: this was included in the original commit of the file, so there's no history to explain it, and I think if 512Ki wasn't good enough for us we would see that in other services like controller before we saw it in crunch-dispatch-local.

As long as we're remotely vaguely on the same page about that, looks good to me.

21640-max-nofile @ e672f484160faae900fb7f7e281d06952fd35d28 -- developer-run-tests: #4245

(Turns out I did run the tests but missed posting the link.)

Brett Smith wrote in #note-12:

This change actually does lower the limit for crunch-dispatch-local

Agree. Given absence of commentary in git, I'm assuming 1000000 was chosen the same way 65536 was chosen ("a number much bigger than 1024") and it doesn't indicate any suspicion that the default hard limit 524288 was too low.