Bug #11547
Updated by Tom Clegg almost 7 years ago
arvados-cwl-runner created a container request that was about 75 MB of JSON. This caused huge memory bloat in the API server, paralyzed requests to both the containers and container_requests table (details on #11546), and resulted in memory bloat of other services as well, such as crunch-dispatch-slurm: <pre> $ ps axv | grep crunch-dispatch-slurm PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 43366 ? Sl 34:03 2248 0 3676584 2918600 10.1 /usr/bin/crunch-dispatch-slurm </pre> crunch-dispatch-slurm does not use arvados-cwl-runner shouldn't produce huge container requests, but the "mounts" field, so it should not load it (or any other potentially large field it doesn't use). The API server should use shouldn't accept them, either. Add a more efficient procedure to achieve atomic "lock" and "unlock" (currently this process involves an unnecessary ActiveRecord load/save cycle). configurable limit on the size of serialization fields. If Suggest setting the API default limit to 4 MB (with testing to ensure that performance is still very slow after these changes, determine what makes it slow, and add some limits stil acceptable with suitable defaults. In any case, arvados-cwl-runner should not produce huge container requests. 4 MB of JSON).