Bug #11547

Updated by Peter Amstutz over 4 years ago

arvados-cwl-runner created a container request that was about 75 MB of JSON. This caused huge memory bloat in the API server, paralyzed requests to both the containers and container_requests table (details on #11546), and resulted in memory bloat of other services as well, such as crunch-dispatch-slurm:

<pre>
$ ps axv | grep crunch-dispatch-slurm
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
43366 ? Sl 34:03 2248 0 3676584 2918600 10.1 /usr/bin/crunch-dispatch-slurm
</pre>

arvados-cwl-runner shouldn't produce huge container requests, but the API server shouldn't accept them, either. Add a configurable limit on the size of serialization fields.

Suggest setting the default limit to 4 MB (with testing to ensure that performance is stil acceptable with 4 MB of JSON).

Back