Bug #5748
Updated by Ward Vandewege about 9 years ago
We have some keepstore servers that have r/o disks (only). After running for a few hours, the keepstored process becomes bloated and pegged at (almost) 100% cpu. It still responds to requests, but very slowly.
<pre>
root 24414 60.4 59.0 2491440 2243984 ? Sl 14:21 263:13 \_ keepstore -enforce-permissions=false -permission-key-file=/etc/keepstore/permission.key -data-manager-token-file=/etc/keepstore/datamanager.key -listen=:25107 -serialize=true -volumes=/...
</pre>
All CPU time is %user, not iowait:
<pre>
# iostat -x
Linux 3.16-0.bpo.2-amd64 (keep5.su92l.arvadosapi.com) 04/15/2015 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
6.24 0.00 1.13 0.21 0.00 92.42
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1.30 0.02 2.09 0.54 26.92 26.07 0.04 21.36 8.53 21.49 1.72 0.36
sdb 0.00 0.02 0.06 1.48 3.94 186.27 247.27 0.17 108.47 6.69 112.33 2.07 0.32
sdc 0.00 0.02 0.05 1.49 3.18 186.98 246.86 0.17 109.06 7.23 112.59 2.08 0.32
dm-0 0.00 0.00 0.06 1.50 3.94 186.27 244.24 0.25 157.84 8.05 163.45 2.09 0.32
dm-1 0.00 0.00 0.05 1.51 3.18 186.98 243.76 0.29 184.28 8.66 190.28 2.09 0.33
sdd 0.00 0.03 0.01 2.01 0.05 255.84 253.40 0.07 35.18 14.01 35.27 2.08 0.42
dm-2 0.00 0.00 0.01 2.04 0.05 255.84 249.51 0.26 129.09 29.02 129.51 2.06 0.42
</pre>
Version:
<pre>
# dpkg -l |grep keepstore
ii keepstore 0.1.20150413210544.0f119f7 amd64 Keepstore is the Keep storage daemon, accessible to clients on the LAN
</pre>