Project

General

Profile

Actions

Bug #5748

closed

[Keep] sometimes, Keep sucks up 100% cpu and becomes really slow after a while

Added by Ward Vandewege almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
-

Description

We have some keepstore servers that have r/o disks (only). After running for a few hours, the keepstored process becomes bloated (high ram) and pegged at (almost) 100% cpu. It still responds to requests, but very slowly. A slow memory leak?

root     24414 60.4 59.0 2491440 2243984 ?     Sl   14:21 263:13      \_ keepstore -enforce-permissions=false -permission-key-file=/etc/keepstore/permission.key -data-manager-token-file=/etc/keepstore/datamanager.key -listen=:25107 -serialize=true -volumes=/...

All CPU time is %user, not iowait:

# iostat -x
Linux 3.16-0.bpo.2-amd64 (keep5.su92l.arvadosapi.com)     04/15/2015     _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.24    0.00    1.13    0.21    0.00   92.42

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     1.30    0.02    2.09     0.54    26.92    26.07     0.04   21.36    8.53   21.49   1.72   0.36
sdb               0.00     0.02    0.06    1.48     3.94   186.27   247.27     0.17  108.47    6.69  112.33   2.07   0.32
sdc               0.00     0.02    0.05    1.49     3.18   186.98   246.86     0.17  109.06    7.23  112.59   2.08   0.32
dm-0              0.00     0.00    0.06    1.50     3.94   186.27   244.24     0.25  157.84    8.05  163.45   2.09   0.32
dm-1              0.00     0.00    0.05    1.51     3.18   186.98   243.76     0.29  184.28    8.66  190.28   2.09   0.33
sdd               0.00     0.03    0.01    2.01     0.05   255.84   253.40     0.07   35.18   14.01   35.27   2.08   0.42
dm-2              0.00     0.00    0.01    2.04     0.05   255.84   249.51     0.26  129.09   29.02  129.51   2.06   0.42

Version:

# dpkg -l |grep keepstore
ii  keepstore                               0.1.20150413210544.0f119f7    amd64        Keepstore is the Keep storage daemon, accessible to clients on the LAN


Subtasks 4 (0 open4 closed)

Task #5933: Configurable limit on block buffers allocated at a given timeResolvedTom Clegg05/06/2015Actions
Task #5925: Review 5748-keepstore-leakResolvedBrett Smith05/06/2015Actions
Task #5950: review 5748-max-buffersResolvedPeter Amstutz05/06/2015Actions
Task #6031: Review 5748-max-buffers-leakResolved05/06/2015Actions

Related issues

Blocks Arvados - Bug #5662: [FUSE] cd/ls sometimes takes too long on su92lClosed04/03/2015Actions
Blocks Arvados - Bug #5713: [FUSE] File access sometimes takes too long on su92lNewTom MorrisActions
Actions

Also available in: Atom PDF