[FUSE] high memory consumption (possible leak) in long-running arv-mount
|Target version:||Arvados Future Sprints|
|Story points||-||Remaining (hours)||0.00 hour|
|Velocity based estimate||-|
We have a (little used) arv-mount that has been running since 6th September.
It was started with the command line:
`/usr/bin/python2.7 /usr/bin/arv-mount /tmp/keep_jr17`
Since no `--file-cache` or `--directory-cache` options were given, those should have been the defaults of 256MiB and 128MiB. If I start a new arv-cache also with defaults and then read some large data through it and exercise some large directories (such as doing a find in `by_tag`), I am able to get memory usage up to 514MB, which seems reasonable.
However, the arv-mount that has been running for the past 77 days is now taking up 15GB of RAM!
I suspect this issue might be related to the increasing memory usage I observed and reported in #10535 when the python SDK test suite got stuck in a tight PollClient loop forever (where "forever" is until it ran the system out of memory).
#4 Updated by Peter Amstutz 3 months ago
- This might be related/due to https://dev.arvados.org/issues/11158 → it is that it is trying to enumerate the entire home directory and it uses up all memory trying to store the full contents.
- Cache management clears releases unused Collection objects. However, those Collection objects may have prefetch threads. If they don't get stopped, they will leak. *
#8 Updated by Peter Amstutz 3 months ago
Lucas Di Pentima wrote:
The thread stopping code was added on a
CollectionDirectoryBasesubclass, is it possible for this problem to happen with
TmpCollectionDirectoryobjects too? Maybe it’s better to do the thread stopping on
CollectionDirectoryBase objects are used to hold
Subcollection objects, which don't have a
TmpCollectionDirectory are not candidates for cache eviction (persisted() is False). The
finalize() method already calls
The difference between
finalize() is that
clear() is called when we want to evict an inode's cached contents, whereas
finalize() is called when the inode will be deleted entirely.