Arvados

Some theories:

• This might be related/due to https://dev.arvados.org/issues/11158 → it is that it is trying to enumerate the entire home directory and it uses up all memory trying to store the full contents.
• Cache management clears releases unused Collection objects. However, those Collection objects may have prefetch threads. If they don't get stopped, they will leak. *

10584-fuse-stop-threads

Ensure get/put threads are stopped before releasing reference to Collection object. Unclear if this is the source of the problem, but seems like a good idea regardless.

The thread stopping code was added on a CollectionDirectoryBase subclass, is it possible for this problem to happen with TmpCollectionDirectory objects too? Maybe it’s better to do the thread stopping on CollectionDirectoryBase?

Lucas Di Pentima wrote:

The thread stopping code was added on a CollectionDirectoryBase subclass, is it possible for this problem to happen with TmpCollectionDirectory objects too? Maybe it’s better to do the thread stopping on CollectionDirectoryBase?

CollectionDirectoryBase objects are used to hold Subcollection objects, which don't have a stop_threads() method.

TmpCollectionDirectory are not candidates for cache eviction (persisted() is False). The finalize() method already calls stop_threads().

The difference between clear() and finalize() is that clear() is called when we want to evict an inode's cached contents, whereas finalize() is called when the inode will be deleted entirely.

Ok, so this looks good to me. Thanks!

• Look at user interaction history with keep
• Track metrics
• Instrumentation to report memory usage / ownership

Might be worth running the "retry PUT" test many times in a row. At least once I've seen the test suite get stuck there using lots of memory.