Bug #10445
[SDKs] Fix memory leak in Python SDK Collection class
0%
Description
Currently, each new CollectionReader creates its own API client, and Keep client, and block cache unless the caller supplies an API client object (and Keep client object?). If a caller creates 10 CollectionReaders and reads 64 MiB from each one, the program will use 640 MiB. Possibly due to HTTP KeepAlive behavior, Python does not reclaim memory even if the caller unreferences the CollectionReaders. For example, this script leaks memory and network connections:
import arvados
uuid = '......'
for i in range(20):
cr = arvados.collection.CollectionReader(uuid)
for fn in cr:
f = cr.open(fn)
f.read()
f.close()
Proposed improvement:
Share block caches between auto-instantiated API clients that use the same settings.
Related issues
History
#1
Updated by Tom Clegg over 5 years ago
- File leaktest.py leaktest.py added
#2
Updated by Peter Amstutz over 5 years ago
The read cache is actually in KeepClient. Reads go through the block manager, but sharing the read cache is a matter of using a shared keepclient object.
(Technically, you could even create multiple KeepClient objects and initialize them with the same KeepBlockCache).
#3
Updated by Tom Clegg over 5 years ago
- Description updated (diff)
#4
Updated by Tom Morris over 4 years ago
- Target version set to Arvados Future Sprints
#5
Updated by Ward Vandewege 11 months ago
- Target version deleted (
Arvados Future Sprints)