Story #7832: [FUSE] [SDKs] Define __slots__ to improve memory utilization - Arvados

Story #7832

[FUSE] [SDKs] Define slots to improve memory utilization

Added by Brett Smith about 4 years ago. Updated about 4 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Peter Amstutz

Category:

SDKs

Target version:

2015-12-02 sprint

Start date:

11/20/2015

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

0.5

Description

Both arv-get and FUSE use a lot of memory. Idea: define __slots__ for classes where it will noticeably improve memory utilization.

First class candidates: objects that are both instantiated commonly and long-lived. The Collection classes, File classes, Range, and LocatorAndRange in the SDK. The Directory and File classes in FUSE.

Second class candidates: objects that are instantiated commonly but with shorter lives. KeepService, ThreadLimiter, RetryLoop.

Subtasks

Associated revisions

Revision f486405f
Added by Peter Amstutz about 4 years ago

Merge branch '7832-pysdk-use-slots' refs #7832

History

#1 Updated by Brett Smith about 4 years ago

Target version set to Arvados Future Sprints

#2 Updated by Peter Amstutz about 4 years ago

7832-pysdk-use-slots

Adds slots to Range, LocatorAndRange, and CacheSlot. These are the obvious low hanging fruit, and together they reduce memory usage by half on a very complex manifest (215dd32873bfa002aa0387c6794e4b2c+54081534)

master:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter    21943 10.4  9.3 1312088 1139036 pts/3 Sl+  16:58   0:36 python get.py

7832-pysdk-use-slots

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter    26853 16.5  4.9 781648 608492 pts/3   Sl+  17:04   0:37 python get.py