Story #4305

[Node Manager] Investigate and try to address RAM use

Added by Brett Smith over 2 years ago. Updated 4 months ago.

Status:ClosedStart date:10/23/2014
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Node Manager
Target version:-
Story points1.0
Velocity based estimate-

Description

Node Manager uses ~1GiB of RAM on our production clusters. Investigate why it's so large, and try to bring it down.

When I run it on my desktop in local testing mode with the dummy driver, it takes ~200MiB. This makes me think apache-libcloud's EC2 driver is responsible for a good chunk of this. But that's just a theory that needs proving.

History

#1 Updated by Brett Smith over 2 years ago

Another idea that occurred to me: arvnodeman.computenode.ec2 loads information about node sizes, images, security groups, etc. by listing them all and finding the one with the matching name. It shouldn't be holding on to the full list, but maybe the underlying ec2 driver is caching it, or maybe it's just hanging around without being garbage collected.

This approach is illustrated in the apache-libcloud tutorials (literally on the project's front page), so I figured it was best, but I didn't investigate very deeply. If there's a way to look up these items directly by their ID, that might make a noticeable dent on RAM use.

#2 Updated by Ward Vandewege over 2 years ago

  • Story points set to 1.0

#3 Updated by Peter Amstutz over 1 year ago

Node manager running on Azure consumes significantly less memory. This suggests that maybe the libcloud EC2 drive is doing something silly, or that we are using it in a silly way.

#4 Updated by Tom Clegg 4 months ago

  • Status changed from New to Closed

#5 Updated by Tom Clegg 4 months ago

  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF