Story #4305

[Node Manager] Investigate and try to address RAM use

Added by Brett Smith over 2 years ago. Updated 20 days ago.

Status:ClosedStart date:10/23/2014
Priority:NormalDue date:
Assignee:-% Done:


Category:Node Manager
Target version:-
Story points1.0
Velocity based estimate-


Node Manager uses ~1GiB of RAM on our production clusters. Investigate why it's so large, and try to bring it down.

When I run it on my desktop in local testing mode with the dummy driver, it takes ~200MiB. This makes me think apache-libcloud's EC2 driver is responsible for a good chunk of this. But that's just a theory that needs proving.


#1 Updated by Brett Smith over 2 years ago

Another idea that occurred to me: arvnodeman.computenode.ec2 loads information about node sizes, images, security groups, etc. by listing them all and finding the one with the matching name. It shouldn't be holding on to the full list, but maybe the underlying ec2 driver is caching it, or maybe it's just hanging around without being garbage collected.

This approach is illustrated in the apache-libcloud tutorials (literally on the project's front page), so I figured it was best, but I didn't investigate very deeply. If there's a way to look up these items directly by their ID, that might make a noticeable dent on RAM use.

#2 Updated by Ward Vandewege over 2 years ago

  • Story points set to 1.0

#3 Updated by Peter Amstutz over 1 year ago

Node manager running on Azure consumes significantly less memory. This suggests that maybe the libcloud EC2 drive is doing something silly, or that we are using it in a silly way.

#4 Updated by Tom Clegg 21 days ago

  • Status changed from New to Closed

#5 Updated by Tom Clegg 20 days ago

  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF