Story #4305

[Node Manager] Investigate and try to address RAM use

Added by Brett Smith over 3 years ago. Updated 10 months ago.

Assigned To:
Node Manager
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


Node Manager uses ~1GiB of RAM on our production clusters. Investigate why it's so large, and try to bring it down.

When I run it on my desktop in local testing mode with the dummy driver, it takes ~200MiB. This makes me think apache-libcloud's EC2 driver is responsible for a good chunk of this. But that's just a theory that needs proving.


#1 Updated by Brett Smith over 3 years ago

Another idea that occurred to me: arvnodeman.computenode.ec2 loads information about node sizes, images, security groups, etc. by listing them all and finding the one with the matching name. It shouldn't be holding on to the full list, but maybe the underlying ec2 driver is caching it, or maybe it's just hanging around without being garbage collected.

This approach is illustrated in the apache-libcloud tutorials (literally on the project's front page), so I figured it was best, but I didn't investigate very deeply. If there's a way to look up these items directly by their ID, that might make a noticeable dent on RAM use.

#2 Updated by Ward Vandewege over 3 years ago

  • Story points set to 1.0

#3 Updated by Peter Amstutz over 2 years ago

Node manager running on Azure consumes significantly less memory. This suggests that maybe the libcloud EC2 drive is doing something silly, or that we are using it in a silly way.

#4 Updated by Tom Clegg over 1 year ago

  • Status changed from New to Closed

#5 Updated by Tom Clegg over 1 year ago

  • Target version deleted (Arvados Future Sprints)

#6 Updated by Nico C├ęsar 10 months ago

see #12163

Also available in: Atom PDF