Project

General

Profile

Actions

Bug #9223

closed

[Node manager] Uses huge amount of RAM on AWS

Added by Peter Amstutz almost 8 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
Node Manager
Target version:
-
Story points:
-

Description

When node manager starts on AWS, it quicky uses a huge amount of RAM (2 GiB) despite not doing anything.

I've tracked this down to arvnodeman.computenode.driver.ec2.ComputeNodeDriver._init_image_id which calls libcloud.compute.drivers.ec2.BaseEC2NodeDriver.list_images. It seems that the problem is (a) it is listing a huge number of images (many thousands) and (b) it is retaining a lot of memory even for image records that should be GC'd.

We are currently using libcloud 0.16 on AWS which is an very out of date version.


Related issues

Related to Arvados - Bug #12055: [node manager] ec2 set tags on createResolvedLucas Di Pentima08/16/2017Actions
Actions #1

Updated by Peter Amstutz almost 8 years ago

  • Description updated (diff)
  • Category set to Node Manager
Actions #2

Updated by Peter Amstutz almost 8 years ago

As a first try I suggest updating libcloud to the latest version we have packaged (0.20 I think?)

Actions #3

Updated by Tom Morris over 7 years ago

The current version of libcloud is 1.2.1. We should definitely not be using a <1.0 release when 1.0+ is available.

https://pypi.python.org/pypi/apache-libcloud/1.2.1

Actions #4

Updated by Peter Amstutz over 7 years ago

The complication is that we're using a fork of libcloud which adds the Azure support that we need. We should try to get that merged upstream but requires allocating some engineering time to move it through the process.

Actions #5

Updated by Nico César over 6 years ago

  • Status changed from New to In Progress

resolved in #12163

Actions #6

Updated by Peter Amstutz over 5 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF