Bug #11583

[crunch-run] Fix excessive memory use

Added by Tom Clegg over 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

The crunch-run test suite seems to have started leaking memory in #9132. RSS climbs steadily with each test case. Leaking memory on each docker invocation wouldn't be a big problem in real life, since each crunch-run process only invokes one container. However, this could be a clue about the second leak.

Even in versions before #9132, TestWriteLogsLarge consumes multiple gigabytes of memory. The test loop itself doesn't seem to allocate a significant amount of memory. If the problem is that crunch-run tries to buffer everything up to the next newline in memory regardless of length, an errant job could (e.g., with a progress bar) cause crunch-run to allocate an arbitrary amount of memory, too.

    for i := 0; i < 2000000; i++ {
        cr.CrunchLog.Printf("Hello %d", i)
    }

Related issues

Related to Arvados - Bug #12447: crunch-run memory leakResolved10/13/2017

Related to Arvados - Bug #13100: [crunch-run] Replace custom manifest-writing code with collectionFSResolved03/15/2018

Associated revisions

Revision 05bf879d
Added by Tom Clegg over 1 year ago

Merge branch '13100-crunch-run-memory'

refs #13100
refs #11583

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

Revision 95be914a
Added by Tom Clegg over 1 year ago

Merge branch '13100-crunch-run-output'

fixes #13100
fixes #11583
fixes #12606
refs #13048

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg over 2 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg over 2 years ago

Adding \n to the TestWriteLogsLarge Printf format doesn't seem to make any difference, suggesting this isn't just about long lines.

#3 Updated by Tom Morris almost 2 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Tom Clegg over 1 year ago

  • Related to Bug #13100: [crunch-run] Replace custom manifest-writing code with collectionFS added

#5 Updated by Tom Clegg over 1 year ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

#6 Updated by Tom Morris over 1 year ago

  • Target version changed from Arvados Future Sprints to 2018-04-25 Sprint

#7 Updated by Tom Morris about 1 year ago

  • Release set to 13

Also available in: Atom PDF