Project

General

Profile

Actions

Idea #11309

closed

[Crunch2] crunchstat-summary --container UUID should summarize container logs

Added by Tom Clegg about 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/16/2017
Due date:
Story points:
-

Subtasks 1 (0 open1 closed)

Task #11894: Review 11309-crunchstat-summary-containersResolvedLucas Di Pentima08/16/2017Actions

Related issues

Related to Arvados - Bug #10359: [crunchstat-summary] Limit concurrency to keep memory use under controlResolvedTom Clegg10/26/2016Actions
Related to Arvados - Bug #12196: [crunchstat-summary] avoid opening too many files at once when working on a large container treeResolvedTom Clegg08/30/2017Actions
Actions #1

Updated by Tom Morris about 7 years ago

  • Assigned To set to Tom Morris
  • Target version set to Arvados Future Sprints
Actions #2

Updated by Tom Morris almost 7 years ago

  • Tracker changed from Bug to Idea
  • Target version changed from Arvados Future Sprints to 2017-07-05 sprint
Actions #3

Updated by Tom Morris almost 7 years ago

  • Target version changed from 2017-07-05 sprint to 2017-07-19 sprint
Actions #4

Updated by Tom Morris almost 7 years ago

  • Target version changed from 2017-07-19 sprint to 2017-07-05 sprint
Actions #5

Updated by Tom Morris almost 7 years ago

  • Target version changed from 2017-07-05 sprint to 2017-07-19 sprint
Actions #6

Updated by Tom Morris almost 7 years ago

  • Target version changed from 2017-07-19 sprint to 2017-08-02 sprint
Actions #7

Updated by Tom Morris over 6 years ago

  • Target version changed from 2017-08-02 sprint to 2017-08-16 sprint
Actions #8

Updated by Tom Clegg over 6 years ago

  • Assigned To changed from Tom Morris to Tom Clegg
Actions #9

Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress
Actions #10

Updated by Tom Clegg over 6 years ago

11309-crunchstat-summary-containers @ dc140f2c277a7571669f5bdf492bac6448a0fba0

I added "--container UUID" and "--container-request UUID" options to make it more obvious, but under the hood they're the same as "--job": they figure out the object type from the UUID.

I also dealt with #10359 because it was easy: added a --threads N option, default 8.

The report is unwieldy for a container with lots of "child" containers. Might be better to sort them by name (or creation time of container?) instead of UUID.

Actions #11

Updated by Tom Clegg over 6 years ago

11309-crunchstat-summary-containers @ f04ff34e47cc5ff1e625e9a5c1b1b0cbeb810d8c

Child containers are now sorted by creation time of container request.

Actions #12

Updated by Tom Clegg over 6 years ago

  • Target version changed from 2017-08-16 sprint to 2017-08-30 Sprint
Actions #13

Updated by Lucas Di Pentima over 6 years ago

A couple comments:

  • Failing test: test_container (tests.test_examples.SummarizeContainer)
  • File tools/crunchstat-summary/crunchstat_summary/summarizer.py
    • Line 437: Is there a case where process could be a dict? If so, what happens with the uuid variable?
Actions #14

Updated by Tom Clegg over 6 years ago

Lucas Di Pentima wrote:

  • Failing test: test_container (tests.test_examples.SummarizeContainer)

Oops, updated fake API record to include created_at.

  • File tools/crunchstat-summary/crunchstat_summary/summarizer.py
    • Line 437: Is there a case where process could be a dict? If so, what happens with the uuid variable?

This "pass already-retrieved object instead of just uuid" didn't end up getting used here but I was thinking it might end up being useful in #10472. But yes, it was broken; I've fixed it, and changed the arg to "process_or_uuid" to make it readable.

11309-crunchstat-summary-containers @ c77cb0091a8488eae52341920e8e32beb8fc47dc

Actions #15

Updated by Lucas Di Pentima over 6 years ago

This LGTM, thanks.

Actions #16

Updated by Tom Clegg over 6 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF