Story #11309

[Crunch2] crunchstat-summary --container UUID should summarize container logs

Added by Tom Clegg almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
-
Target version:
2017-08-30 Sprint
Start date:
08/16/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Subtasks

Task #11894: Review 11309-crunchstat-summary-containersResolvedLucas Di Pentima

Related issues

Related to Arvados - Bug #10359: [crunchstat-summary] Limit concurrency to keep memory use under controlResolved10/26/2016

Related to Arvados - Bug #12196: [crunchstat-summary] avoid opening too many files at once when working on a large container treeResolved08/30/2017

Associated revisions

Revision 89359cee
Added by Tom Clegg over 3 years ago

Merge branch '11309-crunchstat-summary-containers'

refs #11309

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

Revision 3389110d (diff)
Added by Tom Clegg over 3 years ago

Update test to match new log message.

refs #11309

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Morris almost 4 years ago

  • Assigned To set to Tom Morris
  • Target version set to Arvados Future Sprints

#2 Updated by Tom Morris over 3 years ago

  • Tracker changed from Bug to Story
  • Target version changed from Arvados Future Sprints to 2017-07-05 sprint

#3 Updated by Tom Morris over 3 years ago

  • Target version changed from 2017-07-05 sprint to 2017-07-19 sprint

#4 Updated by Tom Morris over 3 years ago

  • Target version changed from 2017-07-19 sprint to 2017-07-05 sprint

#5 Updated by Tom Morris over 3 years ago

  • Target version changed from 2017-07-05 sprint to 2017-07-19 sprint

#6 Updated by Tom Morris over 3 years ago

  • Target version changed from 2017-07-19 sprint to 2017-08-02 sprint

#7 Updated by Tom Morris over 3 years ago

  • Target version changed from 2017-08-02 sprint to 2017-08-16 sprint

#8 Updated by Tom Clegg over 3 years ago

  • Assigned To changed from Tom Morris to Tom Clegg

#9 Updated by Tom Clegg over 3 years ago

  • Status changed from New to In Progress

#10 Updated by Tom Clegg over 3 years ago

11309-crunchstat-summary-containers @ dc140f2c277a7571669f5bdf492bac6448a0fba0

I added "--container UUID" and "--container-request UUID" options to make it more obvious, but under the hood they're the same as "--job": they figure out the object type from the UUID.

I also dealt with #10359 because it was easy: added a --threads N option, default 8.

The report is unwieldy for a container with lots of "child" containers. Might be better to sort them by name (or creation time of container?) instead of UUID.

#11 Updated by Tom Clegg over 3 years ago

11309-crunchstat-summary-containers @ f04ff34e47cc5ff1e625e9a5c1b1b0cbeb810d8c

Child containers are now sorted by creation time of container request.

#12 Updated by Tom Clegg over 3 years ago

  • Target version changed from 2017-08-16 sprint to 2017-08-30 Sprint

#13 Updated by Lucas Di Pentima over 3 years ago

A couple comments:

  • Failing test: test_container (tests.test_examples.SummarizeContainer)
  • File tools/crunchstat-summary/crunchstat_summary/summarizer.py
    • Line 437: Is there a case where process could be a dict? If so, what happens with the uuid variable?

#14 Updated by Tom Clegg over 3 years ago

Lucas Di Pentima wrote:

  • Failing test: test_container (tests.test_examples.SummarizeContainer)

Oops, updated fake API record to include created_at.

  • File tools/crunchstat-summary/crunchstat_summary/summarizer.py
    • Line 437: Is there a case where process could be a dict? If so, what happens with the uuid variable?

This "pass already-retrieved object instead of just uuid" didn't end up getting used here but I was thinking it might end up being useful in #10472. But yes, it was broken; I've fixed it, and changed the arg to "process_or_uuid" to make it readable.

11309-crunchstat-summary-containers @ c77cb0091a8488eae52341920e8e32beb8fc47dc

#15 Updated by Lucas Di Pentima over 3 years ago

This LGTM, thanks.

#16 Updated by Tom Clegg over 3 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF