Project

General

Profile

Actions

Bug #12690

closed

crunchstat-summary --container-request does not report on keep i/o

Added by Bryan Cosca over 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

Crunch-run reports the arv-mount stats in arv-mount.txt in the logging collection, but crunchstat-summary doesn't know to look there.

Actions #1

Updated by Bryan Cosca over 6 years ago

  • Subject changed from crunchstat-summary does not report on keep i/o to crunchstat-summary --container-request does not report on keep i/o
Actions #2

Updated by Peter Amstutz over 6 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Morris over 6 years ago

  • Target version set to To Be Groomed
Actions #4

Updated by Tom Morris over 6 years ago

  • Target version changed from To Be Groomed to 2017-12-20 Sprint
Actions #5

Updated by Tom Morris over 6 years ago

  • Assigned To set to Tom Morris
Actions #6

Updated by Tom Morris over 6 years ago

  • Status changed from New to In Progress
Actions #7

Updated by Peter Amstutz over 6 years ago

  • Target version changed from 2017-12-20 Sprint to 2017-12-06 Sprint
Actions #8

Updated by Peter Amstutz over 6 years ago

The arv-mount.txt stats file was just added recently.

Unfortunately this means if you run crunchstat-summary on an older run, you get lots of errors:

Exception in thread Thread-402:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 526, in run_and_release
    return target(*args, **kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 74, in run
    self._run(logdata)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 78, in _run
    for line in logdata:
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/reader.py", line 33, in __iter__
    self._readers.append(collection.open(filename))
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20171010180436-py2.7.egg/arvados/collection.py", line 660, in open
    raise IOError(errno.ENOENT, "File not found", path)
IOError: [Errno 2] File not found: 'arv-mount.txt'

It needs to handle missing stats files.

Actions #9

Updated by Tom Morris over 6 years ago

Is the job/collection buried some place in that stack trace? I'm not seeing it.

Actions #10

Updated by Tom Morris over 6 years ago

  • Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint
Actions #11

Updated by Tom Morris about 6 years ago

  • Target version changed from 2017-12-20 Sprint to 2018-01-17 Sprint
Actions #12

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-01-17 Sprint to 2018-01-31 Sprint
Actions #13

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-01-31 Sprint to 2018-02-14 Sprint
Actions #14

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-02-14 Sprint to 2018-02-28 Sprint
Actions #15

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-02-28 Sprint to 2018-03-14 Sprint
Actions #16

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-03-14 Sprint to 2018-03-28 Sprint
Actions #17

Updated by Tom Morris about 6 years ago

  • Target version changed from 2018-03-28 Sprint to 2018-04-11 Sprint
Actions #18

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-04-11 Sprint to 2018-04-25 Sprint
Actions #19

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-04-25 Sprint to 2018-05-09 Sprint
Actions #20

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-05-09 Sprint to 2018-05-23 Sprint
Actions #21

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-05-23 Sprint to 2018-06-06 Sprint
Actions #22

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-06-06 Sprint to 2018-06-20 Sprint
Actions #23

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-06-20 Sprint to 2018-07-03 Sprint
Actions #24

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-07-03 Sprint to 2018-07-18 Sprint
Actions #25

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-07-18 Sprint to 2018-08-01 Sprint
Actions #26

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-08-01 Sprint to 2018-08-15 Sprint
Actions #27

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-08-15 Sprint to 2018-09-05 Sprint
Actions #28

Updated by Peter Amstutz over 5 years ago

It's not looking for the "crunchstat:" prefix on the arv-mount log lines and so it is trying to parse error messages:

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:21:35.090831222Z out of memory

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:22:04.094908800Z out of memory
Actions #29

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-09-05 Sprint to 2018-09-19 Sprint
Actions #30

Updated by Tom Morris over 5 years ago

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

Actions #31

Updated by Peter Amstutz over 5 years ago

Tom Morris wrote:

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

Every line in this file has "crunchstat:" after the timestamp and before the actual stats.

https://collections.e51c5.arvadosapi.com/c=10e2b6c05210cf58317b8d904c02f729-489/_/arv-mount.txt?disposition=inline

Actions #32

Updated by Tom Morris over 5 years ago

  • Target version changed from 2018-09-19 Sprint to 2018-10-03 Sprint
Actions #33

Updated by Peter Amstutz over 5 years ago

  • Target version changed from 2018-10-03 Sprint to 2018-10-17 sprint
Actions #34

Updated by Peter Amstutz over 5 years ago

  • Assigned To changed from Tom Morris to Peter Amstutz
Actions #35

Updated by Peter Amstutz over 5 years ago

  • Status changed from In Progress to Resolved
Actions #36

Updated by Tom Morris over 5 years ago

  • Release set to 14
Actions

Also available in: Atom PDF