Bug #12690

crunchstat-summary --container-request does not report on keep i/o

Added by Bryan Cosca about 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
12/05/2017
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

Crunch-run reports the arv-mount stats in arv-mount.txt in the logging collection, but crunchstat-summary doesn't know to look there.

Associated revisions

Revision 35e38be6 (diff)
Added by Tom Clegg over 1 year ago

Fix wrong event_type when getting buffered live logs for crunch2.

refs #12690

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

Revision 04b9d6e9
Added by Peter Amstutz about 1 year ago

Merge branch '12690-12748-crunchstat-summary'

refs #12690
refs #12748
refs #13759

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Bryan Cosca about 2 years ago

  • Subject changed from crunchstat-summary does not report on keep i/o to crunchstat-summary --container-request does not report on keep i/o

#2 Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)

#3 Updated by Tom Morris about 2 years ago

  • Target version set to To Be Groomed

#4 Updated by Tom Morris about 2 years ago

  • Target version changed from To Be Groomed to 2017-12-20 Sprint

#5 Updated by Tom Morris about 2 years ago

  • Assigned To set to Tom Morris

#6 Updated by Tom Morris about 2 years ago

  • Status changed from New to In Progress

#7 Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2017-12-20 Sprint to 2017-12-06 Sprint

#8 Updated by Peter Amstutz about 2 years ago

The arv-mount.txt stats file was just added recently.

Unfortunately this means if you run crunchstat-summary on an older run, you get lots of errors:

Exception in thread Thread-402:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 526, in run_and_release
    return target(*args, **kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 74, in run
    self._run(logdata)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 78, in _run
    for line in logdata:
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/reader.py", line 33, in __iter__
    self._readers.append(collection.open(filename))
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20171010180436-py2.7.egg/arvados/collection.py", line 660, in open
    raise IOError(errno.ENOENT, "File not found", path)
IOError: [Errno 2] File not found: 'arv-mount.txt'

It needs to handle missing stats files.

#9 Updated by Tom Morris about 2 years ago

Is the job/collection buried some place in that stack trace? I'm not seeing it.

#10 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint

#11 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2017-12-20 Sprint to 2018-01-17 Sprint

#12 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2018-01-17 Sprint to 2018-01-31 Sprint

#13 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2018-01-31 Sprint to 2018-02-14 Sprint

#14 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2018-02-14 Sprint to 2018-02-28 Sprint

#15 Updated by Tom Morris almost 2 years ago

  • Target version changed from 2018-02-28 Sprint to 2018-03-14 Sprint

#16 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-03-14 Sprint to 2018-03-28 Sprint

#17 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-03-28 Sprint to 2018-04-11 Sprint

#18 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-04-11 Sprint to 2018-04-25 Sprint

#19 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-04-25 Sprint to 2018-05-09 Sprint

#20 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-05-09 Sprint to 2018-05-23 Sprint

#21 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-05-23 Sprint to 2018-06-06 Sprint

#22 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-06-06 Sprint to 2018-06-20 Sprint

#23 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-06-20 Sprint to 2018-07-03 Sprint

#24 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-07-03 Sprint to 2018-07-18 Sprint

#25 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-07-18 Sprint to 2018-08-01 Sprint

#26 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-08-01 Sprint to 2018-08-15 Sprint

#27 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-08-15 Sprint to 2018-09-05 Sprint

#28 Updated by Peter Amstutz over 1 year ago

It's not looking for the "crunchstat:" prefix on the arv-mount log lines and so it is trying to parse error messages:

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:21:35.090831222Z out of memory

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:22:04.094908800Z out of memory

#29 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-09-05 Sprint to 2018-09-19 Sprint

#30 Updated by Tom Morris over 1 year ago

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

#31 Updated by Peter Amstutz over 1 year ago

Tom Morris wrote:

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

Every line in this file has "crunchstat:" after the timestamp and before the actual stats.

https://collections.e51c5.arvadosapi.com/c=10e2b6c05210cf58317b8d904c02f729-489/_/arv-mount.txt?disposition=inline

#32 Updated by Tom Morris about 1 year ago

  • Target version changed from 2018-09-19 Sprint to 2018-10-03 Sprint

#33 Updated by Peter Amstutz about 1 year ago

  • Target version changed from 2018-10-03 Sprint to 2018-10-17 sprint

#34 Updated by Peter Amstutz about 1 year ago

  • Assigned To changed from Tom Morris to Peter Amstutz

#35 Updated by Peter Amstutz about 1 year ago

  • Status changed from In Progress to Resolved

#36 Updated by Tom Morris about 1 year ago

  • Release set to 14

Also available in: Atom PDF