Arvados

The arv-mount.txt stats file was just added recently.

Unfortunately this means if you run crunchstat-summary on an older run, you get lots of errors:

Exception in thread Thread-402:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 526, in run_and_release
    return target(*args, **kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 74, in run
    self._run(logdata)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/summarizer.py", line 78, in _run
    for line in logdata:
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/crunchstat_summary-0.1.20171205000254-py2.7.egg/crunchstat_summary/reader.py", line 33, in __iter__
    self._readers.append(collection.open(filename))
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20171010180436-py2.7.egg/arvados/collection.py", line 660, in open
    raise IOError(errno.ENOENT, "File not found", path)
IOError: [Errno 2] File not found: 'arv-mount.txt'

It needs to handle missing stats files.

Is the job/collection buried some place in that stack trace? I'm not seeing it.

It's not looking for the "crunchstat:" prefix on the arv-mount log lines and so it is trying to parse error messages:

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:21:35.090831222Z out of memory

Error parsing value u'of' (stat u'memory', category u'out'): ValueError("invalid literal for int() with base 10: 'of'",)
2018-08-16T03:22:04.094908800Z out of memory

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

Tom Morris wrote:

Finally getting back to looking at this. arv-mount.txt doesn't use the crunchstat: prefix. Was this arv-mount.txt or crunchstat.txt? Do you have a link to the job so that I can use it for testing? (I can probably work around it even without the prefix).

Every line in this file has "crunchstat:" after the timestamp and before the actual stats.

https://collections.e51c5.arvadosapi.com/c=10e2b6c05210cf58317b8d904c02f729-489/_/arv-mount.txt?disposition=inline