Actions
Feature #13403
open[crunch-run] Cancel container on FUSE error
Story points:
2.0
Release:
Release relationship:
Auto
Description
https://dev.arvados.org/issues/13377#note-4
The fact that we kept the container alive for 7 hours retrying seems like a problem, too. In the context of a container, if arv-mount gives up on a fuse request and returns an error analogous to "filesystem is corrupt / disk is dead" to the caller, should we automatically fail the container?
If arv-mount has a block read error (or really anything that will get turned into EIO by FUSE), crunch-run should cancel the container (the container request may retry by existing logic, though).
Proposed design:
- arv-mount emits a well known error string when it returns a major file system error (EIO)
- crunch-run monitors arv-mount output and looks for major file system errors or out or memory error (MemoryError)
- on seeing the error from arv-mount, crunch-run logs an error of its own and cancels the container.
Updated by Peter Amstutz over 6 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz over 6 years ago
- Status changed from In Progress to New
Updated by Tom Morris over 6 years ago
- Subject changed from Cancel container on FUSE error to [crunch-run] Cancel container on FUSE error
- Target version changed from To Be Groomed to Arvados Future Sprints
Updated by Peter Amstutz over 3 years ago
- Target version deleted (
Arvados Future Sprints)
Actions