Arvados

Currently, the "arv-mount --unmount" strategy is to kill the arv-mount process and disconnect the mount point as quickly as possible, minimizing risk of getting stuck. It makes no effort to flush writes, wait for processes to close any open files, etc.

This is the behavior desired by crunch-run and crunch-job: in the container/job setting, by the time arv-mount --unmount gets called, anything worth saving has already been saved, and all processes using the mount are dead/abandoned/irrelevant.

However, in other situations like interactive use on a shell node, it makes more sense to flush writes and wait for files to close, i.e., do a clean unmount. This graceful option should be the default, like it is in "umount".

Proposed change:

• arv-mount --force --unmount path invokes the current forceful behavior (and crunch-job and crunch-run are updated to use this)
• arv-mount --unmount path does a "nice" shutdown, waiting (if necessary) for all processes to close all files, and doing whatever else is necessary to ensure arv-mount flushes all writes to Keep before exiting. If --timeout is given, and that time is exceeded, error out without trying anything more forceful.

To avoid deployment races, this change can be made in three stages:

1. arv-mount accepts a --force flag, and behaves forcefully whether or not it's given
2. crunch-run and crunch-job pass the --force flag
3. update crunch-run to behave non-forcefully when --force isn't given