Bug #11693
closed
[Crunch2] Correctly handle symlinks to input files in output dir
Added by Peter Amstutz over 7 years ago.
Updated over 7 years ago.
Description
Crunch v1 and cwltool allow the output directory to have symlinks to input files, which are then efficiently propagated to output. Unfortunately, crunch-run fails in this case. Some scripts rely on this behavior. It needs to support dereferencing symlinks and adding them to the output collection.
- Description updated (diff)
- Assigned To set to Peter Amstutz
Reviewed a340487a7
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
Just a minor comment and a couple of questions:
- File
services/crunch-run/crunchrun.go:962
: outputSuffix
already assigned on L952
- What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”
bit on L958
). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building the binds
list on L924
?
Lucas Di Pentima wrote:
Reviewed a340487a7
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
Just a minor comment and a couple of questions:
- File
services/crunch-run/crunchrun.go:962
: outputSuffix
already assigned on L952
Copy and paste error. Fixed.
- What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”
bit on L958
). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building the binds
list on L924
?
Inputs are always collections. Symlinks to files in temporary directories isn't supported.
Filtering the binds array to just collections is a good idea. Fixed.
Now at aceb2d1ed239fa82fcb8bb352b632a8d92251dac
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:f26f70d0a60798065c5f7a5cb91b95587cc9e9ef.
Also available in: Atom
PDF