Bug #11693
closed
[Crunch2] Correctly handle symlinks to input files in output dir
Description
Crunch v1 and cwltool allow the output directory to have symlinks to input files, which are then efficiently propagated to output. Unfortunately, crunch-run fails in this case. Some scripts rely on this behavior. It needs to support dereferencing symlinks and adding them to the output collection.
Updated by Lucas Di Pentima almost 8 years ago
Reviewed a340487a7
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
- File
services/crunch-run/crunchrun.go:962:outputSuffixalready assigned onL952 - What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”bit onL958). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building thebindslist onL924?
Updated by Peter Amstutz almost 8 years ago
Lucas Di Pentima wrote:
Reviewed a340487a7
Just a minor comment and a couple of questions:
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
- File
services/crunch-run/crunchrun.go:962:outputSuffixalready assigned onL952
Copy and paste error. Fixed.
- What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”bit onL958). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building thebindslist onL924?
Inputs are always collections. Symlinks to files in temporary directories isn't supported.
Filtering the binds array to just collections is a good idea. Fixed.
Updated by Peter Amstutz almost 8 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:f26f70d0a60798065c5f7a5cb91b95587cc9e9ef.