Arvados

The buggy behavior is here:

process.py:1109 mergedirs()

This function was designed to clean up the contents of "listing" or "secondaryFiles" to be suitable for materialization, where there can only be one file with a given 'basename' but there could be two or more directory literals with the same name that can be safely merged into a single one.

It also handles the case of two or more file entries with the same basename that don't refer to the same file.

However, this method is both buggy, and used inappropriately.

• It handles the case of two files that have different locations but the same basename, but does not check the location of Directories, so if you have two directories with different locations, one of them gets forgotten.
• It should only be applied to "listing" and "secondaryFiles". scandeps applies it to every level regardless of whether enforcing unique basenames makes sense.

scandeps should selectively apply it only to secondaryFiles and listing, or possibly not at all. I'm not sure what problem this was originally intended to solve.

I think I remember what this was for.

If you had two dependent files which were in subdirectories:

- class: File
  basename: file1
  location: foo/file1
- class: File
  basename: file2
  location: foo/file2

If you apply "nesting" to mimic the source directory structure, you get this:

- class: Directory
  basename: foo
  listing:
    - class: File
      basename: file1
      location: foo/file1
- class: Directory
  basename: foo
  listing:
    - class: File
      basename: file2
      location: foo/file2

mergedirs() turns it into this:

- class: Directory
  basename: foo
  listing:
    - class: File
      basename: file1
      location: foo/file1
    - class: File
      basename: file2
      location: foo/file2

The bug is in the cwltool scandeps / mergedir function:

https://github.com/common-workflow-language/cwltool/pull/1615

Building a new cwltool release with the bug fix.

Will have a branch that updates to the new release.

18723-cwl-upload @ 0494b40a03348e584dbc255d93615a4ef01afaf9

developer-run-tests: #2908

18723-cwl-upload @ e2b3ed63a292537ec7bf37a904ad2925d6afa0bf

Added integration test