Bug #11788

[arv-put] should preserve directory structure when given directory references

Added by Tom Morris over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/30/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Neither of these examples worked to create a collection containing files with the subdirectory references to the Config and Interop directories preserved:

arv-put Config InterOp SampleSheet.csv
arv-put Config/ InterOp/ SampleSheet.csv

Instead the contents of both subdirectories were flattened into the top level.

If given a directory reference, that directory should be included in the paths stored in the resulting collection.


Subtasks

Task #11826: Review 11788-arvput-dir-referencesResolvedLucas Di Pentima

Associated revisions

Revision 5598bbc3
Added by Lucas Di Pentima over 4 years ago

Merge branch '11788-arvput-dir-references'
Closes #11788

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Lucas Di Pentima over 4 years ago

  • Assigned To set to Lucas Di Pentima

#2 Updated by Tom Clegg over 4 years ago

Might be good if this API change could be toggled by the user. For example, if there is a file foo/bar/x.txt,
  • rsync foo/bar/ dest:baz/ creates dest:baz/x.txt (analogous to existing arv-put behavior)
  • tar cf - foo/bar/ creates a tarball with foo/bar/x.txt (analogous to proposed arv-put behavior)

#3 Updated by Lucas Di Pentima over 4 years ago

  • Status changed from New to In Progress

#4 Updated by Lucas Di Pentima over 4 years ago

Updates at cf9874c59
Test run: https://ci.curoverse.com/job/developer-run-tests/334/

Fixed the specific case that when passing a directory reference to arv-put, it skips uploading the directory and instead goes ahead uploading its contents directly.

So, now the directory uploads behave like this:
  • "arv-put relative/dirpath1 another/relative/dirpath2/ afile.txt" → ./afile.txt, ./path1/..., ./path2/...
  • "arv-put /absolute/path/of/some/dir" → ./dir/...

To make arv-put behave more like rsync, I think we could add a new argument --destination-path, so that all referenced dirs/files are uploaded inside that named subcollection, allowing the previous behavior (dir flattening) by passing "--destination-path /", for example.

#5 Updated by Tom Clegg over 4 years ago

Offline discussion concluded we should behave like rsync. I think that means, if the current directory is "/cwd" and the following local files exist:
  • /cwd/dir1/file1.txt
  • /cwd/dir2/file2.txt
We should have:
arv-put args resulting collection contents
dir1 /dir1/file1.txt
/cwd/dir1 /dir1/file1.txt
dir1/ /file1.txt
/cwd/dir1/ /file1.txt
dir1 dir2 /dir1/file1.txt
/dir2/file2.txt
dir1 dir2/ /dir1/file1.txt
/file2.txt
/cwd/dir1 /cwd/dir2/ /dir1/file1.txt
/file2.txt

If/when we add a "--relative" flag, it should work like this:

arv-put args resulting collection contents
--relative dir1 /dir1/file1.txt
--relative /cwd/dir1 /cwd/dir1/file1.txt
--relative dir1/ /dir1/file1.txt
--relative /cwd/dir1/ /cwd/dir1/file1.txt
--relative dir1 dir2 /dir1/file1.txt
/dir2/file2.txt
--relative dir1 dir2/ /dir1/file1.txt
/dir2/file2.txt
--relative /cwd/dir1 /cwd/dir2/ /cwd/dir1/file1.txt
/cwd/dir2/file2.txt
--relative /cwd/dir1 /cwd/./dir2/ /cwd/dir1/file1.txt
/dir2/file2.txt

#6 Updated by Lucas Di Pentima over 4 years ago

Updates at 9ca7acc39
Test run: https://ci.curoverse.com/job/developer-run-tests/335/

Fixed code to cover both behaviors (with/without trailing slash). Updated tests & help message.

#7 Updated by Tom Clegg over 4 years ago

LGTM, thanks!

#8 Updated by Lucas Di Pentima over 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:5598bbc3c71da60b7b7a665b36495b957d6a3c52.

Also available in: Atom PDF