Bug #11788: [arv-put] should preserve directory structure when given directory references - Arvados

Actions

Copy link

Bug #11788

closed

[arv-put] should preserve directory structure when given directory references

Added by Tom Morris over 7 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Lucas Di Pentima

Category:

Target version:

2017-06-21 sprint

Story points:

Description

Neither of these examples worked to create a collection containing files with the subdirectory references to the Config and Interop directories preserved:

arv-put Config InterOp SampleSheet.csv
arv-put Config/ InterOp/ SampleSheet.csv

Instead the contents of both subdirectories were flattened into the top level.

If given a directory reference, that directory should be included in the paths stored in the resulting collection.

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Lucas Di Pentima over 7 years ago

Assigned To set to Lucas Di Pentima

Actions

Copy link

Updated by Tom Clegg over 7 years ago

Might be good if this API change could be toggled by the user. For example, if there is a file foo/bar/x.txt,

rsync foo/bar/ dest:baz/ creates dest:baz/x.txt (analogous to existing arv-put behavior)
tar cf - foo/bar/ creates a tarball with foo/bar/x.txt (analogous to proposed arv-put behavior)

Actions

Copy link

Updated by Lucas Di Pentima over 7 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Lucas Di Pentima over 7 years ago

Updates at cf9874c59
Test run: https://ci.curoverse.com/job/developer-run-tests/334/

Fixed the specific case that when passing a directory reference to arv-put, it skips uploading the directory and instead goes ahead uploading its contents directly.

So, now the directory uploads behave like this:

"arv-put relative/dirpath1 another/relative/dirpath2/ afile.txt" → ./afile.txt, ./path1/..., ./path2/...
"arv-put /absolute/path/of/some/dir" → ./dir/...

To make arv-put behave more like rsync, I think we could add a new argument --destination-path, so that all referenced dirs/files are uploaded inside that named subcollection, allowing the previous behavior (dir flattening) by passing "--destination-path /", for example.

Actions

Copy link

Updated by Tom Clegg over 7 years ago

Offline discussion concluded we should behave like rsync. I think that means, if the current directory is "/cwd" and the following local files exist:

/cwd/dir1/file1.txt
/cwd/dir2/file2.txt

We should have:

arv-put args	resulting collection contents
dir1	/dir1/file1.txt
/cwd/dir1	/dir1/file1.txt
dir1/	/file1.txt
/cwd/dir1/	/file1.txt
dir1 dir2	/dir1/file1.txt /dir2/file2.txt
dir1 dir2/	/dir1/file1.txt /file2.txt
/cwd/dir1 /cwd/dir2/	/dir1/file1.txt /file2.txt

If/when we add a "--relative" flag, it should work like this:

arv-put args	resulting collection contents
--relative dir1	/dir1/file1.txt
--relative /cwd/dir1	/cwd/dir1/file1.txt
--relative dir1/	/dir1/file1.txt
--relative /cwd/dir1/	/cwd/dir1/file1.txt
--relative dir1 dir2	/dir1/file1.txt /dir2/file2.txt
--relative dir1 dir2/	/dir1/file1.txt /dir2/file2.txt
--relative /cwd/dir1 /cwd/dir2/	/cwd/dir1/file1.txt /cwd/dir2/file2.txt
--relative /cwd/dir1 /cwd/./dir2/	/cwd/dir1/file1.txt /dir2/file2.txt