Arvados

When copying a pipeline template with the (default) recursive option, I expected that everything I needed to run the pipeline would be copied. However, when I do this I find that I am missing some keep collections that are required.

For example:

arv-copy --src qr1hi --dst 7lnae --project-uuid 7lnae-j7d0g-9r429yt0rcrdl8x --dst-git-repo jr17/test qr1hi-p5p6p-6xnh5h0zlkl6t1d
WARNING: 'arvados' is an alias for 'qr1hi-s0uqq-izl5prv9uxccky9'
remote: Counting objects: 70315, done.
remote: Compressing objects: 100% (29048/29048), done.
remote: Total 70315 (delta 50423), reused 52681 (delta 37249)
Receiving objects: 100% (70315/70315), 16.47 MiB | 5.89 MiB/s, done.
Resolving deltas: 100% (50423/50423), done.
WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
Counting objects: 14913, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (3351/3351), done.
Writing objects: 100% (14450/14450), 4.89 MiB, done.
Total 14450 (delta 10753), reused 13801 (delta 10259)
To git@git.arvados.internal.sanger.ac.uk:jr17/test.git

• [new branch] git_git_qr1hi_arvadosapi_com_arvados_git_cbccf62a8f58b71f88606f66dbf57e2243caf0b9 -> git_git_qr1hi_arvadosapi_com_arvados_git_cbccf62a8f58b71f88606f66dbf57e2243caf0b9
  WARNING: 'bcosc.gatk2' is an alias for 'qr1hi-s0uqq-cxccfw9wwjcwd4y'
  remote: Counting objects: 79, done.
  remote: Compressing objects: 100% (65/65), done.
  remote: Total 79 (delta 30), reused 0 (delta 0)
  Receiving objects: 100% (79/79), 22.37 KiB, done.
  Resolving deltas: 100% (30/30), done.
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Counting objects: 16, done.
  Delta compression using up to 32 threads.
  Compressing objects: 100% (14/14), done.
  Writing objects: 100% (16/16), 7.92 KiB, done.
  Total 16 (delta 2), reused 14 (delta 0)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_7fb3bdce205d4d07301c50579b98e5b3ae084e79 -> git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_7fb3bdce205d4d07301c50579b98e5b3ae0
  84e79
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Counting objects: 7, done.
  Delta compression using up to 32 threads.
  Compressing objects: 100% (4/4), done.
  Writing objects: 100% (6/6), 1.98 KiB, done.
  Total 6 (delta 0), reused 4 (delta 0)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_cf745004a2844daf5ee662b356a4cc27fa567780 -> git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_cf745004a2844daf5ee662b356a4cc27fa567780
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Counting objects: 13, done.
  Delta compression using up to 32 threads.
  Compressing objects: 100% (10/10), done.
  Writing objects: 100% (11/11), 6.77 KiB, done.
  Total 11 (delta 2), reused 6 (delta 1)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_8df83f10ef188fdf21fc6c3b9b6be5cea56b9af3 -> git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_8df83f10ef188fdf21fc6c3b9b6be5cea56b9af3
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Counting objects: 11, done.
  Delta compression using up to 32 threads.
  Compressing objects: 100% (7/7), done.
  Writing objects: 100% (10/10), 4.41 KiB, done.
  Total 10 (delta 1), reused 6 (delta 1)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_d0a436be483e5c834595ac0993ca75b5ae4b15df -> git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_d0a436be483e5c834595ac0993ca75b5ae4b15df
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Total 0 (delta 0), reused 0 (delta 0)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_arvados_git_2717e707ce048dd9b47754d620663a76256958cf -> git_git_qr1hi_arvadosapi_com_arvados_git_2717e707ce048dd9b47754d620663a76256958cf
  WARNING: 'jr17/test' is an alias for '7lnae-s0uqq-k3t2lc8k3g068zu'
  Counting objects: 40, done.
  Delta compression using up to 32 threads.
  Compressing objects: 100% (16/16), done.
  Writing objects: 100% (39/39), 11.05 KiB, done.
  Total 39 (delta 15), reused 38 (delta 15)
  To git@git.arvados.internal.sanger.ac.uk:jr17/test.git
• [new branch] git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_aaf5045813ba80123cb3a065d80d70389b30e543 -> git_git_qr1hi_arvadosapi_com_bcosc_gatk2_git_aaf5045813ba80123cb3a065d80d70389b30e543
  2015-07-24 21:39:32 arvados.arv-copy¹⁴¹³⁹ INFO:
  2015-07-24 21:39:32 arvados.arv-copy¹⁴¹³⁹ INFO: Success: created copy with uuid 7lnae-p5p6p-nk2q6rlp939etwo

Now when I go to this new copy (7lnae-p5p6p-nk2q6rlp939etwo) in workbench, I find that a number of components have parameter/values in red. For example:
extract-reference input 25b68283b442c1a921ac826296103426+9636
bwa-index bwa_tbz 8b6e2c4916133e1d859c9e812861ce13+70
bwa-aln samtools_tgz c777e23cf13e5d5906abfdc08d84bfdb+74
picard-gatk2-prep picard-zip 687f74675c6a0e925dec619cc2bec25f+77

It seems like perhaps everything stored in git was copied, but collections stored in keep were not copied?