Arvados

The template is not in order. My pipeline template tb05z-p5p6p-bzn8uoope28ytmo goes allelotype_run > run_lobSTR > sort_bam > index_sort when it should go run_lobSTR > sort_bam > index_sort > allelotype_run

I ran into this copying gatk3

arv-copy --src qr1hi --dst tb05z --dst-git-repo bcosc.bu --project-uuid tb05z-j7d0g-h748bqiwxbr5do2 qr1hi-p5p6p-8584ylktoumirn0
d0136bc494c21f79fc1b6a390561e6cb+2778: 3247M / 3247M 100.0%
Traceback (most recent call last):
File "/usr/local/bin/arv-copy", line 4, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 114, in main
src_arv, dst_arv, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 252, in copy_pipeline_template
pt = copy_collections(pt, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 298, in copy_collections
obj = arvados.util.portable_data_hash_pattern.sub(copy_collection_fn, obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 288, in copy_collection_fn
dst_col = copy_collection(src_id, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 469, in copy_collection
return dst.collections().create(body=c, ensure_unique_name=True).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 135, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
arvados.errors.ApiError: <HttpError 422 when requesting https://tb05z.arvadosapi.com/arvados/v1/collections?ensure_unique_name=true&alt=json returned "Portable data hash does not match computed hash 7eb6160977c6161fee4f365765d33801+2370">

Reviewing ae41687

• All the new API calls need to respect args.retries. Effectively, calling .execute() without any arguments is a bug.
• The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
  

  srccol = src.collections.list(…)
  items = srccol.get('items', [])
  c = None
  # for loop to search items for a matching owner_uuid
  if not c:
      # for loop to search items for one with a name
  if not c:
      if items:
          c = items[0]
      else:
          # warning
  

Thanks.

Brett Smith wrote:

Reviewing ae41687

• All the new API calls need to respect args.retries. Effectively, calling .execute() without any arguments is a bug.

Added num_retries to all the execute() calls in arv-copy.

• The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
  [...]

Sure. Done.

Thanks.

Reviewing 8356273.

Peter Amstutz wrote:

Brett Smith wrote:

• The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
  [...]

Sure. Done.

The second loop (find any collection with a name) has a comment under the for line that's copied+pasted from the first loop, and not correct here. Remove it, and this is good to merge. Thanks.