Bug #5305

Arv-copy issues

Added by Bryan Cosca almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
SDKs
Target version:
Start date:
02/24/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

arv-copy --src qr1hi --dst tb05z --dst-git-repo bcosc.bu --project-uuid tb05z-j7d0g-8vvizxhqnqj24rv qr1hi-p5p6p-9pkaxt6qjnkxhhu
remote: Counting objects: 60606, done.
remote: Compressing objects: 100% (22834/22834), done.
remote: Total 60606 (delta 43180), reused 48798 (delta 34196)
Receiving objects: 100% (60606/60606), 11.84 MiB | 9.35 MiB/s, done.
Resolving deltas: 100% (43180/43180), done.
Counting objects: 41467, done.
Compressing objects: 100% (10624/10624), done.
Writing objects: 100% (41467/41467), 6.54 MiB, done.
Total 41467 (delta 28760), reused 40860 (delta 28435)
To :bcosc.bu.git * [new branch] git_git_qr1hi_arvadosapi_com_arvados_git_462fbba4ab742a72a3cf057dc06610a51af6b0f0 -> git_git_qr1hi_arvadosapi_com_arvados_git_462fbba4ab742a72a3cf057dc06610a51af6b0f0
qr1hi-4zz18-1xkkjw497ovvp24: 5990M / 5990M 100.0%
2015-02-24 23:02:07 arvados.arv-copy5817 INFO:
2015-02-24 23:02:07 arvados.arv-copy5817 INFO: Success: created copy with uuid tb05z-p5p6p-bzn8uoope28ytm

It does not put the copy into the right project. Instead of going to lobstr v3 it went to my home.

Also, names of collections are not preserved, the names are "none"

I'll be adding onto here if I find more things.


Subtasks

Task #5341: Review 5305-arv-copy-fixesResolvedPeter Amstutz

Task #5324: Fix copy issuesResolved


Related issues

Related to Arvados - Bug #5310: [SDK] arv-copy breaks when copying by PDH and multiple collections have the same PDHResolved02/26/2015

Associated revisions

Revision 6cd3b6fb
Added by Peter Amstutz over 5 years ago

Merge branch '5305-arv-copy-fixes' closes #5305

History

#1 Updated by Bryan Cosca almost 6 years ago

The template is not in order. My pipeline template tb05z-p5p6p-bzn8uoope28ytmo goes allelotype_run > run_lobSTR > sort_bam > index_sort when it should go run_lobSTR > sort_bam > index_sort > allelotype_run

#2 Updated by Bryan Cosca almost 6 years ago

I ran into this copying gatk3

arv-copy --src qr1hi --dst tb05z --dst-git-repo bcosc.bu --project-uuid tb05z-j7d0g-h748bqiwxbr5do2 qr1hi-p5p6p-8584ylktoumirn0
d0136bc494c21f79fc1b6a390561e6cb+2778: 3247M / 3247M 100.0%
Traceback (most recent call last):
File "/usr/local/bin/arv-copy", line 4, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 114, in main
src_arv, dst_arv, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 252, in copy_pipeline_template
pt = copy_collections(pt, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in copy_collections
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 302, in <dictcomp>
return {v: copy_collections(obj[v], src, dst, args) for v in obj}
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 298, in copy_collections
obj = arvados.util.portable_data_hash_pattern.sub(copy_collection_fn, obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 288, in copy_collection_fn
dst_col = copy_collection(src_id, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 469, in copy_collection
return dst.collections().create(body=c, ensure_unique_name=True).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 135, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
arvados.errors.ApiError: <HttpError 422 when requesting https://tb05z.arvadosapi.com/arvados/v1/collections?ensure_unique_name=true&alt=json returned "Portable data hash does not match computed hash 7eb6160977c6161fee4f365765d33801+2370">

#3 Updated by Ward Vandewege almost 6 years ago

  • Target version set to Bug Triage

#4 Updated by Peter Amstutz almost 6 years ago

  • Target version changed from Bug Triage to 2015-03-11 sprint

#5 Updated by Peter Amstutz almost 6 years ago

  • Category set to SDKs
  • Assigned To set to Peter Amstutz

#6 Updated by Brett Smith over 5 years ago

Reviewing ae41687

  • All the new API calls need to respect args.retries. Effectively, calling .execute() without any arguments is a bug.
  • The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
    srccol = src.collections.list()
    items = srccol.get('items', [])
    c = None
    # for loop to search items for a matching owner_uuid
    if not c:
        # for loop to search items for one with a name
    if not c:
        if items:
            c = items[0]
        else:
            # warning
    

Thanks.

#7 Updated by Peter Amstutz over 5 years ago

Brett Smith wrote:

Reviewing ae41687

  • All the new API calls need to respect args.retries. Effectively, calling .execute() without any arguments is a bug.

Added num_retries to all the execute() calls in arv-copy.

  • The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
    [...]

Sure. Done.

Thanks.

#8 Updated by Brett Smith over 5 years ago

Reviewing 8356273.

Peter Amstutz wrote:

Brett Smith wrote:

  • The new name-finding heuristics in copy_collection() are a little difficult to follow with multiple layers of branching, and a couple of large if blocks followed by short else clauses. What do you think about a structure like this?
    [...]

Sure. Done.

The second loop (find any collection with a name) has a comment under the for line that's copied+pasted from the first loop, and not correct here. Remove it, and this is good to merge. Thanks.

#9 Updated by Peter Amstutz over 5 years ago

  • Status changed from New to Resolved

Applied in changeset arvados|commit:6cd3b6fb1c143017c6bca4db899c8c6ddd237572.

#10 Updated by Ward Vandewege over 5 years ago

  • Story points set to 0.5

Also available in: Atom PDF