Bug #16138
closedarv-copy can't copy collections
Description
arvados.errors.ApiError: <HttpError 422 when requesting https://9tee4.arvadosapi.com/arvados/v1/collections?ensure_unique_name=true&alt=json returned "request failed: http://localhost:8000/arvados/v1/collections: 422 Unprocessable Entity: #<ActiveModel::UnknownAttributeError: unknown attribute 'etag' for Collection.> (req-2noq740r4utmyb3ss2px)">
Updated by Peter Amstutz almost 5 years ago
- Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint
Updated by Peter Amstutz over 4 years ago
- Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint
Updated by Peter Amstutz over 4 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima over 4 years ago
Some comments & questions:
- Do we need more testing on this command?
- There’re more references to “pipeline” on argument descriptions, and code comments.
- It seems there’s more pipeline related code:
- Line 273:
migrate_components_filters()
- Line 410:
migrate_jobspec()
- Line 452:
copy_git_repos()
- Line 754:
copy_docker_images()
(it looks for acomponents
key, so it seems pipeline-specific)
- Line 273:
- Is the whilelisted
portable_data_hash
field necessary? Isn’t the PDH computed from others?
Updated by Peter Amstutz over 4 years ago
- Target version deleted (
2020-03-25 Sprint) - Category deleted (
SDKs)
Lucas Di Pentima wrote:
Some comments & questions:
- Do we need more testing on this command?
There's no real testing of arv-copy right now. It would require either setting up an environment with two clusters (which used to be impossible without arvbox, with arvados-server boot maybe easier now) or using mocks for everything (which probably wouldn't have detected the original bug, thus would be low-quality tests).
So I was hoping to just leave it the way it is. Eventually arv-copy will be rewritten in Go and have real testing.
- There’re more references to “pipeline” on argument descriptions, and code comments.
Removed most of them I think.
- It seems there’s more pipeline related code:
- Line 273:
migrate_components_filters()
- Line 410:
migrate_jobspec()
- Line 452:
copy_git_repos()
- Line 754:
copy_docker_images()
(it looks for acomponents
key, so it seems pipeline-specific)
Done.
- Is the whilelisted
portable_data_hash
field necessary? Isn’t the PDH computed from others?
Copying it through ensures that manifest text wasn't corrupted when it was rewritten.
Updated by Peter Amstutz over 4 years ago
16138-fix-arv-copy @ 20e09ab0726d7f0edb4333277c3403f995b960ce
Updated by Lucas Di Pentima over 4 years ago
I've tested it manually with 2 arvbox instances, it worked OK.
There're some more pipeline references that may need cleaning, other than that, it LGTM.
$ grep -in pipeline ./sdk/python/build/lib/arvados/commands/arv_copy.py 103: help='The name of the destination git repository. Required when copying a pipeline recursively.') 106: help='The UUID of the project at the destination to which the pipeline should be copied.') 121: description='Copy a pipeline instance, template, workflow, or collection from one Arvados instance to another.', 471: # as the root item (usually a pipeline) being copied. 710:# It returns a string such as 'Collection', 'PipelineInstance', etc.
Updated by Peter Amstutz over 4 years ago
- Target version set to 2020-04-08 Sprint
Updated by Peter Amstutz over 4 years ago
- Status changed from In Progress to Resolved