Project

General

Profile

Actions

Bug #16138

closed

arv-copy can't copy collections

Added by Peter Amstutz about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Description

arvados.errors.ApiError: <HttpError 422 when requesting https://9tee4.arvadosapi.com/arvados/v1/collections?ensure_unique_name=true&alt=json returned "request failed: http://localhost:8000/arvados/v1/collections: 422 Unprocessable Entity: #<ActiveModel::UnknownAttributeError: unknown attribute 'etag' for Collection.> (req-2noq740r4utmyb3ss2px)">

Subtasks 1 (0 open1 closed)

Task #16233: Review 16138-fix-arv-copyResolvedPeter Amstutz03/20/2020Actions
Actions #1

Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint
Actions #3

Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint
Actions #4

Updated by Peter Amstutz about 4 years ago

  • Assigned To set to Peter Amstutz
Actions #5

Updated by Peter Amstutz about 4 years ago

  • Status changed from New to In Progress
Actions #6

Updated by Lucas Di Pentima about 4 years ago

Some comments & questions:

  • Do we need more testing on this command?
  • There’re more references to “pipeline” on argument descriptions, and code comments.
  • It seems there’s more pipeline related code:
    • Line 273: migrate_components_filters()
    • Line 410: migrate_jobspec()
    • Line 452: copy_git_repos()
    • Line 754: copy_docker_images() (it looks for a components key, so it seems pipeline-specific)
  • Is the whilelisted portable_data_hash field necessary? Isn’t the PDH computed from others?
Actions #7

Updated by Peter Amstutz about 4 years ago

  • Target version deleted (2020-03-25 Sprint)
  • Category deleted (SDKs)

Lucas Di Pentima wrote:

Some comments & questions:

  • Do we need more testing on this command?

There's no real testing of arv-copy right now. It would require either setting up an environment with two clusters (which used to be impossible without arvbox, with arvados-server boot maybe easier now) or using mocks for everything (which probably wouldn't have detected the original bug, thus would be low-quality tests).

So I was hoping to just leave it the way it is. Eventually arv-copy will be rewritten in Go and have real testing.

  • There’re more references to “pipeline” on argument descriptions, and code comments.

Removed most of them I think.

  • It seems there’s more pipeline related code:
    • Line 273: migrate_components_filters()
    • Line 410: migrate_jobspec()
    • Line 452: copy_git_repos()
    • Line 754: copy_docker_images() (it looks for a components key, so it seems pipeline-specific)

Done.

  • Is the whilelisted portable_data_hash field necessary? Isn’t the PDH computed from others?

Copying it through ensures that manifest text wasn't corrupted when it was rewritten.

Actions #9

Updated by Lucas Di Pentima about 4 years ago

I've tested it manually with 2 arvbox instances, it worked OK.
There're some more pipeline references that may need cleaning, other than that, it LGTM.

$ grep -in pipeline ./sdk/python/build/lib/arvados/commands/arv_copy.py
103:        help='The name of the destination git repository. Required when copying a pipeline recursively.')
106:        help='The UUID of the project at the destination to which the pipeline should be copied.')
121:        description='Copy a pipeline instance, template, workflow, or collection from one Arvados instance to another.',
471:            # as the root item (usually a pipeline) being copied.
710:#    It returns a string such as 'Collection', 'PipelineInstance', etc.
Actions #10

Updated by Peter Amstutz about 4 years ago

  • Target version set to 2020-04-08 Sprint
Actions #11

Updated by Peter Amstutz about 4 years ago

  • Status changed from In Progress to Resolved
Actions #12

Updated by Peter Amstutz about 4 years ago

  • Release set to 30
Actions

Also available in: Atom PDF