Bug #9049

[SDKs] `arv copy` does not adjust component filters, causing unpredictable runs

Added by Brett Smith over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
-
Target version:
Start date:
05/17/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

A user writes a pipeline template following our Git strategy for pipeline development. They set filters to determine what jobs are eligible for reuse, including a repository filter.

Another user uses arv copy to copy this template somewhere else. The destination Git repository has a different name. However, arv copy does not adjust the filters.

At this point, the pipeline template won't run as intended on the destination cluster. Best case, there is no Git repository on the destination cluster with the same name as the source repository, so the run fails almost immediately. Worst case, there is a Git repository with the same name on the destination cluster, and then something unpredictable happens.


Subtasks

Task #9225: Review 9049-arv-copy-filters-wipResolvedBrett Smith

Associated revisions

Revision 2a40242e
Added by Brett Smith over 5 years ago

Merge branch '9049-arv-copy-filters-wip'

Closes #9049, #9225.

History

#1 Updated by Brett Smith over 5 years ago

  • Subject changed from [SDKs] `arv copy` does not adjust script_filters to [SDKs] `arv copy` does not adjust component filters, causing unpredictable runs

#2 Updated by Brett Smith over 5 years ago

  • Description updated (diff)

#3 Updated by Brett Smith over 5 years ago

arv copy should do sanity checks on the filters early on.

We should detect the common case of "repository is the component's repository, and script_version is a commit hash," and do the translation correctly.

If it detects filters that it can't translate reliably (they refer to another repository, they have a symbolic script_version filter that isn't being copied, something else), it should immediately abort, let the user know what the problem is, and offer the option to do the copy anyway with some flag (--force?).

If the user uses the --force flag, copy everything verbatim. The user is expected to fix them on the destination cluster.

#4 Updated by Brett Smith over 5 years ago

  • Status changed from New to In Progress
  • Assigned To set to Brett Smith

#5 Updated by Brett Smith over 5 years ago

(10:06:47 AM) Me: tomclegg: https://dev.arvados.org/issues/9049 suggests having `arv-copy --force` mean "copy a pipeline template's filters verbatim, even if they're not meaningful on the destination." arv-copy already has --force; it means "upload a collection to destination Keep even if it apparently already exists."
(10:07:26 AM) Me: --force-filters? Maybe we can follow the pattern set by apt, and have a whole family of --force-foo options, and then eventually a --force-all.
(10:07:47 AM) tomclegg: --force-filters ... yeah.
(10:08:01 AM) tomclegg: brett: sgtm.

#6 Updated by Brett Smith over 5 years ago

  • Target version set to 2016-05-25 sprint

#7 Updated by Nico C├ęsar over 5 years ago

test 227b7d4fe9e0c65d9e2148af5c10632c60271822

I read the code and learned about contextlib.contextmanager. that was new to me.

a minor change is that I think functools isn't being used
all the rest LGTM for merge into master.

#8 Updated by Brett Smith over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:2a40242e7f47841c02b4eb9f23a9fafc230b37c4.

Also available in: Atom PDF