Bug #9049
closed
[SDKs] `arv copy` does not adjust component filters, causing unpredictable runs
Description
A user writes a pipeline template following our Git strategy for pipeline development. They set filters to determine what jobs are eligible for reuse, including a repository filter.
Another user uses arv copy to copy this template somewhere else. The destination Git repository has a different name. However, arv copy does not adjust the filters.
At this point, the pipeline template won't run as intended on the destination cluster. Best case, there is no Git repository on the destination cluster with the same name as the source repository, so the run fails almost immediately. Worst case, there is a Git repository with the same name on the destination cluster, and then something unpredictable happens.
Updated by Brett Smith over 8 years ago
- Subject changed from [SDKs] `arv copy` does not adjust script_filters to [SDKs] `arv copy` does not adjust component filters, causing unpredictable runs
Updated by Brett Smith over 8 years ago
arv copy should do sanity checks on the filters early on.
We should detect the common case of "repository is the component's repository, and script_version is a commit hash," and do the translation correctly.
If it detects filters that it can't translate reliably (they refer to another repository, they have a symbolic script_version filter that isn't being copied, something else), it should immediately abort, let the user know what the problem is, and offer the option to do the copy anyway with some flag (--force?).
If the user uses the --force flag, copy everything verbatim. The user is expected to fix them on the destination cluster.
Updated by Brett Smith over 8 years ago
- Status changed from New to In Progress
- Assigned To set to Brett Smith
Updated by Brett Smith over 8 years ago
(10:06:47 AM) Me: tomclegg: https://dev.arvados.org/issues/9049 suggests having `arv-copy --force` mean "copy a pipeline template's filters verbatim, even if they're not meaningful on the destination." arv-copy already has --force; it means "upload a collection to destination Keep even if it apparently already exists."
(10:07:26 AM) Me: --force-filters? Maybe we can follow the pattern set by apt, and have a whole family of --force-foo options, and then eventually a --force-all.
(10:07:47 AM) tomclegg: --force-filters ... yeah.
(10:08:01 AM) tomclegg: brett: sgtm.
Updated by Nico César over 8 years ago
test 227b7d4fe9e0c65d9e2148af5c10632c60271822
I read the code and learned about contextlib.contextmanager. that was new to me.
a minor change is that I think functools isn't being used
all the rest LGTM for merge into master.
Updated by Brett Smith over 8 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:2a40242e7f47841c02b4eb9f23a9fafc230b37c4.