Bug #6095

[SDKs] Make arv-copy preserve pipeline template component order

Added by Brett Smith over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
SDKs
Target version:
Start date:
05/20/2015
Due date:
% Done:

100%

Estimated time:
(Total: 2.00 h)
Story points:
0.5

Description

Implementation steps:

  1. Add an OrderedJsonModel class to arvados.api. This is a subclass of googleapiclient.model.JsonModel. It overrides the de/serialize methods. When it calls a json load/dump method, it passes in collections.OrderedDict as the object_hook to preserve order (refer to the json module documentation).
  2. Ensure that whenever arv-copy instantiates an API client, it passes in an OrderedJsonModel instance as the model keyword argument. Refer to the apiclient.discovery PyDoc.

This will make arv-copy do the right thing with pipeline templates, and give us the right API hooks to extend this functionality to other Python clients easily.

We should not make OrderedJsonModel the default model because the space overhead is noticeable, and not necessary for most applications.


Subtasks

Task #6117: Review 6095-arv-copy-preserve-object-order-wipResolvedBrett Smith


Related issues

Related to Arvados - Story #2992: [Workbench] Support "display_order" (integer) key for pipeline template components. Obey when rendering pipeline summary/details in Workbench.Rejected03/13/2019

Associated revisions

Revision 872cc3d7
Added by Brett Smith about 4 years ago

Merge branch '6095-arv-copy-preserve-object-order-wip'

Closes #6095, #6117.

Revision 45fcd8ae (diff)
Added by Brett Smith about 4 years ago

6095: Stop demoting OrderedDicts to dicts in arv-copy.

History: first there was 79564b0ac7d03327cc351bbd6df544ab1f776380.
This preserved the order of copied pipeline templates, but that's in
part because it stopped recursing through those templates.
1b8caff3ad598744e4a0379b01fc95ca4838caa0 fixed the recursion, but then
started losing the order again. This retains the order by ensuring we
copy OrderedDicts as OrderedDicts.

Refs #6095, #7001.

Revision dac77d5d (diff)
Added by Brett Smith about 4 years ago

6095: Brown paper bag fix the last commit.

I swear I tried to test before I pushed, but I used --no-recursive,
which meant we never got to this code. That also explains why we
didn't catch the lack of recursion earlier. Apologies. Refs #6095.

History

#1 Updated by Brett Smith over 4 years ago

  • Description updated (diff)

#2 Updated by Brett Smith over 4 years ago

  • Subject changed from [DRAFT] [SDKs] Make arv-copy preserve pipeline template component order to [SDKs] Make arv-copy preserve pipeline template component order

#3 Updated by Brett Smith over 4 years ago

  • Assigned To set to Brett Smith

#4 Updated by Brett Smith about 4 years ago

  • Status changed from New to In Progress

#5 Updated by Brett Smith about 4 years ago

6095-arv-copy-preserve-object-order-wip is up for review. A couple of notes:

  • If you look at the original JsonModel source, you may wonder about the whole simplejson module business. If you follow the source all the way down, you'll find that it uses the standard json module whenever that's available. Since we're pretty heavily committed to Python 2.7 at this point, I went ahead and hardcoded that for simplicity.
  • Suggested test by hand: arv-copy --src [instance --dst [same] --no-recursive --project-uuid [project uuid] [pipeline template uuid]

#6 Updated by Tom Clegg about 4 years ago

LGTM @ 2f64531

#7 Updated by Brett Smith about 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:872cc3d79d99f5fc4b26b970c3d269ca363ea27c.

Also available in: Atom PDF