Story #15093

Work with Ops to decide best DB migration strategy for collection file count & size

Added by Tom Morris 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-

Related issues

Related to Arvados - Story #14484: [API Server] Return collection size and number of files in collection recordResolved03/28/2019

Blocked by Arvados - Story #15180: [Spike] Test migration on production databaseResolved

Associated revisions

Revision 0d8adf9b (diff)
Added by Ward Vandewege about 2 months ago

Move the population of the new columns on the collections table to a standalone
script that should be run separate from the migration. Add a note to the
upgrade documentation along those lines. Make the script not blow up on
collections with invalid manifests, but rather just skip them.

refs #15093
refs #14484

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 6817a411 (diff)
Added by Ward Vandewege about 2 months ago

Address review comments.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision d7775a8a
Added by Ward Vandewege about 2 months ago

Merge branch '15093-move-collection-update-out-of-band' into 1.4-dev

closes #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 10c07dad (diff)
Added by Ward Vandewege about 2 months ago

Update the 'upgrading' documentation to reflect the v1.4.0 release, and warn
about the db migration that can take some time during upgrade.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 6f17bfca (diff)
Added by Ward Vandewege about 2 months ago

Move the population of the new columns on the collections table to a standalone
script that should be run separate from the migration. Add a note to the
upgrade documentation along those lines. Make the script not blow up on
collections with invalid manifests, but rather just skip them.

refs #15093
refs #14484

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 7f68130d (diff)
Added by Ward Vandewege about 2 months ago

Address review comments.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 65c51513 (diff)
Added by Ward Vandewege about 2 months ago

Update the 'upgrading' documentation to reflect the v1.4.0 release, and warn
about the db migration that can take some time during upgrade.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Tom Morris 3 months ago

  • Status changed from New to In Progress

#2 Updated by Tom Morris 3 months ago

  • Assigned To set to Eric Biagiotti

#3 Updated by Tom Morris 3 months ago

  • Related to Story #14484: [API Server] Return collection size and number of files in collection record added

#4 Updated by Tom Morris 3 months ago

  • Status changed from In Progress to New
  • Assigned To deleted (Eric Biagiotti)
  • Target version changed from 2019-04-24 Sprint to To Be Groomed

#5 Updated by Tom Morris 3 months ago

  • Blocked by Story #15180: [Spike] Test migration on production database added

#6 Updated by Ward Vandewege about 2 months ago

  • Target version changed from To Be Groomed to 2019-06-05 Sprint
  • Assigned To set to Ward Vandewege
  • Status changed from New to In Progress
  • Category set to API

#7 Updated by Ward Vandewege about 2 months ago

15093-move-collection-update-out-of-band up for review at 0d8adf9bb22e169fb47d64ca19045283ff5fafd1

#8 Updated by Eric Biagiotti about 2 months ago

Ward Vandewege wrote:

15093-move-collection-update-out-of-band up for review at 0d8adf9bb22e169fb47d64ca19045283ff5fafd1

Just a minor note. Because of the addition of the file_count=0 in the where clause, the distinct_pdh_count query could potentially return a number greater than the total number of pdhs processed and the logging in group_pdhs_for_multiple_transactions would be inaccurate. This doesn't affect actual collection processing though. Could just add the same where clause to distinct_pdh_count query.

Otherwise, this LGTM.

#9 Updated by Ward Vandewege about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

Also available in: Atom PDF