Story #15093

Work with Ops to decide best DB migration strategy for collection file count & size

Added by Tom Morris 12 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Story points:
-

Related issues

Related to Arvados - Story #14484: [API Server] Return collection size and number of files in collection recordResolved03/28/2019

Blocked by Arvados - Story #15180: [Spike] Test migration on production databaseResolved

Associated revisions

Revision 0d8adf9b (diff)
Added by Ward Vandewege 10 months ago

Move the population of the new columns on the collections table to a standalone
script that should be run separate from the migration. Add a note to the
upgrade documentation along those lines. Make the script not blow up on
collections with invalid manifests, but rather just skip them.

refs #15093
refs #14484

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 6817a411 (diff)
Added by Ward Vandewege 10 months ago

Address review comments.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision d7775a8a
Added by Ward Vandewege 10 months ago

Merge branch '15093-move-collection-update-out-of-band' into 1.4-dev

closes #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 10c07dad (diff)
Added by Ward Vandewege 10 months ago

Update the 'upgrading' documentation to reflect the v1.4.0 release, and warn
about the db migration that can take some time during upgrade.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 6f17bfca (diff)
Added by Ward Vandewege 10 months ago

Move the population of the new columns on the collections table to a standalone
script that should be run separate from the migration. Add a note to the
upgrade documentation along those lines. Make the script not blow up on
collections with invalid manifests, but rather just skip them.

refs #15093
refs #14484

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 7f68130d (diff)
Added by Ward Vandewege 10 months ago

Address review comments.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 65c51513 (diff)
Added by Ward Vandewege 10 months ago

Update the 'upgrading' documentation to reflect the v1.4.0 release, and warn
about the db migration that can take some time during upgrade.

refs #15093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Tom Morris 12 months ago

  • Status changed from New to In Progress

#2 Updated by Tom Morris 12 months ago

  • Assigned To set to Eric Biagiotti

#3 Updated by Tom Morris 12 months ago

  • Related to Story #14484: [API Server] Return collection size and number of files in collection record added

#4 Updated by Tom Morris 12 months ago

  • Status changed from In Progress to New
  • Assigned To deleted (Eric Biagiotti)
  • Target version changed from 2019-04-24 Sprint to To Be Groomed

#5 Updated by Tom Morris 11 months ago

  • Blocked by Story #15180: [Spike] Test migration on production database added

#6 Updated by Ward Vandewege 10 months ago

  • Category set to API
  • Status changed from New to In Progress
  • Assigned To set to Ward Vandewege
  • Target version changed from To Be Groomed to 2019-06-05 Sprint

#7 Updated by Ward Vandewege 10 months ago

15093-move-collection-update-out-of-band up for review at 0d8adf9bb22e169fb47d64ca19045283ff5fafd1

#8 Updated by Eric Biagiotti 10 months ago

Ward Vandewege wrote:

15093-move-collection-update-out-of-band up for review at 0d8adf9bb22e169fb47d64ca19045283ff5fafd1

Just a minor note. Because of the addition of the file_count=0 in the where clause, the distinct_pdh_count query could potentially return a number greater than the total number of pdhs processed and the logging in group_pdhs_for_multiple_transactions would be inaccurate. This doesn't affect actual collection processing though. Could just add the same where clause to distinct_pdh_count query.

Otherwise, this LGTM.

#9 Updated by Ward Vandewege 10 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF