Project

General

Profile

Actions

Idea #15180

closed

[Spike] Test migration on production database

Added by Tom Morris almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Eric Biagiotti
Category:
-
Target version:
Story points:
0.5
Release relationship:
Auto

Related issues

Blocks Arvados - Idea #15093: Work with Ops to decide best DB migration strategy for collection file count & sizeResolvedWard VandewegeActions
Actions #1

Updated by Tom Morris almost 5 years ago

  • Private changed from Yes to No
Actions #2

Updated by Tom Morris almost 5 years ago

  • Parent task deleted (#15093)
Actions #3

Updated by Tom Morris almost 5 years ago

  • Tracker changed from Task to Idea
  • Target version changed from To Be Groomed to 2019-05-08 Sprint
Actions #4

Updated by Tom Morris almost 5 years ago

  • Blocks Idea #15093: Work with Ops to decide best DB migration strategy for collection file count & size added
Actions #5

Updated by Eric Biagiotti almost 5 years ago

The RecomputeFileNamesIndex migration (#13752) in 1.3.1 on e51c5 took 77 min to run #15008. The RecomputeFileNamesIndex migration is similar to #14484 in that each collection is modified. The difference being that #14484 also uses the Ruby sdk to parse manifest text and retrieve the file count and total file size.

Actions #6

Updated by Eric Biagiotti almost 5 years ago

  • Status changed from New to In Progress

Ran the migration on my machine using the production database from e51c5 with 4228696 collections (2555374 distinct pdhs).

The migration took 291 minutes. Approximately 871896 collections per hour.

My machine:

CPU(s):              8
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
Model name:          Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

Additionally, as a very rough estimate, parsing took ~.003 milliseconds per manifest, while each database commit took ~6 milliseconds.

Actions #7

Updated by Eric Biagiotti almost 5 years ago

  • Status changed from In Progress to Resolved
Actions #8

Updated by Tom Morris almost 5 years ago

  • Release set to 15
Actions

Also available in: Atom PDF