Story #15180

[Spike] Test migration on production database

Added by Tom Morris 8 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
0.5
Release relationship:
Auto

Related issues

Blocks Arvados - Story #15093: Work with Ops to decide best DB migration strategy for collection file count & sizeResolved

History

#1 Updated by Tom Morris 8 months ago

  • Private changed from Yes to No

#2 Updated by Tom Morris 8 months ago

  • Parent task deleted (#15093)

#3 Updated by Tom Morris 8 months ago

  • Tracker changed from Task to Story
  • Target version changed from To Be Groomed to 2019-05-08 Sprint

#4 Updated by Tom Morris 8 months ago

  • Blocks Story #15093: Work with Ops to decide best DB migration strategy for collection file count & size added

#5 Updated by Eric Biagiotti 8 months ago

The RecomputeFileNamesIndex migration (#13752) in 1.3.1 on e51c5 took 77 min to run #15008. The RecomputeFileNamesIndex migration is similar to #14484 in that each collection is modified. The difference being that #14484 also uses the Ruby sdk to parse manifest text and retrieve the file count and total file size.

#6 Updated by Eric Biagiotti 7 months ago

  • Status changed from New to In Progress

Ran the migration on my machine using the production database from e51c5 with 4228696 collections (2555374 distinct pdhs).

The migration took 291 minutes. Approximately 871896 collections per hour.

My machine:

CPU(s):              8
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
Model name:          Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

Additionally, as a very rough estimate, parsing took ~.003 milliseconds per manifest, while each database commit took ~6 milliseconds.

#7 Updated by Eric Biagiotti 7 months ago

  • Status changed from In Progress to Resolved

#8 Updated by Tom Morris 7 months ago

  • Release set to 15

Also available in: Atom PDF