Arvados

For example, some output from this afternoon shows that in two consecutive runs the number of overreplicated blocks does not change (it seems to be stuck at 4737089, which is the same number it had as of the run last night). I would have expected it would have moved those to the trash and they would no longer be showing up as overreplicated once they are in the trash.

2016/06/19 15:12:46 ===
2016/06/19 15:12:46 12 replicas (6 blocks, 402731660 bytes) lost (0=have<want)
2016/06/19 15:12:46 1231 replicas (163 blocks, 80390127068 bytes) underreplicated (0<have<want)
2016/06/19 15:12:46 1141920 replicas (570080 blocks, 71769882700386 bytes) just right (have=want)
2016/06/19 15:12:46 4737089 replicas (4728845 blocks, 230577300632292 bytes) overreplicated (have>want>0)
2016/06/19 15:12:46 25722 replicas (8648 blocks, 2071796729 bytes) unreferenced (have>want=0, new)
2016/06/19 15:12:46 1170608 replicas (692946 blocks, 600101925167 bytes) garbage (have>want=0, old)
2016/06/19 15:12:46 ===
2016/06/19 15:12:46 10601328 replicas (5299094 blocks, 531960816569846 bytes) total commitment (excluding unreferenced)
2016/06/19 15:12:46 16533504 replicas (6000682 blocks, 763059498065306 bytes) total usage
2016/06/19 15:12:46 ===

2016/06/19 18:12:38 ===
2016/06/19 18:12:38 12 replicas (6 blocks, 402731660 bytes) lost (0=have<want)
2016/06/19 18:12:38 1231 replicas (163 blocks, 80390127068 bytes) underreplicated (0<have<want)
2016/06/19 18:12:38 1141920 replicas (570080 blocks, 71769882700386 bytes) just right (have=want)
2016/06/19 18:12:38 4737089 replicas (4728845 blocks, 230577300632292 bytes) overreplicated (have>want>0)
2016/06/19 18:12:38 25565 replicas (8595 blocks, 2071752841 bytes) unreferenced (have>want=0, new)
2016/06/19 18:12:38 1170765 replicas (692999 blocks, 600101969055 bytes) garbage (have>want=0, old)
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 10601328 replicas (5299094 blocks, 531960816569846 bytes) total commitment (excluding unreferenced)
2016/06/19 18:12:38 16533504 replicas (6000682 blocks, 763059498065306 bytes) total usage
2016/06/19 18:12:38 ===

Here is the full output from that run:

root@humgen-01-01:~# keep-balance -config /root/keep-balance.json -commit-pulls -commit-trash
2016/06/19 12:16:01 starting up: will scan every 10m0s and on SIGUSR1
2016/06/19 12:16:01 Run: start
2016/06/19 12:16:01 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 12:16:01 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.459981ms
2016/06/19 12:16:01 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.036692ms
2016/06/19 12:16:01 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.562848ms
2016/06/19 12:16:01 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.56567ms
2016/06/19 12:16:01 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.916922ms
2016/06/19 12:16:01 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.922641ms
2016/06/19 12:16:01 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.953074ms
2016/06/19 12:16:01 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.911003ms
2016/06/19 12:16:01 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.822891ms
2016/06/19 12:16:01 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.875824ms
2016/06/19 12:16:01 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.937196ms
2016/06/19 12:16:01 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: took 2.899201ms
2016/06/19 12:16:01 GetCurrentState: start
2016/06/19 12:16:02 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:02 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 12:16:05 collections: 0/3273174
2016/06/19 12:16:17 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): add 1337686 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): add 1346446 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): add 1337822 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): add 1338775 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): add 1339176 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): add 1312482 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): add 1338270 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): add 1340508 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): add 1335581 replicas to map
2016/06/19 12:16:18 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): add 1334745 replicas to map
2016/06/19 12:16:19 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:21 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:21 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:24 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): add 1373440 replicas to map
2016/06/19 12:16:26 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:26 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:27 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:28 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:29 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:30 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:31 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:16:32 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:17:15 collections: 25000/3273174
2016/06/19 12:18:25 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): add 1798573 replicas to map
2016/06/19 12:18:28 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 12:18:39 collections: 50000/3273174
2016/06/19 12:20:02 collections: 75000/3273174
2016/06/19 12:21:21 collections: 100000/3273174
2016/06/19 12:22:43 collections: 125000/3273174
2016/06/19 12:24:04 collections: 150000/3273174
2016/06/19 12:25:33 collections: 175000/3273174
2016/06/19 12:26:52 collections: 200000/3273174
2016/06/19 12:28:14 collections: 225000/3273174
2016/06/19 12:29:35 collections: 250000/3273174
2016/06/19 12:31:00 collections: 275000/3273174
2016/06/19 12:32:20 collections: 300000/3273174
2016/06/19 12:33:38 collections: 324238/3273174
2016/06/19 12:35:02 collections: 343038/3273174
2016/06/19 12:36:11 collections: 368038/3273174
2016/06/19 12:37:40 collections: 393038/3273174
2016/06/19 12:39:00 collections: 418038/3273174
2016/06/19 12:40:21 collections: 443038/3273174
2016/06/19 12:41:46 collections: 468038/3273174
2016/06/19 12:43:05 collections: 493038/3273174
2016/06/19 12:44:27 collections: 518038/3273174
2016/06/19 12:45:47 collections: 543038/3273174
2016/06/19 12:47:03 collections: 568038/3273174
2016/06/19 12:48:25 collections: 593038/3273174
2016/06/19 12:49:50 collections: 618038/3273174
2016/06/19 12:51:11 collections: 643038/3273174
2016/06/19 12:52:39 collections: 668038/3273174
2016/06/19 12:54:06 collections: 693038/3273174
2016/06/19 12:55:22 collections: 718038/3273174
2016/06/19 12:56:46 collections: 743038/3273174
2016/06/19 12:58:11 collections: 768038/3273174
2016/06/19 12:59:28 collections: 793038/3273174
2016/06/19 13:00:56 collections: 818038/3273174
2016/06/19 13:02:17 collections: 843038/3273174
2016/06/19 13:03:38 collections: 868038/3273174
2016/06/19 13:04:58 collections: 893038/3273174
2016/06/19 13:06:15 collections: 918038/3273174
2016/06/19 13:07:34 collections: 943038/3273174
2016/06/19 13:08:56 collections: 968038/3273174
2016/06/19 13:10:16 collections: 993038/3273174
2016/06/19 13:11:35 collections: 1018038/3273174
2016/06/19 13:12:52 collections: 1043038/3273174
2016/06/19 13:14:15 collections: 1068038/3273174
2016/06/19 13:15:37 collections: 1093038/3273174
2016/06/19 13:16:55 collections: 1118038/3273174
2016/06/19 13:18:15 collections: 1143038/3273174
2016/06/19 13:19:38 collections: 1168038/3273174
2016/06/19 13:20:55 collections: 1193038/3273174
2016/06/19 13:22:13 collections: 1218038/3273174
2016/06/19 13:23:29 collections: 1243038/3273174
2016/06/19 13:24:48 collections: 1268038/3273174
2016/06/19 13:26:07 collections: 1293038/3273174
2016/06/19 13:27:29 collections: 1318038/3273174
2016/06/19 13:28:49 collections: 1343038/3273174
2016/06/19 13:30:10 collections: 1368038/3273174
2016/06/19 13:31:34 collections: 1393038/3273174
2016/06/19 13:32:52 collections: 1418038/3273174
2016/06/19 13:34:11 collections: 1443038/3273174
2016/06/19 13:35:36 collections: 1468038/3273174
2016/06/19 13:36:52 collections: 1493038/3273174
2016/06/19 13:38:08 collections: 1518038/3273174
2016/06/19 13:39:33 collections: 1543038/3273174
2016/06/19 13:40:54 collections: 1568038/3273174
2016/06/19 13:42:16 collections: 1593038/3273174
2016/06/19 13:43:31 collections: 1618038/3273174
2016/06/19 13:44:51 collections: 1643038/3273174
2016/06/19 13:46:13 collections: 1668038/3273174
2016/06/19 13:47:31 collections: 1693038/3273174
2016/06/19 13:48:46 collections: 1718038/3273174
2016/06/19 13:50:04 collections: 1743038/3273174
2016/06/19 13:51:23 collections: 1768038/3273174
2016/06/19 13:52:42 collections: 1793038/3273174
2016/06/19 13:54:02 collections: 1818038/3273174
2016/06/19 13:55:26 collections: 1843038/3273174
2016/06/19 13:56:47 collections: 1868038/3273174
2016/06/19 13:58:04 collections: 1893038/3273174
2016/06/19 13:59:25 collections: 1918038/3273174
2016/06/19 14:00:42 collections: 1943038/3273174
2016/06/19 14:02:00 collections: 1968038/3273174
2016/06/19 14:03:19 collections: 1993038/3273174
2016/06/19 14:04:36 collections: 2018038/3273174
2016/06/19 14:05:55 collections: 2043038/3273174
2016/06/19 14:07:13 collections: 2068038/3273174
2016/06/19 14:08:36 collections: 2093038/3273174
2016/06/19 14:09:54 collections: 2118038/3273174
2016/06/19 14:11:09 collections: 2143038/3273174
2016/06/19 14:12:25 collections: 2168038/3273174
2016/06/19 14:13:44 collections: 2193038/3273174
2016/06/19 14:15:05 collections: 2218038/3273174
2016/06/19 14:16:27 collections: 2243038/3273174
2016/06/19 14:17:47 collections: 2268038/3273174
2016/06/19 14:19:10 collections: 2293038/3273174
2016/06/19 14:20:28 collections: 2318038/3273174
2016/06/19 14:21:44 collections: 2343038/3273174
2016/06/19 14:23:00 collections: 2368038/3273174
2016/06/19 14:24:17 collections: 2393038/3273174
2016/06/19 14:25:37 collections: 2418038/3273174
2016/06/19 14:26:55 collections: 2443038/3273174
2016/06/19 14:28:11 collections: 2468038/3273174
2016/06/19 14:29:26 collections: 2493038/3273174
2016/06/19 14:30:47 collections: 2518038/3273174
2016/06/19 14:32:06 collections: 2543038/3273174
2016/06/19 14:33:21 collections: 2568038/3273174
2016/06/19 14:34:41 collections: 2593038/3273174
2016/06/19 14:35:58 collections: 2618038/3273174
2016/06/19 14:37:20 collections: 2643038/3273174
2016/06/19 14:38:37 collections: 2668038/3273174
2016/06/19 14:39:50 collections: 2693038/3273174
2016/06/19 14:41:11 collections: 2718038/3273174
2016/06/19 14:42:30 collections: 2743038/3273174
2016/06/19 14:43:49 collections: 2768038/3273174
2016/06/19 14:45:06 collections: 2793038/3273174
2016/06/19 14:46:22 collections: 2818038/3273174
2016/06/19 14:47:44 collections: 2843038/3273174
2016/06/19 14:49:05 collections: 2868038/3273174
2016/06/19 14:50:22 collections: 2893038/3273174
2016/06/19 14:51:40 collections: 2918038/3273174
2016/06/19 14:52:57 collections: 2943038/3273174
2016/06/19 14:54:17 collections: 2968038/3273174
2016/06/19 14:55:32 collections: 2993038/3273174
2016/06/19 14:56:53 collections: 3018038/3273174
2016/06/19 14:58:10 collections: 3043038/3273174
2016/06/19 14:59:27 collections: 3068038/3273174
2016/06/19 15:00:47 collections: 3093038/3273174
2016/06/19 15:02:05 collections: 3118038/3273174
2016/06/19 15:03:19 collections: 3143038/3273174
2016/06/19 15:04:36 collections: 3168038/3273174
2016/06/19 15:05:57 collections: 3193038/3273174
2016/06/19 15:07:14 collections: 3218038/3273174
2016/06/19 15:08:34 collections: 3243038/3273174
2016/06/19 15:09:50 collections: 3268038/3273174
2016/06/19 15:10:06 collections: 3273174/3273174
2016/06/19 15:10:06 collections: 3273174/3273174
2016/06/19 15:11:26 GetCurrentState: took 2h55m24.782781041s
2016/06/19 15:11:26 ComputeChangeSets: start
2016/06/19 15:12:42 ComputeChangeSets: took 1m16.270671548s
2016/06/19 15:12:46 ===
2016/06/19 15:12:46 12 replicas (6 blocks, 402731660 bytes) lost (0=have<want)
2016/06/19 15:12:46 1231 replicas (163 blocks, 80390127068 bytes) underreplicated (0<have<want)
2016/06/19 15:12:46 1141920 replicas (570080 blocks, 71769882700386 bytes) just right (have=want)
2016/06/19 15:12:46 4737089 replicas (4728845 blocks, 230577300632292 bytes) overreplicated (have>want>0)
2016/06/19 15:12:46 25722 replicas (8648 blocks, 2071796729 bytes) unreferenced (have>want=0, new)
2016/06/19 15:12:46 1170608 replicas (692946 blocks, 600101925167 bytes) garbage (have>want=0, old)
2016/06/19 15:12:46 ===
2016/06/19 15:12:46 10601328 replicas (5299094 blocks, 531960816569846 bytes) total commitment (excluding unreferenced)
2016/06/19 15:12:46 16533504 replicas (6000682 blocks, 763059498065306 bytes) total usage
2016/06/19 15:12:46 ===
2016/06/19 15:12:46 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1548, Trashes:23248}
2016/06/19 15:12:46 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1563, Trashes:23118}
2016/06/19 15:12:46 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:36345, Trashes:85366}
2016/06/19 15:12:46 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:14318, Trashes:20328}
2016/06/19 15:12:46 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:2940, Trashes:23171}
2016/06/19 15:12:46 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:4553, Trashes:23289}
2016/06/19 15:12:46 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1712, Trashes:484655}
2016/06/19 15:12:46 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1641, Trashes:23102}
2016/06/19 15:12:46 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1476, Trashes:23302}
2016/06/19 15:12:46 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1522, Trashes:22855}
2016/06/19 15:12:46 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1511, Trashes:23367}
2016/06/19 15:12:46 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1495, Trashes:23349}
2016/06/19 15:12:46 ===
2016/06/19 15:12:46 Replication level distribution (counting N replicas on a single server as N):
2016/06/19 15:12:46  0:       6 #######
2016/06/19 15:12:46  1:  390568 ##################################################
2016/06/19 15:12:46  2:  697270 ####################################################
2016/06/19 15:12:46  3: 4905403 ###########################################################
2016/06/19 15:12:46  4:    6282 ##################################
2016/06/19 15:12:46  5:     952 ##########################
2016/06/19 15:12:46  6:      30 #############
2016/06/19 15:12:46  7:       1 ##
2016/06/19 15:12:46  8:       0
2016/06/19 15:12:46  9:       0
2016/06/19 15:12:46 10:       0
2016/06/19 15:12:46 11:       0
2016/06/19 15:12:46 12:     176 ####################
2016/06/19 15:12:46 ===
2016/06/19 15:12:47 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 15:12:47 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send pull list: took 18.294418ms
2016/06/19 15:12:47 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send pull list: took 19.28437ms
2016/06/19 15:12:47 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send pull list: took 20.657787ms
2016/06/19 15:12:47 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send pull list: took 20.805979ms
2016/06/19 15:12:47 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: took 25.161251ms
2016/06/19 15:12:47 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send pull list: took 25.9737ms
2016/06/19 15:12:47 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send pull list: took 27.55602ms
2016/06/19 15:12:47 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send pull list: Put http://humgen-02-03.internal.sanger.ac.uk:25107/pull: EOF
2016/06/19 15:12:47 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: took 28.358467ms
2016/06/19 15:12:47 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: Put http://humgen-03-03.internal.sanger.ac.uk:25107/pull: EOF
2016/06/19 15:12:47 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send pull list: took 34.643279ms
2016/06/19 15:12:47 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send pull list: took 49.676402ms
2016/06/19 15:12:47 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send pull list: took 150.482078ms
2016/06/19 15:12:47 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send pull list: took 359.143541ms
2016/06/19 15:12:47 Run: took 2h56m46.019896237s
2016/06/19 15:12:47 run failed: z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: Put http://humgen-03-03.internal.sanger.ac.uk:25107/pull: EOF
2016/06/19 15:12:47 timer went off
2016/06/19 15:12:47 starting next run
2016/06/19 15:12:47 Run: start
2016/06/19 15:12:47 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 15:12:47 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.192673ms
2016/06/19 15:12:47 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.181356ms
2016/06/19 15:12:47 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.195518ms
2016/06/19 15:12:47 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.390754ms
2016/06/19 15:12:47 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.345774ms
2016/06/19 15:12:47 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.398906ms
2016/06/19 15:12:47 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.431792ms
2016/06/19 15:12:47 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.546558ms
2016/06/19 15:12:47 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.516724ms
2016/06/19 15:12:47 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.555333ms
2016/06/19 15:12:47 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.729169ms
2016/06/19 15:12:47 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: took 2.074129ms
2016/06/19 15:12:47 GetCurrentState: start
2016/06/19 15:12:48 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:48 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 15:12:51 collections: 0/3273174
2016/06/19 15:13:02 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): add 1334745 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): add 1312482 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): add 1337686 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): add 1338270 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): add 1337822 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): add 1338775 replicas to map
2016/06/19 15:13:02 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): add 1339176 replicas to map
2016/06/19 15:13:03 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): add 1335581 replicas to map
2016/06/19 15:13:03 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): add 1373440 replicas to map
2016/06/19 15:13:03 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:03 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): add 1346446 replicas to map
2016/06/19 15:13:03 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): add 1340508 replicas to map
2016/06/19 15:13:04 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:05 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:08 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:09 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:10 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:10 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:11 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:12 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:13 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:13:13 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:14:07 collections: 25000/3273174
2016/06/19 15:14:18 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): add 1798573 replicas to map
2016/06/19 15:14:20 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 15:15:33 collections: 50000/3273174
2016/06/19 15:16:58 collections: 75000/3273174
2016/06/19 15:18:25 collections: 100000/3273174
2016/06/19 15:19:45 collections: 125000/3273174
2016/06/19 15:21:11 collections: 150000/3273174
2016/06/19 15:22:39 collections: 175000/3273174
2016/06/19 15:23:59 collections: 200000/3273174
2016/06/19 15:25:22 collections: 225000/3273174
2016/06/19 15:26:48 collections: 250000/3273174
2016/06/19 15:28:07 collections: 275000/3273174
2016/06/19 15:29:27 collections: 300000/3273174
2016/06/19 15:30:52 collections: 324238/3273174
2016/06/19 15:32:16 collections: 343038/3273174
2016/06/19 15:33:22 collections: 368038/3273174
2016/06/19 15:34:43 collections: 393038/3273174
2016/06/19 15:36:06 collections: 418038/3273174
2016/06/19 15:37:33 collections: 443038/3273174
2016/06/19 15:38:52 collections: 468038/3273174
2016/06/19 15:40:15 collections: 493038/3273174
2016/06/19 15:41:39 collections: 518038/3273174
2016/06/19 15:43:02 collections: 543038/3273174
2016/06/19 15:44:27 collections: 568038/3273174
2016/06/19 15:45:47 collections: 593038/3273174
2016/06/19 15:47:13 collections: 618038/3273174
2016/06/19 15:48:32 collections: 643038/3273174
2016/06/19 15:49:57 collections: 668038/3273174
2016/06/19 15:51:22 collections: 693038/3273174
2016/06/19 15:52:44 collections: 718038/3273174
2016/06/19 15:54:03 collections: 743038/3273174
2016/06/19 15:55:21 collections: 768038/3273174
2016/06/19 15:56:44 collections: 793038/3273174
2016/06/19 15:58:12 collections: 818038/3273174
2016/06/19 15:59:33 collections: 843038/3273174
2016/06/19 16:00:55 collections: 868038/3273174
2016/06/19 16:02:15 collections: 893038/3273174
2016/06/19 16:03:33 collections: 918038/3273174
2016/06/19 16:05:00 collections: 943038/3273174
2016/06/19 16:06:20 collections: 968038/3273174
2016/06/19 16:07:43 collections: 993038/3273174
2016/06/19 16:09:08 collections: 1018038/3273174
2016/06/19 16:10:38 collections: 1043038/3273174
2016/06/19 16:11:56 collections: 1068038/3273174
2016/06/19 16:13:18 collections: 1093038/3273174
2016/06/19 16:14:41 collections: 1118038/3273174
2016/06/19 16:16:05 collections: 1143038/3273174
2016/06/19 16:17:25 collections: 1168038/3273174
2016/06/19 16:18:46 collections: 1193038/3273174
2016/06/19 16:20:05 collections: 1218038/3273174
2016/06/19 16:21:32 collections: 1243038/3273174
2016/06/19 16:22:55 collections: 1268038/3273174
2016/06/19 16:24:15 collections: 1293038/3273174
2016/06/19 16:25:42 collections: 1318038/3273174
2016/06/19 16:27:03 collections: 1343038/3273174
2016/06/19 16:28:23 collections: 1368038/3273174
2016/06/19 16:29:41 collections: 1393038/3273174
2016/06/19 16:31:00 collections: 1418038/3273174
2016/06/19 16:32:22 collections: 1443038/3273174
2016/06/19 16:33:40 collections: 1468038/3273174
2016/06/19 16:34:59 collections: 1493038/3273174
2016/06/19 16:36:24 collections: 1518038/3273174
2016/06/19 16:37:44 collections: 1543038/3273174
2016/06/19 16:39:05 collections: 1568038/3273174
2016/06/19 16:40:24 collections: 1593038/3273174
2016/06/19 16:41:48 collections: 1618038/3273174
2016/06/19 16:43:08 collections: 1643038/3273174
2016/06/19 16:44:33 collections: 1668038/3273174
2016/06/19 16:45:52 collections: 1693038/3273174
2016/06/19 16:47:15 collections: 1718038/3273174
2016/06/19 16:48:35 collections: 1743038/3273174
2016/06/19 16:49:58 collections: 1768038/3273174
2016/06/19 16:51:20 collections: 1793038/3273174
2016/06/19 16:52:41 collections: 1818038/3273174
2016/06/19 16:54:02 collections: 1843038/3273174
2016/06/19 16:55:22 collections: 1868038/3273174
2016/06/19 16:56:45 collections: 1893038/3273174
2016/06/19 16:58:02 collections: 1918038/3273174
2016/06/19 16:59:23 collections: 1943038/3273174
2016/06/19 17:00:43 collections: 1968038/3273174
2016/06/19 17:02:08 collections: 1993038/3273174
2016/06/19 17:03:26 collections: 2018038/3273174
2016/06/19 17:04:40 collections: 2043038/3273174
2016/06/19 17:06:01 collections: 2068038/3273174
2016/06/19 17:07:20 collections: 2093038/3273174
2016/06/19 17:08:39 collections: 2118038/3273174
2016/06/19 17:09:56 collections: 2143038/3273174
2016/06/19 17:11:22 collections: 2168038/3273174
2016/06/19 17:12:41 collections: 2193038/3273174
2016/06/19 17:13:58 collections: 2218038/3273174
2016/06/19 17:15:21 collections: 2243038/3273174
2016/06/19 17:16:39 collections: 2268038/3273174
2016/06/19 17:17:58 collections: 2293038/3273174
2016/06/19 17:19:23 collections: 2318038/3273174
2016/06/19 17:20:39 collections: 2343038/3273174
2016/06/19 17:22:01 collections: 2368038/3273174
2016/06/19 17:23:22 collections: 2393038/3273174
2016/06/19 17:24:49 collections: 2418038/3273174
2016/06/19 17:26:13 collections: 2443038/3273174
2016/06/19 17:27:34 collections: 2468038/3273174
2016/06/19 17:28:54 collections: 2493038/3273174
2016/06/19 17:30:09 collections: 2518038/3273174
2016/06/19 17:31:27 collections: 2543038/3273174
2016/06/19 17:32:45 collections: 2568038/3273174
2016/06/19 17:34:02 collections: 2593038/3273174
2016/06/19 17:35:24 collections: 2618038/3273174
2016/06/19 17:36:47 collections: 2643038/3273174
2016/06/19 17:38:07 collections: 2668038/3273174
2016/06/19 17:39:23 collections: 2693038/3273174
2016/06/19 17:40:39 collections: 2718038/3273174
2016/06/19 17:41:58 collections: 2743038/3273174
2016/06/19 17:43:16 collections: 2768038/3273174
2016/06/19 17:44:35 collections: 2793038/3273174
2016/06/19 17:45:57 collections: 2818038/3273174
2016/06/19 17:47:18 collections: 2843038/3273174
2016/06/19 17:48:37 collections: 2868038/3273174
2016/06/19 17:49:56 collections: 2893038/3273174
2016/06/19 17:51:17 collections: 2918038/3273174
2016/06/19 17:52:38 collections: 2943038/3273174
2016/06/19 17:54:01 collections: 2968038/3273174
2016/06/19 17:55:21 collections: 2993038/3273174
2016/06/19 17:56:39 collections: 3018038/3273174
2016/06/19 17:57:59 collections: 3043038/3273174
2016/06/19 17:59:18 collections: 3068038/3273174
2016/06/19 18:00:35 collections: 3093038/3273174
2016/06/19 18:01:49 collections: 3118038/3273174
2016/06/19 18:03:04 collections: 3143038/3273174
2016/06/19 18:04:24 collections: 3168038/3273174
2016/06/19 18:05:43 collections: 3193038/3273174
2016/06/19 18:07:05 collections: 3218038/3273174
2016/06/19 18:08:22 collections: 3243038/3273174
2016/06/19 18:09:38 collections: 3268038/3273174
2016/06/19 18:09:52 collections: 3273174/3273174
2016/06/19 18:09:52 collections: 3273174/3273174
2016/06/19 18:11:15 GetCurrentState: took 2h58m27.595042209s
2016/06/19 18:11:15 ComputeChangeSets: start
2016/06/19 18:12:34 ComputeChangeSets: took 1m19.342601722s
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 12 replicas (6 blocks, 402731660 bytes) lost (0=have<want)
2016/06/19 18:12:38 1231 replicas (163 blocks, 80390127068 bytes) underreplicated (0<have<want)
2016/06/19 18:12:38 1141920 replicas (570080 blocks, 71769882700386 bytes) just right (have=want)
2016/06/19 18:12:38 4737089 replicas (4728845 blocks, 230577300632292 bytes) overreplicated (have>want>0)
2016/06/19 18:12:38 25565 replicas (8595 blocks, 2071752841 bytes) unreferenced (have>want=0, new)
2016/06/19 18:12:38 1170765 replicas (692999 blocks, 600101969055 bytes) garbage (have>want=0, old)
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 10601328 replicas (5299094 blocks, 531960816569846 bytes) total commitment (excluding unreferenced)
2016/06/19 18:12:38 16533504 replicas (6000682 blocks, 763059498065306 bytes) total usage
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:4553, Trashes:23291}
2016/06/19 18:12:38 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1712, Trashes:484663}
2016/06/19 18:12:38 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1641, Trashes:23110}
2016/06/19 18:12:38 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1476, Trashes:23309}
2016/06/19 18:12:38 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:36345, Trashes:85368}
2016/06/19 18:12:38 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:14318, Trashes:20329}
2016/06/19 18:12:38 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1522, Trashes:22858}
2016/06/19 18:12:38 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1511, Trashes:23370}
2016/06/19 18:12:38 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1495, Trashes:23353}
2016/06/19 18:12:38 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1548, Trashes:23254}
2016/06/19 18:12:38 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:1563, Trashes:23120}
2016/06/19 18:12:38 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): ChangeSet{Pulls:2940, Trashes:23178}
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 Replication level distribution (counting N replicas on a single server as N):
2016/06/19 18:12:38  0:       6 #######
2016/06/19 18:12:38  1:  390568 ##################################################
2016/06/19 18:12:38  2:  697270 ####################################################
2016/06/19 18:12:38  3: 4905403 ###########################################################
2016/06/19 18:12:38  4:    6282 ##################################
2016/06/19 18:12:38  5:     952 ##########################
2016/06/19 18:12:38  6:      30 #############
2016/06/19 18:12:38  7:       1 ##
2016/06/19 18:12:38  8:       0
2016/06/19 18:12:38  9:       0
2016/06/19 18:12:38 10:       0
2016/06/19 18:12:38 11:       0
2016/06/19 18:12:38 12:     176 ####################
2016/06/19 18:12:38 ===
2016/06/19 18:12:38 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send pull list: start
2016/06/19 18:12:38 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send pull list: took 18.714115ms
2016/06/19 18:12:38 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send pull list: took 18.576846ms
2016/06/19 18:12:38 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send pull list: took 19.002479ms
2016/06/19 18:12:38 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send pull list: took 19.383481ms
2016/06/19 18:12:38 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send pull list: took 22.965699ms
2016/06/19 18:12:38 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: took 24.329261ms
2016/06/19 18:12:38 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: Put http://humgen-01-01.internal.sanger.ac.uk:25107/pull: EOF
2016/06/19 18:12:38 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send pull list: took 25.162295ms
2016/06/19 18:12:38 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send pull list: took 25.905864ms
2016/06/19 18:12:38 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send pull list: took 34.18427ms
2016/06/19 18:12:38 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send pull list: took 55.276046ms
2016/06/19 18:12:38 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send pull list: took 182.975728ms
2016/06/19 18:12:39 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send pull list: took 391.756587ms
2016/06/19 18:12:39 Run: took 2h59m51.275338725s
2016/06/19 18:12:39 run failed: z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send pull list: Put http://humgen-01-01.internal.sanger.ac.uk:25107/pull: EOF
2016/06/19 18:12:39 timer went off
2016/06/19 18:12:39 starting next run
2016/06/19 18:12:39 Run: start
2016/06/19 18:12:39 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: start
2016/06/19 18:12:39 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.211381ms
2016/06/19 18:12:39 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.302391ms
2016/06/19 18:12:39 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.344222ms
2016/06/19 18:12:39 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.338358ms
2016/06/19 18:12:39 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.511689ms
2016/06/19 18:12:39 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.519639ms
2016/06/19 18:12:39 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.563933ms
2016/06/19 18:12:39 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): send trash list: took 1.586532ms
2016/06/19 18:12:39 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.47872ms
2016/06/19 18:12:39 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.577564ms
2016/06/19 18:12:39 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): send trash list: took 1.577052ms
2016/06/19 18:12:39 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): send trash list: took 1.728638ms
2016/06/19 18:12:39 GetCurrentState: start
2016/06/19 18:12:39 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:39 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): retrieve index
2016/06/19 18:12:43 collections: 0/3273174
2016/06/19 18:12:53 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): add 1312482 replicas to map
2016/06/19 18:12:53 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): add 1334745 replicas to map
2016/06/19 18:12:53 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): add 1338775 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): add 1339176 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): add 1337822 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): add 1335581 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): add 1338270 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): add 1337686 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): add 1340508 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): add 1346446 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): add 1373440 replicas to map
2016/06/19 18:12:54 z8ta6-bi6l4-nynctbmdi8nj6v0 (humgen-01-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:12:56 z8ta6-bi6l4-3kqkr5lgow2uogm (humgen-03-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:12:56 z8ta6-bi6l4-sg7xxak114gh1j0 (humgen-02-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:12:58 z8ta6-bi6l4-a1pntf0wx8vfr5v (humgen-03-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:12:59 z8ta6-bi6l4-stmnte9yvd2gh6o (humgen-04-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:00 z8ta6-bi6l4-yxhkoekmnv5czf3 (humgen-02-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:00 z8ta6-bi6l4-w3rpndae62qwwre (humgen-02-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:01 z8ta6-bi6l4-4b0e02ad7mk84ye (humgen-01-01.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:02 z8ta6-bi6l4-lhps1yuzszk0315 (humgen-04-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:02 z8ta6-bi6l4-kijrzcy3zkflg3s (humgen-01-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:06 z8ta6-bi6l4-az89xled1ycwnpb (humgen-04-03.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:13:52 collections: 25000/3273174
2016/06/19 18:14:59 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): add 1798573 replicas to map
2016/06/19 18:15:00 z8ta6-bi6l4-ph34sug9wmnom07 (humgen-03-02.internal.sanger.ac.uk:25107, disk): done
2016/06/19 18:15:14 collections: 50000/3273174
2016/06/19 18:16:29 collections: 75000/3273174
...