Bug #7254
Updated by Brett Smith over 8 years ago
Doing a simple arv-put that specifies --replication different than the default will ignore the replication for Keep blocks. They'll still be replicated to Keep at the default level. Here's a demonstration on the PGP tutorial data. The debug output shows it writing each block to two Keep services (each of two ports on keep0):
<pre>
brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 var-GS000016015-ASM.tsv.bz2
0M / 216M 0.0% 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospx
uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u
'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'
service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.c
om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvado
sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-t
pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service
_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}}
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}]
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/']
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/204e43b8a1185621ca55a94839582e6f
2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/
64M / 216M 29.5% 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvadosapi.com:25108/']
2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03
2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/b9677abbac956bd3e86b1deb28dfac03
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/
128M / 216M 59.1% 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/']
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/fc15aff2a762b13f521baf042140acec
2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/
192M / 216M 88.6% 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/']
2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd
2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/323d2a3ce20370c4ca1d3462a344f8fd
2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/
2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/
216M / 216M 100.0%
Collection saved as 'Saved at 2015-09-10 13:49:04 UTC by brett@shell.4xphq.arvadosapi.com'
4xphq-4zz18-q0u4nb9z66joxs6
</pre>
Same thing happens with more-than-default replication (e.g., --replication=3). @replication_desired@ is set correctly on the output collection, but Keep blocks are always replicated twice.
It works if you add --no-resume:
<pre>
brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 --no-resume var-GS000016015-ASM.tsv.bz2
0M / 216M 0.0% 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', u'4xphq-bi6l4-ospx
uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': u
'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': u'
service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, u'keep0.4xphq.arvadosapi.c
om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvado
sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'4xphq-t
pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': u'service
_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}}
2015-09-10 13:52:49 arvados.keep[16939] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4x
phq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid':
u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk',
u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados#keepService', u'arvados#
keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'created_
at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv',
u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'
service_host': u'keep0.4xphq.arvadosapi.com'}]
2015-09-10 13:52:49 arvados.keep[16939] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 'http://keep0.4xphq.arvadosapi.com:25107/'
]
2015-09-10 13:52:49 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/
/keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:52:49 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f
2015-09-10 13:52:50 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http://
keep0.4xphq.arvadosapi.com:25107/
64M / 216M 29.5% 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvadosapi.com:25108/'] 'http://keep0.4xphq.arvad
osapi.com:25108/']
2015-09-10 13:52:51 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/
/keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:52:51 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03
2015-09-10 13:52:52 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http://
keep0.4xphq.arvadosapi.com:25107/
128M / 216M 59.1% 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 'http://keep0.4xphq.arva
dosapi.com:25107/']
2015-09-10 13:52:53 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/
/keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:52:53 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec
2015-09-10 13:52:54 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/
192M / 216M 88.6% 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/']
2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/
2015-09-10 13:52:55 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd
2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/
216M / 216M 100.0%
Collection saved as 'Saved at 2015-09-10 13:52:48 UTC by brett@shell.4xphq.arvadosapi.com'
4xphq-4zz18-lxei9i58nv4gvo7
</pre>