Project

General

Profile

Bug #7254

Updated by Brett Smith almost 7 years ago

Doing a simple arv-put that specifies --replication different than the default will ignore the replication for Keep blocks.    They'll still be replicated to Keep at the default level.    Here's a demonstration on the PGP tutorial data.    The debug output shows it writing each block to two Keep services (each of two ports on keep0): 

 <pre> 
 brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 var-GS000016015-ASM.tsv.bz2 
 0M / 216M 0.0% 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospx 
 uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u 
 'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u' 
 service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.c 
 om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvado 
 sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-t 
 pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service 
 _ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}} 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}] 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/204e43b8a1185621ca55a94839582e6f 
 2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 64M / 216M 29.5% 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvadosapi.com:25108/'] 
 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03 
 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/b9677abbac956bd3e86b1deb28dfac03 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 128M / 216M 59.1% 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/fc15aff2a762b13f521baf042140acec 
 2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 192M / 216M 88.6% 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 
 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd 
 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/323d2a3ce20370c4ca1d3462a344f8fd 
 2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/ 
 2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 
 216M / 216M 100.0% 
 Collection saved as 'Saved at 2015-09-10 13:49:04 UTC by brett@shell.4xphq.arvadosapi.com' 
 4xphq-4zz18-q0u4nb9z66joxs6 
 </pre> 

 Same thing happens with more-than-default replication (e.g., --replication=3).    @replication_desired@ is set correctly on the output collection, but Keep blocks are always replicated twice. 

 It works if you add --no-resume: 

 <pre> 
 brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 --no-resume var-GS000016015-ASM.tsv.bz2 
 0M / 216M 0.0% 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', u'4xphq-bi6l4-ospx 
 uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': u 
 'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': u' 
 service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, u'keep0.4xphq.arvadosapi.c 
 om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvado 
 sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'4xphq-t 
 pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': u'service 
 _ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}} 
 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4x 
 phq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': 
  u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', 
  u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados#keepService', u'arvados# 
 keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'created_ 
 at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv',  
 u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u' 
 service_host': u'keep0.4xphq.arvadosapi.com'}] 
 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 'http://keep0.4xphq.arvadosapi.com:25107/' 
 ] 
 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/ 
 /keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f 
 2015-09-10 13:52:50 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:// 
 keep0.4xphq.arvadosapi.com:25107/ 
 64M / 216M 29.5% 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvadosapi.com:25108/'] 'http://keep0.4xphq.arvad 
 osapi.com:25108/'] 
 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/ 
 /keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03 
 2015-09-10 13:52:52 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:// 
 keep0.4xphq.arvadosapi.com:25107/ 
 128M / 216M 59.1% 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 'http://keep0.4xphq.arva 
 dosapi.com:25107/'] 
 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ http:/ 
 /keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec 
 2015-09-10 13:52:54 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 
 192M / 216M 88.6% 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 
 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 
 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd 
 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 
 216M / 216M 100.0% 
  
 Collection saved as 'Saved at 2015-09-10 13:52:48 UTC by brett@shell.4xphq.arvadosapi.com' 
 4xphq-4zz18-lxei9i58nv4gvo7 
 </pre> 

Back