Actions
Bug #12475
closed
keepproxy panic: send on closed channel
Story points:
-
Description
Our keepproxy crashes when we try to run a heavy workload on it (consisting of 40 clients each of which does an arv-put followed by an arv-get to verify md5). It is currently running on an instance with 4 cores and 16GB of RAM, which is very low for this workload, but I don't think we should expect keepproxy to panic in this situation.
Here is the end of keepproxy output during one of these panics (it seems to do it within a few minutes of starting up this workload):
2017/10/22 19:20:03 172.17.125.24:37929 GET /a84821554454fe8f7596caaf9a61cccd+67108864+Aed63b730e725c629b40e07e553eaf37933256698@59ff62cf 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/a84821554454fe8f7596caaf9a61cccd+67108864+Aed63b730e725c629b40e07e553eaf37933256698@59ff62cf <nil>
2017/10/22 19:20:03 172.17.125.24:39898 GET /94b41e42d11131cb972797d1e6fff08d+67108864+A6949359fcffe7369baffbbf6c756785b1ebee296@59ff6388 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/94b41e42d11131cb972797d1e6fff08d+67108864+A6949359fcffe7369baffbbf6c756785b1ebee296@59ff6388 <nil>
2017/10/22 19:20:04 172.17.125.20:58495 GET /bcdf290b01a8cb009863b7a7f7c5aa1a+67108864+A69da494c07a84ae0a7aa5b22313fb7755878311a@59ff63ab 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/bcdf290b01a8cb009863b7a7f7c5aa1a+67108864+A69da494c07a84ae0a7aa5b22313fb7755878311a@59ff63ab <nil>
2017/10/22 19:20:04 172.17.125.24:38643 GET /46367ef0ae85032ddddf2b09bb1c9279+67108864+A2009776510367dd137fce6a786b656a2c977da39@59ff6314 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/46367ef0ae85032ddddf2b09bb1c9279+67108864+A2009776510367dd137fce6a786b656a2c977da39@59ff6314 <nil>
2017/10/22 19:20:04 172.17.125.20:57082 PUT /b6bb371abd73e72268e143a3be0bc685 200 67108864 2 3 b6bb371abd73e72268e143a3be0bc685+67108864+Aaa372c94f2b2ad30d068737188ed7e63297ad7d8@59ff6464 <nil>
2017/10/22 19:20:04 172.17.125.17:43708 PUT /1fd14f1cf5f52b710c81053ea2570a2f 200 67108864 2 3 1fd14f1cf5f52b710c81053ea2570a2f+67108864+A0fb506e498ee15e1b2c06666e01f8aff28211a50@59ff6464 <nil>
2017/10/22 19:20:04 172.17.125.23:23723 GET /ade35c0b85f67985a43ce8058b3ffe4d+67108864+Adf1930a6d6e4bf5e971ec9a0f893393780eedd22@59ff630a 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/ade35c0b85f67985a43ce8058b3ffe4d+67108864+Adf1930a6d6e4bf5e971ec9a0f893393780eedd22@59ff630a <nil>
2017/10/22 19:20:04 172.17.125.24:38216 PUT /5069a264a8766477da5ac90970b3254a 200 67108864 2 3 5069a264a8766477da5ac90970b3254a+67108864+A33c0a8e2dcad8580035ffa2ad71614db01aee537@59ff6464 <nil>
2017/10/22 19:20:04 172.17.125.17:43067 PUT /7f25b9337609917bd6c3af526fd9331b 200 67108864 2 3 7f25b9337609917bd6c3af526fd9331b+67108864+A51c536ee6ae0821af5156933ea6f4cf5a0ac3f34@59ff6464 <nil>
2017/10/22 19:20:05 172.17.125.24:41786 PUT /34ef3fccf17eef9b8c86246711ac00b6 200 67108864 2 3 34ef3fccf17eef9b8c86246711ac00b6+67108864+A76d706a3c52fe376585cc38490520728e23a9346@59ff6465 <nil>
panic: send on closed channel
goroutine 26589 [running]:
git.curoverse.com/arvados.git/sdk/go/streamer.handleReadRequest(0x0, 0xe98, 0xc4d097e660, 0xc5a0b00000, 0x4000000, 0x4000000, 0x0, 0x0, 0x0)
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/transfer.go:170 +0x218
git.curoverse.com/arvados.git/sdk/go/streamer.(*AsyncStream).transfer(0xc500aa8840, 0x8a6300, 0xc4a086f6e0)
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/transfer.go:244 +0x51d
created by git.curoverse.com/arvados.git/sdk/go/streamer.AsyncStreamFromReader
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/streamer.go:74 +0x1d0
Updated by Joshua Randall over 6 years ago
I've now turned on debug output from keepproxy and have found what I think is triggering the panic.
2017/10/22 20:39:13 172.17.125.17:33697 PUT /c443d77d21ac01d1db6a49c5b5564ef9 200 67108864 2 3 c443d77d21ac01d1db6a49c5b5564ef9+67108864+A35aa77ab599af1d660b61eb6db2796a43ef8aab6@59ff76f1 <nil>
2017/10/22 20:39:13 DEBUG: [3aecfd39] Begin upload 6048b6ecbfb782adbdd73b45dad1ea94 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:13 DEBUG: [3aecfd39] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:13 172.17.125.20:45773 GET /64027d993c50e14bfa14582d50b6ca4a+67108864+A6c73a19af5a16905da1385e0af51f66d67426f3e@59ff764b 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/64027d993c50e14bfa14582d50b6ca4a+67108864+A6c73a19af5a16905da1385e0af51f66d67426f3e@59ff764b <nil>
2017/10/22 20:39:13 172.17.125.23:12702 GET /abceca0347087e512c3fee752fb99b83+67108864+A8807bacade63b3bdfdd2f41ff693b33b36be41b5@59ff7666 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/abceca0347087e512c3fee752fb99b83+67108864+A8807bacade63b3bdfdd2f41ff693b33b36be41b5@59ff7666 <nil>
2017/10/22 20:39:13 DEBUG: [596a35f5] Begin upload 1382fb94280e24d491ab0570846cdec2 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:13 DEBUG: [596a35f5] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:13 DEBUG: [31f19953] Begin upload ff0c76b19b771312512f056b47a3dabe to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:13 DEBUG: [31f19953] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:14 DEBUG: [1e01ead6] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/d1b344f70399f7487de7e644131f24e5 success
2017/10/22 20:39:14 172.17.125.24:29272 PUT /d1b344f70399f7487de7e644131f24e5 200 67108864 2 3 d1b344f70399f7487de7e644131f24e5+67108864+A2b3ed154eaac27debd452a13fe238a2d5e564b4a@59ff76f1 <nil>
2017/10/22 20:39:14 DEBUG: [722869e7] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/5a8b3fe75ed73db23e4904837fa50c34 success
2017/10/22 20:39:14 172.17.125.22:55825 PUT /5a8b3fe75ed73db23e4904837fa50c34 200 67108864 2 3 5a8b3fe75ed73db23e4904837fa50c34+67108864+A66985fdc6adfefadc3f1f6982a0d3d5ec401ad57@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [787f7535] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/018ae2f5e70e810947c56355fe4d5ea9 success
2017/10/22 20:39:14 172.17.125.17:34308 PUT /018ae2f5e70e810947c56355fe4d5ea9 200 67108864 2 3 018ae2f5e70e810947c56355fe4d5ea9+67108864+Ae9c967f508a73a48f546bfd1e27b76a27c323356@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [2f260b4d] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/44e9cdb47dddaea203565c266680876b success
2017/10/22 20:39:14 172.17.125.24:27782 PUT /44e9cdb47dddaea203565c266680876b 200 67108864 2 3 44e9cdb47dddaea203565c266680876b+67108864+Ae5a2fd55effc7faa83feff06ec6e730045fbe1fd@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [00c517e2] Begin upload 7d59a417af3fc8954c61c5d8c0c2d835 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:14 DEBUG: [00c517e2] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:14 DEBUG: [1cc118c5] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/8fb53f488c78773f9e36bb8e17f6de74 success
2017/10/22 20:39:14 172.17.125.24:24392 PUT /8fb53f488c78773f9e36bb8e17f6de74 200 67108864 2 3 8fb53f488c78773f9e36bb8e17f6de74+67108864+A4ea7536da08c5d5fb43549c5006fa1d4d1e65b80@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [3dc47a92] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/4b96d0a6b71d138aaa1e37a10faa3e6c success
2017/10/22 20:39:14 172.17.125.24:26276 PUT /4b96d0a6b71d138aaa1e37a10faa3e6c 200 67108864 2 3 4b96d0a6b71d138aaa1e37a10faa3e6c+67108864+A23463bcdc26367929b6c55f93b99d4f63c76ffb1@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [6d563ddd] Begin upload 4cec86373c63e3b7a1c07bf7a6475581 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:14 DEBUG: [6d563ddd] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:14 DEBUG: [5afc10dc] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/b2c65727d691eca2f6000d5e0d4eec61 success
2017/10/22 20:39:14 172.17.125.19:63593 PUT /b2c65727d691eca2f6000d5e0d4eec61 200 67108864 2 3 b2c65727d691eca2f6000d5e0d4eec61+67108864+Adaad8d9da1d124202dc163faac56199a41c7b41a@59ff76f2 <nil>
2017/10/22 20:39:14 DEBUG: [6a0dcbda] Begin upload 29710486d4ddafb9581b169735387f84 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:14 DEBUG: [6a0dcbda] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:15 DEBUG: [70bd590f] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/28f38d511759a09443c7be70438336a6 success
2017/10/22 20:39:15 172.17.125.23:12359 PUT /28f38d511759a09443c7be70438336a6 200 67108864 2 3 28f38d511759a09443c7be70438336a6+67108864+Ae3e8388c485bfe8a87263ef296b31d3a0e00a4fa@59ff76f3 <nil>
2017/10/22 20:39:15 172.17.125.22:58676 GET /93f786cf46f2c85532aefcc76295cb3c+67108864+Af2065ccc9b6a446570edfe3fc29cf829ff5a6526@59ff75d3 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/93f786cf46f2c85532aefcc76295cb3c+67108864+Af2065ccc9b6a446570edfe3fc29cf829ff5a6526@59ff75d3 <nil>
2017/10/22 20:39:15 DEBUG: [0309fe74] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/59d79982c7aff685b2a17d52a0d984b3 success
2017/10/22 20:39:15 172.17.125.20:45324 PUT /59d79982c7aff685b2a17d52a0d984b3 200 67108864 2 3 59d79982c7aff685b2a17d52a0d984b3+67108864+Afd6160be3d8fbf1faf9235d9a52c66c2d1014f9a@59ff76f3 <nil>
2017/10/22 20:39:15 DEBUG: [372f9462] Begin upload a2183c5c8e62e92ebb72a2af93237fb3 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:15 DEBUG: [372f9462] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:15 DEBUG: [47c5f68f] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/d25a84d6b2bfcd1ed5a06bba42de7e3d success
2017/10/22 20:39:15 172.17.125.17:34195 PUT /d25a84d6b2bfcd1ed5a06bba42de7e3d 200 67108864 2 3 d25a84d6b2bfcd1ed5a06bba42de7e3d+67108864+Afe54508df7a5901b59523cdc1b952499b40b05d7@59ff76f3 <nil>
2017/10/22 20:39:15 172.17.125.22:61658 GET /de8fd0e0c5df1ceec279573c1d4c0196+67108864+Ab6996172a419e44f97b433166f783ba5f96bfc92@59ff76f1 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/de8fd0e0c5df1ceec279573c1d4c0196+67108864+Ab6996172a419e44f97b433166f783ba5f96bfc92@59ff76f1 <nil>
2017/10/22 20:39:15 DEBUG: [4c688f2b] Begin upload 6726b71be0987314605286a07d07e14e to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:15 DEBUG: [4c688f2b] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:15 DEBUG: [35e9d112] Begin upload f110b9cf040df51f96d46ecbbbea7e17 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:15 DEBUG: [35e9d112] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:15 DEBUG: [04a30500] Begin upload 86643dc55fef61f90d4488312adf6c86 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:15 DEBUG: [04a30500] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:15 DEBUG: [6b9a8b13] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/a01a07707851b9795a16300c02392635 success
2017/10/22 20:39:15 172.17.125.20:46595 PUT /a01a07707851b9795a16300c02392635 200 67108864 2 3 a01a07707851b9795a16300c02392635+67108864+A9c6ff515474fc28dd735ecfc8619b5f18c05c9e7@59ff76f3 <nil>
2017/10/22 20:39:15 DEBUG: [3ae09adb] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/a326a09b022616b133427ed85453f9c4 success
2017/10/22 20:39:15 172.17.125.21:56269 PUT /a326a09b022616b133427ed85453f9c4 200 67108864 2 3 a326a09b022616b133427ed85453f9c4+67108864+Adcfc2b22d1caae064d3d63ca7959f73bb16441f5@59ff76f3 <nil>
2017/10/22 20:39:15 172.17.125.17:33253 GET /6348808dbcc7ff03fcfa2f527400524d+67108864+Ae6f57a017e6d32bab22345a50cf63d3d6c92992e@59ff7635 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/6348808dbcc7ff03fcfa2f527400524d+67108864+Ae6f57a017e6d32bab22345a50cf63d3d6c92992e@59ff7635 <nil>
2017/10/22 20:39:16 172.17.125.20:45773 GET /9eb4d48eed556149881f21b538bbaac3+67108864+A576a21d0d00c1c48a0e7cda02414932dac651000@59ff764b 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/9eb4d48eed556149881f21b538bbaac3+67108864+A576a21d0d00c1c48a0e7cda02414932dac651000@59ff764b <nil>
2017/10/22 20:39:16 172.17.125.17:35246 GET /184fb1eb9d3d21f6862ae2f27251c6ff+67108864+A05632b386a9b7687a8d57d8df798fccf209c8b12@59ff76e7 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/184fb1eb9d3d21f6862ae2f27251c6ff+67108864+A05632b386a9b7687a8d57d8df798fccf209c8b12@59ff76e7 <nil>
2017/10/22 20:39:16 DEBUG: [6f139653] Begin upload dca8b7a4fbcf22f7566f8b6b0c1aa9da to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [6f139653] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 DEBUG: [7553ccaa] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/94a83b2cd51284441b41262b045eee2c success
2017/10/22 20:39:16 172.17.125.19:63360 PUT /94a83b2cd51284441b41262b045eee2c 200 67108864 2 3 94a83b2cd51284441b41262b045eee2c+67108864+A4cf9a57e3471a1eb51e5f361ae99e5361e0bb1c7@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [6fbfda28] Begin upload 9312a2fb29e9853714cbb93e0b4808b0 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [6fbfda28] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 DEBUG: [313ef7ea] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/bed93b8d6fbf920932d5d8900f9133cc success
2017/10/22 20:39:16 172.17.125.17:33695 PUT /bed93b8d6fbf920932d5d8900f9133cc 200 67108864 2 3 bed93b8d6fbf920932d5d8900f9133cc+67108864+A623249eae99363afbf4c7946e304f8aa29fd45fe@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [25fc6b77] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/8e7b79f239438c50b8fa93e0994564b9 success
2017/10/22 20:39:16 172.17.125.20:45552 PUT /8e7b79f239438c50b8fa93e0994564b9 200 67108864 2 3 8e7b79f239438c50b8fa93e0994564b9+67108864+A270970a966f0bd0ad18bde78d6e6cee96d295fbf@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [79f862c7] Begin upload 2b0c5901ca68c80014a1f639a3964491 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [79f862c7] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 172.17.125.23:12702 GET /39708b63c23ae859df75983bcf5c7ecc+67108864+Acff0464a07b723e79263ddfcdfbc05868a65c2a8@59ff7666 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/39708b63c23ae859df75983bcf5c7ecc+67108864+Acff0464a07b723e79263ddfcdfbc05868a65c2a8@59ff7666 <nil>
2017/10/22 20:39:16 DEBUG: [3625d700] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/0b959c8f1cd05b70c6a5e8c80af8ae5f success
2017/10/22 20:39:16 172.17.125.19:62432 PUT /0b959c8f1cd05b70c6a5e8c80af8ae5f 200 67108864 2 3 0b959c8f1cd05b70c6a5e8c80af8ae5f+67108864+Ac308caf037f9fd2808c6faff9b876a53676a4ef6@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [39a4a9b9] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/30887aeb9c853d53c9c4ccdf496e517f success
2017/10/22 20:39:16 172.17.125.22:59782 PUT /30887aeb9c853d53c9c4ccdf496e517f 200 67108864 2 3 30887aeb9c853d53c9c4ccdf496e517f+67108864+Aa7838b06296f367bfd2421cbf3cce4b154f5d685@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [5b0e5f20] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/687da9b16673205e84477a2a2e1bd242 success
2017/10/22 20:39:16 172.17.125.17:33230 PUT /687da9b16673205e84477a2a2e1bd242 200 67108864 2 3 687da9b16673205e84477a2a2e1bd242+67108864+A5265538be379e4d285158a761e75268cfae938d5@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [34d4d523] Begin upload 188346e76485d249bacfe1c39002ab51 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [34d4d523] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 DEBUG: [680ecda2] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/2b443bd7e27a11d6f6d76f3cb516b112 success
2017/10/22 20:39:16 172.17.125.17:33004 PUT /2b443bd7e27a11d6f6d76f3cb516b112 200 67108864 2 3 2b443bd7e27a11d6f6d76f3cb516b112+67108864+A4a6e9a3190a3fedadbac9b321e0f9551b7e3823e@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [2dc49cbf] Begin upload c8e3764c006facc7813381d2d63224ab to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [2dc49cbf] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 DEBUG: [110111b4] Begin upload 5c1889a86321cb57d91fab323a3db49d to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:16 DEBUG: [110111b4] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:16 DEBUG: [7499b682] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/3a59de1cf51453edea776373048f1a86 success
2017/10/22 20:39:16 172.17.125.24:28252 PUT /3a59de1cf51453edea776373048f1a86 200 67108864 2 3 3a59de1cf51453edea776373048f1a86+67108864+A9cfd2035673d316482079bb76ce05b7e219ccbd8@59ff76f4 <nil>
2017/10/22 20:39:16 DEBUG: [563d4b3d] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/028bcd9eacf12c0dbd2379af435ff445 success
2017/10/22 20:39:16 172.17.125.24:27673 PUT /028bcd9eacf12c0dbd2379af435ff445 200 67108864 2 3 028bcd9eacf12c0dbd2379af435ff445+67108864+Aa5c1884b2c4ef8f30c49fa376813979c45c6013f@59ff76f4 <nil>
2017/10/22 20:39:17 DEBUG: [386d1206] Begin upload 182f1a026db5e32657bdc3314de23dac to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [386d1206] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [0a962a34] Begin upload 164e2e1c019934d8a62d930d464786b6 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [0a962a34] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [3aecfd39] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/6048b6ecbfb782adbdd73b45dad1ea94 success
2017/10/22 20:39:17 172.17.125.19:62386 PUT /6048b6ecbfb782adbdd73b45dad1ea94 200 67108864 2 3 6048b6ecbfb782adbdd73b45dad1ea94+67108864+A640c9c333a0247a32001f2bb8ce0dff2a4eb87b5@59ff76f5 <nil>
2017/10/22 20:39:17 DEBUG: [36323a26] Begin upload de96e79af7e438b76c821b5990ece34b to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [36323a26] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [1bb0203e] Begin upload 848283177e9f1b85eb206e35ba74c7eb to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [1bb0203e] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [5a72046b] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/f522f74884f09e46cf50a4eed6c06af4 success
2017/10/22 20:39:17 172.17.125.17:32424 PUT /f522f74884f09e46cf50a4eed6c06af4 200 67108864 2 3 f522f74884f09e46cf50a4eed6c06af4+67108864+A60778f738dbb246a71fb640cd92c419acbe674e7@59ff76f5 <nil>
2017/10/22 20:39:17 DEBUG: [384c00cf] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/35b81a92d256b2f45c9954c60aaa3050 success
2017/10/22 20:39:17 172.17.125.24:26735 PUT /35b81a92d256b2f45c9954c60aaa3050 200 67108864 2 3 35b81a92d256b2f45c9954c60aaa3050+67108864+A7059142ff75e068369d11ef946193c6b2288c7b3@59ff76f5 <nil>
2017/10/22 20:39:17 DEBUG: [30564887] Begin upload 6f9b637e5084edb3181caf859d2de7fb to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [30564887] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 172.17.125.22:58676 GET /6f2828e90ba26f937b9d501422304eaa+67108864+Ae946c84c3ded60468ccfd9c68bfd9c9ccd95a9a6@59ff75d3 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/6f2828e90ba26f937b9d501422304eaa+67108864+Ae946c84c3ded60468ccfd9c68bfd9c9ccd95a9a6@59ff75d3 <nil>
2017/10/22 20:39:17 DEBUG: [4a20b813] Begin upload 5771ff83b0c5034c9ba77196be6bb3dd to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [4a20b813] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [2f1c4fbd] Begin upload 4d2df4ace9cc48b879736cfb290fc331 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [2f1c4fbd] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 DEBUG: [769d9b96] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/0e055506574353bd2bfb89b2a611aefe success
2017/10/22 20:39:17 172.17.125.17:34480 PUT /0e055506574353bd2bfb89b2a611aefe 200 67108864 2 3 0e055506574353bd2bfb89b2a611aefe+67108864+Ab36d34847054fa06aad2bee98c3f632ee2517f99@59ff76f5 <nil>
2017/10/22 20:39:17 DEBUG: [660dcaba] Begin upload c910ad03375f23f935065b34cf611adf to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:17 DEBUG: [660dcaba] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:17 172.17.125.22:61658 GET /18df61fc2df6d5a0d7b81792373a6bcd+67108864+Adb7dd2ce4b4f7c4addca3d9b70484b2e11e6ae9e@59ff76f1 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/18df61fc2df6d5a0d7b81792373a6bcd+67108864+Adb7dd2ce4b4f7c4addca3d9b70484b2e11e6ae9e@59ff76f1 <nil>
2017/10/22 20:39:18 DEBUG: [05658cea] Begin upload ae1402739ed4ce1fcb326a6bbb867498 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:18 DEBUG: [05658cea] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:18 DEBUG: [56b4ae10] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/e69c26d400d17683bbca0f93ca182977 success
2017/10/22 20:39:18 172.17.125.19:61344 PUT /e69c26d400d17683bbca0f93ca182977 200 67108864 2 3 e69c26d400d17683bbca0f93ca182977+67108864+A573101824e8ef3064f5be7be387b5b899baf8e2a@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [5338961a] Begin upload d43def6df99fd08ae5b6d0a2628f1956 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:18 DEBUG: [5338961a] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:18 DEBUG: [0e58daa3] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/c6782a42c8f93aac0fea828c1f0fd896 success
2017/10/22 20:39:18 172.17.125.20:45396 PUT /c6782a42c8f93aac0fea828c1f0fd896 200 67108864 2 3 c6782a42c8f93aac0fea828c1f0fd896+67108864+A9e0509df216b6c509a5aa9f833b52d7624dc7a7c@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [3f1a74d7] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/5a5bc6fe10ea6caabb69601b562d30e6 success
2017/10/22 20:39:18 172.17.125.21:56497 PUT /5a5bc6fe10ea6caabb69601b562d30e6 200 67108864 2 3 5a5bc6fe10ea6caabb69601b562d30e6+67108864+Ae5d73aeda6189d207f08459de5085d071cf6ecff@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [1acef10e] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/f4981001b8be86bcaf7b1655bb38eda5 success
2017/10/22 20:39:18 172.17.125.21:56175 PUT /f4981001b8be86bcaf7b1655bb38eda5 200 67108864 2 3 f4981001b8be86bcaf7b1655bb38eda5+67108864+A2f63f6b11f8f8a0a81d54a7e311c33bdcc68e27b@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [4ecec703] Begin upload 6b59072f5a387a420edc95faf3b9672e to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:18 DEBUG: [4ecec703] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:18 DEBUG: [6032cbf1] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/5cb3a1d5379729366aee3b8ee08401e9 success
2017/10/22 20:39:18 172.17.125.21:55727 PUT /5cb3a1d5379729366aee3b8ee08401e9 200 67108864 2 3 5cb3a1d5379729366aee3b8ee08401e9+67108864+A6d3166636359174492fe6c7ccb011b621be8f46b@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [00c517e2] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/7d59a417af3fc8954c61c5d8c0c2d835 success
2017/10/22 20:39:18 172.17.125.20:46696 PUT /7d59a417af3fc8954c61c5d8c0c2d835 200 67108864 2 3 7d59a417af3fc8954c61c5d8c0c2d835+67108864+A6551b075f0e0cf65ac2b381a0556a03daf05840a@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [306ccaac] Begin upload ed84ec5c27a25f497492f0675cd7023f to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:18 DEBUG: [306ccaac] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:18 172.17.125.20:45773 GET /42b6acbf0944c8671031f422a8e7f43c+67108864+A83883ffb2fd251b148ad407a8984ee254f22b54b@59ff764b 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/42b6acbf0944c8671031f422a8e7f43c+67108864+A83883ffb2fd251b148ad407a8984ee254f22b54b@59ff764b <nil>
2017/10/22 20:39:18 DEBUG: [7de195d2] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/b9ec8be2c49e81ecc00d7e0b35919f78 success
2017/10/22 20:39:18 172.17.125.20:46747 PUT /b9ec8be2c49e81ecc00d7e0b35919f78 200 67108864 2 3 b9ec8be2c49e81ecc00d7e0b35919f78+67108864+A13bc64989a7a919f499330cf44574db386499eaf@59ff76f6 <nil>
2017/10/22 20:39:18 172.17.125.23:12702 GET /8739c483dfec28a2696dfd4c014d1994+67108864+A145229899647f28c66474a244e07dd2055042d09@59ff7666 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/8739c483dfec28a2696dfd4c014d1994+67108864+A145229899647f28c66474a244e07dd2055042d09@59ff7666 <nil>
2017/10/22 20:39:18 DEBUG: [596a35f5] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/1382fb94280e24d491ab0570846cdec2 success
2017/10/22 20:39:18 172.17.125.17:33442 PUT /1382fb94280e24d491ab0570846cdec2 200 67108864 2 3 1382fb94280e24d491ab0570846cdec2+67108864+A9d151eecef3bb7f8708319e7cf5ff95a3ab44bfe@59ff76f6 <nil>
2017/10/22 20:39:18 DEBUG: [386d1206] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/182f1a026db5e32657bdc3314de23dac success
2017/10/22 20:39:18 172.17.125.19:62432 PUT /182f1a026db5e32657bdc3314de23dac 200 67108864 2 3 182f1a026db5e32657bdc3314de23dac+67108864+A67f6a65c16d59690fcdabfb36e07807fccc186b1@59ff76f6 <nil>
2017/10/22 20:39:19 DEBUG: [581f9a7d] Begin upload f048f1fab182ecee65d937b8fa374d7c to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [581f9a7d] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [79f862c7] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/2b0c5901ca68c80014a1f639a3964491 success
2017/10/22 20:39:19 172.17.125.19:63593 PUT /2b0c5901ca68c80014a1f639a3964491 200 67108864 2 3 2b0c5901ca68c80014a1f639a3964491+67108864+A9043d1173369264e3265ca4bbbd1ff35f7a8205d@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [37087479] Begin upload 51f23eabff83c4cf9fd2b617bf25322d to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [37087479] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [4c688f2b] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/6726b71be0987314605286a07d07e14e success
2017/10/22 20:39:19 172.17.125.23:12359 PUT /6726b71be0987314605286a07d07e14e 200 67108864 2 3 6726b71be0987314605286a07d07e14e+67108864+Abbccdc5d39739c924dc2d335e11ceb725c99d3c0@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [48e18c58] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/7dc6b62a1793a85b486c11f7e141b6b1 success
2017/10/22 20:39:19 172.17.125.20:47002 PUT /7dc6b62a1793a85b486c11f7e141b6b1 200 67108864 2 3 7dc6b62a1793a85b486c11f7e141b6b1+67108864+A3996dcdd9070f7a5919aa8b0433e62003cb28965@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [6e29a4fa] Begin upload db6a2f5afe68976160037e10173c43a1 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [6e29a4fa] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 172.17.125.17:33253 GET /0bfa3f03863a5ad45cf3d6847dfc521f+67108864+Af3e71341fe07329c9308807ee44f43daa45cbced@59ff7635 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/0bfa3f03863a5ad45cf3d6847dfc521f+67108864+Af3e71341fe07329c9308807ee44f43daa45cbced@59ff7635 <nil>
2017/10/22 20:39:19 DEBUG: [3f28ba25] Begin upload f857e77637f6cbbb66f107fce106014c to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [3f28ba25] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [2ec2524f] Begin upload f276914bb6bdf12874928361957e9840 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [2ec2524f] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [6a0dcbda] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/29710486d4ddafb9581b169735387f84 success
2017/10/22 20:39:19 172.17.125.24:27782 PUT /29710486d4ddafb9581b169735387f84 200 67108864 2 3 29710486d4ddafb9581b169735387f84+67108864+A0b6d05158a3c40173d6002666021bd8805ce64d1@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [36c451af] Begin upload 86098957907292145813c56f6a089b1c to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [36c451af] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [1956785b] Begin upload 1321be9afd333c182d5ac2aa5bdc03ed to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [1956785b] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [68bfe66a] Begin upload a943edfc74d1ab77f70ed18df7837136 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [68bfe66a] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 172.17.125.17:35246 GET /b784faf3d40cfba3cde8c3e65b450b0d+67108864+Ae96463ef4d3b1517e602b88c161240c10b10dbdf@59ff76e7 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/b784faf3d40cfba3cde8c3e65b450b0d+67108864+Ae96463ef4d3b1517e602b88c161240c10b10dbdf@59ff76e7 <nil>
2017/10/22 20:39:19 DEBUG: [4fa1f683] Begin upload c3f8d51541521454f11046c5739dec84 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [4fa1f683] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:19 DEBUG: [04a30500] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/86643dc55fef61f90d4488312adf6c86 success
2017/10/22 20:39:19 172.17.125.24:29272 PUT /86643dc55fef61f90d4488312adf6c86 200 67108864 2 3 86643dc55fef61f90d4488312adf6c86+67108864+Aba5f0da37000167868c101b210cbaee95769e9d9@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [35e9d112] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/f110b9cf040df51f96d46ecbbbea7e17 success
2017/10/22 20:39:19 172.17.125.24:24392 PUT /f110b9cf040df51f96d46ecbbbea7e17 200 67108864 2 3 f110b9cf040df51f96d46ecbbbea7e17+67108864+Aab46e0c9bd9e3becba9332824f6ee5e1dd7b9e9d@59ff76f7 <nil>
2017/10/22 20:39:19 DEBUG: [6bece2cc] Begin upload f88d02adc2e6981fc727435506b54093 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:19 DEBUG: [6bece2cc] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 172.17.125.22:58676 GET /08bc4586c9fe5b9dd8d163bdb6d82225+67108864+A554e5c80f9adb7fa897b3b06a58f0e905a239609@59ff75d3 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/08bc4586c9fe5b9dd8d163bdb6d82225+67108864+A554e5c80f9adb7fa897b3b06a58f0e905a239609@59ff75d3 <nil>
2017/10/22 20:39:20 DEBUG: [17a2a0f5] Begin upload 228b1cb34700f57d316b2ba88a067a2b to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [17a2a0f5] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 DEBUG: [372f9462] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/a2183c5c8e62e92ebb72a2af93237fb3 success
2017/10/22 20:39:20 172.17.125.17:34308 PUT /a2183c5c8e62e92ebb72a2af93237fb3 200 67108864 2 3 a2183c5c8e62e92ebb72a2af93237fb3+67108864+Af5fbb5f7db8f093d6c6031c73b2c195cb24bc2b2@59ff76f8 <nil>
2017/10/22 20:39:20 DEBUG: [00c1a6cf] Begin upload 0f5277163c5eaf736481f14c2de65cdb to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [00c1a6cf] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 DEBUG: [78667eac] Begin upload 3ddb10859d8dc3026003c02302cdaeed to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [78667eac] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 172.17.125.22:61658 GET /62640cfbfb5f691631cfbd8502b27e59+67108864+Aef1c0b64e7b678a7f314a4118a3f03424d818d13@59ff76f1 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/62640cfbfb5f691631cfbd8502b27e59+67108864+Aef1c0b64e7b678a7f314a4118a3f03424d818d13@59ff76f1 <nil>
2017/10/22 20:39:20 DEBUG: [5338961a] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/d43def6df99fd08ae5b6d0a2628f1956 success
2017/10/22 20:39:20 172.17.125.19:62386 PUT /d43def6df99fd08ae5b6d0a2628f1956 200 67108864 2 3 d43def6df99fd08ae5b6d0a2628f1956+67108864+Ac30a780e2e9b59e511eb38a125bbefd191c50922@59ff76f8 <nil>
2017/10/22 20:39:20 DEBUG: [34d4d523] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/188346e76485d249bacfe1c39002ab51 success
2017/10/22 20:39:20 172.17.125.20:45324 PUT /188346e76485d249bacfe1c39002ab51 200 67108864 2 3 188346e76485d249bacfe1c39002ab51+67108864+A5c8cabaa073e31017be7332923f28bcbe5a9f07e@59ff76f8 <nil>
2017/10/22 20:39:20 DEBUG: [693636b6] Begin upload bae7ca0651527d84d17c65b77751033a to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [693636b6] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 DEBUG: [040051c4] Begin upload a878dd89d76fd6cd0da20fcd8d9e0e6a to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [040051c4] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 172.17.125.23:12702 GET /fdaa448cd6e73e49ce697e0db9055302+67108864+Ac9d403df1305dbd9962991abd58b5f6c353a6f1c@59ff7666 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/fdaa448cd6e73e49ce697e0db9055302+67108864+Ac9d403df1305dbd9962991abd58b5f6c353a6f1c@59ff7666 <nil>
2017/10/22 20:39:20 DEBUG: [325ccfaf] Begin upload f589a0c1c9294d487123cb6c39827bd3 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:20 DEBUG: [325ccfaf] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:20 172.17.125.20:45773 GET /61d4781e886bc7a977c0f96b50dc40ae+67108864+A9b365f91e1f9ad086d74571c5f5bf16fcf75958e@59ff764b 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/61d4781e886bc7a977c0f96b50dc40ae+67108864+A9b365f91e1f9ad086d74571c5f5bf16fcf75958e@59ff764b <nil>
2017/10/22 20:39:20 DEBUG: [6f139653] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/dca8b7a4fbcf22f7566f8b6b0c1aa9da success
2017/10/22 20:39:20 172.17.125.17:34195 PUT /dca8b7a4fbcf22f7566f8b6b0c1aa9da 200 67108864 2 3 dca8b7a4fbcf22f7566f8b6b0c1aa9da+67108864+Ad60891eaee50e5d1f3b02a0503ccf6766e12afdc@59ff76f8 <nil>
2017/10/22 20:39:21 DEBUG: [6fbfda28] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/9312a2fb29e9853714cbb93e0b4808b0 success
2017/10/22 20:39:21 172.17.125.24:26276 PUT /9312a2fb29e9853714cbb93e0b4808b0 200 67108864 2 3 9312a2fb29e9853714cbb93e0b4808b0+67108864+A316c4b9f02b2c6c2b58508aca31cfb30105e6a42@59ff76f8 <nil>
2017/10/22 20:39:21 DEBUG: [6a0c20e6] Begin upload 3328043e0bf23ae6f653168dd8a6c63c to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:21 DEBUG: [6a0c20e6] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:21 DEBUG: [30564887] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/6f9b637e5084edb3181caf859d2de7fb success
2017/10/22 20:39:21 172.17.125.24:28252 PUT /6f9b637e5084edb3181caf859d2de7fb 200 67108864 2 3 6f9b637e5084edb3181caf859d2de7fb+67108864+A30eeb0c6892849b10a13cff1a38e2ac2785c8d6f@59ff76f9 <nil>
2017/10/22 20:39:21 DEBUG: [519269fa] Begin upload 5413d1598dec02a0b8413a24f1a923e1 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:21 DEBUG: [519269fa] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:21 DEBUG: [4df73f5b] Begin upload 2507cd58ed7550ec98a3f3d6ab3ef58f to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:21 DEBUG: [4df73f5b] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:21 DEBUG: [4a20b813] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/5771ff83b0c5034c9ba77196be6bb3dd success
2017/10/22 20:39:21 172.17.125.24:27673 PUT /5771ff83b0c5034c9ba77196be6bb3dd 200 67108864 2 3 5771ff83b0c5034c9ba77196be6bb3dd+67108864+Aa9aa4e34ff2f400917e218846816b92731da3eb5@59ff76f9 <nil>
2017/10/22 20:39:21 DEBUG: [2a66aa8b] Begin upload 95c8a06065d492de21f088ae87aadedc to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:21 DEBUG: [2a66aa8b] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:22 DEBUG: [2f1c4fbd] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/4d2df4ace9cc48b879736cfb290fc331 success
2017/10/22 20:39:22 172.17.125.19:63360 PUT /4d2df4ace9cc48b879736cfb290fc331 200 67108864 2 3 4d2df4ace9cc48b879736cfb290fc331+67108864+A05bd438057c17b009f5ea201b634ae42d0751136@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [2dd8bafe] Begin upload 363298367993b806fa779147d68ffc47 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:22 DEBUG: [2dd8bafe] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:22 DEBUG: [4ecec703] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/6b59072f5a387a420edc95faf3b9672e success
2017/10/22 20:39:22 172.17.125.17:34480 PUT /6b59072f5a387a420edc95faf3b9672e 200 67108864 2 3 6b59072f5a387a420edc95faf3b9672e+67108864+A1b18ea3325a8ca245957a37de10eb43c42a39bb7@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [6d563ddd] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/4cec86373c63e3b7a1c07bf7a6475581 success
2017/10/22 20:39:22 172.17.125.22:55825 PUT /4cec86373c63e3b7a1c07bf7a6475581 200 67108864 2 3 4cec86373c63e3b7a1c07bf7a6475581+67108864+A6f01801e20a57315633ef5ba4cee70eaaf1f13c1@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [0a093b1c] Begin upload a92db9f6d83ded839b1ec70dc10201fa to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:22 DEBUG: [0a093b1c] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:22 DEBUG: [306ccaac] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/ed84ec5c27a25f497492f0675cd7023f success
2017/10/22 20:39:22 172.17.125.24:26735 PUT /ed84ec5c27a25f497492f0675cd7023f 200 67108864 2 3 ed84ec5c27a25f497492f0675cd7023f+67108864+Af96d65b909f630270abf17d2debc229fa944ca55@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [6c16f1e3] Begin upload d0f95bec51dca44c81505734587b278b to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:22 DEBUG: [6c16f1e3] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:22 DEBUG: [0a962a34] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/164e2e1c019934d8a62d930d464786b6 success
2017/10/22 20:39:22 172.17.125.20:45552 PUT /164e2e1c019934d8a62d930d464786b6 200 67108864 2 3 164e2e1c019934d8a62d930d464786b6+67108864+A93f288f01b21050d98f897bfabfee283da6094c1@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [36c451af] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/86098957907292145813c56f6a089b1c success
2017/10/22 20:39:22 172.17.125.19:62432 PUT /86098957907292145813c56f6a089b1c 200 67108864 2 3 86098957907292145813c56f6a089b1c+67108864+Afa1701de73a2505fe8408938b4efb0040aaac8d1@59ff76fa <nil>
2017/10/22 20:39:22 DEBUG: [2fc2d620] Begin upload 0be9860e4a6ea28bf6228110a7715df2 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:22 DEBUG: [2fc2d620] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:22 172.17.125.22:61658 GET /d093e5f5d4b78b3bee49bc13a7107ee1+67108864+A1eaab7e6992424e7b410c0ac2f200cd7f021a366@59ff76f1 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/d093e5f5d4b78b3bee49bc13a7107ee1+67108864+A1eaab7e6992424e7b410c0ac2f200cd7f021a366@59ff76f1 <nil>
2017/10/22 20:39:22 172.17.125.17:33253 GET /9c222cfb2bdaa2048c56dd71a5a1e653+67108864+A065477259934271bdd4dff621c7f55d57fc83a21@59ff7635 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/9c222cfb2bdaa2048c56dd71a5a1e653+67108864+A065477259934271bdd4dff621c7f55d57fc83a21@59ff7635 <nil>
2017/10/22 20:39:22 DEBUG: [660dcaba] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/c910ad03375f23f935065b34cf611adf success
2017/10/22 20:39:22 172.17.125.17:33695 PUT /c910ad03375f23f935065b34cf611adf 200 67108864 2 3 c910ad03375f23f935065b34cf611adf+67108864+A3fb257df484cf037ad362170acc9d85796be9289@59ff76fa <nil>
2017/10/22 20:39:23 172.17.125.22:58676 GET /13774ba5099562059b4d7116c8d879b3+67108864+A5d6b199a7cb90e1401b9c4ae704cc33aedb3cafa@59ff75d3 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/13774ba5099562059b4d7116c8d879b3+67108864+A5d6b199a7cb90e1401b9c4ae704cc33aedb3cafa@59ff75d3 <nil>
2017/10/22 20:39:23 172.17.125.17:35246 GET /327f03247105c026e9baf7c371c2f3e6+67108864+Ae71eca58bdc3c21f3c47aa7afeb8ef896c0d2c07@59ff76e7 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/327f03247105c026e9baf7c371c2f3e6+67108864+Ae71eca58bdc3c21f3c47aa7afeb8ef896c0d2c07@59ff76e7 <nil>
2017/10/22 20:39:23 DEBUG: [502012f3] Begin upload cde60adfa1c82458d0e98eb33d11f301 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:23 DEBUG: [502012f3] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:23 DEBUG: [31d73e17] Begin upload 4d06bc93ec4ef0527d5261d9d1544dad to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:23 DEBUG: [31d73e17] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:23 DEBUG: [110111b4] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/5c1889a86321cb57d91fab323a3db49d success
2017/10/22 20:39:23 172.17.125.20:46595 PUT /5c1889a86321cb57d91fab323a3db49d 200 67108864 2 3 5c1889a86321cb57d91fab323a3db49d+67108864+A17833f37c48e3a0533f3a3cab2073a1e8c6c23c9@59ff76fb <nil>
2017/10/22 20:39:23 DEBUG: [1bb0203e] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/848283177e9f1b85eb206e35ba74c7eb success
2017/10/22 20:39:23 172.17.125.17:33230 PUT /848283177e9f1b85eb206e35ba74c7eb 200 67108864 2 3 848283177e9f1b85eb206e35ba74c7eb+67108864+Ab6d39773dda684cedef74a25ebfb0fa89c2606e8@59ff76fb <nil>
2017/10/22 20:39:23 DEBUG: [7174e154] Begin upload d699db1cdb4041b586d1ad0e6eb24c1a to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:23 DEBUG: [7174e154] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:23 DEBUG: [78667eac] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/3ddb10859d8dc3026003c02302cdaeed success
2017/10/22 20:39:23 172.17.125.23:12359 PUT /3ddb10859d8dc3026003c02302cdaeed 200 67108864 2 3 3ddb10859d8dc3026003c02302cdaeed+67108864+A02d275af7e5c45c7d7e0913bb7c465f46fc015b7@59ff76fb <nil>
2017/10/22 20:39:23 DEBUG: [23b2ca3f] Begin upload d17d48a7e447a25611156499953393d1 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:23 DEBUG: [23b2ca3f] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:23 172.17.125.23:12702 GET /e0fed9c95a6b3e770d07f09a8f051365+67108864+Ac104d019a1a0c67eae38b91fd14d0effcf3143d0@59ff7666 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/e0fed9c95a6b3e770d07f09a8f051365+67108864+Ac104d019a1a0c67eae38b91fd14d0effcf3143d0@59ff7666 <nil>
2017/10/22 20:39:23 DEBUG: [6e29a4fa] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/db6a2f5afe68976160037e10173c43a1 success
2017/10/22 20:39:23 172.17.125.19:61344 PUT /db6a2f5afe68976160037e10173c43a1 200 67108864 2 3 db6a2f5afe68976160037e10173c43a1+67108864+A4e5064c9d8c1ffc8b3653f56db545fc64a06400a@59ff76fb <nil>
2017/10/22 20:39:23 DEBUG: [68bfe66a] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/a943edfc74d1ab77f70ed18df7837136 success
2017/10/22 20:39:23 172.17.125.21:56497 PUT /a943edfc74d1ab77f70ed18df7837136 200 67108864 2 3 a943edfc74d1ab77f70ed18df7837136+67108864+A50f2108bba05ce3ca4207fba5b5df42f2dee097e@59ff76fb <nil>
2017/10/22 20:39:23 172.17.125.20:45773 GET /0b8e9baee9db04a2eacc7607374d85b9+67108864+A1cfec5bd7f55af2d708ed1aeccec3ad3b524f03e@59ff764b 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/0b8e9baee9db04a2eacc7607374d85b9+67108864+A1cfec5bd7f55af2d708ed1aeccec3ad3b524f03e@59ff764b <nil>
2017/10/22 20:39:23 DEBUG: [40f4b138] Begin upload df75f166f1a300305decd846dac39107 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:23 DEBUG: [40f4b138] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:23 DEBUG: [040051c4] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/a878dd89d76fd6cd0da20fcd8d9e0e6a success
2017/10/22 20:39:23 172.17.125.19:63593 PUT /a878dd89d76fd6cd0da20fcd8d9e0e6a 200 67108864 2 3 a878dd89d76fd6cd0da20fcd8d9e0e6a+67108864+A2a3825b46e9f455b851798f85eef5697368eecaa@59ff76fb <nil>
2017/10/22 20:39:23 DEBUG: [581f9a7d] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/f048f1fab182ecee65d937b8fa374d7c success
2017/10/22 20:39:23 172.17.125.17:32424 PUT /f048f1fab182ecee65d937b8fa374d7c 200 67108864 2 3 f048f1fab182ecee65d937b8fa374d7c+67108864+Aeed5bd4fdc8eb1b72128b970454c41c8b174ea7f@59ff76fb <nil>
2017/10/22 20:39:24 DEBUG: [2dc49cbf] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/c8e3764c006facc7813381d2d63224ab success
2017/10/22 20:39:24 172.17.125.21:56269 PUT /c8e3764c006facc7813381d2d63224ab 200 67108864 2 3 c8e3764c006facc7813381d2d63224ab+67108864+Ac8dbbacc717bc0ffd2c0c51c031cf21fbbcfa5e9@59ff76fc <nil>
2017/10/22 20:39:24 DEBUG: [12fcafe7] Begin upload ce37a7515455a0eb3eaa41bb0df23d8c to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [12fcafe7] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [50e29c7d] Begin upload 98447371696c00bd385a6186d8f67758 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [50e29c7d] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [74f42041] Begin upload c3cc637a2852f1b874e1fe89e8a0cafb to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [74f42041] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [6bece2cc] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/f88d02adc2e6981fc727435506b54093 success
2017/10/22 20:39:24 172.17.125.20:46696 PUT /f88d02adc2e6981fc727435506b54093 200 67108864 2 3 f88d02adc2e6981fc727435506b54093+67108864+Ab57bfc7f09ffa1c933791951e7298549ea373b28@59ff76fc <nil>
2017/10/22 20:39:24 DEBUG: [75dcaa38] Begin upload 78248c4044d54726a56e009ce189f438 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [75dcaa38] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [6a0c20e6] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/3328043e0bf23ae6f653168dd8a6c63c success
2017/10/22 20:39:24 172.17.125.19:62386 PUT /3328043e0bf23ae6f653168dd8a6c63c 200 67108864 2 3 3328043e0bf23ae6f653168dd8a6c63c+67108864+A3bf88f14e18e747e8a7bd51d89c0756cac5887ce@59ff76fc <nil>
2017/10/22 20:39:24 DEBUG: [00c1a6cf] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/0f5277163c5eaf736481f14c2de65cdb success
2017/10/22 20:39:24 172.17.125.24:27782 PUT /0f5277163c5eaf736481f14c2de65cdb 200 67108864 2 3 0f5277163c5eaf736481f14c2de65cdb+67108864+Ac52f7fac1ea145e992c64156097b16be28fd994c@59ff76fc <nil>
2017/10/22 20:39:24 DEBUG: [3eb9c218] Begin upload c995a3eae5d93066f7f34d6e09221c2a to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [3eb9c218] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [1ae47164] Begin upload 5eb4e5ff77d9ab5931948a03bab94708 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:24 DEBUG: [1ae47164] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:24 DEBUG: [2a66aa8b] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/95c8a06065d492de21f088ae87aadedc success
2017/10/22 20:39:24 172.17.125.17:34195 PUT /95c8a06065d492de21f088ae87aadedc 200 67108864 2 3 95c8a06065d492de21f088ae87aadedc+67108864+Ac9ddb668b6ab608e6186580e3732b7a4aa260c48@59ff76fc <nil>
2017/10/22 20:39:25 DEBUG: [46066f5f] Begin upload 2741b8adabc3b1ae18571afc5ee4cdf4 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [46066f5f] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [693636b6] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/bae7ca0651527d84d17c65b77751033a success
2017/10/22 20:39:25 172.17.125.24:24392 PUT /bae7ca0651527d84d17c65b77751033a 200 67108864 2 3 bae7ca0651527d84d17c65b77751033a+67108864+A49d61356c5b4729edd8d25e4713bcb3dfafc1bd5@59ff76fd <nil>
2017/10/22 20:39:25 172.17.125.22:61658 GET /1fe86b55070e78c1edf1a263f558d53a+67108864+A34254bb577586b0760bd24d0994a8d11c952a1f1@59ff76f1 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/1fe86b55070e78c1edf1a263f558d53a+67108864+A34254bb577586b0760bd24d0994a8d11c952a1f1@59ff76f1 <nil>
2017/10/22 20:39:25 DEBUG: [3f28ba25] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/f857e77637f6cbbb66f107fce106014c success
2017/10/22 20:39:25 172.17.125.21:56175 PUT /f857e77637f6cbbb66f107fce106014c 200 67108864 2 3 f857e77637f6cbbb66f107fce106014c+67108864+Ac9846b88a42d546318bdc3ebee0fa0c74474929d@59ff76fd <nil>
2017/10/22 20:39:25 DEBUG: [325ccfaf] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/f589a0c1c9294d487123cb6c39827bd3 success
2017/10/22 20:39:25 172.17.125.17:34308 PUT /f589a0c1c9294d487123cb6c39827bd3 200 67108864 2 3 f589a0c1c9294d487123cb6c39827bd3+67108864+A990600dd1e6ae522c14b442f4457fd48c0e2a073@59ff76fd <nil>
2017/10/22 20:39:25 DEBUG: [49e77f22] Begin upload 1b6a7665da64710b084f615f8f793f75 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [49e77f22] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [00f69542] Begin upload c9952058f8074c33a6fe1d126e30e556 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [00f69542] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [36323a26] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/de96e79af7e438b76c821b5990ece34b success
2017/10/22 20:39:25 172.17.125.17:33004 PUT /de96e79af7e438b76c821b5990ece34b 200 67108864 2 3 de96e79af7e438b76c821b5990ece34b+67108864+A4c3eb148a4945609d8f63e905173d67f65e10e19@59ff76fd <nil>
2017/10/22 20:39:25 DEBUG: [0a3f3241] Begin upload 762a3868556cba374c66ffb40f870776 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [0a3f3241] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [00420059] Begin upload 95ad9aec7bface02e80e16971b91df1e to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [00420059] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 172.17.125.22:58676 GET /dffe2f3f9c8e156e8469124d06dc5917+67108864+A6546af03e8d1ab6c4c4ef7d913054bf41ce80f65@59ff75d3 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/dffe2f3f9c8e156e8469124d06dc5917+67108864+A6546af03e8d1ab6c4c4ef7d913054bf41ce80f65@59ff75d3 <nil>
2017/10/22 20:39:25 DEBUG: [4fa1f683] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/c3f8d51541521454f11046c5739dec84 success
2017/10/22 20:39:25 172.17.125.17:33442 PUT /c3f8d51541521454f11046c5739dec84 200 67108864 2 3 c3f8d51541521454f11046c5739dec84+67108864+A80c23d296cb6d1482dd44bbf469684779dcbef78@59ff76fd <nil>
2017/10/22 20:39:25 DEBUG: [31d73e17] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/4d06bc93ec4ef0527d5261d9d1544dad success
2017/10/22 20:39:25 172.17.125.19:62432 PUT /4d06bc93ec4ef0527d5261d9d1544dad 200 67108864 2 3 4d06bc93ec4ef0527d5261d9d1544dad+67108864+Ae990b7a6fa468cc163360d8868ef15cf2c6056ef@59ff76fd <nil>
2017/10/22 20:39:25 DEBUG: [666ac7be] Begin upload 22588075c5a908fcefa7b2b9d8956aa4 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [666ac7be] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [0264a137] Begin upload 096cf06df657ade8874bdf6edb642ae8 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [0264a137] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:25 DEBUG: [2164d5ca] Begin upload 937d70862799aa2eabacaa1be7c3b696 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:25 DEBUG: [2164d5ca] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [344f28c1] Begin upload 162f775b734e48763acb7aa3dbb783d0 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [344f28c1] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [2fc2d620] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/0be9860e4a6ea28bf6228110a7715df2 success
2017/10/22 20:39:26 172.17.125.17:34480 PUT /0be9860e4a6ea28bf6228110a7715df2 200 67108864 2 3 0be9860e4a6ea28bf6228110a7715df2+67108864+A6e07eee62939251c65e36609f6063f2524701be6@59ff76fe <nil>
2017/10/22 20:39:26 172.17.125.23:12702 GET /2319d6128c6fc7de10b5ef1dc69c9ff4+67108864+Ae0ca5bef9dd70fd83e2e2a3f546f9ec121c6e0bc@59ff7666 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/2319d6128c6fc7de10b5ef1dc69c9ff4+67108864+Ae0ca5bef9dd70fd83e2e2a3f546f9ec121c6e0bc@59ff7666 <nil>
2017/10/22 20:39:26 DEBUG: [2ec2524f] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/f276914bb6bdf12874928361957e9840 success
2017/10/22 20:39:26 172.17.125.21:55727 PUT /f276914bb6bdf12874928361957e9840 200 67108864 2 3 f276914bb6bdf12874928361957e9840+67108864+A324224e5384ed240fc373b61e4250b590e1f89c0@59ff76fe <nil>
2017/10/22 20:39:26 DEBUG: [249f67d4] Begin upload fc2c33d70a4a70ec9b30033c232be597 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [249f67d4] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [365a99d8] Begin upload 9ee94f09da0963fd4bab3ebb305de27d to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [365a99d8] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 172.17.125.17:35246 GET /82748fd429928c0599b3f5de2b4bd25e+67108864+Ac029ab84be9d0cdf4f48604cc9b20ec516edfd5f@59ff76e7 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/82748fd429928c0599b3f5de2b4bd25e+67108864+Ac029ab84be9d0cdf4f48604cc9b20ec516edfd5f@59ff76e7 <nil>
2017/10/22 20:39:26 DEBUG: [6c16f1e3] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/d0f95bec51dca44c81505734587b278b success
2017/10/22 20:39:26 172.17.125.24:27673 PUT /d0f95bec51dca44c81505734587b278b 200 67108864 2 3 d0f95bec51dca44c81505734587b278b+67108864+Ad0f5d5faa489adc8d3bf1e6beb835c2fed8cb7df@59ff76fe <nil>
2017/10/22 20:39:26 DEBUG: [4df73f5b] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/2507cd58ed7550ec98a3f3d6ab3ef58f success
2017/10/22 20:39:26 172.17.125.24:29272 PUT /2507cd58ed7550ec98a3f3d6ab3ef58f 200 67108864 2 3 2507cd58ed7550ec98a3f3d6ab3ef58f+67108864+A275816406e9d7bed7ab3e15755e402b405502e75@59ff76fe <nil>
2017/10/22 20:39:26 DEBUG: [2be57741] Begin upload fb8b9c5c03e4b3b780ce823c0aba484d to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [2be57741] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [74f42041] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/c3cc637a2852f1b874e1fe89e8a0cafb success
2017/10/22 20:39:26 172.17.125.23:12359 PUT /c3cc637a2852f1b874e1fe89e8a0cafb 200 67108864 2 3 c3cc637a2852f1b874e1fe89e8a0cafb+67108864+A201d9ab4a89acdc5c79dcdffb95816a2f29b3edb@59ff76fe <nil>
2017/10/22 20:39:26 DEBUG: [17ef40f3] Begin upload baa92696c6d34188ab58203ea0d16123 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [17ef40f3] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [486fe79a] Begin upload fc37ffd893eaa29404eb0893b9cce531 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [486fe79a] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:26 DEBUG: [1956785b] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/1321be9afd333c182d5ac2aa5bdc03ed success
2017/10/22 20:39:26 172.17.125.20:46747 PUT /1321be9afd333c182d5ac2aa5bdc03ed 200 67108864 2 3 1321be9afd333c182d5ac2aa5bdc03ed+67108864+A160ca353303016bc2533fd10c81423483cd449ba@59ff76fe <nil>
2017/10/22 20:39:26 172.17.125.20:45773 GET /d24cd1f1e24dbe0120d888a0542c12ab+67108864+A9b90bf68ae9d88e9d11859c70e25f01112268776@59ff764b 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/d24cd1f1e24dbe0120d888a0542c12ab+67108864+A9b90bf68ae9d88e9d11859c70e25f01112268776@59ff764b <nil>
2017/10/22 20:39:26 DEBUG: [5c4848aa] Begin upload d58b35a5c9e1523ce02dc9bc0b92a484 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:26 DEBUG: [5c4848aa] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:27 DEBUG: [519269fa] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/5413d1598dec02a0b8413a24f1a923e1 success
2017/10/22 20:39:27 172.17.125.20:45324 PUT /5413d1598dec02a0b8413a24f1a923e1 200 67108864 2 3 5413d1598dec02a0b8413a24f1a923e1+67108864+A376b4ad213ff4a6b44c91831b728c3ecbbaf7661@59ff76fe <nil>
2017/10/22 20:39:27 172.17.125.17:33253 GET /abfb2d5c7e7d8561b422c914d86a81d4+67108864+A4240fea05b547937ddefee41e9cd6a29840d929a@59ff7635 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/abfb2d5c7e7d8561b422c914d86a81d4+67108864+A4240fea05b547937ddefee41e9cd6a29840d929a@59ff7635 <nil>
2017/10/22 20:39:27 DEBUG: [2dd8bafe] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/363298367993b806fa779147d68ffc47 success
2017/10/22 20:39:27 172.17.125.24:28252 PUT /363298367993b806fa779147d68ffc47 200 67108864 2 3 363298367993b806fa779147d68ffc47+67108864+Aaefe24b9dc8d89646b1919ab5ba53b36be33439c@59ff76ff <nil>
2017/10/22 20:39:27 DEBUG: [41164389] Begin upload 81d3709350b0c15db672089d566770b7 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:27 DEBUG: [41164389] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:27 DEBUG: [7174e154] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/d699db1cdb4041b586d1ad0e6eb24c1a success
2017/10/22 20:39:27 172.17.125.24:26735 PUT /d699db1cdb4041b586d1ad0e6eb24c1a 200 67108864 2 3 d699db1cdb4041b586d1ad0e6eb24c1a+67108864+A21c0fe75df5ac8a408375fb541dde7d6df12f0b7@59ff76ff <nil>
2017/10/22 20:39:27 DEBUG: [2e47a60c] Begin upload 089fc7679f3c5b3ec2087c0941ef9690 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:27 DEBUG: [2e47a60c] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:27 DEBUG: [470a1178] Begin upload 8b8e21f13cf94dd12197801f4c62b0e9 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:27 DEBUG: [470a1178] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:27 DEBUG: [46066f5f] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/2741b8adabc3b1ae18571afc5ee4cdf4 success
2017/10/22 20:39:27 172.17.125.21:56497 PUT /2741b8adabc3b1ae18571afc5ee4cdf4 200 67108864 2 3 2741b8adabc3b1ae18571afc5ee4cdf4+67108864+A88ca074f708999a90ad590d27b740203d69f5dad@59ff76ff <nil>
2017/10/22 20:39:27 172.17.125.22:61658 GET /326c94525c6b57d0baa083dce9b8b00c+67108864+Abde936941b26618dba19cd4af8f352230de3c4d0@59ff76f1 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/326c94525c6b57d0baa083dce9b8b00c+67108864+Abde936941b26618dba19cd4af8f352230de3c4d0@59ff76f1 <nil>
2017/10/22 20:39:27 DEBUG: [40f4b138] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/df75f166f1a300305decd846dac39107 success
2017/10/22 20:39:27 172.17.125.19:63360 PUT /df75f166f1a300305decd846dac39107 200 67108864 2 3 df75f166f1a300305decd846dac39107+67108864+Ab2b4734d5a5ccdeb86eafeeec7d9410767267770@59ff76ff <nil>
2017/10/22 20:39:27 DEBUG: [70c9e943] Begin upload 2c147b80cf80873a2a70b43eeffdaa48 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:27 DEBUG: [70c9e943] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:27 DEBUG: [7eeedcfd] Begin upload 9c538c0707f7edfd780c56b87335b0a3 to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:27 DEBUG: [7eeedcfd] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:28 DEBUG: [0a093b1c] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/a92db9f6d83ded839b1ec70dc10201fa success
2017/10/22 20:39:28 172.17.125.24:26276 PUT /a92db9f6d83ded839b1ec70dc10201fa 200 67108864 2 3 a92db9f6d83ded839b1ec70dc10201fa+67108864+Aa6fe72b9b898d51f0e69c6358fddbc2752112f8f@59ff7700 <nil>
2017/10/22 20:39:28 DEBUG: [3eb9c218] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/c995a3eae5d93066f7f34d6e09221c2a success
2017/10/22 20:39:28 172.17.125.19:61344 PUT /c995a3eae5d93066f7f34d6e09221c2a 200 67108864 2 3 c995a3eae5d93066f7f34d6e09221c2a+67108864+Aa0381863a730a4f3c554ec6db55ef967ca61a0dd@59ff7700 <nil>
2017/10/22 20:39:28 DEBUG: [3234c7ea] Begin upload fd1dc6d980df0783238d5d119cb49fd6 to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:28 DEBUG: [3234c7ea] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:28 DEBUG: [4770608c] Begin upload 0083d9648a610f055d4bf249ff1a02cf to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:28 DEBUG: [4770608c] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:28 172.17.125.22:58676 GET /3ececf9feb6db1afb2575bfd63344f0e+67108864+A11059452b857a8279a16f33c0cff8d08ad0931a0@59ff75d3 200 67108864 67108864 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/3ececf9feb6db1afb2575bfd63344f0e+67108864+A11059452b857a8279a16f33c0cff8d08ad0931a0@59ff75d3 <nil>
2017/10/22 20:39:28 172.17.125.23:12702 GET /0c93bd0b9cb399c8d16b937a27f65625+67108864+A8c80c7e540a5543a03964d5905322e922775935d@59ff7666 200 67108864 67108864 http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/0c93bd0b9cb399c8d16b937a27f65625+67108864+A8c80c7e540a5543a03964d5905322e922775935d@59ff7666 <nil>
2017/10/22 20:39:28 DEBUG: [237b4975] Begin upload 9a6234eef12de53e1b0ce6cd0d064fda to http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:28 DEBUG: [237b4975] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:28 DEBUG: [49e77f22] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/1b6a7665da64710b084f615f8f793f75 success
2017/10/22 20:39:28 172.17.125.19:63593 PUT /1b6a7665da64710b084f615f8f793f75 200 67108864 2 3 1b6a7665da64710b084f615f8f793f75+67108864+A43d32f2f490e9ab687eabbf65b7b0165b6831080@59ff7700 <nil>
2017/10/22 20:39:28 DEBUG: [31f19953] Upload failed http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/ff0c76b19b771312512f056b47a3dabe error: Put http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/ff0c76b19b771312512f056b47a3dabe: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017/10/22 20:39:28 DEBUG: [31f19953] Begin upload ff0c76b19b771312512f056b47a3dabe to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:28 DEBUG: [31f19953] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:28 DEBUG: [00420059] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/95ad9aec7bface02e80e16971b91df1e success
2017/10/22 20:39:28 172.17.125.17:32424 PUT /95ad9aec7bface02e80e16971b91df1e 200 67108864 2 3 95ad9aec7bface02e80e16971b91df1e+67108864+A2a3904916e459ef5a2590dc5486e6f71bd28280d@59ff7700 <nil>
2017/10/22 20:39:28 DEBUG: [4fdf5f23] Begin upload abdf6e1de668184433bfbf52ea13494b to http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107
2017/10/22 20:39:28 DEBUG: [4fdf5f23] Replicas remaining to write: 2 active uploads: 1
2017/10/22 20:39:29 DEBUG: [17a2a0f5] Upload http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/228b1cb34700f57d316b2ba88a067a2b success
2017/10/22 20:39:29 172.17.125.20:47002 PUT /228b1cb34700f57d316b2ba88a067a2b 200 67108864 2 3 228b1cb34700f57d316b2ba88a067a2b+67108864+Ab84eacc308bdf1c082f59fb4f40012b1450e1a23@59ff7700 <nil>
2017/10/22 20:39:29 DEBUG: [12fcafe7] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/ce37a7515455a0eb3eaa41bb0df23d8c success
2017/10/22 20:39:29 172.17.125.20:46595 PUT /ce37a7515455a0eb3eaa41bb0df23d8c 200 67108864 2 3 ce37a7515455a0eb3eaa41bb0df23d8c+67108864+A4b41db30e64cdb841ccf12d5ec88b1e9297e494a@59ff7701 <nil>
2017/10/22 20:39:29 DEBUG: [502012f3] Upload http://arvados-keep1-ncucu.node.delta-hgiarvados.consul:25107/cde60adfa1c82458d0e98eb33d11f301 success
2017/10/22 20:39:29 172.17.125.22:55825 PUT /cde60adfa1c82458d0e98eb33d11f301 200 67108864 2 3 cde60adfa1c82458d0e98eb33d11f301+67108864+A9e89ca959a4a0b0264afb85bc8916a48253d06a0@59ff7701 <nil>
panic: send on closed channel
goroutine 15977 [running]:
git.curoverse.com/arvados.git/sdk/go/streamer.handleReadRequest(0x0, 0xe98, 0xc420863260, 0xc47f8d4000, 0x4000000, 0x4000000, 0x0, 0x0, 0x8a9d00)
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/transfer.go:170 +0x218
git.curoverse.com/arvados.git/sdk/go/streamer.(*AsyncStream).transfer(0xc4207282c0, 0x8a6300, 0xc4205f90b0)
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/transfer.go:244 +0x51d
created by git.curoverse.com/arvados.git/sdk/go/streamer.AsyncStreamFromReader
/tmp/tmp.JuAvTIlWhz/src/git.curoverse.com/arvados.git/sdk/go/streamer/streamer.go:74 +0x1d0
I tried this a second time and the "Upload failed" "net/http: request canceled (Client.Timeout exceeded while awaiting headers)" line also occurred in that instance, so it seems that somehow a failure to contact one of the keepstores within the timeout may be triggering a close on a channel which is being sent to by another goroutine.
Updated by Joshua Randall over 6 years ago
We have increased the number of keepproxy and keepstore services in our cluster which works around this issue, but it remains a bug with keepproxy which in no case should be expected to panic and crash when it cannot successfully contact a keepstore.
Updated by Tom Morris
Updated by Lucas Di Pentima over 6 years ago
- Status changed from New to In Progress
- Assigned To set to Lucas Di Pentima
Updated by Lucas Di Pentima over 6 years ago
After adding lots of logging and running a test case with keepclient's HTTPClient Timeout set to 1 microsecond to force a timeout error, I'm getting what it seems to be a racy behavior that I'm not sure is the same or related to this report.
What I'm seeing is that StreamReader.Close() is called once when httpClient().Do() (at support.go file) doesn't have any error, but when a timeout error is forced, Close() is called several times, and some of the times, it produces panics related to "close on closed channel" or the one that is reported here: "send on closed channel". Some other times I get a segmentation violation error.
I looked for similar reports related to net/http package but couldn't find anything meaningful yet.
What I'm doing to force an error is modify TestPutHR test on keepclient_test.go file, changing the timeout this way:
kc.httpClient().(*http.Client).Timeout = time.Microsecond
...while trying to write a ~60 MB []byte
Updated by Lucas Di Pentima over 6 years ago
- Target version changed from 2017-11-22 Sprint to 2017-12-06 Sprint
Updated by Tom Clegg over 6 years ago
- Assigned To changed from Lucas Di Pentima to Tom Clegg
The behavior of a second call to Close() is "undefined", but that doesn't mean it's allowed to crash. There are code paths in the http library that close the request body twice.
Updated by Tom Clegg over 6 years ago
12475-asyncbuf @ 4f1a135e93df78bb833dff32562efe713c6f690e
- complete rewrite of sdk/go/streamer as sdk/go/asyncbuf. Much simpler implementation and interface: 450→100 lines, no extra goroutines, no need to close readers, option to supply initial data and/or buffer space at creation time just like bytes.Buffer.
- keepproxy test that times out with the old code, but succeeds now (this is the closest I've come to reproducing the crash).
Updated by Lucas Di Pentima over 6 years ago
Just one question:
On file sdk/go/asyncbuf/buf.go, line 90: Is the “|| len(p) == 0” part for cases when the reader passes a new buffer? If that’s the case, shouldn’t “r.read” be reset?
Updated by Tom Clegg over 6 years ago
len(p)==0 means the caller asked to read into a zero-length buffer. I figured it would be pointless to block and wait for more data. But this meant reading into a zero-length buffer would never return io.EOF. I've updated it so Read(buf[:0]) never blocks, but does signal EOF as promised by https://golang.org/pkg/io/#Reader: after a Read that reaches EOF, "The next Read should return 0, EOF."
Updated by Joshua Randall over 6 years ago
I re-ran the workload that triggered the panic and verified it was still doing so. I then built keepproxy from the 12475-asyncbuf branch and re-ran the workload again. The fix seems to have resolved the panic issue, but another issue has arisen in its place.
The workload consists of 30 parallel workers uploading data to keep, then immediately downloading the data to verify the md5.
So far, 7 have had an error during write to keep and 80 have had an error during the read back from keep. None have succeeded.
The write errors are all 413 Request Entity Too Large:
arvados.errors.KeepWriteError: failed to write 7eef0c02d3d7482f5300ff2cb9ea99d7 (wanted 2 copies but wrote 0): service http://arvados-keepproxy-ncucu-02.hgi.sanger.ac.uk:25107/ responded with 413 HTTP/1.0 413 Request Entity Too Large ; service http://arvados-keepproxy-ncucu-01.hgi.sanger.ac.uk:25107/ responded with 413 HTTP/1.0 413 Request Entity Too Large
The read errors are all "transfer closed with XXXXXXXX bytes remaining to read:
arvados.errors.KeepReadError: failed to read dc44317018f467814ce4ed414bb63739+67108864+Aa607aa1e3ae81ea02fdc89135cb244108c7e9056@5a2d3e12: service http://arvados-keepproxy-ncucu-02.hgi.sanger.ac.uk:25107/ responded with 0 (18, 'transfer closed with 2285026 bytes remaining to read'); service http://arvados-keepproxy-ncucu-01.hgi.sanger.ac.uk:25107/ responded with 0 (18, 'transfer closed with 49384122 bytes remaining to read')
The keepproxy logs does have some timeouts in its logs:
2017-11-26_17:14:09.23062 2017/11/26 17:14:09 172.17.91.44,172.17.55.130:58232 GET /140fabfbbbc0df3d21a5a7c05e3e8913+67108864+A1ac948fab8993057b2aade3328ef3dd528acfb44@5a2d6928 200 671088 64 31392458 http://arvados-keep0-ncucu.node.delta-hgiarvados.consul:25107/140fabfbbbc0df3d21a5a7c05e3e8913+67108864+A1ac948fab8993057b2aade3328ef3dd528acfb44@5a2d6928 net/http: request canceled (Client.Timeout exceeded while reading body)
Updated by Tom Clegg over 6 years ago
Josh, I wonder if you need to configure keepproxy with a longer timeout. The default 15 seconds, although the install guide recommends a 300 second timeout for Nginx upstream.
# /etc/arvados/keepproxy/keepproxy.yml Client: APIHost: zzzzz.arvadosapi.com AuthToken: anon_token_goes_here Listen: :25107 Timeout: 300s
Updated by Tom Clegg over 6 years ago
- Status changed from In Progress to Feedback
Updated by Joshua Randall over 6 years ago
nginx has the recommended timeout:
proxy_read_timeout 300s;
But there is no timeout specified for keepproxy itself. Will add that now and give it a try.
Updated by Joshua Randall over 6 years ago
With the 300s timeout on keepproxy, the previous issue has gone away and I am now seeing a new error: "Operation too slow. Less than 32768 bytes/sec transferred the last 64 seconds":
Traceback (most recent call last):
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/bin/arv-put", line 4, in <module>
__import__('pkg_resources').run_script('arvados-python-client==0.1.20151119172035', 'arv-put')
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 735, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1652, in run_script
exec(code, namespace, namespace)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/EGG-INFO/scripts/arv-put", line 4, in <module>
main()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/commands/put.py", line 484, in main
path, max_manifest_depth=args.max_manifest_depth)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/commands/put.py", line 334, in write_directory_tree
path, stream_name, max_manifest_depth)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 216, in write_directory_tree
self.do_queued_work()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 144, in do_queued_work
self._work_file()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 157, in _work_file
self.write(buf)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 471, in write
return super(ResumableCollectionWriter, self).write(data)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 227, in write
self.flush_data()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/commands/put.py", line 310, in flush_data
super(ArvPutCollectionWriter, self).flush_data()
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/collection.py", line 264, in flush_data
copies=self.replication))
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/retry.py", line 153, in num_retries_setter
return orig_func(self, *args, **kwargs)
File "/software/hgi/pkglocal/arvados-sdk-cli-git-20151119-76683cd/arvados-python-venv/lib/python2.7/site-packages/arvados_python_client-0.1.20151119172035-py2.7.egg/arvados/keep.py", line 1061, in put
data_hash, copies, thread_limiter.done()), service_errors, label="service")
arvados.errors.KeepWriteError: failed to write e9bd7f5ff85505b0da7c8abdb6433037 (wanted 2 copies but wrote 0): service http://arvados-keepproxy-ncucu-01.hgi.sanger.ac.uk:25107/ responded with 0 (28, 'Operation too slow. Less than 32768 bytes/sec transferred the last 64 seconds'); service http://arvados-keepproxy-ncucu-03.hgi.sanger.ac.uk:25107/ responded with 502 HTTP/1.0 502 Bad Gateway
; service http://arvados-keepproxy-ncucu-02.hgi.sanger.ac.uk:25107/ responded with 0 (28, 'Operation too slow. Less than 32768 bytes/sec transferred the last 64 seconds'); service http://arvados-keepproxy-ncucu-04.hgi.sanger.ac.uk:25107/ responded with 502 HTTP/1.0 502 Bad Gateway
Not sure if it matters, but clients are currently accessing keepproxy directly on port 25107 rather than via nginx.
Updated by Joshua Randall over 6 years ago
After quite a lot of searching, I see that error is actually coming from pycurl on the client side (that could probably be made more clear in the reported error, which makes it seem like the server actually responded with a 502). Something is really wrong if connections are not delivering 32kB/s - I'll investigate the connection more in-depth.
Updated by Joshua Randall over 6 years ago
Ok, it looks like the asyncbuf changes have resolved all of the issues. Other issues I was having appear to be because the clients were accidentally pulling in some http_proxy env vars so the connections were going through a web cache (and overloading it, I guess).
Actions