Project

General

Profile

Actions

Bug #13067

closed

"Uh oh" While uploading output files ... bad address

Added by Joshua Randall about 6 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
-

Description

We have a container that repeatedly fails "while uploading output files" after the container itself has apparently succeeded:

2018-02-13T10:47:28.405139093Z Waiting for container to finish 
2018-02-13T10:58:08.265385511Z Container exited with code: 0 
2018-02-13T10:58:08.266390114Z Uploading 00/13/745a3b7f3937b3f0d58b8fb8b96d (1303 bytes) 
2018-02-13T10:58:08.268423270Z Uploading 00/19/79586db1b719cca6d3dc0e50e71c (2795 bytes) 
2018-02-13T10:58:08.270223330Z Uploading 00/21/932312a252327115aef207a22260 (1037 bytes) 
2018-02-13T10:58:08.272155070Z Uploading 00/21/de79f69c7e6f7be872209251d8a2 (1291 bytes) 
2018-02-13T10:58:08.272514690Z Uploading 00/40/1a0950f66bdb8079810e7cd7ea17 (1301 bytes) 
2018-02-13T10:58:08.274532596Z Uploading 00/70/9b8f2f15aa3aa482a992d1d87b60 (3354 bytes) 
2018-02-13T10:58:08.276624212Z Uploading 00/71/e65ee429afd318ef699ad315c4b4 (3349 bytes) 
2018-02-13T10:58:08.281484141Z Uploading 00/9f/b96b4ee6d64732dac95a1f380f41 (198005 bytes) 
2018-02-13T10:58:08.284373979Z Uploading 00/a6/63d257ac9d4d6eabae95f6fb6d8d (3517 bytes) 
2018-02-13T10:58:08.286329164Z Uploading 00/a9/379bf89f4556f66988ab0193b4c8 (3841 bytes) 
2018-02-13T10:58:08.288339941Z Uploading 00/b5/a7cce8e5acc98b508e4d39091f15 (3354 bytes) 
2018-02-13T10:58:08.290243279Z Uploading 00/ea/d3e9dd86f6d5e7d499e285b509b9 (1312 bytes) 
2018-02-13T10:58:08.292314091Z Uploading 00/f1/80d0f4f2be5d3a79232ebf1f4150 (1200 bytes) 
2018-02-13T10:58:08.294269577Z Uploading 00/fd/36e8b22143b0adfbe025e2d5d2b4 (3506 bytes) 
2018-02-13T10:58:08.296324612Z Uploading 01/02/bc3c8078cc16bddfa125f005370a (1396 bytes) 
2018-02-13T10:58:08.298141915Z Uploading 01/07/37c4caab4d5e629753ad40bee3bf (8632 bytes) 
2018-02-13T10:58:08.300011513Z Uploading 01/2d/f8747a01987e4e356ba886a5abc6 (118 bytes) 
2018-02-13T10:58:08.301899534Z Uploading 01/36/d8cfbbe9fd393a1e699753f7f7a3 (3323 bytes) 
2018-02-13T10:58:08.303731504Z Uploading 01/81/d1738fbbb95deba33bf46e646c6c (109 bytes) 
2018-02-13T10:58:08.305649650Z Uploading 01/94/f66b5128ba6d3eb9fec43ee985c7 (3332 bytes) 
2018-02-13T10:58:08.310300310Z Uploading 01/98/ad2dfad702357fb849a43500deaa (2400 bytes) 
2018-02-13T10:58:08.312137508Z Uploading 01/b0/01b48d8d89f7671231f8cd45ca05 (2979 bytes) 
2018-02-13T10:58:08.314075928Z Uploading 01/bb/e99249b489660daa17fa9f958d10 (1726 bytes) 
2018-02-13T10:58:08.315896739Z Uploading 01/c1/108b447f8fc1ac66a83a185cf320 (1113 bytes) 
2018-02-13T10:58:08.317669710Z Uploading 01/cd/0df602495b044b2c214d69a60aa2 (3503 bytes) 
2018-02-13T10:58:08.319449499Z Uploading 01/e9/ba63de5b8b6e1a07efb0b1179f20 (1297 bytes) 
2018-02-13T10:58:08.321308413Z Uploading 02/0a/82dc938b6e53e904f477268c7d5c (1453 bytes) 
2018-02-13T10:58:08.323245085Z Uploading 02/1a/0164c68fd77f51bead9e2c4e9cea (10556 bytes) 
2018-02-13T10:58:08.325110854Z Uploading 02/34/68d3dca8b3726de68eb85c7d2301 (1370 bytes) 
2018-02-13T10:58:08.326949368Z Uploading 02/5a/05973cf7594618e0b7f75b9275ed (132 bytes) 
2018-02-13T10:58:08.328878613Z Uploading 02/80/145eb36a86910fb481636aee0329 (119 bytes) 
2018-02-13T10:58:08.330714408Z Uploading 02/90/d7c3b24a19eefc6a89bc41146c39 (2676 bytes) 
2018-02-13T10:58:08.332569542Z Uploading 02/a4/0e206c78446d368b6001622aa540 (3410 bytes) 
2018-02-13T10:58:08.334415253Z Uploading 02/ae/1b32f6268509dbfab828c8339a36 (3517 bytes) 
2018-02-13T10:58:08.336334390Z Uploading 02/b4/6fcce463bf80c626c51f5d36bab3 (3086 bytes) 
2018-02-13T10:58:08.338245323Z Uploading 02/b5/04f4187bebdd1b90e7af2e356c13 (132 bytes) 
2018-02-13T10:58:08.340092245Z Uploading 02/bd/84b3d7b9d16ec8546efc41b4d0c5 (2917 bytes) 
2018-02-13T10:58:08.341993640Z Uploading 02/cb/7169934dce464116e8644638b22d (2677 bytes) 
2018-02-13T10:58:08.343787961Z Uploading 02/de/644cfd20b00dbf3730793f839102 (4678 bytes) 
2018-02-13T10:58:08.345828072Z Uploading 03/1c/851664e31b2c17337fd6f9004858 (180857866 bytes) 
2018-02-13T10:58:09.080000950Z Uploading 03/23/f4f5194c96dc1bcb919f06960a20 (2128 bytes) 
2018-02-13T10:58:09.082740234Z Uploading 03/34/6468330af3c0b6afb3ff57a8fc9e (100 bytes) 
2018-02-13T10:58:09.085426733Z Uploading 03/38/a9928d761051eb989aba5a478df1 (1138 bytes) 
2018-02-13T10:58:09.087994627Z Uploading 03/43/aa684954b84cff10724f5d1624e1 (91 bytes) 
2018-02-13T10:58:09.090663894Z Uploading 03/55/34e6b4f7a7b2b20506a41290b8de (3502 bytes) 
2018-02-13T10:58:09.096895302Z Uploading 03/55/7dc0d8d0509da30549dc16031c51 (7832 bytes) 
2018-02-13T10:58:09.097245597Z Uploading 03/5e/ac523101444b286c852a129bb0f0 (2644 bytes) 
2018-02-13T10:58:09.101616516Z Uploading 03/6d/24e0ed6b0b50f1cfbd3f587c4cc2 (1234 bytes) 
2018-02-13T10:58:09.104380060Z Uploading 03/70/2f960713a7c42469b033b7ef05ec (3387 bytes) 
2018-02-13T10:58:09.106884284Z Uploading 03/70/a3a0105a0683cbd802403418f170 (1154 bytes) 
2018-02-13T10:58:09.107240324Z Uploading 03/85/be87eb49df4c59d7487495e3b1b4 (3095 bytes) 
2018-02-13T10:58:09.109864781Z Uploading 03/86/df1d3476e6649f919195cc072fc7 (155864 bytes) 
2018-02-13T10:58:09.113873743Z Uploading 03/97/65c983a107557b8c3ec0a025e44c (1162 bytes) 
2018-02-13T10:58:09.116762569Z Uploading 03/ab/659ca0a3d031ed57f5b71c40b7bb (2066 bytes) 
2018-02-13T10:58:09.119561946Z Uploading 03/c6/c36060cd6c9f9b4fd0b096ff40e2 (586476 bytes) 
2018-02-13T10:58:09.124625097Z Uploading 03/cd/b0fa6bc0f56dc7266334229fca85 (3455 bytes) 
2018-02-13T10:58:09.127457339Z Uploading 03/ed/78804fcfe33fb202c22d20e1ed05 (96924 bytes) 
2018-02-13T10:58:09.130466463Z Uploading 03/ee/83997251cdeeec6269f547238122 (3270 bytes) 
2018-02-13T10:58:09.133121336Z Uploading 03/f3/627a2f7242721c8ba740808d67d3 (1040 bytes) 
2018-02-13T10:58:09.135752154Z Uploading 04/0a/6029ad56ae20245dd1757c313f19 (1379 bytes) 
2018-02-13T10:58:09.138409005Z Uploading 04/54/9369e1197c626669a10164613635 (2168 bytes) 
2018-02-13T10:58:09.141061575Z Uploading 04/57/c7224cf704690ef28f6a00aa772f (106 bytes) 
2018-02-13T10:58:09.143688794Z Uploading 04/5d/b17b3bfbaa2bd06f68e2e508b326 (135 bytes) 
2018-02-13T10:58:09.146399565Z Uploading 04/69/343a7677dac01c66bf219aa0611d (2872 bytes) 
2018-02-13T10:58:09.148870182Z Uploading 04/a7/0773f36888dde7766a58c3a92de1 (1614 bytes) 
2018-02-13T10:58:09.151562380Z Uploading 04/ad/661ad55cf80b26d65b537fa28426 (1198 bytes) 
2018-02-13T10:58:09.154265703Z Uploading 04/ae/c18ad103fe46337c6643d0eaca0c (4379 bytes) 
2018-02-13T10:58:09.156982897Z Uploading 04/bd/45b790033b8311ed85dea3015e01 (1208 bytes) 
2018-02-13T10:58:09.159610601Z Uploading 04/e3/6a548c184b404c00b7841ba45394 (3508 bytes) 
2018-02-13T10:58:09.162411608Z Uploading 04/e6/ca7eabe96be35a18eb4c0675071d (111737 bytes) 
2018-02-13T10:58:09.165523879Z Uploading 04/fd/5dc748b4e7815316b0cccea3cba0 (1007 bytes) 
2018-02-13T10:58:09.168311247Z Uploading 05/0d/f236c1fa3a63c31f33b73308c168 (3173 bytes) 
2018-02-13T10:58:09.170938613Z Uploading 05/2b/d75a7a62c49f1a50245553b38755 (127 bytes) 
2018-02-13T10:58:09.173834715Z Uploading 05/2f/146ade5330135e5381db9c779f75 (3892 bytes) 
2018-02-13T10:58:09.176580713Z Uploading 05/3e/a4692acbe04aa98a6f331417531f (2029 bytes) 
2018-02-13T10:58:09.179263322Z Uploading 05/7f/5c85ad1f09e82b0f04bbc54f9283 (3227 bytes) 
2018-02-13T10:58:09.182058713Z Uploading 05/b1/945756fa1adf0d4547bcd12f01ce (1012 bytes) 
2018-02-13T10:58:09.184600028Z Uploading 05/c2/d3d1fe5c9569c6f982e7978b0d6d (1258 bytes) 
2018-02-13T10:58:09.187445752Z Uploading 05/cd/e0965d9330b1048d4d3c3d3d700c (1242 bytes) 
2018-02-13T10:58:09.190342978Z Uploading 05/d7/18c7b92a89c2a4b961ee6c0b008c (3473 bytes) 
2018-02-13T10:58:09.193236624Z Uploading 05/dc/0384da2f751afe549a9bfdbc3037 (3105 bytes) 
2018-02-13T10:58:09.195835702Z Uploading 05/e4/2e07ae267531d1c66f7da4ad58d9 (117 bytes) 
2018-02-13T10:58:09.198736650Z Uploading 05/f2/acb99e398b98f272d297727f28d1 (3320 bytes) 
2018-02-13T10:58:09.203347641Z Uploading 05/f9/88a4a7180849c6c81602c29279eb (2303 bytes) 
2018-02-13T10:58:09.206434242Z Uploading 05/fc/8de1a02c971099db853c790d3fca (3152 bytes) 
2018-02-13T10:58:09.209236984Z Uploading 06/0b/9bb36bc05042b371bfbfc6618e3c (9572 bytes) 
2018-02-13T10:58:09.212239344Z Uploading 06/23/13b0d5e67fee9d60100c7938397d (1069 bytes) 
2018-02-13T10:58:09.215605315Z Uploading 06/2e/3bbeb41213d0e68f944e1911f058 (1213 bytes) 
2018-02-13T10:58:09.218422630Z Uploading 06/30/c8622e65da828a153d66ab29b298 (1672 bytes) 
2018-02-13T10:58:09.221125796Z Uploading 06/33/58c8e7f81361b959efab7b3f15cc (223995 bytes) 
2018-02-13T10:58:09.224846039Z Uploading 06/7a/ad52b901e2666ca5ae78bc788671 (118 bytes) 
2018-02-13T10:58:09.227485429Z Uploading 06/84/fd43ee5428bc529561b33db2f2d0 (3323 bytes) 
2018-02-13T10:58:09.230496229Z Uploading 06/93/c472fdc0225c133374a15973bec3 (7368 bytes) 
2018-02-13T10:58:09.233508274Z Uploading 06/b5/a586c070a38af486a0ae25dac08e (1320 bytes) 
2018-02-13T10:58:09.236133056Z Uploading 06/b6/6d904a7902fb764a3e661b9c8bea (1572 bytes) 
2018-02-13T10:58:09.239140894Z Uploading 06/b8/80357cb8d04e7715100462e6e58d (2676 bytes) 
2018-02-13T10:58:09.241977200Z Uploading 06/c6/aff700dd7ddd935fd83f2def16e9 (3389 bytes) 
2018-02-13T10:58:09.244870716Z Uploading 06/c8/1b1f3014d187114393c990dbb3d1 (1199 bytes) 
2018-02-13T10:58:09.249683976Z Uploading 06/cb/f126247d89664a4faebad130fe9c (40103 bytes) 
2018-02-13T10:58:09.257188265Z Uploading 06/cd/979553eb7045f14feac8e6f2e2bf (1360 bytes) 
2018-02-13T10:58:09.260099904Z Uploading 06/e8/78f736f352fe167cc033f030b492 (2254 bytes) 
2018-02-13T10:58:09.262997195Z Uploading 06/eb/26d8459f3e43735ded45aa07a2e4 (1847 bytes) 
2018-02-13T10:58:09.265688651Z Uploading 07/0b/4678e22501029c2e3297115216bc (5674 bytes) 
2018-02-13T10:58:09.268475097Z Uploading 07/18/91675e2ba4476534e0035ef13cef (2157 bytes) 
2018-02-13T10:58:09.270917842Z Uploading 07/1e/a5ccf103c1e5bc64476b69297cae (1055 bytes) 
2018-02-13T10:58:09.274283315Z Uploading 07/24/eae831018e8ab18798c02eaa4f1a (1096 bytes) 
2018-02-13T10:58:09.276968365Z Uploading 07/2e/a3e53c79f3d00e1f1a7b492b0a8f (3136 bytes) 
2018-02-13T10:58:09.279667695Z Uploading 07/34/ed3f7e1aed9828f22b347664e98e (2885 bytes)
2018-02-13T10:58:09.282325310Z Uploading 07/40/173db9ffd264d728f32784845cd7 (180915260 bytes) 
2018-02-13T10:58:11.517665213Z Uh oh 
2018-02-13T10:58:11.518534703Z While uploading output files: read /tmp/527909571/07/40/173db9ffd264d728f32784845cd7: bad address 
2018-02-13T10:58:11.520830284Z Cancelled 
2018-02-13T10:58:12.192661163Z Running [arv-mount --unmount-timeout=8 --unmount /tmp/keep337770665] 
2018-02-13T10:58:12.225128495Z Arv-mount exit error: signal: killed 
2018-02-13T10:58:12.225185604Z While cleaning up arv-mount directory /tmp/keep337770665: remove /tmp/keep337770665: device or resource busy 
2018-02-13T10:58:12.560288194Z crunch-run finished

Actions #1

Updated by Joshua Randall about 6 years ago

After tracing this (to `io.Copy()`) It seems likely this is somehow related to memory cgroup limits which have recently been applied.

Actions #2

Updated by Tom Morris almost 6 years ago

If this was tracked down to an out of memory condition, can it be closed?

Actions #3

Updated by Peter Amstutz about 4 years ago

  • Status changed from New to Closed
Actions

Also available in: Atom PDF