Project

General

Profile

Actions

Bug #9843

closed

[Testing] FUSE tests sometimes hang

Added by Peter Amstutz over 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
-

Description

https://ci.curoverse.com/job/run-tests-services-fuse/69/console

test_tmp_snapshots (tests.test_tmp_collection.TmpCollectionTest) ... [keep1] 2016/08/23 22:48:02 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000658s 0.000650s 0.000009s 200 86 "OK" 
[keep0] 2016/08/23 22:48:02 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000858s 0.000852s 0.000006s 200 86 "OK" 
[keep1] 2016/08/23 22:48:02 [[::1]:45259] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000436s 0.000429s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:02 [[::1]:59964] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000602s 0.000597s 0.000006s 200 86 "OK" 
[keep0] 2016/08/23 22:48:02 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.002765s 0.002750s 0.000015s 200 86 "OK" 
[keep1] 2016/08/23 22:48:02 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.004229s 0.004223s 0.000006s 200 86 "OK" 
[keep0] 2016/08/23 22:48:02 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000342s 0.000335s 0.000007s 200 86 "OK" 
[keep1] 2016/08/23 22:48:02 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.026206s 0.026194s 0.000012s 200 86 "OK" 
[keep1] 2016/08/23 22:48:02 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.030500s 0.030485s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:02 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.030799s 0.030791s 0.000008s 200 86 "OK" 
[keep1] 2016/08/23 22:48:02 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000410s 0.000398s 0.000011s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.083586s 0.083573s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000616s 0.000603s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009820s 0.009809s 0.000011s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000350s 0.000343s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000450s 0.000392s 0.000057s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000459s 0.000451s 0.000009s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009906s 0.009893s 0.000014s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009985s 0.009972s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.010542s 0.010530s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.011556s 0.010449s 0.001108s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.012206s 0.012197s 0.000009s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.010836s 0.010821s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.010675s 0.010660s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000462s 0.000448s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.013127s 0.013114s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000301s 0.000295s 0.000006s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000278s 0.000273s 0.000005s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011132s 0.011117s 0.000015s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011588s 0.011582s 0.000006s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000354s 0.000342s 0.000012s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009689s 0.009677s 0.000012s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.010275s 0.010263s 0.000012s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.012181s 0.012168s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011158s 0.011143s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.012465s 0.012455s 0.000009s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000493s 0.000483s 0.000009s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009490s 0.009472s 0.000018s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.004301s 0.004285s 0.000016s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.010401s 0.010387s 0.000014s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000518s 0.000506s 0.000012s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009417s 0.009406s 0.000011s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.000473s 0.000463s 0.000010s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009573s 0.009559s 0.000015s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000413s 0.000407s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000495s 0.000489s 0.000006s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009701s 0.009684s 0.000018s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.010009s 0.009995s 0.000014s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.004252s 0.004237s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.012884s 0.012870s 0.000014s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.000628s 0.000619s 0.000008s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.009735s 0.009722s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.009907s 0.009892s 0.000015s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011379s 0.011372s 0.000007s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45257] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011366s 0.011355s 0.000011s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59962] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011163s 0.011152s 0.000011s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.010564s 0.010550s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.012965s 0.012958s 0.000008s 200 86 "OK" 
[keep1] 2016/08/23 22:48:03 [[::1]:45259] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.010881s 0.010868s 0.000014s 200 86 "OK" 
[keep0] 2016/08/23 22:48:03 [[::1]:59964] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.011822s 0.011817s 0.000005s 200 86 "OK" 
ok
test_two_tmp (tests.test_tmp_collection.TmpCollectionTest) ... [keep1] 2016/08/23 22:48:05 [[::1]:45269] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.002983s 0.002967s 0.000016s 200 86 "OK" 
[keep0] 2016/08/23 22:48:05 [[::1]:59974] PUT acbd18db4cc2f85cedef654fccc4a4d8 3 0.015186s 0.015174s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:05 [[::1]:59976] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.012685s 0.012672s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:05 [[::1]:45271] PUT 37b51d194a7513e45b56f6524f2d51f2 3 0.013135s 0.013129s 0.000006s 200 86 "OK" 
ok
Sent SIGTERM to 18524 (/data/1/jenkins/workspace/run-tests-services-fuse/tmp/keep0.pid)
[keep0] 2016/08/23 22:48:07 caught signal: terminated
[keep0] 2016/08/23 22:48:07 keepstore exiting, pid 18524
Sent SIGTERM to 18537 (/data/1/jenkins/workspace/run-tests-services-fuse/tmp/keep1.pid)
[keep1] 2016/08/23 22:48:07 caught signal: terminated
[keep1] 2016/08/23 22:48:07 keepstore exiting, pid 18537
[keep0] 2016/08/23 22:48:10 keepstore starting, pid 19722
[keep0] 2016/08/23 22:48:10 Using volume [UnixVolume /tmp/tmpgkVxnA] (writable=true)
[keep0] 2016/08/23 22:48:10 never-delete is not set. Warning: the relevant features in keepstore and data manager have not been extensively tested. You should leave this option alone unless you can afford to lose data.
[keep0] 2016/08/23 22:48:10 -max-requests <1 or not specified; defaulting to maxBuffers * 2 == 256
[keep0] 2016/08/23 22:48:10 listening at :38597
[keep1] 2016/08/23 22:48:11 keepstore starting, pid 19768
[keep1] 2016/08/23 22:48:11 Using volume [UnixVolume /tmp/tmppqnoNx] (writable=true)
[keep1] 2016/08/23 22:48:11 never-delete is not set. Warning: the relevant features in keepstore and data manager have not been extensively tested. You should leave this option alone unless you can afford to lose data.
[keep1] 2016/08/23 22:48:11 -max-requests <1 or not specified; defaulting to maxBuffers * 2 == 256
[keep1] 2016/08/23 22:48:11 listening at :34601
test_cache_spill (tests.test_cache.TmpCollectionTest) ... [keep0] 2016/08/23 22:48:12 [[::1]:59912] PUT 16a7cbedc0cf22673283940f5bb78f36 9 0.004757s 0.004746s 0.000011s 200 86 "OK" 
[keep1] 2016/08/23 22:48:12 [[::1]:39985] PUT 16a7cbedc0cf22673283940f5bb78f36 9 0.005996s 0.005989s 0.000008s 200 86 "OK" 
[keep0] 2016/08/23 22:48:12 [[::1]:59912] PUT f2b7e87217fb5a47afcc84f91dac3bd6 6 0.006730s 0.006716s 0.000014s 200 86 "OK" 
[keep1] 2016/08/23 22:48:12 [[::1]:39985] PUT f2b7e87217fb5a47afcc84f91dac3bd6 6 0.007925s 0.007918s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:13 [[::1]:59918] PUT 5e1a431e9dfe4470e969095d8c79fa37 9 0.028305s 0.028289s 0.000016s 200 86 "OK" 
[keep1] 2016/08/23 22:48:13 [[::1]:39989] PUT 5e1a431e9dfe4470e969095d8c79fa37 9 0.033839s 0.033828s 0.000011s 200 86 "OK" 
[keep1] 2016/08/23 22:48:13 [[::1]:39989] PUT ff7fef68670bc4d2aa216ae2fe29cc7d 6 0.076493s 0.076481s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:13 [[::1]:59918] PUT ff7fef68670bc4d2aa216ae2fe29cc7d 6 0.072024s 0.072010s 0.000014s 200 86 "OK" 
[keep1] 2016/08/23 22:48:13 [[::1]:40001] PUT 3b889b56b99273e33aa7397c5d6f5e8a 9 0.002646s 0.002635s 0.000012s 200 86 "OK" 
[keep0] 2016/08/23 22:48:13 [[::1]:59931] PUT 3b889b56b99273e33aa7397c5d6f5e8a 9 0.011761s 0.011748s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:14 [[::1]:40004] PUT b77064f9e68b606834e4bccfeb645926 6 0.008003s 0.007988s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:14 [[::1]:59933] PUT b77064f9e68b606834e4bccfeb645926 6 0.015654s 0.015640s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:14 [[::1]:59938] PUT 3a91839b46ec89abab6e063cb0b11950 9 0.011365s 0.011350s 0.000015s 200 86 "OK" 
[keep1] 2016/08/23 22:48:14 [[::1]:40009] PUT 3a91839b46ec89abab6e063cb0b11950 9 0.012238s 0.012229s 0.000009s 200 86 "OK" 
[keep1] 2016/08/23 22:48:14 [[::1]:40009] PUT e592be5eacd4a01e1f7f7a420f7693d2 6 0.000595s 0.000586s 0.000009s 200 86 "OK" 
[keep0] 2016/08/23 22:48:14 [[::1]:59938] PUT e592be5eacd4a01e1f7f7a420f7693d2 6 0.001345s 0.001338s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:15 [[::1]:59947] PUT 3a9a566b982a349db60fff39e2dfcd9d 9 0.006289s 0.006275s 0.000013s 200 86 "OK" 
[keep1] 2016/08/23 22:48:15 [[::1]:40020] PUT 3a9a566b982a349db60fff39e2dfcd9d 9 0.003170s 0.003159s 0.000011s 200 86 "OK" 
[keep1] 2016/08/23 22:48:15 [[::1]:40023] PUT 4718bb80a947650f768661962cdb8591 6 0.000527s 0.000519s 0.000008s 200 86 "OK" 
[keep0] 2016/08/23 22:48:15 [[::1]:59950] PUT 4718bb80a947650f768661962cdb8591 6 0.010735s 0.010723s 0.000012s 200 86 "OK" 
[keep1] 2016/08/23 22:48:15 [[::1]:40029] PUT 4bfebd5ffef6c5ace06039f38245a9c1 9 0.013030s 0.013016s 0.000015s 200 86 "OK" 
[keep0] 2016/08/23 22:48:15 [[::1]:59958] PUT 4bfebd5ffef6c5ace06039f38245a9c1 9 0.018535s 0.018522s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:15 [[::1]:59958] PUT 5320ad51278a512afb3f880cbbd0f70c 6 0.000524s 0.000514s 0.000010s 200 86 "OK" 
[keep1] 2016/08/23 22:48:15 [[::1]:40029] PUT 5320ad51278a512afb3f880cbbd0f70c 6 0.011390s 0.011377s 0.000013s 200 86 "OK" 
[keep0] 2016/08/23 22:48:16 [[::1]:59963] PUT 0ad3b66970ba9d3964230c9332224d94 9 0.000622s 0.000614s 0.000008s 200 86 "OK" 
[keep1] 2016/08/23 22:48:16 [[::1]:40036] PUT 0ad3b66970ba9d3964230c9332224d94 9 0.000471s 0.000464s 0.000006s 200 86 "OK" 
[keep1] 2016/08/23 22:48:16 [[::1]:40036] PUT 5fc84b0d9e55c001e39540afc77c465a 6 0.013588s 0.013577s 0.000010s 200 86 "OK" 
[keep0] 2016/08/23 22:48:16 [[::1]:59963] PUT 5fc84b0d9e55c001e39540afc77c465a 6 0.015459s 0.015449s 0.000010s 200 86 "OK" 
[keep1] 2016/08/23 22:48:17 [[::1]:40046] PUT e65f9aeaa3cb91e54a9c046dd7f92de4 9 0.000659s 0.000649s 0.000010s 200 86 "OK" 
[keep0] 2016/08/23 22:48:17 [[::1]:59973] PUT e65f9aeaa3cb91e54a9c046dd7f92de4 9 0.011325s 0.011307s 0.000018s 200 86 "OK" 
[keep1] 2016/08/23 22:48:17 [[::1]:40046] PUT 5b36a2183b880ca29e01bd23559e7080 6 0.000651s 0.000644s 0.000007s 200 86 "OK" 
[keep0] 2016/08/23 22:48:17 [[::1]:59973] PUT 5b36a2183b880ca29e01bd23559e7080 6 0.013226s 0.013212s 0.000015s 200 86 "OK" 
2016-08-23 22:48:19 arvados.arv-mount[8859] WARNING: Mount.__exit__: llfuse thread still alive 0.100000s after umount -- abandoning and exiting anyway
FAIL
Sent SIGTERM to 19722 (/data/1/jenkins/workspace/run-tests-services-fuse/tmp/keep0.pid)
[keep0] 2016/08/23 22:48:21 caught signal: terminated
[keep0] 2016/08/23 22:48:21 keepstore exiting, pid 19722
Sent SIGTERM to 19768 (/data/1/jenkins/workspace/run-tests-services-fuse/tmp/keep1.pid)
[keep1] 2016/08/23 22:48:21 caught signal: terminated
[keep1] 2016/08/23 22:48:21 keepstore exiting, pid 19768
Actions #1

Updated by Peter Amstutz over 7 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Clegg over 6 years ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF