Bug #18078

RuntimeError: dictionary changed size during iteration

Added by Peter Amstutz about 2 months ago. Updated 4 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
FUSE
Target version:
Start date:
09/02/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

2021-09-01T08:49:28.425746193Z 2021-09-01 04:49:28 arvados.arvados_fuse8318 ERROR: Unhandled exception during FUSE operation
2021-09-01T08:49:28.425746193Z Traceback (most recent call last):
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 325, in catch_exceptions_wrapper
2021-09-01T08:49:28.425746193Z return orig_func(self, *args, **kwargs)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 633, in open
2021-09-01T08:49:28.425746193Z self.inodes.touch(p)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 274, in touch
2021-09-01T08:49:28.425746193Z self.inode_cache.touch(entry)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 232, in touch
2021-09-01T08:49:28.425746193Z self.manage(obj)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 226, in manage
2021-09-01T08:49:28.425746193Z self.cap_cache()
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 210, in cap_cache
2021-09-01T08:49:28.425746193Z self._remove(ent, True)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 176, in _remove
2021-09-01T08:49:28.425746193Z if obj.has_ref(True):
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/fusedir.py", line 174, in has_ref
2021-09-01T08:49:28.425746193Z for v in self._entries.itervalues():
2021-09-01T08:49:28.425746193Z RuntimeError: dictionary changed size during iteration
2021-09-01T08:49:30.578235405Z crunchstat: keepcalls 0 put 598 get -- interval 10.0000 seconds 0 put 585 get


Subtasks

Task #18084: Review 18078-reacquire-fuse-lockResolvedPeter Amstutz

History

#1 Updated by Peter Amstutz about 2 months ago

  • Category set to FUSE
  • Description updated (diff)
  • Subject changed from FUSE error to RuntimeError: dictionary changed size during iteration

#2 Updated by Peter Amstutz about 2 months ago

  • Assigned To set to Peter Amstutz

#3 Updated by Peter Amstutz about 2 months ago

18078-reacquire-fuse-lock @ 025e6f91c13bdeb78585ce2c7a6d70c532eb8d7b

Reacquire llfuse lock before populating collection directory

Updates the test for generating "conflict" files.

I don't have a test case for the reported bug, however this is the
only place I could find where _entries was being modified without the
lock held. By reacquiring the lock, it should no longer be possible
to read _entries at the same time as it is being updated.

Also clean up a few "from future" bits that are no longer needed
because we're python 3 only.

https://ci.arvados.org/view/Developer/job/developer-run-tests/2669/

#4 Updated by Peter Amstutz about 2 months ago

  • Status changed from New to In Progress

#5 Updated by Lucas Di Pentima about 2 months ago

The code changes LGTM, but the test changes are having some issues.

#6 Updated by Peter Amstutz about 2 months ago

Lucas Di Pentima wrote:

The code changes LGTM, but the test changes are having some issues.

Huh, it seems to be failing when I run the whole test suite, but not when I run just one test on its own. I'll have investigate.

#7 Updated by Peter Amstutz about 2 months ago

Customer had a really quick turnaround to deploy the branch for testing, and are reporting it is working much better. So we just need to resolve the testing issue.

#9 Updated by Lucas Di Pentima about 2 months ago

LGTM, thanks!

#10 Updated by Peter Amstutz about 2 months ago

  • Status changed from In Progress to Resolved

#11 Updated by Peter Amstutz about 1 month ago

  • Release set to 43

#12 Updated by Lucas Di Pentima 4 days ago

  • Release changed from 43 to 42

Also available in: Atom PDF