Bug #18078
closed
RuntimeError: dictionary changed size during iteration
Description
2021-09-01T08:49:28.425746193Z 2021-09-01 04:49:28 arvados.arvados_fuse8318 ERROR: Unhandled exception during FUSE operation
2021-09-01T08:49:28.425746193Z Traceback (most recent call last):
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 325, in catch_exceptions_wrapper
2021-09-01T08:49:28.425746193Z return orig_func(self, *args, **kwargs)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 633, in open
2021-09-01T08:49:28.425746193Z self.inodes.touch(p)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 274, in touch
2021-09-01T08:49:28.425746193Z self.inode_cache.touch(entry)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 232, in touch
2021-09-01T08:49:28.425746193Z self.manage(obj)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 226, in manage
2021-09-01T08:49:28.425746193Z self.cap_cache()
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 210, in cap_cache
2021-09-01T08:49:28.425746193Z self._remove(ent, True)
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 176, in _remove
2021-09-01T08:49:28.425746193Z if obj.has_ref(True):
2021-09-01T08:49:28.425746193Z File "/usr/share/python2.7/dist/python-arvados-fuse/lib/python2.7/site-packages/arvados_fuse/fusedir.py", line 174, in has_ref
2021-09-01T08:49:28.425746193Z for v in self._entries.itervalues():
2021-09-01T08:49:28.425746193Z RuntimeError: dictionary changed size during iteration
2021-09-01T08:49:30.578235405Z crunchstat: keepcalls 0 put 598 get -- interval 10.0000 seconds 0 put 585 get
Updated by Peter Amstutz over 3 years ago
- Category set to FUSE
- Description updated (diff)
- Subject changed from FUSE error to RuntimeError: dictionary changed size during iteration
Updated by Peter Amstutz over 3 years ago
18078-reacquire-fuse-lock @ 025e6f91c13bdeb78585ce2c7a6d70c532eb8d7b
Reacquire llfuse lock before populating collection directory
Updates the test for generating "conflict" files.
I don't have a test case for the reported bug, however this is the
only place I could find where _entries was being modified without the
lock held. By reacquiring the lock, it should no longer be possible
to read _entries at the same time as it is being updated.
Also clean up a few "from future" bits that are no longer needed
because we're python 3 only.
Updated by Peter Amstutz over 3 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima over 3 years ago
The code changes LGTM, but the test changes are having some issues.
Updated by Peter Amstutz over 3 years ago
Lucas Di Pentima wrote:
The code changes LGTM, but the test changes are having some issues.
Huh, it seems to be failing when I run the whole test suite, but not when I run just one test on its own. I'll have investigate.
Updated by Peter Amstutz over 3 years ago
Customer had a really quick turnaround to deploy the branch for testing, and are reporting it is working much better. So we just need to resolve the testing issue.
Updated by Peter Amstutz over 3 years ago
18078-reacquire-fuse-lock @ 0f815282ed307b5cda9ab77a5343d3dc3c349bb1
Updated by Peter Amstutz over 3 years ago
- Status changed from In Progress to Resolved