Bug #8712

[FUSE] Read-only collections appear empty after they are evicted from the cache

Added by Abram Connelly over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
FUSE
Target version:
Start date:
03/15/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

On one of my shell nodes (lightning-dev1.su92l) I cannot see files that should be there through my keep mount in a collection. They show up in workbench and logging into another shell node, the files are visible.

On lightning-dev1:

abram@lightning-dev1:~$ hostname
lightning-dev1.shell.su92l.arvadosapi.com
abram@lightning-dev1:~$ cd keep/by_id/a2198905927b8d24e162abe42e541f37+173
abram@lightning-dev1:~/keep/by_id/a2198905927b8d24e162abe42e541f37+173$ ls
abram@lightning-dev1:~/keep/by_id/a2198905927b8d24e162abe42e541f37+173$ ls -la
total 1
dr-xr-xr-x 1 abram abram 0 Jan  1  1970 .
dr-xr-xr-x 1 abram abram 0 Feb 22 20:40 ..
abram@lightning-dev1:~/keep/by_id/a2198905927b8d24e162abe42e541f37+173$

On abe.shell.su92l:

abram@abe.su92l:~$ hostname
abe.shell.su92l.arvadosapi.com
abram@abe.su92l:~$ cd keep/by_id/a2198905927b8d24e162abe42e541f37+173
abram@abe.su92l:~/keep/by_id/a2198905927b8d24e162abe42e541f37+173$ ls -la
total 57016
dr-xr-xr-x 1 abram abram        0 Jan  1  1970 .
dr-xr-xr-x 1 abram abram        0 Feb 22 22:50 ..
-r-xr-xr-x 1 abram abram    32302 Jan  1  1970 assembly.00.hg19.fw.fwi
-r-xr-xr-x 1 abram abram 58305282 Jan  1  1970 assembly.00.hg19.fw.gz
-r-xr-xr-x 1 abram abram    41784 Jan  1  1970 assembly.00.hg19.fw.gz.gzi
-r-xr-xr-x 1 abram abram     2712 Jan  1  1970 README
abram@abe.su92l:~/keep/by_id/a2198905927b8d24e162abe42e541f37+173$

Subtasks

Task #8744: InvestigateResolvedPeter Amstutz

Task #8745: Review 8712-fuse-cache-reload-bugResolvedNico César


Related issues

Has duplicate Arvados - Bug #8874: [FUSE] by_id cached subdirectories silently stop having contentsDuplicate04/04/2016

Associated revisions

Revision b8ae9ce6
Added by Nico César over 5 years ago

Merge branch '8712-fuse-cache-reload-bug'

closes #8712

History

#1 Updated by Brett Smith over 5 years ago

  • Category set to FUSE
  • Assigned To set to Peter Amstutz
  • Target version set to 2016-03-30 sprint

#2 Updated by Brett Smith over 5 years ago

  • Target version changed from 2016-03-30 sprint to 2016-04-13 sprint

#3 Updated by Brett Smith over 5 years ago

  • Subject changed from Disappearing files in keep mount to [FUSE?] Disappearing files in keep mount

#4 Updated by Peter Amstutz over 5 years ago

The bug is when a read-only collection has been spilled from the cache and then needs to be reloaded:

2016-04-04 10:25:02 arvados.arvados_fuse[10263] DEBUG: Updating None
2016-04-04 10:25:02 arvados.arvados_fuse[10263] ERROR: arv-mount b1eb3a7572b5fbbf04335bbf0ade7209+85: error
Traceback (most recent call last):
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_fuse-0.1.20160322160927-py2.7.egg/arvados_fuse/fusedir.py", line 410, in update
    self.collection.update()
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20160322001610-py2.7.egg/arvados/arvfile.py", line 238, in synchronized_wrapper
    return orig_func(self, *args, **kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20160322001610-py2.7.egg/arvados/retry.py", line 153, in num_retries_setter
    return orig_func(self, *args, **kwargs)
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20160322001610-py2.7.egg/arvados/collection.py", line 1228, in update
    self.apply(baseline.diff(other))
  File "/home/peter/work/scripts/venv/local/lib/python2.7/site-packages/arvados_python_client-0.1.20160322001610-py2.7.egg/arvados/arvfile.py", line 389, in must_be_writable_wrapper
    raise IOError(errno.EROFS, "Collection is read-only.")
IOError: [Errno 30] Collection is read-only.

#5 Updated by Nico César over 5 years ago

reviewing 55778ad90a9b869f514e729b3f85c12383f4e2a7

I'm confused about if clear() should return a boolean or not...

because:

(class Directory) clear() line 166 does a

   if not oldentries[n].clear(force)

and _entries dict could contain a CollectionDirectory correct?

if that's the case code should be something like:

    def clear(self, force=False):
        rtn = super(CollectionDirectoryBase, self).clear(force)
        self.collection = None
        return rtn

and tests should include more complex examples of a full Directory._entries

#6 Updated by Brett Smith over 5 years ago

  • Subject changed from [FUSE?] Disappearing files in keep mount to [FUSE] Read-only collections appear empty after they are evicted from the cache

#7 Updated by Peter Amstutz over 5 years ago

  • Status changed from New to In Progress
  • Story points set to 0.5

#8 Updated by Nico César over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:b8ae9ce66c24055f3bf66b0956e15c44c9dd51f0.

Also available in: Atom PDF