Project

General

Profile

Actions
Copy link

Bug #8874

closed

[FUSE] by_id cached subdirectories silently stop having contents

Added by Joshua Randall almost 7 years ago. Updated almost 7 years ago.

Status:
Duplicate
Priority:
Normal
Assigned To:
-
Category:
FUSE
Target version:
-
Start date:
04/04/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

A keep mount on humgen-01-01 started failing today: 

root@humgen-01-01:/# ls /keep
by_id  by_tag  home  README  shared
root@humgen-01-01:/# ls /keep/by_id
07dd95202bc58bb32b4f92b6cc623220+4548821  200f1e42494725e2a8354e1e40bcd79a+958      2a3b26379cb547fac7e13a140516e845+4551649  c5941f347c9e7f4c70be88dab1bd2897+280      README                       z8ta6-4zz18-2uaolpmc02bft0m  z8ta6-4zz18-6vzh013prjzhjug
1596b7f287a34d1ec9e19db7b2887156+4547039  24f519adbe3fdc9d4ea6e4ff6f6fa304+4142184  73661cc19da872179b979ca46ece8186+45498    f07c98fa47044331af1a0ac5f29d55b8+4549464  z8ta6-4zz18-2qcqdawjhc2788o  z8ta6-4zz18-3i7q35yophj46pm  z8ta6-4zz18-909ga3vvw0p7hut
root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2qcqdawjhc2788o
root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894
hs37d5.dict  hs37d5.fa.amb  hs37d5.fa.bwt   hs37d5.fa.fai       hs37d5.fa.gc_stats.old                                        hs37d5.fa.index  hs37d5.fa.rbwt  hs37d5.fa.rsa  hs37d5.vcfdict
hs37d5.fa    hs37d5.fa.ann  hs37d5.fa.dict  hs37d5.fa.gc_stats  hs37d5.fa.gc_stats.targeted-98e7918adbfd0e759220c2b9b3e99803  hs37d5.fa.pac    hs37d5.fa.rpac  hs37d5.fa.sa
root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894
hs37d5.dict  hs37d5.fa.amb  hs37d5.fa.bwt   hs37d5.fa.fai       hs37d5.fa.gc_stats.old                                        hs37d5.fa.index  hs37d5.fa.rbwt  hs37d5.fa.rsa  hs37d5.vcfdict
hs37d5.fa    hs37d5.fa.ann  hs37d5.fa.dict  hs37d5.fa.gc_stats  hs37d5.fa.gc_stats.targeted-98e7918adbfd0e759220c2b9b3e99803  hs37d5.fa.pac    hs37d5.fa.rpac  hs37d5.fa.sa
root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464
root@humgen-01-01:/# for subdir in /keep/by_id/*; do ls ${subdir}; done
/keep/by_id/README
root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894
root@humgen-01-01:/#

It looks like an attempt to `ls` one of the existing cached subdirectories of `by_id` causes the others to lose their contents, although it doesn't seem to happen every time:

root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/07dd95202bc58bb32b4f92b6cc623220+4548821
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/1596b7f287a34d1ec9e19db7b2887156+4547039
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/200f1e42494725e2a8354e1e40bcd79a+958
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/24f519adbe3fdc9d4ea6e4ff6f6fa304+4142184
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/2a3b26379cb547fac7e13a140516e845+4551649
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/73661cc19da872179b979ca46ece8186+45498
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/c5941f347c9e7f4c70be88dab1bd2897+280
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2qcqdawjhc2788o
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2uaolpmc02bft0m
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+619
ls: cannot access /keep/by_id/a83bd4e5a26a64612322f21515d93bab+619: No such file or directory
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-3i7q35yophj46pm
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-6vzh013prjzhjug
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-909ga3vvw0p7hut
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
hs38DH.dict  hs38DH.fa.alt  hs38DH.fa.ann  hs38DH.fa.dict  hs38DH.fa.gc_stats  hs38DH.fa.log.e  hs38DH.fa.nhr  hs38DH.fa.nsq  hs38DH.fa.phr  hs38DH.fa.psq
hs38DH.fa    hs38DH.fa.amb  hs38DH.fa.bwt  hs38DH.fa.fai   hs38DH.fa.log       hs38DH.fa.log.o  hs38DH.fa.nin  hs38DH.fa.pac  hs38DH.fa.pin  hs38DH.fa.sa
root@humgen-01-01:/# for subdir in /keep/by_id/*; do ls ${subdir}; done
/keep/by_id/README
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190
root@humgen-01-01:/#

This is "bad" because it results in collections appearing to have no contents rather than giving an error.

Related issues

Is duplicate of Arvados - Bug #8712: [FUSE] Read-only collections appear empty after they are evicted from the cacheResolvedPeter Amstutz03/15/2016

Actions
Actions
Copy link
 #1

Updated by Joshua Randall almost 7 years ago

Not just an `ls` problem - arv-mount asserts that the files do not exist:

root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190/hs38DH.fa
ls: cannot access /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190/hs38DH.fa: No such file or directory

Actions
Copy link
 #2

Updated by Joshua Randall almost 7 years ago

Additional notes:
- some (~5) of the by_id subdirectories would have had ~40000 files each.
- earlier today (during the life of this arv-mount process) one of our twelve keep servers crashed and has since been rebooted

Actions
Copy link
 #3

Updated by Ward Vandewege almost 7 years ago

Some more detail:

Installed versions:

ii  python-arvados-fuse              0.1.20160210120515-1                 The Keep FUSE driver
ii  python-arvados-python-client     0.1.20160204181731-1                 The Arvados Python SDK
Actions
Copy link
 #4

Updated by Brett Smith almost 7 years ago

  • Subject changed from arv-mount by_id cached subdirectories silently stop having contents to [FUSE] by_id cached subdirectories silently stop having contents
Actions
Copy link
 #5

Updated by Brett Smith almost 7 years ago

  • Target version set to Arvados Future Sprints
Actions
Copy link
 #6

Updated by Brett Smith almost 7 years ago

  • Status changed from New to Duplicate
  • Target version deleted (Arvados Future Sprints)

Given the recent diagnosis of #8712, we feel confident this is a duplicate of that.

Actions
Copy link
 #7

Updated by Brett Smith almost 7 years ago

  • Target version set to Arvados Future Sprints
Actions
Copy link
 #8

Updated by Brett Smith almost 7 years ago

  • Target version deleted (Arvados Future Sprints)
Actions
Copy link

Also available in: Atom PDF