Bug #8874
closed[FUSE] by_id cached subdirectories silently stop having contents
0%
Description
A keep mount on humgen-01-01 started failing today:
root@humgen-01-01:/# ls /keep by_id by_tag home README shared root@humgen-01-01:/# ls /keep/by_id 07dd95202bc58bb32b4f92b6cc623220+4548821 200f1e42494725e2a8354e1e40bcd79a+958 2a3b26379cb547fac7e13a140516e845+4551649 c5941f347c9e7f4c70be88dab1bd2897+280 README z8ta6-4zz18-2uaolpmc02bft0m z8ta6-4zz18-6vzh013prjzhjug 1596b7f287a34d1ec9e19db7b2887156+4547039 24f519adbe3fdc9d4ea6e4ff6f6fa304+4142184 73661cc19da872179b979ca46ece8186+45498 f07c98fa47044331af1a0ac5f29d55b8+4549464 z8ta6-4zz18-2qcqdawjhc2788o z8ta6-4zz18-3i7q35yophj46pm z8ta6-4zz18-909ga3vvw0p7hut root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464 root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2qcqdawjhc2788o root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894 hs37d5.dict hs37d5.fa.amb hs37d5.fa.bwt hs37d5.fa.fai hs37d5.fa.gc_stats.old hs37d5.fa.index hs37d5.fa.rbwt hs37d5.fa.rsa hs37d5.vcfdict hs37d5.fa hs37d5.fa.ann hs37d5.fa.dict hs37d5.fa.gc_stats hs37d5.fa.gc_stats.targeted-98e7918adbfd0e759220c2b9b3e99803 hs37d5.fa.pac hs37d5.fa.rpac hs37d5.fa.sa root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894 hs37d5.dict hs37d5.fa.amb hs37d5.fa.bwt hs37d5.fa.fai hs37d5.fa.gc_stats.old hs37d5.fa.index hs37d5.fa.rbwt hs37d5.fa.rsa hs37d5.vcfdict hs37d5.fa hs37d5.fa.ann hs37d5.fa.dict hs37d5.fa.gc_stats hs37d5.fa.gc_stats.targeted-98e7918adbfd0e759220c2b9b3e99803 hs37d5.fa.pac hs37d5.fa.rpac hs37d5.fa.sa root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464 root@humgen-01-01:/# for subdir in /keep/by_id/*; do ls ${subdir}; done /keep/by_id/README root@humgen-01-01:/# ls /keep/by_id/1b447d1aa006cde57df03042736750b9+7894 root@humgen-01-01:/#
It looks like an attempt to `ls` one of the existing cached subdirectories of `by_id` causes the others to lose their contents, although it doesn't seem to happen every time:
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/07dd95202bc58bb32b4f92b6cc623220+4548821 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/1596b7f287a34d1ec9e19db7b2887156+4547039 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/200f1e42494725e2a8354e1e40bcd79a+958 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/24f519adbe3fdc9d4ea6e4ff6f6fa304+4142184 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/2a3b26379cb547fac7e13a140516e845+4551649 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/73661cc19da872179b979ca46ece8186+45498 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/c5941f347c9e7f4c70be88dab1bd2897+280 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/f07c98fa47044331af1a0ac5f29d55b8+4549464 root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2qcqdawjhc2788o root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-2uaolpmc02bft0m root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+619 ls: cannot access /keep/by_id/a83bd4e5a26a64612322f21515d93bab+619: No such file or directory root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-3i7q35yophj46pm root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-6vzh013prjzhjug root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# ls /keep/by_id/z8ta6-4zz18-909ga3vvw0p7hut root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 hs38DH.dict hs38DH.fa.alt hs38DH.fa.ann hs38DH.fa.dict hs38DH.fa.gc_stats hs38DH.fa.log.e hs38DH.fa.nhr hs38DH.fa.nsq hs38DH.fa.phr hs38DH.fa.psq hs38DH.fa hs38DH.fa.amb hs38DH.fa.bwt hs38DH.fa.fai hs38DH.fa.log hs38DH.fa.log.o hs38DH.fa.nin hs38DH.fa.pac hs38DH.fa.pin hs38DH.fa.sa root@humgen-01-01:/# for subdir in /keep/by_id/*; do ls ${subdir}; done /keep/by_id/README root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190 root@humgen-01-01:/#
This is "bad" because it results in collections appearing to have no contents rather than giving an error.
Related issues
Updated by Joshua Randall almost 7 years ago
Not just an `ls` problem - arv-mount asserts that the files do not exist:
root@humgen-01-01:/# ls /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190/hs38DH.fa ls: cannot access /keep/by_id/a83bd4e5a26a64612322f21515d93bab+6190/hs38DH.fa: No such file or directory
Updated by Joshua Randall almost 7 years ago
Additional notes:
- some (~5) of the by_id subdirectories would have had ~40000 files each.
- earlier today (during the life of this arv-mount process) one of our twelve keep servers crashed and has since been rebooted
Updated by Ward Vandewege almost 7 years ago
Some more detail:
Installed versions:
ii python-arvados-fuse 0.1.20160210120515-1 The Keep FUSE driver ii python-arvados-python-client 0.1.20160204181731-1 The Arvados Python SDK
Updated by Brett Smith almost 7 years ago
- Subject changed from arv-mount by_id cached subdirectories silently stop having contents to [FUSE] by_id cached subdirectories silently stop having contents
Updated by Brett Smith almost 7 years ago
- Target version set to Arvados Future Sprints
Updated by Brett Smith almost 7 years ago
- Status changed from New to Duplicate
- Target version deleted (
Arvados Future Sprints)
Given the recent diagnosis of #8712, we feel confident this is a duplicate of that.
Updated by Brett Smith almost 7 years ago
- Target version set to Arvados Future Sprints
Updated by Brett Smith almost 7 years ago
- Target version deleted (
Arvados Future Sprints)