Arvados

No big mystery here - the API call to list the links does not specify a limit and does not check items_available, but seems to just assume the API server returned everything: https://github.com/curoverse/arvados/blob/master/services/fuse/arvados_fuse/fusedir.py#L671-L679

I've confirmed that adding `limit=1000` to the API list query (fusedir.py:673) results in a `by_tag` directory with 1000 entries. So, one aspect to the solution to this would be to implement a loop that retrieves all available tags (although in my opinion it would be better to do that within the python SDK a la #8502 as it is common pattern that is needed often).

673c673
<                 select=['name'], distinct=True
---
>                 select=['name'], distinct=True, limit=1000

I also checked differences in memory usage with a limit of 100 vs 1000 (because it looks like TagsDirectory creates a TagDirectory object for each tag and I wasn't sure how expensive those are).

# arv-mount /tmp/keep_root
# cat /proc/$(ps auxwww |grep keep_root | grep -v grep | awk '{print $2}')/status | grep '^VmRSS'
VmRSS:     31604 kB
# ls /tmp/keep_root/by_tag | wc -l
100
# cat /proc/$(ps auxwww |grep keep_root | grep -v grep | awk '{print $2}')/status | grep '^VmRSS'
VmRSS:     35156 kB
# umount /tmp/keep_root
# # change to limit=1000
# arv-mount /tmp/keep_root
# cat /proc/$(ps auxwww |grep keep_root | grep -v grep | awk '{print $2}')/status | grep '^VmRSS'
VmRSS:     31592 kB
# ls /tmp/keep_root/by_tag | wc -l
1000
# cat /proc/$(ps auxwww |grep keep_root | grep -v grep | awk '{print $2}')/status | grep '^VmRSS'
VmRSS:     36984 kB

100 tag limit
35156 - 31604 kB = 3552 kB

1000 tag limit
36984 - 31592 kB = 5392 kB

Expected memory cost for each additional tag: (5392 - 3552 kB) / (1000-100) tags = 2.04 kB/tag

Given this, it seems reasonable to retain the current behavior of pre-populating TagDirectory objects unless there are more than in the vicinity of 100000-10000000 tags (at the 1m tag mark, the expected memory usage for arv-mount would be ~2GB, which seems excessive for a client that isn't likely to actually need 99.999% of those objects). I'm not understandingwhy the TagDirectory object is not simply created if and when a request is made for it. I think the only useful piece of state that it has is `self.tag` which is the name of the tag to use in the API query to get the associated collections. Since that is equal to the `name` that would be passed to the `lookup` function prior to access, I do not see any point in prepopulating those objects - it would be just as fast to have a general TagDirectory object that (like MagicDirectory) waits for the `lookup` call to do the actual API server query and prepare a list of associated CollectionDirectory objects (which also don't seem like they need to be prepopulated).

I would also note that it appears that the list of collections that a tag is associated with is also subject to the same issue regarding only doing a single API list query and thus it would also only show the first 100 collections associated with a particular tag.

IMHO clients shouldn't really be using polling to check if there is new data - ideally some mechanism (such as etag) would allow a client to ask the API server if the resource it is accessing has changed without having to actually perform a query at all.

Hey Josh, what do you think about making the tags directory "magic" similarly to the by_id directory, so the directory is only instantiated when you attempt to access it? This would remove the ability to use "ls" to list tags, but it sounds like that wouldn't be a problem for your use case. Or maybe we should fix the "tags" directory to have a full listing, but add an alternate "magic_tags" directory?

New behavior for by_tag/ dir. Allow 'cd' into any tag directory that exists on the server, even if it isn't returned in the listing. Also increase listing limit.

I am having difficulty running fuse tests in my development env. So, assuming tests are passing to you, LGTM