Idea #6309
closed
[Crunch] Support a runtime_constraint that specifies how much RAM arv-mount should use for cache
Added by Brett Smith over 9 years ago.
Updated about 9 years ago.
Description
For jobs that access many files simultaneously, FUSE's default block cache is usually not sufficient, and will thrash regularly. This can lead to blocks downloaded multiple times, and degraded performance.
If there's evidence that it will substantially benefit production pipelines, add a runtime_constraint to jobs so they can specify an argument for the --file-cache option of their FUSE mount.
Implementation¶
- When the job specifies a
keep_cache_mb_per_task runtime constraint, crunch-job calls arv-mount with that value in the --file-cache switch (converting units as needed).
- Document this new runtime constraint in the Jobs schema API reference.
- Subject changed from [Maybe] [Crunch] Support a runtime_constraint that specifies how much RAM arv-mount should use for cache to [Maybe] [Crunch] Support a runtime_constraint that specifies how much RAM arv-mount should use for cache
- Description updated (diff)
- Category set to Crunch
- Description updated (diff)
- Target version changed from Arvados Future Sprints to Deferred
Bryan's benchmarks indicate that this change doesn't help performance, at least for the cases we're worried about. Deferring this until there's real demand or we have CWL instead, whichever comes first.
- Target version deleted (
Deferred)
- Target version set to 2015-12-02 sprint
- Subject changed from [Maybe] [Crunch] Support a runtime_constraint that specifies how much RAM arv-mount should use for cache to [Crunch] Support a runtime_constraint that specifies how much RAM arv-mount should use for cache
- Description updated (diff)
- Assigned To set to Peter Amstutz
6309-file-cache-runtime-constraint ready for review
- Status changed from New to In Progress
Reviewing 292b28f.
- Could $arv_file_cache just be the empty string when the constraint is not specified? That way, we'll just get arv-mount's default, without duplicating it.
- The documentation update duplicates the row about min_ram_mb_task. Please clean that up.
Thanks.
Brett Smith wrote:
Reviewing 292b28f.
- Could $arv_file_cache just be the empty string when the constraint is not specified? That way, we'll just get arv-mount's default, without duplicating it.
Done.
- The documentation update duplicates the row about min_ram_mb_task. Please clean that up.
Fixed.
Now at dcab751
Peter Amstutz wrote:
Now at dcab751
This is good to merge. Thanks.
- Status changed from In Progress to Resolved
- % Done changed from 67 to 100
Applied in changeset arvados|commit:8788c145b860e19a1f04c4dc6abdcda14c859403.
Also available in: Atom
PDF
Updated by 
Updated by