Project

General

Profile

Actions

Feature #19847

closed

Choose disk cache size based on RAM request

Added by Peter Amstutz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Target version:
Story points:
0.5
Release relationship:
Auto

Description

Instead of applying a flat 8 GB disk cache size it should be proportional to the task size.

After giving it some thought, let's try

2 GiB <= the size of the RAM request <= 32 GiB


Subtasks 1 (0 open1 closed)

Task #19850: Review 19847-cwl-disk-cache-sizeResolvedPeter Amstutz12/07/2022Actions
Actions #1

Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
  • Subject changed from Choose disk cache size based on ram/cpu count to Choose disk cache size based on RAM request
Actions #3

Updated by Peter Amstutz over 1 year ago

19847-cwl-disk-cache-size @ 0531cc9f35e78a7be1eec7eb96fb6cc668ebcefa

  • Default disk cache heuristic computed by arv-mount is now 2 GiB <= the size of the RAM request <= 32 GiB
  • Added arvados-cwl-runner CWL extension to explicitly select between ram and disk cache
  • Fixed crunch-run to pass --ram-cache to explicitly select RAM cache when specified by runtime_constraints

developer-run-tests: #3402

Actions #4

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-12-07 Sprint to 2022-12-21 Sprint
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Story points set to 0.5
Actions #6

Updated by Tom Clegg over 1 year ago

This seems to mean the DefaultKeepCacheDisk config is essentially a boolean (all non-zero values just mean "enable disk cache, choose size automatically") when it comes to containers submitted by arvados-cwl-runner -- is that right? Would this be worth mentioning in the config comment?

Actions #7

Updated by Peter Amstutz over 1 year ago

Tom Clegg wrote in #note-6:

This seems to mean the DefaultKeepCacheDisk config is essentially a boolean (all non-zero values just mean "enable disk cache, choose size automatically") when it comes to containers submitted by arvados-cwl-runner -- is that right? Would this be worth mentioning in the config comment?

If I add that to the config comment would this be a LGTM?

Actions #8

Updated by Tom Clegg over 1 year ago

It does seem like a weird knob. I suppose that's OK if it's documented. But I wonder if it's worth trying to make it less weird.
  • rename config to MinimumKeepCacheDisk, instead of using hard-coded 2 GiB in a-c-r?
  • config sets the default disk cache size as % RAM size instead of # bytes?
  • auto-size logic moves to railsapi so it applies to top-level a-c-r containers too?
Actions #9

Updated by Tom Clegg over 1 year ago

would you be able to implement the behavior you want on the rails api side and then I'll update the a-c-r branch?

I did both sides, but kept the a-c-r updates in a separate commit.

19847-cwl-disk-cache-size @ 2c8bbcd431eb3aee1b4fd650b8268979b8da556c -- developer-run-tests: #3421

retry wb1 developer-run-tests-apps-workbench-integration: #3680

Actions #10

Updated by Peter Amstutz over 1 year ago

  • Release set to 47
Actions #11

Updated by Peter Amstutz over 1 year ago

Tom Clegg wrote in #note-9:

would you be able to implement the behavior you want on the rails api side and then I'll update the a-c-r branch?

I did both sides, but kept the a-c-r updates in a separate commit.

19847-cwl-disk-cache-size @ 2c8bbcd431eb3aee1b4fd650b8268979b8da556c -- developer-run-tests: #3421

retry wb1 developer-run-tests-apps-workbench-integration: #3680

Your implementation LGTM, if you're happy I'm happy.

Actions #12

Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF