Project

General

Profile

Actions

Feature #10231

closed

[Crunch2] Support setting keep_cache in runtime_constraints

Added by Peter Amstutz over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
Crunch
Target version:
Story points:
0.5
Release:
Release relationship:
Auto

Description

Support keep_cache_ram field in crunch v2 runtime_constraints.

Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.

Add API server config container_default_keep_cache_ram (default 268435456)

Update API server to copy provided [minimum] keep_cache_ram from container request to container, or set to Rails.configuration.container_default_keep_cache_ram if not provided in container request.

Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.


Subtasks 1 (0 open1 closed)

Task #10256: Review branch 10231-keep-cache-runtime-constraintsResolvedRadhika Chippada10/27/2016Actions
Actions #1

Updated by Tom Morris over 7 years ago

  • Assigned To set to Radhika Chippada
  • Target version set to 2016-10-26 sprint
  • Story points set to 0.5
Actions #2

Updated by Peter Amstutz over 7 years ago

  • Description updated (diff)
  • Assigned To deleted (Radhika Chippada)
  • Target version deleted (2016-10-26 sprint)
Actions #3

Updated by Peter Amstutz over 7 years ago

  • Category set to Crunch
  • Assigned To set to Radhika Chippada
  • Target version set to 2016-10-26 sprint
Actions #4

Updated by Tom Clegg over 7 years ago

  • Description updated (diff)
Actions #5

Updated by Radhika Chippada over 7 years ago

Peter,

Please detail how to address the following items:

  • Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.

In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?

resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20
  • Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.

Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?

    my $arv_file_cache = "";
    if (defined($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'})) {
      $arv_file_cache = "--file-cache=" . ($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'} * 1024 * 1024);
    }
Actions #6

Updated by Peter Amstutz over 7 years ago

Radhika Chippada wrote:

Peter,

Please detail how to address the following items:

  • Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.

In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?

resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20

No. It's part of the RuntimeConstraints hint.

Replace logger.warn with runtime_constraints["keep_cache_ram"] = @runtime_req["keep_cache"]

        runtime_req, _ = get_feature(self, "http://arvados.org/cwl#RuntimeConstraints")
        if runtime_req:
            logger.warn("RuntimeConstraints not yet supported by container API")
  • Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.

Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?

No, this story is to add the feature to crunch v2, not change anything in crunch v1.

Actions #7

Updated by Radhika Chippada over 7 years ago

Peter:

“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”

Can you please clarify which file and what needs to be updated for this? Thanks.

Actions #8

Updated by Peter Amstutz over 7 years ago

Radhika Chippada wrote:

Peter:

“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”

Can you please clarify which file and what needs to be updated for this? Thanks.

arvados/services/crunch-run/crunchrun.go

Actions #9

Updated by Radhika Chippada over 7 years ago

  • Status changed from New to In Progress
Actions #10

Updated by Radhika Chippada over 7 years ago

  • Target version changed from 2016-10-26 sprint to 2016-11-09 sprint
Actions #11

Updated by Peter Amstutz over 7 years ago

Reviewing 10231-keep-cache-runtime-constraints @ 61b286dcf40fa85319cceb69e16660aebc71b942

Need to add keep_cache_ram to documentation arvados/doc/_includes/_container_runtime_constraints.liquid

In crunchrun.go, can we add --file-cache earlier in the command line. Having it appear after the non-optional mount point argument is a little bit confusing and possibly may not work. I suggest moving it to be above line 256 for bind, mnt := range runner.Container.Mounts {

In arvcontainer.py, the "keep_cache" parameter is optional, so you need to check if it is present before setting it . e.g. if "keep_cache" in runtime_req: (compare arvjob.py)

Actions #13

Updated by Radhika Chippada over 7 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:3cb54d529632df28dd14d1973f70d5fdbd98e2bb.

Actions

Also available in: Atom PDF