Arvados

Peter,

Please detail how to address the following items:

• Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.

In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?

resources = self.builder.resources
        if resources is not None:
            runtime_constraints["vcpus"] = resources.get("cores", 1)
            runtime_constraints["ram"] = resources.get("ram") * 2**20

• Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.

Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?

    my $arv_file_cache = "";
    if (defined($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'})) {
      $arv_file_cache = "--file-cache=" . ($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'} * 1024 * 1024);
    }

Radhika Chippada wrote:

Peter,

Please detail how to address the following items:

• Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.

In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?

resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20

No. It's part of the RuntimeConstraints hint.

Replace logger.warn with runtime_constraints["keep_cache_ram"] = @runtime_req["keep_cache"]

        runtime_req, _ = get_feature(self, "http://arvados.org/cwl#RuntimeConstraints")
        if runtime_req:
            logger.warn("RuntimeConstraints not yet supported by container API")

• Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.

Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?

No, this story is to add the feature to crunch v2, not change anything in crunch v1.

Peter:

“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”

Can you please clarify which file and what needs to be updated for this? Thanks.

Radhika Chippada wrote:

Peter:

“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”

Can you please clarify which file and what needs to be updated for this? Thanks.

arvados/services/crunch-run/crunchrun.go

Reviewing 10231-keep-cache-runtime-constraints @ 61b286dcf40fa85319cceb69e16660aebc71b942

Need to add keep_cache_ram to documentation arvados/doc/_includes/_container_runtime_constraints.liquid

In crunchrun.go, can we add --file-cache earlier in the command line. Having it appear after the non-optional mount point argument is a little bit confusing and possibly may not work. I suggest moving it to be above line 256 for bind, mnt := range runner.Container.Mounts {

In arvcontainer.py, the "keep_cache" parameter is optional, so you need to check if it is present before setting it . e.g. if "keep_cache" in runtime_req: (compare arvjob.py)

cc3857d9fdcd7588af329a9b2452e0ef09eb68a4 LGTM