Feature #10231
closed[Crunch2] Support setting keep_cache in runtime_constraints
Description
Support keep_cache_ram
field in crunch v2 runtime_constraints.
Update arvados-cwl-runner
to set keep_cache_ram
in runtime_constraints
for ArvadosContainer.
Add API server config container_default_keep_cache_ram
(default 268435456)
Update API server to copy provided [minimum] keep_cache_ram from container request to container, or set to Rails.configuration.container_default_keep_cache_ram
if not provided in container request.
Update crunch-run
to get keep_cache_ram
from runtime_constraints
and pass it to arv-mount --file-cache
.
Updated by Tom Morris almost 8 years ago
- Assigned To set to Radhika Chippada
- Target version set to 2016-10-26 sprint
- Story points set to 0.5
Updated by Peter Amstutz almost 8 years ago
- Description updated (diff)
- Assigned To deleted (
Radhika Chippada) - Target version deleted (
2016-10-26 sprint)
Updated by Peter Amstutz almost 8 years ago
- Category set to Crunch
- Assigned To set to Radhika Chippada
- Target version set to 2016-10-26 sprint
Updated by Radhika Chippada almost 8 years ago
Peter,
Please detail how to address the following items:
- Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.
In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?
resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20
- Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.
Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?
my $arv_file_cache = ""; if (defined($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'})) { $arv_file_cache = "--file-cache=" . ($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'} * 1024 * 1024); }
Updated by Peter Amstutz almost 8 years ago
Radhika Chippada wrote:
Peter,
Please detail how to address the following items:
- Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.
In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?
resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20
No. It's part of the RuntimeConstraints
hint.
Replace logger.warn
with runtime_constraints["keep_cache_ram"] = @runtime_req["keep_cache"]
runtime_req, _ = get_feature(self, "http://arvados.org/cwl#RuntimeConstraints") if runtime_req: logger.warn("RuntimeConstraints not yet supported by container API")
- Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.
Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?
No, this story is to add the feature to crunch v2, not change anything in crunch v1.
Updated by Radhika Chippada almost 8 years ago
Peter:
“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”
Can you please clarify which file and what needs to be updated for this? Thanks.
Updated by Peter Amstutz almost 8 years ago
Radhika Chippada wrote:
Peter:
“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”
Can you please clarify which file and what needs to be updated for this? Thanks.
arvados/services/crunch-run/crunchrun.go
Updated by Radhika Chippada almost 8 years ago
- Status changed from New to In Progress
Updated by Radhika Chippada almost 8 years ago
- Target version changed from 2016-10-26 sprint to 2016-11-09 sprint
Updated by Peter Amstutz almost 8 years ago
Reviewing 10231-keep-cache-runtime-constraints @ 61b286dcf40fa85319cceb69e16660aebc71b942
Need to add keep_cache_ram
to documentation arvados/doc/_includes/_container_runtime_constraints.liquid
In crunchrun.go, can we add --file-cache earlier in the command line. Having it appear after the non-optional mount point argument is a little bit confusing and possibly may not work. I suggest moving it to be above line 256 for bind, mnt := range runner.Container.Mounts {
In arvcontainer.py, the "keep_cache" parameter is optional, so you need to check if it is present before setting it . e.g. if "keep_cache" in runtime_req:
(compare arvjob.py)
Updated by Peter Amstutz almost 8 years ago
Updated by Radhika Chippada almost 8 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:3cb54d529632df28dd14d1973f70d5fdbd98e2bb.