Feature #10231
closed
[Crunch2] Support setting keep_cache in runtime_constraints
Added by Peter Amstutz almost 8 years ago.
Updated almost 8 years ago.
Assigned To:
Radhika Chippada
Release relationship:
Auto
Description
Support keep_cache_ram
field in crunch v2 runtime_constraints.
Update arvados-cwl-runner
to set keep_cache_ram
in runtime_constraints
for ArvadosContainer.
Add API server config container_default_keep_cache_ram
(default 268435456)
Update API server to copy provided [minimum] keep_cache_ram from container request to container, or set to Rails.configuration.container_default_keep_cache_ram
if not provided in container request.
Update crunch-run
to get keep_cache_ram
from runtime_constraints
and pass it to arv-mount --file-cache
.
- Assigned To set to Radhika Chippada
- Target version set to 2016-10-26 sprint
- Story points set to 0.5
- Description updated (diff)
- Assigned To deleted (
Radhika Chippada)
- Target version deleted (
2016-10-26 sprint)
- Category set to Crunch
- Assigned To set to Radhika Chippada
- Target version set to 2016-10-26 sprint
- Description updated (diff)
Peter,
Please detail how to address the following items:
- Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.
In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?
resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20
- Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.
Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?
my $arv_file_cache = "";
if (defined($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'})) {
$arv_file_cache = "--file-cache=" . ($Job->{'runtime_constraints'}->{'keep_cache_mb_per_task'} * 1024 * 1024);
}
Radhika Chippada wrote:
Peter,
Please detail how to address the following items:
- Update arvados-cwl-runner to set keep_cache_ram in runtime_constraints for ArvadosContainer.
In arvcontainer.py, ram and vcpus are addressed as follows. Should "keep_cache_ram" be addressed the same way as "ram" here?
resources = self.builder.resources
if resources is not None:
runtime_constraints["vcpus"] = resources.get("cores", 1)
runtime_constraints["ram"] = resources.get("ram") * 2**20
No. It's part of the RuntimeConstraints
hint.
Replace logger.warn
with runtime_constraints["keep_cache_ram"] = @runtime_req["keep_cache"]
runtime_req, _ = get_feature(self, "http://arvados.org/cwl#RuntimeConstraints")
if runtime_req:
logger.warn("RuntimeConstraints not yet supported by container API")
- Update crunch-run to get keep_cache_ram from runtime_constraints and pass it to arv-mount --file-cache.
Does this need updating the following in sdk/cli/bin/crunch-job? Replace "keep_cache_mb_per_task" with "keep_cache_ram" ?
No, this story is to add the feature to crunch v2, not change anything in crunch v1.
Peter:
“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”
Can you please clarify which file and what needs to be updated for this? Thanks.
Radhika Chippada wrote:
Peter:
“No, this story is to add the feature to crunch v2, not change anything in crunch v1.”
Can you please clarify which file and what needs to be updated for this? Thanks.
arvados/services/crunch-run/crunchrun.go
- Status changed from New to In Progress
- Target version changed from 2016-10-26 sprint to 2016-11-09 sprint
Reviewing 10231-keep-cache-runtime-constraints @ 61b286dcf40fa85319cceb69e16660aebc71b942
Need to add keep_cache_ram
to documentation arvados/doc/_includes/_container_runtime_constraints.liquid
In crunchrun.go, can we add --file-cache earlier in the command line. Having it appear after the non-optional mount point argument is a little bit confusing and possibly may not work. I suggest moving it to be above line 256 for bind, mnt := range runner.Container.Mounts {
In arvcontainer.py, the "keep_cache" parameter is optional, so you need to check if it is present before setting it . e.g. if "keep_cache" in runtime_req:
(compare arvjob.py)
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:3cb54d529632df28dd14d1973f70d5fdbd98e2bb.
Also available in: Atom
PDF