Project

General

Profile

Actions

Feature #8558

closed

[CWL] arvados-cwl-runner propagates ResourceRequirement

Added by Peter Amstutz almost 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
03/08/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

Propagate draft-3 ResourceRequirement (cores/ram) to crunch runtime_constraints (min_cores_per_node, min_ram_mb_per_node) in arvados-cwl-runner.

References:

https://common-workflow-language.github.io/draft-3/CommandLineTool.html#ResourceRequirement

http://doc.arvados.org/api/schema/Job.html

In arvados/sdk/cwl/arvados_cwl/__init__.py ArvadosJob.run() you can acces self.builder.resources which is a dict containing keys for "cores", "ram", "tmpdirSize" and "outdirSize". These need to be translated to the appropriate arvados keys in runtime_constraints.


Subtasks 1 (0 open1 closed)

Task #8611: Review branch: 8558-cwl-propagate-resource-reqResolvedRadhika Chippada03/08/2016

Actions
Actions #1

Updated by Brett Smith almost 7 years ago

  • Target version set to 2016-03-16 sprint
Actions #2

Updated by Peter Amstutz almost 7 years ago

  • Subject changed from [CWL] arvados-cwl-runner propagates ResourceRequirements to [CWL] arvados-cwl-runner propagates ResourceRequirement
  • Description updated (diff)
Actions #3

Updated by Brett Smith almost 7 years ago

  • Story points set to 0.5
Actions #4

Updated by Peter Amstutz almost 7 years ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz almost 7 years ago

  • Description updated (diff)
Actions #6

Updated by Brett Smith almost 7 years ago

  • Assigned To set to Radhika Chippada
Actions #7

Updated by Radhika Chippada almost 7 years ago

  • Added coresMin and ramMin from builder resources into runtime_constraints. Per the documentation, these can be long or string typed and hence I converted them into integer before adding to runtime_constraints (our documentation says we use integers).
  • In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.
Actions #8

Updated by Radhika Chippada almost 7 years ago

  • Status changed from New to In Progress
Actions #9

Updated by Brett Smith almost 7 years ago

Radhika Chippada wrote:

  • In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.

The constraint min_scratch_mb_per_node is pretty closely analogous to tmpdirSize. Both specify how much working disk space the job should be able to write to while it runs.

We don't have anything like outdirSize, so that shouldn't be handled, you're right.

Actions #10

Updated by Peter Amstutz almost 7 years ago

The resources dict contains keys "cores", "ram", "tmpdirSize" and "outdirSize". This is generated by evalResources: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L305

min_scratch_mb_per_node should be the sum of tmpdirSize and outdirSize because currently we use the same partition for both scratch space and output staging.

Python style note, if "cores" in resources is a more idiomatic way of writing if "cores" in resources.keys() and probably a bit more efficient (resources.keys() returns a list).

Actions #11

Updated by Radhika Chippada almost 7 years ago

  • Arvados branch 8558-cwl-propagate-resource-req:
    • Updated to use the correct names (tmpdirSize instead of tmpdirMin etc) and other suggested updates.
    • Updated a couple typos in the tests. Also, updated the tests to expect fields from default resources when one or more are omitted.
  • Added branch 8558-add-cwl-jenkins in arvados-jenkins
Actions #12

Updated by Peter Amstutz over 6 years ago

            if "cores" in resources:
                try:
                    runtime_constraints["min_cores_per_node"] = int(resources["cores"])
                except:
                    runtime_constraints["min_cores_per_node"] = None

This should be written more compactly (the "get" method returns None if the key is unavailable):

  runtime_constraints["min_cores_per_node"] = resources.get("cores")

Similarly (the optional 2nd parameter to the "get" method is the value to return if the key is unavailable):

  runtime_constraints["min_scratch_mb_per_node"] = resources.get("tmpdirSize", 0) + resources.get("outdirSize", 0)
Actions #13

Updated by Peter Amstutz over 6 years ago

Actually, it's a bad idea to set runtime_constraints["min_cores_per_node"] to None, it should probably have a default value as well, eg runtime_constraints["min_cores_per_node"] = resources.get("cores", 1)

(Although in practice the "cores" key should always be there so the default doesn't really matter.)

Actions #14

Updated by Radhika Chippada over 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:a2d5d6b24dde2de391831aa122cfda8e2cb759e0.

Actions

Also available in: Atom PDF