Feature #8558

[CWL] arvados-cwl-runner propagates ResourceRequirement

Added by Peter Amstutz over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
03/08/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

Propagate draft-3 ResourceRequirement (cores/ram) to crunch runtime_constraints (min_cores_per_node, min_ram_mb_per_node) in arvados-cwl-runner.

References:

https://common-workflow-language.github.io/draft-3/CommandLineTool.html#ResourceRequirement

http://doc.arvados.org/api/schema/Job.html

In arvados/sdk/cwl/arvados_cwl/__init__.py ArvadosJob.run() you can acces self.builder.resources which is a dict containing keys for "cores", "ram", "tmpdirSize" and "outdirSize". These need to be translated to the appropriate arvados keys in runtime_constraints.


Subtasks

Task #8611: Review branch: 8558-cwl-propagate-resource-reqResolvedRadhika Chippada

Associated revisions

Revision be8fb727 (diff)
Added by Peter Amstutz over 5 years ago

Self test script for running CWL conformance tests on arvados-cwl-runner
against Arvados arvbox instance. refs #8558

Revision 4bd414c0 (diff)
Added by Peter Amstutz over 5 years ago

Bugfix for CWL conformance test script, refs #8558

Revision e11ee6c4 (diff)
Added by Peter Amstutz over 5 years ago

CWL test select config, refs #8558

Revision 2fe15804
Added by Radhika Chippada over 5 years ago

refs #8558
Merge branch '8558-add-cwl-jenkins'

Revision 2fe15804
Added by Radhika Chippada over 5 years ago

refs #8558
Merge branch '8558-add-cwl-jenkins'

Revision a2d5d6b2
Added by Radhika Chippada over 5 years ago

closes #8558
Merge branch '8558-cwl-propagate-resource-req'

History

#1 Updated by Brett Smith over 5 years ago

  • Target version set to 2016-03-16 sprint

#2 Updated by Peter Amstutz over 5 years ago

  • Subject changed from [CWL] arvados-cwl-runner propagates ResourceRequirements to [CWL] arvados-cwl-runner propagates ResourceRequirement
  • Description updated (diff)

#3 Updated by Brett Smith over 5 years ago

  • Story points set to 0.5

#4 Updated by Peter Amstutz over 5 years ago

  • Description updated (diff)

#5 Updated by Peter Amstutz over 5 years ago

  • Description updated (diff)

#6 Updated by Brett Smith over 5 years ago

  • Assigned To set to Radhika Chippada

#7 Updated by Radhika Chippada over 5 years ago

  • Added coresMin and ramMin from builder resources into runtime_constraints. Per the documentation, these can be long or string typed and hence I converted them into integer before adding to runtime_constraints (our documentation says we use integers).
  • In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.

#8 Updated by Radhika Chippada over 5 years ago

  • Status changed from New to In Progress

#9 Updated by Brett Smith over 5 years ago

Radhika Chippada wrote:

  • In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.

The constraint min_scratch_mb_per_node is pretty closely analogous to tmpdirSize. Both specify how much working disk space the job should be able to write to while it runs.

We don't have anything like outdirSize, so that shouldn't be handled, you're right.

#10 Updated by Peter Amstutz over 5 years ago

The resources dict contains keys "cores", "ram", "tmpdirSize" and "outdirSize". This is generated by evalResources: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L305

min_scratch_mb_per_node should be the sum of tmpdirSize and outdirSize because currently we use the same partition for both scratch space and output staging.

Python style note, if "cores" in resources is a more idiomatic way of writing if "cores" in resources.keys() and probably a bit more efficient (resources.keys() returns a list).

#11 Updated by Radhika Chippada over 5 years ago

  • Arvados branch 8558-cwl-propagate-resource-req:
    • Updated to use the correct names (tmpdirSize instead of tmpdirMin etc) and other suggested updates.
    • Updated a couple typos in the tests. Also, updated the tests to expect fields from default resources when one or more are omitted.
  • Added branch 8558-add-cwl-jenkins in arvados-jenkins

#12 Updated by Peter Amstutz over 5 years ago

            if "cores" in resources:
                try:
                    runtime_constraints["min_cores_per_node"] = int(resources["cores"])
                except:
                    runtime_constraints["min_cores_per_node"] = None

This should be written more compactly (the "get" method returns None if the key is unavailable):

  runtime_constraints["min_cores_per_node"] = resources.get("cores")

Similarly (the optional 2nd parameter to the "get" method is the value to return if the key is unavailable):

  runtime_constraints["min_scratch_mb_per_node"] = resources.get("tmpdirSize", 0) + resources.get("outdirSize", 0)

#13 Updated by Peter Amstutz over 5 years ago

Actually, it's a bad idea to set runtime_constraints["min_cores_per_node"] to None, it should probably have a default value as well, eg runtime_constraints["min_cores_per_node"] = resources.get("cores", 1)

(Although in practice the "cores" key should always be there so the default doesn't really matter.)

#14 Updated by Radhika Chippada over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:a2d5d6b24dde2de391831aa122cfda8e2cb759e0.

Also available in: Atom PDF