Feature #8558
closed
- Target version set to 2016-03-16 sprint
- Subject changed from [CWL] arvados-cwl-runner propagates ResourceRequirements to [CWL] arvados-cwl-runner propagates ResourceRequirement
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Assigned To set to Radhika Chippada
- Added coresMin and ramMin from builder resources into runtime_constraints. Per the documentation, these can be long or string typed and hence I converted them into integer before adding to runtime_constraints (our documentation says we use integers).
- In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.
- Status changed from New to In Progress
Radhika Chippada wrote:
- In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.
The constraint min_scratch_mb_per_node is pretty closely analogous to tmpdirSize. Both specify how much working disk space the job should be able to write to while it runs.
We don't have anything like outdirSize, so that shouldn't be handled, you're right.
The resources
dict contains keys "cores", "ram", "tmpdirSize" and "outdirSize". This is generated by evalResources: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L305
min_scratch_mb_per_node
should be the sum of tmpdirSize
and outdirSize
because currently we use the same partition for both scratch space and output staging.
Python style note, if "cores" in resources
is a more idiomatic way of writing if "cores" in resources.keys()
and probably a bit more efficient (resources.keys() returns a list).
- Arvados branch 8558-cwl-propagate-resource-req:
- Updated to use the correct names (tmpdirSize instead of tmpdirMin etc) and other suggested updates.
- Updated a couple typos in the tests. Also, updated the tests to expect fields from default resources when one or more are omitted.
- Added branch 8558-add-cwl-jenkins in arvados-jenkins
if "cores" in resources:
try:
runtime_constraints["min_cores_per_node"] = int(resources["cores"])
except:
runtime_constraints["min_cores_per_node"] = None
This should be written more compactly (the "get" method returns None if the key is unavailable):
runtime_constraints["min_cores_per_node"] = resources.get("cores")
Similarly (the optional 2nd parameter to the "get" method is the value to return if the key is unavailable):
runtime_constraints["min_scratch_mb_per_node"] = resources.get("tmpdirSize", 0) + resources.get("outdirSize", 0)
Actually, it's a bad idea to set runtime_constraints["min_cores_per_node"]
to None
, it should probably have a default value as well, eg runtime_constraints["min_cores_per_node"] = resources.get("cores", 1)
(Although in practice the "cores" key should always be there so the default doesn't really matter.)
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:a2d5d6b24dde2de391831aa122cfda8e2cb759e0.
Also available in: Atom
PDF