Arvados

• Added coresMin and ramMin from builder resources into runtime_constraints. Per the documentation, these can be long or string typed and hence I converted them into integer before adding to runtime_constraints (our documentation says we use integers).

• In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.

• I verified any syntax errors in the code I added, but could not test. We have absolutely no automated tests!! And, I could not figure out how to at least test manually from https://dev.arvados.org/projects/arvados/wiki/Running_Common_Workflow_Language_%28CWL%29_workflows_on_Arvados

Radhika Chippada wrote:

• In our documentation, I did not find the equivalents of tmpdirSize and outdirSize. Hence, I did not handle these.

The constraint min_scratch_mb_per_node is pretty closely analogous to tmpdirSize. Both specify how much working disk space the job should be able to write to while it runs.

We don't have anything like outdirSize, so that shouldn't be handled, you're right.

The resources dict contains keys "cores", "ram", "tmpdirSize" and "outdirSize". This is generated by evalResources: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L305

min_scratch_mb_per_node should be the sum of tmpdirSize and outdirSize because currently we use the same partition for both scratch space and output staging.

Python style note, if "cores" in resources is a more idiomatic way of writing if "cores" in resources.keys() and probably a bit more efficient (resources.keys() returns a list).

• Arvados branch 8558-cwl-propagate-resource-req:
  • Updated to use the correct names (tmpdirSize instead of tmpdirMin etc) and other suggested updates.
  • Updated a couple typos in the tests. Also, updated the tests to expect fields from default resources when one or more are omitted.

• Added branch 8558-add-cwl-jenkins in arvados-jenkins

            if "cores" in resources:
                try:
                    runtime_constraints["min_cores_per_node"] = int(resources["cores"])
                except:
                    runtime_constraints["min_cores_per_node"] = None

This should be written more compactly (the "get" method returns None if the key is unavailable):

  runtime_constraints["min_cores_per_node"] = resources.get("cores")

Similarly (the optional 2nd parameter to the "get" method is the value to return if the key is unavailable):

  runtime_constraints["min_scratch_mb_per_node"] = resources.get("tmpdirSize", 0) + resources.get("outdirSize", 0)

Actually, it's a bad idea to set runtime_constraints["min_cores_per_node"] to None, it should probably have a default value as well, eg runtime_constraints["min_cores_per_node"] = resources.get("cores", 1)

(Although in practice the "cores" key should always be there so the default doesn't really matter.)