Arvados

From our conversation earlier today: we would be running cwl work flows using:

1. submit mode where runner job is submitted to server
2. in local mode, where we are creating a pipeline instance and attaching the runner job to it
3. from workbench, where a pipeline instance is created from the template

We also talked about work units / processes which would only be pipelines or containers. This seems to be in conflict with how we are handling the submit mode.

Should we be creating a pipeline instance in the case of submit as well?

Please clarify requirements.

Radhika Chippada wrote:

From our conversation earlier today: we would be running cwl work flows using:

1. submit mode where runner job is submitted to server
2. in local mode, where we are creating a pipeline instance and attaching the runner job to it
3. from workbench, where a pipeline instance is created from the template

We also talked about work units / processes which would only be pipelines or containers. This seems to be in conflict with how we are handling the submit mode.

Should we be creating a pipeline instance in the case of submit as well?

Please clarify requirements.

1. In submit mode it should create a pipeline instance with a single component which is the cwl-runner job. the cwl-runner job should list its jobs as components of the cwl-runner job.
2. in local mode, it should create a pipeline add the component jobs to it (the existing behavior remains unchanged)
3. from workbench, there is nothing to do on the pipeline instance (it is created by workbench), and the cwl-runner job should list its jobs as components of the cwl-runner job.

Branch 9275-cwl-runner-creates-jobs at 94d3f4ba addresses the comments from note 6.

94d3f4ba358ac5fef765fb7574d6823ff8c335aa

It should create a pipeline instance when submit is true (case 1) or cwl_runner_job is not in kwargs (case 2). If cwl_runner_job is present we should assume there is already a pipeline instance.

For (1) and (3), it needs to update the 'components' field of Job record. That's missing from the branch. You can add this as an alternate behavior in ArvadosJob.update_pipeline_component. If self.arvrunner.pipeline is not None, keep the existing behavior. Otherwise, update the Job record corresponding to cwl_runner_job on API server and add to job["components"][self.name] = record["uuid"]

In update_pipeline_component(self, record) starting on line 211:

Add if self.arvrunner.pipeline: because it will raise an exception when self.arvrunner.pipeline is None (it looks like that exception is being caught and discarded, but it will prevent it from getting to the 2nd part).

line 222:

                    components['component'+str(len(components))] = record["uuid"]

This should use self.name instead of 'component'+str(len(components))

Line 683:

        if kwargs.get("submit") or "cwl_runner_job" not in kwargs:

On further examination, think we can drop the kwargs.get("submit"), because the case where we don't want to create a pipeline is the case when "cwl_runner_job" in kwargs is True; it should never be the case for cwl_runner_job and submit to be True.

line 714:

                if "cwl_runner_job" in kwargs:
                    uuid = kwargs.get("cwl_runner_job").get('uuid')

This needs to be self.uuid = ...

Make those corrections.

Regarding: use self.name instead of 'component'+str(len(components))

Is self.name guaranteed to be non-null? Otherwise, how about " self.name or 'component'+str(len(components)) "

Line 714: This needs to be self.uuid = ...

Ah. Thanks for noticing it.