Story #9767

[Crunch2][Workbench] Extend pipeline template chooser to browse CWL workflows

Added by Tom Morris over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
08/24/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0
Release:
Release relationship:
Auto

Description

  • In dashboard, when the "Run a pipeline" button is clicked, show pipeline_templates and workflows in the chooser popup.
    • Do we want to rename this button as "Run a process"?
  • When this button is pressed and a workflow is selected:
    • Show the workflow details in the preview panel.
    • When one of the workflows is selected and the "Select inputs" button is pressed: create a ContainerRequest with the data from the workflow and take the user to the container_request show page.

Subtasks

Task #9847: Add a URL to workflow in the container_request#show pageResolvedRadhika Chippada

Task #9880: Review branch 9767-workflow-link-in-container-requestResolvedPeter Amstutz

Task #9811: Review branch 9767-workflows-in-chooserResolvedPeter Amstutz

Associated revisions

Revision 1062d902
Added by Radhika Chippada over 4 years ago

refs #9767
Merge branch '9767-workflows-in-chooser'

Revision a87fd176
Added by Radhika Chippada over 4 years ago

closes #9767
Merge branch '9767-workflow-link-in-container-request'

History

#1 Updated by Tom Morris over 4 years ago

  • Story points set to 2.0

#2 Updated by Tom Clegg over 4 years ago

  • Subject changed from [Crunch2][Workbench] Extend pipeline template display to browse CWL workflows to [Crunch2][Workbench] Extend pipeline template chooser to browse CWL workflows

#3 Updated by Tom Morris over 4 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Radhika Chippada over 4 years ago

  • Description updated (diff)

#5 Updated by Tom Morris over 4 years ago

  • Tracker changed from Bug to Story

#6 Updated by Radhika Chippada over 4 years ago

  • Description updated (diff)
  • Is the workflow yaml from the selected Workflow object going to contain all the necessary information to fill in the required fields of a container_request, namely, command, container_image, output_path, and cwd?
  • Also, what values should be used for the runtime_contraints of the newly created ContainerRequest?
  • Do we need to expand the container_request schema to preserve / remember the selected Workflow uuid in the container_request?
    • Do we need to expand the container_request show page to display a link to the Workflow (similar to a pipeline_instance display)?

#7 Updated by Tom Morris over 4 years ago

  • Story points changed from 2.0 to 3.0

#8 Updated by Tom Morris over 4 years ago

  • Target version changed from Arvados Future Sprints to 2016-08-31 sprint

#9 Updated by Peter Amstutz over 4 years ago

The newly created container request should set up the following mounts:

            "mounts": {
                "/var/lib/cwl/workflow.json": {
                    "kind": "json",
                    "content": {"class": "Workflow", ...},
                },
                "/var/lib/cwl/cwl.input.json": {
                    "kind": "json",
                    "content": {}
                }

(note: edited "/var/lib/cwl/jobs/cwl.input.json" -> "/var/lib/cwl/cwl.input.json")

#10 Updated by Tom Clegg over 4 years ago

Sounds good to me.

#11 Updated by Radhika Chippada over 4 years ago

  • Assigned To set to Radhika Chippada

#12 Updated by Peter Amstutz over 4 years ago

{
  "command": ["arvados-cwl-runner", "--local", "--api=containers", "/var/lib/cwl/workflow.json", "/var/lib/cwl/cwl.input.json"],
  "output_path": "/var/spool/cwl",
  "cwd": "/var/spool/cwl",
  "priority": 1,
  "state": "Uncommitted",
  "container_image": "arvados/jobs",
  "mounts": {
    "/var/lib/cwl/workflow.json": {
      "kind": "json",
      "content": {"class": "Workflow", ...},
    },
    "/var/lib/cwl/cwl.input.json": {
      "kind": "json",
      "content": {}
    },
    "stdout": {
      "kind": "file",
      "path": "/var/spool/cwl/cwl.output.json" 
    },
    "/var/spool/cwl": {
      "kind": "collection",
      "writable": True
    }
  },
  "runtime_constraints": {
    "vcpus": 1,
    "ram": 256000000,
    "API": True
  }
}

#13 Updated by Peter Amstutz over 4 years ago

The above container request is based on the one already in use by arvados-cwl-runner --submit so it should work.

#14 Updated by Radhika Chippada over 4 years ago

  • Status changed from New to In Progress

#15 Updated by Radhika Chippada over 4 years ago

  • Description updated (diff)

#16 Updated by Radhika Chippada over 4 years ago

Please clarify the workflow.json mount point's content attribute. What key name should I use to pass the workflow json? Thanks.

  "mounts": {
    "/var/lib/cwl/workflow.json": {
      "kind": "json",
      "content": {"class": "Workflow", ...},
    },

#17 Updated by Peter Amstutz over 4 years ago

Radhika Chippada wrote:

Please clarify the workflow.json mount point's content attribute. What key name should I use to pass the workflow json? Thanks.

The content of the "workflow" field from the "workflow" record should be assigned to the "content" field under "/var/lib/cwl/workflow.json".

#18 Updated by Radhika Chippada over 4 years ago

Branch 9767-workflows-in-chooser is ready for review at c837f9d95

I manually added workflow yaml to 4xphq-7fd4e-2fkoj19vfnbc6j1, and created the container_request 4xphq-xvhdp-emh6t3cmt038ssa via the dashboard. (And I am able to see the inputs I added in the yaml in #9043 branch updates)

#19 Updated by Peter Amstutz over 4 years ago

In work_units_controller.rb:51

wf_json = JSON.dump(YAML::load(workflow.workflow))

The mounts field is a structured/serialized field, so this should be:

wf_json = YAML::load(workflow.workflow)

For the chooser title, instead of "Choose a pipeline or process to run" suggest this say "Choose a pipeline or workflow to run".

In the chooser, I'm not sure showing the raw text of the workflow is very useful. Perhaps remove the workflow text, and if there is no description, perhaps render "No description provided" so that the box isn't empty?

#20 Updated by Tom Clegg over 4 years ago

Don't forget to add safe_yaml, if Workbench is parsing user-supplied YAML.

#21 Updated by Peter Amstutz over 4 years ago

The container requests page should have a link back to the workflow object, similar to the way the pipeline instance page has a link to the pipeline template.

Suggest setting a default container request name based on the name of the workflow. Something like "%s container" or "%s run" (where %s = the workflow name).

#22 Updated by Radhika Chippada over 4 years ago

wf_json = JSON.dump(YAML::load(workflow.workflow)) ... wf_json = YAML::load(workflow.workflow)

Updated

For the chooser title, instead of "Choose a pipeline or process to run" suggest this say "Choose a pipeline or workflow to run".

Done

In the chooser, I'm not sure showing the raw text of the workflow is very useful. Perhaps remove the workflow text, and if there is no description, perhaps render "No description provided" so that the box isn't empty?

Removed the workflow display from preview pane

Tom said: Don't forget to add safe_yaml, if Workbench is parsing user-supplied YAML.

Added safe_yaml gem and updated the load_config similar to API load_config

The container requests page should have a link back to the workflow object, similar to the way the pipeline instance page has a link to the pipeline template.

Will do this in a separate branch. Created task #9847

Suggest setting a default container request name based on the name of the workflow. Something like "%s container" or "%s run" (where %s = the workflow name).

Added this

#23 Updated by Peter Amstutz over 4 years ago

The "Run a pipeline..." on project pages should also be updated to display workflows in the chooser dialog.

#24 Updated by Radhika Chippada over 4 years ago

Good catch. Commit a485bcb2 addresses this. Thanks.

#25 Updated by Peter Amstutz over 4 years ago

a485bcb LGTM, thanks.

#26 Updated by Radhika Chippada over 4 years ago

Branch 9767-workflow-link-in-container-request at ac36412 adds a link to workflow in the container_request#show page.

#27 Updated by Peter Amstutz over 4 years ago

9767-workflow-link-in-container-request @ commit:a720aab7961883fe3541dc68ed7ad0d5f91edbee LGTM

#28 Updated by Radhika Chippada over 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 67 to 100

Applied in changeset arvados|commit:a87fd176c3248699cbaaa45b815c8a34b3f5f11b.

Also available in: Atom PDF