Story #15133

Remove crunch v1 (jobs api)

Added by Peter Amstutz 5 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/08/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0

Description

https://dev.arvados.org/projects/arvados-private/wiki/Sunsetting_the_jobs_api

Crunch v1 has been deprecated for a while. Maintaining unused code is becoming a burden.

  • Convert models to be read-only, prevent creating and updating
    • jobs
    • job_tasks
    • pipeline_templates
    • pipeline_instances
  • Delete crunch-dispatch.rb
  • Delete crunch-job
  • Delete deprecated pipeline template documentation from user guide
  • Delete crunch_scripts subdirectory
  • Delete arv-run
  • Delete tests for jobs API that use creating and updating

Subtasks

Task #15506: Review 15133-remove-jobs-apiResolvedPeter Amstutz


Related issues

Blocked by Arvados - Story #15181: [cwl] Remove support for jobs api in a-c-rResolved08/06/2019

Blocked by Arvados - Feature #15087: [Workbench] Show number of queued containers on dashboard (instead of busy/idle nodes)Resolved06/14/2019

Associated revisions

Revision 35658af9
Added by Peter Amstutz about 1 month ago

Merge branch '15133-remove-jobs-api' refs #15133

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision bad30523 (diff)
Added by Peter Amstutz about 1 month ago

Delete crunch-dispatch.rb stub as requested by ops refs #15133

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 5 months ago

  • Subject changed from Remove crunch v1 to Remove crunch v1 (jobs api)
  • Description updated (diff)

#3 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 5 months ago

  • Related to Story #15181: [cwl] Remove support for jobs api in a-c-r added

#5 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 5 months ago

  • Related to Feature #15087: [Workbench] Show number of queued containers on dashboard (instead of busy/idle nodes) added

#7 Updated by Tom Morris 4 months ago

  • Related to deleted (Story #15181: [cwl] Remove support for jobs api in a-c-r)

#8 Updated by Tom Morris 4 months ago

  • Blocked by Story #15181: [cwl] Remove support for jobs api in a-c-r added

#9 Updated by Tom Morris 4 months ago

  • Related to deleted (Feature #15087: [Workbench] Show number of queued containers on dashboard (instead of busy/idle nodes))

#10 Updated by Tom Morris 4 months ago

  • Blocked by Feature #15087: [Workbench] Show number of queued containers on dashboard (instead of busy/idle nodes) added

#11 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)

#12 Updated by Tom Morris 4 months ago

  • Story points set to 3.0
  • Target version changed from To Be Groomed to Arvados Future Sprints

#13 Updated by Tom Morris about 2 months ago

  • Target version changed from Arvados Future Sprints to 2019-08-14 Sprint

#14 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2019-08-14 Sprint to Arvados Future Sprints
  • Description updated (diff)

#15 Updated by Peter Amstutz about 2 months ago

  • Target version changed from Arvados Future Sprints to 2019-08-14 Sprint
  • Assigned To set to Peter Amstutz

#16 Updated by Peter Amstutz about 1 month ago

  • Status changed from New to In Progress

#19 Updated by Lucas Di Pentima about 1 month ago

  • There’re some unit tests that we might be able to remove in tandem with some model methods, for example Jobs & PipelineInstances’ cancel methods+tests. How about avoiding create/delete/updates at the model level?
  • If the above comment is valid, I think we can remove from lib/config several config entries, like Containers.JobsAPI.DefaultDockerImage, Containers.JobsAPI.ReuseJobIfOutputsDiffer, etc.
  • On the docs, there’s a “Job API” section that could be removed/updated at doc/api/execution.html.textile.liquid line 53.
  • Doc file doc/sdk/cli/subcommands.html.textile.liquid (line 344) talks about arv-run-pipeline-instance, if that sub-command doesn’t exist anymore, could this section be removed?
  • Running git grep arv-run-pipeline-instance on the repo’s root dir reveals some other places where seem to be expected to exist.
  • Doing the same as above with “crunch-job” also seem to show that it’s expected to exist for arvbox, and maybe a couple of Dockerfiles?

#20 Updated by Peter Amstutz about 1 month ago

Lucas Di Pentima wrote:

  • There’re some unit tests that we might be able to remove in tandem with some model methods, for example Jobs & PipelineInstances’ cancel methods+tests. How about avoiding create/delete/updates at the model level?
  • If the above comment is valid, I think we can remove from lib/config several config entries, like Containers.JobsAPI.DefaultDockerImage, Containers.JobsAPI.ReuseJobIfOutputsDiffer, etc.

Great idea! I was able to remove another whole batch of tests that don't make sense if create/update is disabled.

  • On the docs, there’s a “Job API” section that could be removed/updated at doc/api/execution.html.textile.liquid line 53.

Removed it.

  • Doc file doc/sdk/cli/subcommands.html.textile.liquid (line 344) talks about arv-run-pipeline-instance, if that sub-command doesn’t exist anymore, could this section be removed?

Removed it.

  • Running git grep arv-run-pipeline-instance on the repo’s root dir reveals some other places where seem to be expected to exist.

Thanks, I cleaned those up as well.

  • Doing the same as above with “crunch-job” also seem to show that it’s expected to exist for arvbox, and maybe a couple of Dockerfiles?

Removed crunch-dispatch (v1) from arvbox.

15133-remove-jobs-api @ 043185f6d16dd41135c255f0f36fb8a75203537b

https://ci.curoverse.com/job/developer-run-tests/1469/

https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/217/

The "integration" part of arvados-cwl-conformance-tests seem to be failing regularly. However they pass when I run them locally. I'm not quite sure what is going on. Unfortunately, since the test runs in a container on an ephemeral compute node, there isn't any additional information to track down the failures.

#22 Updated by Lucas Di Pentima about 1 month ago

This LGTM, except for the fact that when trying out a previously created arvbox with current master (but using an image from this dev branch), I got the following error when trying to start the api:

[...]
2019-08-09_23:44:25.85307 + mkdir -p tmp/cache
2019-08-09_23:44:25.85830 + bundle exec rake db:migrate
2019-08-09_23:44:34.52979 time="2019-08-09T23:44:34.529692101Z" level=warning msg="deprecated or unknown config entry: Clusters.x3o8s.NodeProfiles" 
2019-08-09_23:44:36.18151 rake aborted!
2019-08-09_23:44:36.18155 NameError: uninitialized constant CommitAncestor
2019-08-09_23:44:36.18173 /var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/inflector/methods.rb:268:in `const_get'
2019-08-09_23:44:36.18174 /var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/inflector/methods.rb:268:in `block in constantize'
2019-08-09_23:44:36.18174 /var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/inflector/methods.rb:266:in `each'
2019-08-09_23:44:36.18174 /var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/inflector/methods.rb:266:in `inject'
2019-08-09_23:44:36.18175 /var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/inflector/methods.rb:266:in `constantize'
[...]

Not sure if this also will happen when upgrading other systems.

#24 Updated by Peter Amstutz about 1 month ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF