Project

General

Profile

Feature #4561

Updated by Tom Clegg about 9 years ago

Currently you have two main options for running a job: 
 # Put the program you need in a docker image. Use run-command from the arvados tree to wrap it as a crunch job. 
 # Write a native crunch script in your git tree. 

 The first option forces you to save a new docker image in order to run a new version of your program. Unacceptable! 

 There are several features of run-command that make it convenient and attractive to beginners. However, it forces you to use a totally different approach to developing and running scripts, an approach which prevents you from doing some important things: 
 * Keeping your code in revision control, 
 * Using the same code in more than one pipeline template, 
 * Writing jobs in the programming language of your choice. 

 There is no migration path from a simple run-command job to a non-trivial program, so the developer is forced to choose: live with run-command's custom JSON-based programming language, or abandon the existing pipelines and all of run-command's advantages, and rewrite everything in a normal language like Python. 

 This can be addressed by refactoring run-command as a set of utilities and features, rather than a programming language that can only be used inside crunch jobs. 
 * The JSON programming language should be presented as an interpreter. 
 * Convenience features like "store output dir contents in Keep and set success=true at end of task" should be ported to other SDKs too (most obviously bash), so authors can migrate from the JSON language to a normal language. 
 * It should be possible to provide a crunch script in _any_ language by copying the script itself into the job record. (This makes it possible use run-command to run jobs without touching git.) 

 It should be possible to access run-command's features from wrap a script that lives in your git tree. 
 * Currently, this can be done awkwardly by copying some version of run-command into your own git tree. 
 * With #4027, we can make run-command's features available through the SDK. 
 * #4027 it shouldn't be much of a stretch to include run-command in an SDK package. Then we just need a way to invoke run-command from the installed package rather than requiring it to be in @$CRUNCH_SRC/crunch_scripts/@.

Back