Arvados

Wiki » SDKs »

Python SDK » History » Revision 5

Revision 4 (Brett Smith, 08/28/2014 09:29 AM) → Revision 5/17 (Brett Smith, 08/28/2014 10:33 AM)

h1. Python SDK 

 (design draft) 

 h1. Hypothetical future Crunch scripts 

 We're writing these out with the goal of designing a new SDK for Crunch h3. Example crunch script authors. 

 {{toc}} 

 h2. Example scripts 

 h3. grep+process example with annotations 

 <pre><code class="python"> 
 #!/usr/bin/env python 

 from arvados import CrunchJob 

 import examplelib 
 import re 

 class NormalizeMatchingFiles(CrunchJob): 
     @CrunchJob.task() 
     def grep_files(self): 
         # CrunchJob instantiates input parameters based on the 
         # dataclass attribute.    When we ask for the input parameter, 
         # CrunchJob sees that it's a Collection, and returns a 
         # CollectionReader object. 
         input_coll = self.job_param('input') 
         for filename in input_coll.filenames(): 
             self.grep_file(self.job_param('pattern'), input_coll, filename) 

     @CrunchJob.task() 
     def grep_file(self, pattern, collection, filename): 
         regexp = re.compile(pattern) 
         with collection.open(filename) as in_file: 
             for line in in_file: 
                 if regexp.search(line): 
                     self.normalize(in_file) 
                     break 

     # examplelib is already multi-threaded and will peg the whole 
     # compute node.    These tasks should run sequentially. 
     # When tasks are created, Arvados-specific objects like Collection file 
     # objects are serialized as task parameters.    CrunchJob instantiates 
     # these parameters as real objects when it runs the task. 
     @CrunchJob.task(parallel_with=[]) 
     def normalize(self, coll_file): 
         output = examplelib.frob(coll_file.mount_path()) 
         # self.output is a CollectionWriter.    When this task method finishes, 
         # CrunchJob checks if we wrote anything to it.    If so, it takes care 
         # of finishing the upload process, and sets this task's output to the 
         # Collection UUID. 
         with self.output.open(filename) as out_file: 
             out_file.write(output) 


 if __name__ == '__main__': 
     NormalizeMatchingFiles(task0='grep_files').main() 
 </code></pre> 

 h3. Example from #3603 

 This is the script that Abram used to illustrate #3603. 

 <pre><code class="python"> Notes/todo: 
 #!/usr/bin/env python 

 from arvados import Collection, CrunchJob 
 from subprocess import check_call 

 class Example3603(CrunchJob): 
     @CrunchJob.task() 
     def parse_human_map(self): 
         refpath = self.job_param('REFPATH').name 
         for line in self.job_param('HUMAN_COLLECTION_LIST'): 
             fastj_id, human_id = line.strip().split(',') 
             self.run_ruler(refpath, fastj_id, human_id) 

     @CrunchJob.task() 
     def run_ruler(self, refpath, fastj_id, human_id): 
         check_call(["tileruler", "--crunch", "--noterm", "abv", 
                     "-human", human_id, 
                     "-fastj-path", Collection(fastj_id).mount_path(), 
                     "-lib-path", refpath]) 
         self.output.add('.')    # Or the path where tileruler writes output. 


 if __name__ == '__main__': 
     Example3603(task0='parse_human_map').run() 
 </code></pre> 

 h2. Notes/TODO 

 * Important concurrency limits that job scripts must be able to express: 
 ** Task Z cannot start until all outputs/side effects of tasks W, X, Y are known/complete (e.g., because Z uses WXY's outputs as its inputs). 
 ** Task Y and Z cannot run on the same worker node without interfering with each other (e.g., due to RAM requirements). 
 * In general, the output name is not known until the task is nearly finished. Frequently it is clearer to specify it when the task is queued, though. We should provide a convenient way to do this without any boilerplate in the queued task. 
 * It would be nice to pass an openable object to a task rather than a filename. (i.e., @grep_file()@ shouldn't have to repeat @job_param('input')@; that should be implicit in its argument.) 
 * A second example that uses a "case" and "control" input (e.g., "tumor" and "normal") might help reveal features. 
 * Should get more clear about how the output of the job (as opposed to the output of the last task) is to be set. The obvious way (concatenate all task outputs) should be a one-liner, if not implicit. Either way, it should run in a task rather than being left up to @crunch-job@.