Task #4667
Updated by Brett Smith over 9 years ago
Mostly, this implements the actual work part of #4027, where Crunch installs an SDK version specified by the job record, and uses it to run the Crunch script.
I tested this in a local Docker installation, using a Crunch script that reported its environment, and saved that as its output. In order to do that, I had to fix the Gitolite setup in our Docker setup to make repositories usable (making sure the API server is always SSH-able, and making a couple minor bugfixes to our Gitolite setup script).
The script reported results as expected running outside Docker, inside Docker with no SDK specified, and inside Docker with an SDK specified. That script:
<pre><code class="python">#!/usr/bin/env python
import os
import subprocess
import arvados
from pprint import pprint
this_task = arvados.current_task()
out_coll = arvados.CollectionWriter()
with out_coll.open('environment.txt') as out_file:
pprint(arvados.__file__, out_file)
pprint(os.environ, out_file)
if 'CRUNCH_INSTALL' in os.environ:
pipe = subprocess.Popen(['ls', '-lR', os.environ['CRUNCH_INSTALL']],
stdout=subprocess.PIPE)
for line in pipe.stdout:
out_file.write(line)
pipe.stdout.close()
this_task.set_output(out_coll.finish())
</code></pre>
Please see the commit messages for additional information.