Task #4667

Feature #4027: [Crunch] Accept SDK version as a runtime constraint. Install SDK into the docker container before running tasks.

Review 4027-crunch-sdk-install-wip

Added by Brett Smith about 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
11/24/2014
Due date:
% Done:

100%

Estimated time:
2.00 h

Description

Mostly, this implements the actual work part of #4027, where Crunch installs an SDK version specified by the job record, and uses it to run the Crunch script.

I tested this in a local Docker installation, using a Crunch script that reported its environment, and saved that as its output. In order to do that, I had to fix the Gitolite setup in our Docker setup to make repositories usable (making sure the API server is always SSH-able, and making a couple minor bugfixes to our Gitolite setup script).

The script reported results as expected running outside Docker, inside Docker with no SDK specified, and inside Docker with an SDK specified. That script:

#!/usr/bin/env python

import os
import subprocess

import arvados

from pprint import pprint

this_task = arvados.current_task()
out_coll = arvados.CollectionWriter()
with out_coll.open('environment.txt') as out_file:
    pprint(arvados.__file__, out_file)
    pprint(os.environ, out_file)
    if 'CRUNCH_INSTALL' in os.environ:
        pipe = subprocess.Popen(['ls', '-lR', os.environ['CRUNCH_INSTALL']],
                                stdout=subprocess.PIPE)
        for line in pipe.stdout:
            out_file.write(line)
        pipe.stdout.close()
this_task.set_output(out_coll.finish())

Please see the commit messages for additional information.

Associated revisions

Revision d980949a
Added by Brett Smith about 6 years ago

Merge branch '4027-crunch-sdk-install-wip'

Closes #4027, #4667.

History

#1 Updated by Brett Smith about 6 years ago

  • Description updated (diff)

#2 Updated by Peter Amstutz about 6 years ago

  • Assigned To set to Peter Amstutz

#3 Updated by Peter Amstutz about 6 years ago

  • Status changed from New to In Progress

#4 Updated by Peter Amstutz about 6 years ago

  • Assigned To changed from Peter Amstutz to Brett Smith

#5 Updated by Brett Smith about 6 years ago

  • Assigned To changed from Brett Smith to Peter Amstutz

#6 Updated by Peter Amstutz about 6 years ago

  • Assigned To changed from Peter Amstutz to Brett Smith

#7 Updated by Brett Smith about 6 years ago

  • Assigned To changed from Brett Smith to Peter Amstutz

#8 Updated by Peter Amstutz about 6 years ago

  • Assigned To changed from Peter Amstutz to Brett Smith

#9 Updated by Brett Smith about 6 years ago

  • Assigned To changed from Brett Smith to Peter Amstutz

#10 Updated by Brett Smith about 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
  • Remaining (hours) changed from 2.0 to 0.0

Applied in changeset arvados|commit:d980949ac4c092a44f3b64fb7cbd4a27a49256fb.

Also available in: Atom PDF