Arvados

I have c969d8ec54ce0661d42099a90bd2a0f7756dff90 (branch: 14819-arvados-jobs-on-stretch) ready for review.

#RUN cat gpg --no-tty --import /tmp/1078ECD7.key

This line should probably be deleted?

RUN apt-get install -yq --no-install-recommends gpg procps

Are these necessary?

python-pip python-virtualenv \
    python-dev libgnutls28-dev libcurl4-gnutls-dev \

I wonder if are these still needed now that we have the virtualenv packages? If they are needed, they should be pulled in as dependencies of python-arvados-cwl-runner not listed explicitly.

If we can slim down the image that seems like it would be a good thing.

I've addressed your comments in 37c46ca294ce95242ac424c8375a130d783f6d1e

The latest arvados/jobs image in head is 405MB.

After this commit merges, it will be down to 281MB.

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

Step 15/18 : RUN apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version
 ---> Running in 488310a7c95a
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3.1.20190205182514-1' for 'python-arvados-python-client' was not found
E: Version '1.3.1.20190206223817-1' for 'python-arvados-cwl-runner' was not found
The command '/bin/sh -c apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version' returned a non-zero code: 100

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

    ARVADOS_BUILDING_VERSION="$(git tag -l |sort -V -r |head -n1).$(date -ud "@$git_ts" +%Y%m%d%H%M%S)" 

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

2nd question, are we are officially dropping support for crunch v1 in the next version? I need to know if I should test if this image is capable of running crunch v1 jobs.

Peter Amstutz wrote:

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

[...]

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

Peter Amstutz wrote:

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

[...]

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

In order to avoid breaking crunchv1

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-python-client/bin/python /usr/bin/python

needs to be

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-cwl-runner/bin/python /usr/bin/python

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

Ward Vandewege wrote:

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

LGTM