Feature #14819

Upgrade the arvados/jobs image to Debian Stretch

Added by Ward Vandewege 7 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/27/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Subtasks

Task #14842: Review 14819-arvados-jobs-on-stretchResolvedWard Vandewege

Associated revisions

Revision 494b80a7
Added by Ward Vandewege 6 months ago

Merge branch '14819-arvados-jobs-on-stretch'

closes #14819

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 89c5953f (diff)
Added by Ward Vandewege 5 months ago

Switch to stretch-slim as the base image for arvados/jobs, this results in a
smaller image.

arvados/jobs 1.3.1.20190308210512 2a6c5e4993e7 4 days ago 284MB
arvados/jobs latest 0cfff29120dd 20 minutes ago 232MB

refs #14819

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Ward Vandewege 7 months ago

  • Status changed from New to In Progress

#2 Updated by Ward Vandewege 7 months ago

  • Release deleted (22)

#3 Updated by Ward Vandewege 7 months ago

I have c969d8ec54ce0661d42099a90bd2a0f7756dff90 (branch: 14819-arvados-jobs-on-stretch) ready for review.

#4 Updated by Tom Morris 6 months ago

  • Target version set to 2019-02-27 Sprint

#5 Updated by Peter Amstutz 6 months ago

#RUN cat gpg --no-tty --import /tmp/1078ECD7.key

This line should probably be deleted?

#6 Updated by Peter Amstutz 6 months ago

RUN apt-get install -yq --no-install-recommends gpg procps

Are these necessary?

python-pip python-virtualenv \
    python-dev libgnutls28-dev libcurl4-gnutls-dev \

I wonder if are these still needed now that we have the virtualenv packages? If they are needed, they should be pulled in as dependencies of python-arvados-cwl-runner not listed explicitly.

If we can slim down the image that seems like it would be a good thing.

#7 Updated by Tom Morris 6 months ago

  • Target version changed from 2019-02-27 Sprint to 2019-03-13 Sprint

#8 Updated by Ward Vandewege 6 months ago

I've addressed your comments in 37c46ca294ce95242ac424c8375a130d783f6d1e

The latest arvados/jobs image in head is 405MB.

After this commit merges, it will be down to 281MB.

#9 Updated by Peter Amstutz 6 months ago

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

Step 15/18 : RUN apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version
 ---> Running in 488310a7c95a
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3.1.20190205182514-1' for 'python-arvados-python-client' was not found
E: Version '1.3.1.20190206223817-1' for 'python-arvados-cwl-runner' was not found
The command '/bin/sh -c apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version' returned a non-zero code: 100

#10 Updated by Peter Amstutz 6 months ago

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

    ARVADOS_BUILDING_VERSION="$(git tag -l |sort -V -r |head -n1).$(date -ud "@$git_ts" +%Y%m%d%H%M%S)" 

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

#11 Updated by Peter Amstutz 6 months ago

2nd question, are we are officially dropping support for crunch v1 in the next version? I need to know if I should test if this image is capable of running crunch v1 jobs.

#12 Updated by Ward Vandewege 6 months ago

Peter Amstutz wrote:

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

[...]

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

#13 Updated by Ward Vandewege 6 months ago

Peter Amstutz wrote:

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

[...]

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

#14 Updated by Peter Amstutz 6 months ago

In order to avoid breaking crunchv1

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-python-client/bin/python /usr/bin/python

needs to be

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-cwl-runner/bin/python /usr/bin/python

#15 Updated by Ward Vandewege 6 months ago

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

#16 Updated by Peter Amstutz 6 months ago

Ward Vandewege wrote:

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

LGTM

#17 Updated by Ward Vandewege 6 months ago

  • Status changed from In Progress to Resolved

#18 Updated by Tom Morris 6 months ago

  • Release set to 15

Also available in: Atom PDF