Project

General

Profile

Actions

Feature #14819

closed

Upgrade the arvados/jobs image to Debian Stretch

Added by Ward Vandewege about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Subtasks 1 (0 open1 closed)

Task #14842: Review 14819-arvados-jobs-on-stretchResolvedWard Vandewege02/27/2019Actions

Related issues

Copied to Arvados - Feature #16476: Upgrade the arvados/jobs image to Debian BusterResolvedWard Vandewege02/27/2019Actions
Actions #1

Updated by Ward Vandewege about 5 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Ward Vandewege about 5 years ago

  • Release deleted (22)
Actions #3

Updated by Ward Vandewege about 5 years ago

I have c969d8ec54ce0661d42099a90bd2a0f7756dff90 (branch: 14819-arvados-jobs-on-stretch) ready for review.

Actions #4

Updated by Tom Morris about 5 years ago

  • Target version set to 2019-02-27 Sprint
Actions #5

Updated by Peter Amstutz about 5 years ago

#RUN cat gpg --no-tty --import /tmp/1078ECD7.key

This line should probably be deleted?

Actions #6

Updated by Peter Amstutz about 5 years ago

RUN apt-get install -yq --no-install-recommends gpg procps

Are these necessary?

python-pip python-virtualenv \
    python-dev libgnutls28-dev libcurl4-gnutls-dev \

I wonder if are these still needed now that we have the virtualenv packages? If they are needed, they should be pulled in as dependencies of python-arvados-cwl-runner not listed explicitly.

If we can slim down the image that seems like it would be a good thing.

Actions #7

Updated by Tom Morris about 5 years ago

  • Target version changed from 2019-02-27 Sprint to 2019-03-13 Sprint
Actions #8

Updated by Ward Vandewege about 5 years ago

I've addressed your comments in 37c46ca294ce95242ac424c8375a130d783f6d1e

The latest arvados/jobs image in head is 405MB.

After this commit merges, it will be down to 281MB.

Actions #9

Updated by Peter Amstutz about 5 years ago

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

Step 15/18 : RUN apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version
 ---> Running in 488310a7c95a
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3.1.20190205182514-1' for 'python-arvados-python-client' was not found
E: Version '1.3.1.20190206223817-1' for 'python-arvados-cwl-runner' was not found
The command '/bin/sh -c apt-get install -yq --no-install-recommends nodejs     python-arvados-python-client=$python_sdk_version     python-arvados-cwl-runner=$cwl_runner_version' returned a non-zero code: 100
Actions #10

Updated by Peter Amstutz about 5 years ago

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

    ARVADOS_BUILDING_VERSION="$(git tag -l |sort -V -r |head -n1).$(date -ud "@$git_ts" +%Y%m%d%H%M%S)" 

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

Actions #11

Updated by Peter Amstutz about 5 years ago

2nd question, are we are officially dropping support for crunch v1 in the next version? I need to know if I should test if this image is capable of running crunch v1 jobs.

Actions #12

Updated by Ward Vandewege about 5 years ago

Peter Amstutz wrote:

I'm running into version issues, I don't know if this is something that will happen on jenkins as well or I should be working around it somehow:

[...]

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

Actions #13

Updated by Ward Vandewege about 5 years ago

Peter Amstutz wrote:

1.3.0.20190205182514 exists but 1.3.1.20190205182514 does not.

So the problem is here:

[...]

The latest tag is "1.3.1" but this branch is 3 weeks old, when the branch was created, the latest tag was 1.3.0.

This is an old problem with our versioning scheme that isn't really specific to this branch but does affect reproducibility of builds. Better logic would be get the most recent tag looking on at commits earlier than this one.

But if we don't want to fix that, in the short term if we rebase onto master then I can probably build and test successfully.

This is expected, for testing you need to set ARVADOS_BUILDING_VERSION. After merge, this will work correctly (it picks up the git revision from the tree, and when we run this script, the packages will exist).

Actions #14

Updated by Peter Amstutz about 5 years ago

In order to avoid breaking crunchv1

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-python-client/bin/python /usr/bin/python

needs to be

RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-cwl-runner/bin/python /usr/bin/python
Actions #15

Updated by Ward Vandewege about 5 years ago

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

Actions #16

Updated by Peter Amstutz about 5 years ago

Ward Vandewege wrote:

Peter Amstutz wrote:

In order to avoid breaking crunchv1

[...]

needs to be

[...]

Good catch, pushed at dd969e81dfbe0ff1b60c85841aaf01e51b45c574

LGTM

Actions #17

Updated by Ward Vandewege about 5 years ago

  • Status changed from In Progress to Resolved
Actions #18

Updated by Tom Morris about 5 years ago

  • Release set to 15
Actions #20

Updated by Ward Vandewege almost 4 years ago

  • Copied to Feature #16476: Upgrade the arvados/jobs image to Debian Buster added
Actions

Also available in: Atom PDF