Story #13823

Package Arvados for Ubuntu Bionic Beaver 18.04 LTS

Added by Joshua Randall about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/28/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5
Release relationship:
Auto

Description

It would be helpful to have arvados apt packages for ubuntu bionic, the "new" LTS release.

One motivating reason is that none of the currently supported Debian/Ubuntu releases have a slurm package with functionality new enough to support the priority ranges Arvados uses when large number of containers are run (see issue 13489).

Bionic has slurm 17.11.2 which would alleviate this issue.


Subtasks

Task #14549: ReviewResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #13435: a-c-r and python SDK ruamel.yaml versioningResolved05/07/2018

Related to Arvados - Story #14483: Update ruaml version for a-c-rResolved

Associated revisions

Revision e4fca76d
Added by Ward Vandewege about 1 year ago

Merge branch 'master' into 13823-bionic

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision bc97d806 (diff)
Added by Ward Vandewege about 1 year ago

Add support for package testing to our ubuntu 18.04 packages.

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision ac2cc876
Added by Ward Vandewege about 1 year ago

Merge branch '13823-bionic'

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision a1d78727 (diff)
Added by Ward Vandewege about 1 year ago

Add support for ubuntu 18.04 (aka 'bionic') to run_upload_packages.py

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision e73008e4 (diff)
Added by Ward Vandewege about 1 year ago

We need to build ws 0.4.2 for ubuntu1404 and ubuntu1604

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision 926dcfea (diff)
Added by Ward Vandewege 6 months ago

Ubuntu 18.04 ("bionic") packages are now confirmed working, add it to the list
of supported GNU/Linux distributions.

refs #13823

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Tom Morris about 1 year ago

  • Tracker changed from Bug to Story
  • Subject changed from package arvados for ubuntu bionic to Package Arvados for Ubuntu Bionic Beaver 18.04 LTS
  • Target version set to To Be Groomed

#2 Updated by Tom Clegg about 1 year ago

  • Status changed from New to In Progress

#3 Updated by Tom Clegg about 1 year ago

Some work in progress on 13823-bionic.

easy_install3 has disappeared, and we're supposed to use "pip3 install" instead. This will be easy to migrate if it turns out "pip install" and "pip3 install" work for all older releases, too. Testing that.

#4 Updated by Tom Clegg about 1 year ago

With the pip3 change I can build bionic packages, but the python packages have dependency problems.

"Depends: python-ws4py (< 0.4) but 0.4.2+dfsg1-2 is to be installed"

(python-arvados-python-client depends on ws4py<0.4 because tests failed with 0.4.1 -- but ubuntu:bionic ships 0.4.2)

SDK tests seem to pass with ws4py 0.4.2.

#5 Updated by Tom Clegg about 1 year ago

Same problem with ruamel.yaml: bionic ships 0.15.34-1 but sdk/python requires <0.15.

python-arvados-python-client : Depends: python-ruamel.yaml (< 0.15) but 0.15.34-1 is to be installed

d015f8bd07c4890b3084306a6cdd67aeda7f81f0 introduced the <0.15 part; #13435 suggests we copied it from cwltool, which has since changed to <0.16 in 7f8a62c7b4f63480fb187fc7def4c6e3695e669c.

#6 Updated by Tom Clegg about 1 year ago

13823-bionic @ f8f3944e6ba1f470eedf48aa609a50c780fe4b6d

...successfully produces packages for all distros on my dev box. Haven't tried test-packages.

Incidental changes (see commit comments for more details):
  • switch from cure-fpm to fpm
  • upgrade ws4py and ruamel.yaml python deps

#7 Updated by Tom Clegg about 1 year ago

  • Related to Bug #13435: a-c-r and python SDK ruamel.yaml versioning added

#8 Updated by Tom Clegg about 1 year ago

  • Category deleted (Deployment)
  • Target version deleted (To Be Groomed)

#9 Updated by Tom Morris about 1 year ago

  • Assigned To set to Tom Clegg
  • Target version set to 2018-08-15 Sprint

Apparently this got urgently pulled onto the current sprint.

#10 Updated by Tom Morris about 1 year ago

  • Target version changed from 2018-08-15 Sprint to Arvados Future Sprints

#11 Updated by Ward Vandewege about 1 year ago

I reviewed and made some more improvements on this branch, latest at bc97d806fa1fb15e49cefe819f00089ebc33d376

  • fixed package testing for bionic
  • merged latest master

From my perspective this is now ready for merging. Then:

#12 Updated by Ward Vandewege 12 months ago

This was merged, and the necessary CI changes have been made. Packages have been published to the bionic-dev repo, and will be going forward.

Next step is a functional test.

#13 Updated by Ward Vandewege 11 months ago

  • Release set to 14

#14 Updated by Tom Morris 9 months ago

This was tagged for the upcoming release, but I can't tell if it's done or not. Merged, but needs to be tested still?

#15 Updated by Tom Morris 9 months ago

  • Related to Story #14483: Update ruaml version for a-c-r added

#16 Updated by Tom Morris 9 months ago

  • Target version changed from Arvados Future Sprints to 2018-11-28 Sprint

#17 Updated by Ward Vandewege 9 months ago

Tom Morris wrote:

This was tagged for the upcoming release, but I can't tell if it's done or not. Merged, but needs to be tested still?

Everything is done in this ticket, but there is a known issue with a cwltool/ruamel version incompatibility with bionic, which is being handled in #14483. Once that is merged, this should work, and we can test our 18.04 packaging.

#18 Updated by Ward Vandewege 9 months ago

Well, the ruamel issue is resolved, but there's one more dependency issue, now with python-avro:


root@1aa90ea77c25:/etc/apt/sources.list.d# apt-get install python-schema-salad
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-schema-salad : Depends: python-avro (< 1.8.2) but 1.8.2+dfsg-1 is to be installed
E: Unable to correct problems, you have held broken packages.
root@1aa90ea77c25:/etc/apt/sources.list.d# apt-cache policy python-avro
python-avro:
  Installed: (none)
  Candidate: 1.8.2+dfsg-1
  Version table:
     1.8.2+dfsg-1 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
     1.8.1-2 500
        500 http://apt.arvados.org bionic-dev/main amd64 Packages

#19 Updated by Tom Clegg 9 months ago

  • Assigned To changed from Tom Clegg to Ward Vandewege
I think the next step is one of
  • update schema-salad so it works with avro≥1.8.2
  • update schema-salad to use the avro-cwl fork in python2 (avro-cwl is at 1.8.4, but it's used by schema-salad only when in python3)
  • update a-c-r packaging to use virtualenv for python deps

#20 Updated by Ward Vandewege 9 months ago

  • Target version changed from 2018-11-28 Sprint to 2018-12-12 Sprint

#21 Updated by Ward Vandewege 8 months ago

  • Release changed from 14 to 15

#22 Updated by Tom Morris 8 months ago

  • Target version changed from 2018-12-12 Sprint to 2018-12-21 Sprint

#23 Updated by Tom Morris 8 months ago

  • Target version changed from 2018-12-21 Sprint to 2019-01-16 Sprint

#24 Updated by Tom Morris 7 months ago

  • Target version changed from 2019-01-16 Sprint to 2019-01-30 Sprint

#25 Updated by Ward Vandewege 7 months ago

  • Target version changed from 2019-01-30 Sprint to 2019-02-13 Sprint

#26 Updated by Ward Vandewege 7 months ago

  • Story points set to 0.5

#27 Updated by Ward Vandewege 6 months ago

  • Target version changed from 2019-02-13 Sprint to 2019-02-27 Sprint

#28 Updated by Tom Morris 6 months ago

  • Target version changed from 2019-02-27 Sprint to 2019-03-13 Sprint

#29 Updated by Ward Vandewege 6 months ago

  • Status changed from In Progress to Resolved

Packages have been available for a while, and I've confirmed that they work in our Arvados on Kubernetes environment.

Also available in: Atom PDF