Story #7370

[Deployment] Testing for Python SDK package installation

Added by Peter Amstutz about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
07/15/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

Merge test scripts to arvados-dev to accomplish the following:

  1. That Python SDK packages can be successfully installed on clean a base system using "apt-get" or "yum" install from a local package repository (no errors from package manager)
  2. That the Python SDK packages are minimally usable (can do "import arvados" and "import arvados_fuse" in Python, and "arv-mount --help" and "arv-get --help" in bash)
  3. New Jenkins step following package build that runs these package tests

Out of scope (future development): support for testing packages other than the Python SDK and FUSE driver.


Subtasks

Task #7578: Set up jenkins stepResolvedWard Vandewege

Task #7169: Review 7370-package-install-testing in arvados-devResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #7184: [Deployment] Test distribution packagesNew

Associated revisions

Revision cffee40d (diff)
Added by Ward Vandewege about 6 years ago

Shut up dpkg-scanpackages warnings.

refs #7370

Revision cffee40d (diff)
Added by Ward Vandewege about 6 years ago

Shut up dpkg-scanpackages warnings.

refs #7370

Revision d119e26a
Added by Peter Amstutz about 6 years ago

Merge branch '7370-package-install-testing' closes #7370

Revision d119e26a
Added by Peter Amstutz about 6 years ago

Merge branch '7370-package-install-testing' closes #7370

Revision d3c6950d (diff)
Added by Ward Vandewege about 6 years ago

Fix path issue when running run-build-packages-all-targets.sh from
another directory.

refs #7370

Revision d3c6950d (diff)
Added by Ward Vandewege about 6 years ago

Fix path issue when running run-build-packages-all-targets.sh from
another directory.

refs #7370

Revision 40fff998 (diff)
Added by Ward Vandewege almost 6 years ago

Make sure that the CentOS 6 test script actually does the yum update call.

refs #7370

Revision 40fff998 (diff)
Added by Ward Vandewege almost 6 years ago

Make sure that the CentOS 6 test script actually does the yum update call.

refs #7370

Revision 69e10b5d (diff)
Added by Peter Amstutz almost 6 years ago

Obsoleted by run-build-packages-all-targets.sh --test-packages refs #7370

Revision 69e10b5d (diff)
Added by Peter Amstutz almost 6 years ago

Obsoleted by run-build-packages-all-targets.sh --test-packages refs #7370

Revision 846f42e5 (diff)
Added by Peter Amstutz almost 6 years ago

Remove unnecessary yum update. touch RPM database to work around overlayfs bug.
refs #7370

Revision 846f42e5 (diff)
Added by Peter Amstutz almost 6 years ago

Remove unnecessary yum update. touch RPM database to work around overlayfs bug.
refs #7370

Revision c271c93c (diff)
Added by Brett Smith almost 6 years ago

6638/7370: Force new builds of Python backports with dependencies.

Even though we've declared these dependencies for a while now, Jenkins
has not published packages with them, because without a new upstream
version, fpm believes that there's no new package to build. This
resolves that by building a new iteration of the affected packages.

This is less than ideal, because if a new version is released, we'll
automatically package it with iteration 2. That is not correct, but
it doesn't affect any functionality, and we already have a plan to do
things properly in #6885. So we'll live with "correct functionality,
gross aesthetics" until then.

Ward approved in conversation. Refs #6638, #7370.

History

#1 Updated by Peter Amstutz about 6 years ago

  • Subject changed from [Deployment] Test framework to check that SDK distribution packages dependencies are satisfied. to [Deployment] Testing for Python SDK package installation
  • Description updated (diff)
  • Status changed from New to In Progress
  • Assigned To set to Peter Amstutz
  • Story points set to 1.0

#2 Updated by Tom Clegg about 6 years ago

  • Description updated (diff)

#3 Updated by Brett Smith about 6 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Brett Smith about 6 years ago

  • Target version changed from Arvados Future Sprints to 2015-10-28 sprint

#5 Updated by Ward Vandewege about 6 years ago

This branch can be simplified by reusing existing infrastructure. There is some code in this repository to test the sso packages. It can be run like this:

First build the package:

WORKSPACE=<your-workspace> ./run-build-packages-one-target.sh --target ubuntu1204 --command run-build-packages-sso.sh

Then run the tests:

WORKSPACE=<your-workspace> ./run-build-packages-one-target.sh --target ubuntu1204 --command run-test-packages-sso.sh

This branch should be refactored to follow the same pattern: write an equivalent of run-test-packages-sso.sh that runs the tests you want to run, and leave the docker side of things up to run-build-packages-one-target.sh and run-build-packages-all-targets.sh.

#6 Updated by Brett Smith about 6 years ago

  • Target version changed from 2015-10-28 sprint to Arvados Future Sprints

#7 Updated by Brett Smith about 6 years ago

  • Target version changed from Arvados Future Sprints to 2015-10-28 sprint

#8 Updated by Ward Vandewege about 6 years ago

Reviewing 7370-package-install-testing:

  • pushed one small commit on the branch to make dpkg-scanpackages more quiet if there are many versions of a package available, please include that when you merge.
  • package tests fail for centos6, debian7, debian8 with a variety of errors.

Other than that, LGTM.

#9 Updated by Peter Amstutz about 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados-dev|commit:d119e26a14b0bc143638ca38926a4a0d911fe8f1.

#10 Updated by Peter Amstutz about 6 years ago

  • Status changed from Resolved to In Progress

Reopening because there is still a jenkins task.

#11 Updated by Ward Vandewege almost 6 years ago

OK there is now a test-packages standalone step.

Right now that is running off a one-time copy of the packages built by build-packages (as of just now); once this thing actually passes I'll move it into the build-packages step, before the upload to the apt repository. That way it will share the packages directory (symlinking doesn't work because a symlink doesn't survive a docker volume mount).

Peter, you can already see the CentOS6 failure in

https://ci.curoverse.com/job/test-packages/5/console

In local testing I'm seeing the same, also for the Debian tests. You should be able to reproduce that?

Thanks,
Ward.

#12 Updated by Peter Amstutz almost 6 years ago

The python-llfuse_0.41.1 package on Jenkins was last built Aug 23.
The python-arvados-fuse_0.1.20150916213104 package on Jenkins was last built Sep 16.

When I compare these packages to the ones that I build locally, they are different.

So I think what is going on is that we have made fixes to the package building, but because we don't have package metadata versioning, the actual packages that are automatically built and distributed by Jenkins are not getting rebuilt. Which means the package install testing is correct in identifying that there is a problem.

I logged into the Jenkins server and manually ran the tests for debian7, debian8, ubuntu1204 and ubuntu1404 and they all passed, so it is just the out-of-date centos6 packages that are holding up the build, because the centos6 test runs first and the script is quitting on first failure.

#13 Updated by Ward Vandewege almost 6 years ago

  • Target version changed from 2015-10-28 sprint to 2015-11-11 sprint

#14 Updated by Ward Vandewege almost 6 years ago

  • Status changed from In Progress to Resolved

#15 Updated by Brett Smith almost 6 years ago

Peter Amstutz wrote:

So I think what is going on is that we have made fixes to the package building, but because we don't have package metadata versioning, the actual packages that are automatically built and distributed by Jenkins are not getting rebuilt.

Correct. Fixed manually in c271c93c. New packages now correctly declare their dependencies, and pass the tests.

Also available in: Atom PDF