Idea #7370
closed
[Deployment] Testing for Python SDK package installation
Description
Merge test scripts to arvados-dev to accomplish the following:
- That Python SDK packages can be successfully installed on clean a base system using "apt-get" or "yum" install from a local package repository (no errors from package manager)
- That the Python SDK packages are minimally usable (can do "import arvados" and "import arvados_fuse" in Python, and "arv-mount --help" and "arv-get --help" in bash)
- New Jenkins step following package build that runs these package tests
Out of scope (future development): support for testing packages other than the Python SDK and FUSE driver.
Related issues
Updated by Peter Amstutz over 8 years ago
- Subject changed from [Deployment] Test framework to check that SDK distribution packages dependencies are satisfied. to [Deployment] Testing for Python SDK package installation
- Description updated (diff)
- Status changed from New to In Progress
- Assigned To set to Peter Amstutz
- Story points set to 1.0
Updated by
Updated by Brett Smith over 8 years ago
- Target version set to Arvados Future Sprints
Updated by Brett Smith over 8 years ago
- Target version changed from Arvados Future Sprints to 2015-10-28 sprint
Updated by Ward Vandewege over 8 years ago
This branch can be simplified by reusing existing infrastructure. There is some code in this repository to test the sso packages. It can be run like this:
First build the package:
WORKSPACE=<your-workspace> ./run-build-packages-one-target.sh --target ubuntu1204 --command run-build-packages-sso.shThen run the tests:
WORKSPACE=<your-workspace> ./run-build-packages-one-target.sh --target ubuntu1204 --command run-test-packages-sso.shThis branch should be refactored to follow the same pattern: write an equivalent of run-test-packages-sso.sh that runs the tests you want to run, and leave the docker side of things up to run-build-packages-one-target.sh and run-build-packages-all-targets.sh.
Updated by Brett Smith over 8 years ago
- Target version changed from 2015-10-28 sprint to Arvados Future Sprints
Updated by Brett Smith over 8 years ago
- Target version changed from Arvados Future Sprints to 2015-10-28 sprint
Updated by Ward Vandewege over 8 years ago
Reviewing 7370-package-install-testing:
- pushed one small commit on the branch to make dpkg-scanpackages more quiet if there are many versions of a package available, please include that when you merge.
- package tests fail for centos6, debian7, debian8 with a variety of errors.
Other than that, LGTM.
Updated by Peter Amstutz over 8 years ago
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
Applied in changeset arvados-dev|commit:d119e26a14b0bc143638ca38926a4a0d911fe8f1.
Updated by Peter Amstutz over 8 years ago
- Status changed from Resolved to In Progress
Reopening because there is still a jenkins task.
Updated by Ward Vandewege over 8 years ago
OK there is now a test-packages standalone step.
Right now that is running off a one-time copy of the packages built by build-packages (as of just now); once this thing actually passes I'll move it into the build-packages step, before the upload to the apt repository. That way it will share the packages directory (symlinking doesn't work because a symlink doesn't survive a docker volume mount).
Peter, you can already see the CentOS6 failure in
https://ci.curoverse.com/job/test-packages/5/consoleIn local testing I'm seeing the same, also for the Debian tests. You should be able to reproduce that?
Thanks,
Ward.
Updated by Peter Amstutz over 8 years ago
The python-llfuse_0.41.1 package on Jenkins was last built Aug 23.
The python-arvados-fuse_0.1.20150916213104 package on Jenkins was last built Sep 16.
When I compare these packages to the ones that I build locally, they are different.
So I think what is going on is that we have made fixes to the package building, but because we don't have package metadata versioning, the actual packages that are automatically built and distributed by Jenkins are not getting rebuilt. Which means the package install testing is correct in identifying that there is a problem.
I logged into the Jenkins server and manually ran the tests for debian7, debian8, ubuntu1204 and ubuntu1404 and they all passed, so it is just the out-of-date centos6 packages that are holding up the build, because the centos6 test runs first and the script is quitting on first failure.
Updated by Ward Vandewege over 8 years ago
- Target version changed from 2015-10-28 sprint to 2015-11-11 sprint
Updated by Ward Vandewege over 8 years ago
- Status changed from In Progress to Resolved
Updated by Brett Smith over 8 years ago
Peter Amstutz wrote:
So I think what is going on is that we have made fixes to the package building, but because we don't have package metadata versioning, the actual packages that are automatically built and distributed by Jenkins are not getting rebuilt.
Correct. Fixed manually in c271c93c. New packages now correctly declare their dependencies, and pass the tests.