Actions
Idea #8491
closed
Refactor arvados-dev repository into arvados
Start date:
03/11/2016
Due date:
Story points:
1.0
Description
Move most of arvados-dev repository into main arvados repository. Proposed plan:
a) Use "git subtree split" in arvados-dev to create a standalone jenkins branch. Use "git subtree add" to import it with history into arvados/tools/build
b) Move "arvados/tools/build/run-tests.sh" to "arvados/tools/testing" ; remove it from arvados-dev.git
c) In arvados-dev.git remove "run-build-*", "package-*/", "libcloud-pin"
d) In "arvados/tools/build" remove "run-cwl-tests.sh", "run-deploy.sh", "run-diagnostics-suite.sh", "run-performance-suite.sh", "run-tapestry-tests.sh" "run_upload_packages.py" (these stay in arvados-dev.git)
e) Move "arvados/backports/" to "arvados/tools/build/backports"
Related issues
Updated by Brett Smith about 8 years ago
- Target version set to 2016-03-16 sprint
Updated by Nico César about 8 years ago
I'm in favor of all the rearragement
today we have:
ci:/usr/local/bin# ls -l run* | cut -b 40- run-build-docker-images.sh -> /usr/local/arvados-dev/jenkins/run-build-docker-images.sh run-build-docker-jobs-image.sh -> /usr/local/arvados-dev/jenkins/run-build-docker-jobs-image.sh run-build-packages.sh -> /usr/local/arvados-dev/jenkins/run-build-packages.sh run-build-packages-sso.sh -> /usr/local/arvados-dev/jenkins/run-build-packages-sso.sh run-cwl-tests.sh -> /usr/local/arvados-dev/jenkins/run-cwl-tests.sh run-deploy.sh -> /usr/local/arvados-dev/jenkins/run-deploy.sh run-diagnostics-suite.sh -> /usr/local/arvados-dev/jenkins/run-diagnostics-suite.sh run-performance-suite.sh -> /usr/local/arvados-dev/jenkins/run-performance-suite.sh run-tapestry-tests.sh -> /usr/local/arvados-dev/jenkins/run-tapestry-tests.sh run-tests.sh -> /usr/local/arvados-dev/jenkins/run-tests.sh run_upload_packages.py -> /usr/local/arvados-dev/jenkins/run_upload_packages.py
where /usr/local/arvados-dev is a clone of the repo.
and it's being pulled by a human. that is also able to run them manually to check that everything is aligned with other parts of ci.curoverse.com
- ~jenkins/.ssh/* (private keys to do deploys and github)
- ~jenkins/.config/arvados/*conf (API secret for credential)
- ~jenkins/.dockercfg (dockerhub cerdentials)
(also we make local changes when scripts don't work )
But this is valid for the run-deploy* and run-diagnosis* scripts we as ops don't care much about the build and test jobs.
Updated by Brett Smith about 8 years ago
- Description updated (diff)
Changing the spec to leave run_upload_packages.py in arvados-dev. It includes code that's specific to our servers, and requires setup on the CI server. While you often want to update it when adding a new component, the decision to publish a package is technically separate from the decision to add a component. It's not so critical that this stay in sync with Arvados code, the same way the tests and the package builders need to.
Updated by Tom Clegg about 8 years ago
8491-import-build-tools @ 9155994 (TODO: rebase to master after merging 8345)
a) Use "git subtree split" in arvados-dev to create a standalone jenkins branch. Use "git subtree add" to import it with history into arvados/tools/build
Instead I've brought the whole arvados-dev repo in as a subtree, and then moved files around. The resulting tree is the same, but this way it will still be possible to look up commit hashes from arvados-dev (e.g., mentioned in issue discussions and commit logs).
I've also used just arvados/build, because arvados/tools already has a different kind of tools.
b) Move "arvados/tools/build/run-tests.sh" to "arvados/tools/testing" ; remove it from arvados-dev.git
I've just left run-tests.sh in build/. Seems like we want our build and test recipes to get more integrated, not less, right? E.g., test unpackaged code in source tree, build package, install package, test package after installing.
c) In arvados-dev.git remove "run-build-*", "package-*/", "libcloud-pin"
Holding off on the "remove from arvados-dev" parts until the arvados changes are merged and jenkins stops relying on arvados-dev.
e) Move "arvados/backports/" to "arvados/tools/build/backports"
I'm inclined to leave backports where it is. In any case it doesn't seem related to the current issue, i.e., "should be easy to keep build scripts and code tree in sync when updating/switching branches".
Also added a top level Makefile so you can get started quickly with the traditional
make test
instead of having to learn the Special Snowflake commands right away.
Updated by Tom Clegg about 8 years ago
8491-import-build-tools @ d695d41
The arvados-dev version merged here is arvados-dev|555b039, which includes the "build fuse backports for older OSes" parts of #8345 but not the actual llfuse>=0.42 change.
When this branch merges, these three ways of running tests will be equivalent:
~/arvados$ WORKSPACE=`pwd` ~/arvados-dev/jenkins/run-tests.sh --temp /tmp/XYZ ~/arvados$ WORKSPACE=`pwd` build/run-tests.sh --temp /tmp/XYZ ~/arvados$ make test TESTOPTS='--temp /tmp/XYZ'
At that point we can update our jenkins configs and Hacking docs to use the 2nd or 3rd variant, then delete the duplicated build scripts from the arvados-dev tree.
Updated by Peter Amstutz about 8 years ago
Looks good, trying to run all of test / build packages / test packages end to end, but that is going to take a while.
In the meantime, some suggestions:
- The default "make" rule could be a "help" command that tells you about the available targets and also provides a link to the "prerequisites" page on the wiki. Otherwise when someone types "make" it'll launch into trying to run the tests which is confusing.
- Update README.md with similar information.
- Update arvbox so that it no longer relies on arvados-dev (if you prefer I can do that in a separate branch, though)
Updated by Peter Amstutz about 8 years ago
********** Running services/login-sync tests **********
Run options: -v --seed 61159
# Running:
Creating account adminroot
Creating account active
TestAddUser#test_useradd_error = 0.51 s = F
*** Expect crash in dir_s_mkdir:
Creating account adminroot
Creating account active
TestAddUser#test_useradd_success = 0.48 s = F
Finished in 0.992465s, 2.0152 runs/s, 3.0228 assertions/s.
1) Failure:
TestAddUser#test_useradd_error [/home/peter/work/arvclone/services/login-sync/test/test_add_user.rb:14]:
Expected /useradd -m -c active -s \/bin\/bash -G fuse active/ to match "useradd -m -c adminroot -s /bin/bash -G docker adminroot\nuseradd -m -c active -s /bin/bash -G active\n".
2) Failure:
TestAddUser#test_useradd_success [/home/peter/work/arvclone/services/login-sync/test/test_add_user.rb:32]:
Failed assertion, no message given.
2 runs, 3 assertions, 2 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/home/peter/.rvm/gems/ruby-2.1.1@arvados-tests/gems/rake-10.4.2/lib" "/home/peter/.rvm/gems/ruby-2.1.1@arvados-tests/gems/rake-10.4.2/lib/rake/rake_test_loader.rb" "test/test*.rb" -v]
/home/peter/.rvm/gems/ruby-2.1.1@arvados-tests/bin/ruby_executable_hooks:15:in `eval'
/home/peter/.rvm/gems/ruby-2.1.1@arvados-tests/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)
********** !!!!!! services/login-sync tests FAILED !!!!!! **********
********** End of services/login-sync tests (2s) **********
Updated by Peter Amstutz about 8 years ago
----------------------------------------------------------------------
FAIL: keep-rsync_test.go:337: ServerRequiredSuite.TestErrorDuringRsync_FakeDstKeepservers
keep-rsync_test.go:344:
c.Check(strings.Contains(err.Error(), "no such host"), Equals, true)
... obtained bool = false
... expected bool = true
----------------------------------------------------------------------
FAIL: keep-rsync_test.go:325: ServerRequiredSuite.TestErrorDuringRsync_FakeSrcKeepservers
keep-rsync_test.go:332:
c.Check(strings.Contains(err.Error(), "no such host"), Equals, true)
... obtained bool = false
... expected bool = true
Updated by Tom Clegg about 8 years ago
Peter Amstutz wrote:
- The default "make" rule could be a "help" command that tells you about the available targets and also provides a link to the "prerequisites" page on the wiki. Otherwise when someone types "make" it'll launch into trying to run the tests which is confusing.
Indeed. Added.
- Update README.md with similar information.
README.md links to Hacking, which is still the right place after this ... adding more detailed test/build instructions up front seems like a separate question.
- Update arvbox so that it no longer relies on arvados-dev (if you prefer I can do that in a separate branch, though)
That'd be great, thanks
Updated by Peter Amstutz about 8 years ago
Running test-packages:
START: python27-python-arvados-fuse test on arvados/package-test:centos6
Loaded plugins: fastestmirror
Setting up Install Process
Determining fastest mirrors
* base: mirror.netdepot.com
* extras: mirrors.advancedhosters.com
* rpmforge: repoforge.mirror.constant.com
* rpmforge-extras: repoforge.mirror.constant.com
* rpmforge-testing: repoforge.mirror.constant.com
* updates: mirror.cs.pitt.edu
Resolving Dependencies
--> Running transaction check
---> Package python27-python-arvados-fuse.noarch 0:0.1.20160206004530-1 will be installed
--> Processing Dependency: python27-python-llfuse = 0.41.1 for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Processing Dependency: python27-python-arvados-python-client >= 0.1.20151118035730 for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Processing Dependency: python27-python-ciso8601 for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Processing Dependency: python27-python-daemon for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Processing Dependency: fuse for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Processing Dependency: python27-python for package: python27-python-arvados-fuse-0.1.20160206004530-1.noarch
--> Running transaction check
---> Package fuse.x86_64 0:2.9.2-5 will be installed
---> Package python27-python.x86_64 0:2.7.5-10.el6.centos.alt will be installed
--> Processing Dependency: python27-python-libs(x86-64) = 2.7.5-10.el6.centos.alt for package: python27-python-2.7.5-10.el6.centos.alt.x86_64
--> Processing Dependency: python27-runtime for package: python27-python-2.7.5-10.el6.centos.alt.x86_64
--> Processing Dependency: libpython2.7.so.1.0()(64bit) for package: python27-python-2.7.5-10.el6.centos.alt.x86_64
---> Package python27-python-arvados-python-client.noarch 0:0.1.20160301181511-1 will be installed
--> Processing Dependency: python27-python-google-api-python-client = 1.4.2 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-oauth2client < 2 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-gflags < 3.0 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-pycurl < 7.21.5 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-oauth2client >= 1.4.6 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-pycurl >= 7.19.5.1 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-ws4py for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
--> Processing Dependency: python27-python-httplib2 for package: python27-python-arvados-python-client-0.1.20160301181511-1.noarch
---> Package python27-python-ciso8601.x86_64 0:1.0.1-2 will be installed
---> Package python27-python-daemon.noarch 0:2.1.1-1 will be installed
--> Processing Dependency: python27-python-lockfile >= 0.10 for package: python27-python-daemon-2.1.1-1.noarch
--> Processing Dependency: python27-python-setuptools for package: python27-python-daemon-2.1.1-1.noarch
--> Processing Dependency: python27-python-docutils for package: python27-python-daemon-2.1.1-1.noarch
---> Package python27-python-llfuse.x86_64 0:0.41.1-1 will be installed
--> Running transaction check
---> Package python27-python-docutils.noarch 0:0.11-1.el6.centos.alt will be installed
---> Package python27-python-gflags.noarch 0:2.0-1 will be installed
---> Package python27-python-google-api-python-client.noarch 0:1.4.2-1 will be installed
--> Processing Dependency: python27-python-six >= 1.6.1 for package: python27-python-google-api-python-client-1.4.2-1.noarch
--> Processing Dependency: python27-python-uritemplate >= 0.6 for package: python27-python-google-api-python-client-1.4.2-1.noarch
---> Package python27-python-httplib2.noarch 0:0.9.2-2 will be installed
---> Package python27-python-libs.x86_64 0:2.7.5-10.el6.centos.alt will be installed
---> Package python27-python-lockfile.noarch 0:0.12.2-1 will be installed
---> Package python27-python-oauth2client.noarch 0:1.5.2-1 will be installed
--> Processing Dependency: python27-python-pyasn1-modules >= 0.0.5 for package: python27-python-oauth2client-1.5.2-1.noarch
--> Processing Dependency: python27-python-rsa >= 3.1.4 for package: python27-python-oauth2client-1.5.2-1.noarch
--> Processing Dependency: python27-python-pyasn1 >= 0.1.7 for package: python27-python-oauth2client-1.5.2-1.noarch
---> Package python27-python-pycurl.x86_64 0:7.19.5.3-1 will be installed
---> Package python27-python-setuptools.noarch 0:0.9.8-2.el6.centos.alt will be installed
---> Package python27-python-ws4py.noarch 0:0.3.4-1 will be installed
---> Package python27-runtime.x86_64 0:1.1-16.el6.centos.alt will be installed
--> Running transaction check
---> Package python27-python-pyasn1.noarch 0:0.1.7-1 will be installed
---> Package python27-python-pyasn1-modules.noarch 0:0.0.5-1 will be installed
---> Package python27-python-rsa.noarch 0:3.3-1 will be installed
---> Package python27-python-six.noarch 0:1.10.0-1 will be installed
---> Package python27-python-uritemplate.noarch 0:0.6-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository
Size
================================================================================
Installing:
python27-python-arvados-fuse noarch 0.1.20160206004530-1 localrepo 62 k
Installing for dependencies:
fuse x86_64 2.9.2-5 localrepo 79 k
python27-python x86_64 2.7.5-10.el6.centos.alt scl 80 k
python27-python-arvados-python-client
noarch 0.1.20160301181511-1 localrepo 181 k
python27-python-ciso8601 x86_64 1.0.1-2 localrepo 13 k
python27-python-daemon noarch 2.1.1-1 localrepo 28 k
python27-python-docutils noarch 0.11-1.el6.centos.alt scl 1.6 M
python27-python-gflags noarch 2.0-1 localrepo 70 k
python27-python-google-api-python-client
noarch 1.4.2-1 localrepo 84 k
python27-python-httplib2 noarch 0.9.2-2 localrepo 119 k
python27-python-libs x86_64 2.7.5-10.el6.centos.alt scl 5.5 M
python27-python-llfuse x86_64 0.41.1-1 localrepo 445 k
python27-python-lockfile noarch 0.12.2-1 localrepo 18 k
python27-python-oauth2client noarch 1.5.2-1 localrepo 121 k
python27-python-pyasn1 noarch 0.1.7-1 localrepo 68 k
python27-python-pyasn1-modules noarch 0.0.5-1 localrepo 61 k
python27-python-pycurl x86_64 7.19.5.3-1 localrepo 146 k
python27-python-rsa noarch 3.3-1 localrepo 60 k
python27-python-setuptools noarch 0.9.8-2.el6.centos.alt scl 423 k
python27-python-six noarch 1.10.0-1 localrepo 20 k
python27-python-uritemplate noarch 0.6-1 localrepo 7.2 k
python27-python-ws4py noarch 0.3.4-1 localrepo 62 k
python27-runtime x86_64 1.1-16.el6.centos.alt scl 1.0 M
Transaction Summary
================================================================================
Install 23 Package(s)
Total download size: 10 M
Installed size: 37 M
Downloading Packages:
--------------------------------------------------------------------------------
Total 1.3 MB/s | 10 MB 00:07
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Installing : python27-runtime-1.1-16.el6.centos.alt.x86_64 1/23
Installing : python27-python-libs-2.7.5-10.el6.centos.alt.x86_64 2/23
Installing : python27-python-2.7.5-10.el6.centos.alt.x86_64 3/23
Installing : python27-python-pyasn1-0.1.7-1.noarch 4/23
Installing : python27-python-httplib2-0.9.2-2.noarch 5/23
Installing : python27-python-six-1.10.0-1.noarch 6/23
Installing : python27-python-ciso8601-1.0.1-2.x86_64 7/23
Installing : python27-python-pyasn1-modules-0.0.5-1.noarch 8/23
Installing : python27-python-rsa-3.3-1.noarch 9/23
Installing : python27-python-oauth2client-1.5.2-1.noarch 10/23
Installing : python27-python-docutils-0.11-1.el6.centos.alt.noarch 11/23
Installing : python27-python-gflags-2.0-1.noarch 12/23
Installing : python27-python-lockfile-0.12.2-1.noarch 13/23
Installing : python27-python-setuptools-0.9.8-2.el6.centos.alt.noarch 14/23
Installing : python27-python-daemon-2.1.1-1.noarch 15/23
Installing : python27-python-pycurl-7.19.5.3-1.x86_64 16/23
Installing : python27-python-ws4py-0.3.4-1.noarch 17/23
Installing : python27-python-uritemplate-0.6-1.noarch 18/23
Installing : python27-python-google-api-python-client-1.4.2-1.noarch 19/23
Installing : python27-python-arvados-python-client-0.1.20160301181511 20/23
Installing : python27-python-llfuse-0.41.1-1.x86_64 21/23
Installing : fuse-2.9.2-5.x86_64 22/23
Installing : python27-python-arvados-fuse-0.1.20160206004530-1.noarch 23/23
Verifying : python27-python-pyasn1-0.1.7-1.noarch 1/23
Verifying : python27-python-docutils-0.11-1.el6.centos.alt.noarch 2/23
Verifying : python27-python-pyasn1-modules-0.0.5-1.noarch 3/23
Verifying : python27-python-gflags-2.0-1.noarch 4/23
Verifying : python27-python-lockfile-0.12.2-1.noarch 5/23
Verifying : python27-python-google-api-python-client-1.4.2-1.noarch 6/23
Verifying : python27-python-setuptools-0.9.8-2.el6.centos.alt.noarch 7/23
Verifying : python27-python-oauth2client-1.5.2-1.noarch 8/23
Verifying : python27-python-daemon-2.1.1-1.noarch 9/23
Verifying : python27-python-pycurl-7.19.5.3-1.x86_64 10/23
Verifying : python27-python-arvados-python-client-0.1.20160301181511 11/23
Verifying : python27-python-rsa-3.3-1.noarch 12/23
Verifying : python27-python-2.7.5-10.el6.centos.alt.x86_64 13/23
Verifying : python27-runtime-1.1-16.el6.centos.alt.x86_64 14/23
Verifying : fuse-2.9.2-5.x86_64 15/23
Verifying : python27-python-libs-2.7.5-10.el6.centos.alt.x86_64 16/23
Verifying : python27-python-httplib2-0.9.2-2.noarch 17/23
Verifying : python27-python-ws4py-0.3.4-1.noarch 18/23
Verifying : python27-python-six-1.10.0-1.noarch 19/23
Verifying : python27-python-arvados-fuse-0.1.20160206004530-1.noarch 20/23
Verifying : python27-python-ciso8601-1.0.1-2.x86_64 21/23
Verifying : python27-python-uritemplate-0.6-1.noarch 22/23
Verifying : python27-python-llfuse-0.41.1-1.x86_64 23/23
Installed:
python27-python-arvados-fuse.noarch 0:0.1.20160206004530-1
Dependency Installed:
fuse.x86_64 0:2.9.2-5
python27-python.x86_64 0:2.7.5-10.el6.centos.alt
python27-python-arvados-python-client.noarch 0:0.1.20160301181511-1
python27-python-ciso8601.x86_64 0:1.0.1-2
python27-python-daemon.noarch 0:2.1.1-1
python27-python-docutils.noarch 0:0.11-1.el6.centos.alt
python27-python-gflags.noarch 0:2.0-1
python27-python-google-api-python-client.noarch 0:1.4.2-1
python27-python-httplib2.noarch 0:0.9.2-2
python27-python-libs.x86_64 0:2.7.5-10.el6.centos.alt
python27-python-llfuse.x86_64 0:0.41.1-1
python27-python-lockfile.noarch 0:0.12.2-1
python27-python-oauth2client.noarch 0:1.5.2-1
python27-python-pyasn1.noarch 0:0.1.7-1
python27-python-pyasn1-modules.noarch 0:0.0.5-1
python27-python-pycurl.x86_64 0:7.19.5.3-1
python27-python-rsa.noarch 0:3.3-1
python27-python-setuptools.noarch 0:0.9.8-2.el6.centos.alt
python27-python-six.noarch 0:1.10.0-1
python27-python-uritemplate.noarch 0:0.6-1
python27-python-ws4py.noarch 0:0.3.4-1
python27-runtime.x86_64 0:1.1-16.el6.centos.alt
Complete!
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/arvados_fuse/__init__.py", line 50, in <module>
import llfuse
File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/llfuse/__init__.py", line 15, in <module>
from llfuse.capi import *
ImportError: libfuse.so.2: cannot open shared object file: No such file or directory
ERROR: python27-python-arvados-fuse test on arvados/package-test:centos6 failed with exit status 1
Updated by Tom Clegg about 8 years ago
- Status changed from In Progress to Resolved
Actions