Story #9309

[Deployment] Make Arvados installable on CentOS 7

Added by Brett Smith over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
-
Target version:
Start date:
05/28/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

Branch 1:

  • Build packages
  • Test packages

Branch 2:

  • Upload packages
    • Update the uploader

Non-branch work:

  • Add a Jenkins job
  • Find the out-of-band packages we published for CentOS 6, and publish the same for 7 if needed (Git, SLURM, maybe others)

Branch 3:

  • Update the Install Guide
centos7-test.log.xz (8.84 KB) centos7-test.log.xz At 7961ba6 Brett Smith, 06/01/2016 04:07 PM
centos6-test.log.xz (10.6 KB) centos6-test.log.xz At 7961ba6 Brett Smith, 06/01/2016 04:07 PM

Subtasks

Task #9313: Review 9309-centos-7-packages-wipResolvedWard Vandewege

Task #9314: Review arvados-dev branch 9309-centos7-uplaods-wipResolvedWard Vandewege

Task #9351: Review 9309-centos-7-rails-scripts-wipResolvedBrett Smith

Task #9367: Review 9309-postgresql-install-guide-wipResolvedNico César

Task #9368: Review 9309-centos-7-install-guide-wipResolvedBrett Smith

Task #9393: Review 9309-python-redundant-upload-wipResolvedBrett Smith


Related issues

Related to Arvados - Support #9054: Arvados single sign on server installation on CentOS 7New04/26/2016

Related to Arvados - Bug #8905: [Deployment] arvados-sso-server deb package unnecessarily depends on "postgresql" meta-packageResolved04/07/2016

Associated revisions

Revision 79ad2827
Added by Brett Smith over 3 years ago

Merge branch '9309-centos-7-packages-wip'

Refs #9309. Closes #9313.

Revision 799f3676
Added by Brett Smith over 3 years ago

Merge branch '9309-centos7-uplaods-wip'

Refs #9309.

Revision 5604e411
Added by Brett Smith over 3 years ago

Merge branch '9309-postgresql-install-guide-wip'

Refs #9309. Closes #9367.

Revision 2bf563e0 (diff)
Added by Brett Smith over 3 years ago

9309: Bugfix Ruby source install instructions for CentOS.

  • Add missing `make` dependency.
  • Add `-i` to `sudo gem install` throughout. Red Hat adds /usr/local
    paths to $PATH in `/etc/profile`, so we need `-i` to find `gem`.

Refs #9309.

Revision 5f301325
Added by Brett Smith over 3 years ago

Merge branch '9309-python-redundant-upload-wip'

Refs #9309. Closes #9393.

Revision e47a01ee
Added by Brett Smith over 3 years ago

Merge branch '9309-centos-7-rails-scripts-wip'

Refs #9309. Closes #9351, #9345.

Revision e649689d (diff)
Added by Brett Smith over 3 years ago

9309: Bump Rails package iteration.

To accommodate systemd support in the package scripts.
Refs #9309.

Revision f6d8e101
Added by Brett Smith over 3 years ago

Merge branch '9309-centos-7-install-guide-wip'

Refs #9309. Closes #9368.

History

#1 Updated by Brett Smith over 3 years ago

  • Status changed from New to In Progress
  • Assigned To set to Brett Smith

#2 Updated by Brett Smith over 3 years ago

  • Target version set to 2016-06-08 sprint

9309-centos-7-packages-wip implements branch #1, package building and testing.

#3 Updated by Brett Smith over 3 years ago

arvados-dev branch 9309-centos7-uplaods-wip is branch #2, and super-short.

#4 Updated by Ward Vandewege over 3 years ago

reviewing branch 9309-centos7-uplaods-wip on arvados-dev: LGTM.

#5 Updated by Ward Vandewege over 3 years ago

reviewing 9309-centos-7-packages-wip:

  • this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)
  • line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

#6 Updated by Brett Smith over 3 years ago

Ward Vandewege wrote:

reviewing 9309-centos-7-packages-wip:

  • this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)

Done, although there were no merge conflicts in the process...

  • line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Ugh, no, it means I introduced a bug. I didn't realize we were setting $PACKAGE_NAME in that case. Things happened to work on distros where $PYTHON2_PKG_PREFIX == $PACKAGE_TYPE, but I broke fpm-info.sh on other distros like centos6. I have to add some new scheme here in order to accommodate libcloud, where we can't add fpm-info.sh to the source directory (because it's fundamentally not our repo) and $PACKAGE is not predictable (because it's a random tempdir).

I have reverted to having code closer to master, that adds backports/$PACKAGE_NAME as another place to search.

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

The tests failed because of the CWL package-related brokenness that we fixed in master over the weekend. Now that we're rebased on master, things should be good. Now at 35abd53.

#7 Updated by Ward Vandewege over 3 years ago

Brett Smith wrote:

Ward Vandewege wrote:

reviewing 9309-centos-7-packages-wip:

  • this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)

Done, although there were no merge conflicts in the process...

Great.

  • line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Ugh, no, it means I introduced a bug. I didn't realize we were setting $PACKAGE_NAME in that case. Things happened to work on distros where $PYTHON2_PKG_PREFIX == $PACKAGE_TYPE, but I broke fpm-info.sh on other distros like centos6. I have to add some new scheme here in order to accommodate libcloud, where we can't add fpm-info.sh to the source directory (because it's fundamentally not our repo) and $PACKAGE is not predictable (because it's a random tempdir).

Yeah, ok.

I have reverted to having code closer to master, that adds backports/$PACKAGE_NAME as another place to search.

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

The tests failed because of the CWL package-related brokenness that we fixed in master over the weekend. Now that we're rebased on master, things should be good. Now at 35abd53.

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

Loading fpm overrides from /arvados/backports/python-llfuse/fpm-info.sh
ls: cannot access /arvados/packages/centos7/fuse-devel-[0-9]*.rpm: No such file or directory
Attemping to install build_dep fuse-devel
error: open of fuse-devel failed: No such file or directory

#8 Updated by Brett Smith over 3 years ago

Ward Vandewege wrote:

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

That's not a build failure: check your package directory, and you should have python-llfuse there. These errors come from the part of fpm_build where we try to install build_depends. ls fails because we didn't build fuse-devel locally, and then rpm -i fuse-devel fails because that doesn't even make sense (it should probably say yum install -q -y fuse-devel instead). But the Docker image already has fuse-devel installed, so the main build still succeeds.

In my own run, building the keep-balance package failed... I'm looking into that now.

#9 Updated by Ward Vandewege over 3 years ago

Brett Smith wrote:

Ward Vandewege wrote:

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

That's not a build failure: check your package directory, and you should have python-llfuse there. These errors come from the part of fpm_build where we try to install build_depends. ls fails because we didn't build fuse-devel locally, and then rpm -i fuse-devel fails because that doesn't even make sense (it should probably say yum install -q -y fuse-devel instead). But the Docker image already has fuse-devel installed, so the main build still succeeds.

In my own run, building the keep-balance package failed... I'm looking into that now.

You are right. The build failed on keep-balance for me too, the thing about python-llfuse was not a build failure. I didn't scroll up enough.

#10 Updated by Brett Smith over 3 years ago

It looks like the keep-balance failure is just that the Dockerfile needs to be updated to reflect the new name of the Go tarball. I'm trying that now.

#11 Updated by Brett Smith over 3 years ago

  • Description updated (diff)

#12 Updated by Brett Smith over 3 years ago

9309-centos-7-packages-wip is now properly rebased on master, applying recent build changes to centos7 itself, at 7961ba6. It is now passing all tests; see the attached logs.

#13 Updated by Ward Vandewege over 3 years ago

LGTM now, thanks!

#14 Updated by Brett Smith over 3 years ago

Build process for the extracirricular packages was mostly the same as before in #8005-12, with just minor variations:

<https://codeload.github.com/dun/munge/legacy.tar.gz/880584f51f1a4b3641552bfae9f89e1f1c9a2fda>
    yum install bzip2-devel
    tar -xf munge-0.5.12.tar.gz
    mv dun-munge-*/ munge-0.5.12
    tar -c munge-0.5.12 | xz > munge-0.5.12.tar.xz
    rpmbuild -tb munge-0.5.12.tar.xz
    rpmbuild -ts munge-0.5.12.tar.xz

<http://www.schedmd.com/download/latest/slurm-16.05.0.tar.bz2>
     yum install pam-devel openssl
     rpm -ivh ~/rpmbuild/RPMS/x86_64/munge*.rpm
     rpmbuild -tb slurm-16.05.0.tar.bz2
     rpmbuild -ts slurm-16.05.0.tar.bz2

## runit
     yum -y install rpmdevtools git glibc-static
     git clone https://github.com/imeyer/runit-rpm runit-rpm
     cd runit-rpm/
     git checkout eefeaca0913638f375afe92d03c5b4c5f81128f2
     ./build.sh
     rpmbuild -bs ~/rpmbuild/SPECS/runit.spec

#15 Updated by Brett Smith over 3 years ago

New package build is failing. This seems to be why.

The most commonly recommended workaround in that thread is to use the devicemapper storage backend.

Didn't happen to me on my workstation, running Debian 8 with a Linux 4.3 backport and just a stock Docker configuration.

#16 Updated by Brett Smith over 3 years ago

My workstation is using the devicemapper driver (even though this isn't specified in any config that I've found yet), while ci is using aufs, so that's fundamentally the distinction causing different results.

#17 Updated by Brett Smith over 3 years ago

  • Target version changed from 2016-06-08 sprint to 2016-06-22 sprint
  • Story points set to 2.0

#18 Updated by Brett Smith over 3 years ago

(10:17:27 AM) Me: Do people have strong opinions about adding/splitting off an Install Guide page, before the SSO install page, titled something like "Set up PostgreSQL database(s)"?
(10:17:53 AM) Me: Deployers have a lot of flexibility in the database setup, so splitting that off gives us a little space to cover the options and let people pick what they want at the right time.
(10:18:17 AM) Me: The database setup is also one of the more distro-specific parts of the Install Guide, so splitting it off can help make other pages less fork-y.
(10:19:44 AM) ward: sounds good to me!
(10:20:46 AM) tetron_: agree

#19 Updated by Brett Smith over 3 years ago

9309-postgresql-install-guide-wip is up for review to make the change discussed above. It is pure refactoring; it does not add anything about CentOS 7.

#20 Updated by Brett Smith over 3 years ago

9309-centos-7-install-guide-wip is up for review. I didn't follow literally the entire install guide, but I did go through all the steps that involved installing any of our packages, and setting up stuff around that. I also installed third-party software like Gitolite. But I didn't step through creating API objects and so on; I figure that's high-level enough that it's low-risk.

The weirdest thing is the process for installing the FUSE package. There's a python2-llfuse package in epel that provides llfuse 1.01 or thereabouts, and it obsoletes the python-llfuse package. However, later versions of llfuse have breaking changes, so we really need to have the exact version we've backported. Telling yum to ignore the repository in the meantime seemed like the easiest way forward.

If you want to test things out, you will need to check out 9309-centos-7-rails-scripts-wip and build packages from it. Follow this blog post to create and start a CentOS 7 Docker image running systemd.

The branch probably shouldn't be merged until the rails-scripts branch is merged and packages are actually built (which requires some Jenkins work too).

#21 Updated by Brett Smith over 3 years ago

Brett Smith wrote:

The weirdest thing is the process for installing the FUSE package. There's a python2-llfuse package in epel that provides llfuse 1.01 or thereabouts, and it obsoletes the python-llfuse package. However, later versions of llfuse have breaking changes, so we really need to have the exact version we've backported. Telling yum to ignore the repository in the meantime seemed like the easiest way forward.

The specific problem:

--> Processing Dependency: python-llfuse = 0.41.1 for package: python-arvados-fuse-0.1.20160426220707-2.noarch
Package python-llfuse is obsoleted by python2-llfuse, but obsoleting package does not provide for requirements
Error: Package: python-arvados-fuse-0.1.20160426220707-2.noarch (localrepo)
           Requires: python-llfuse = 0.41.1
           Available: python-llfuse-0.41.1-3.x86_64 (localrepo)
               python-llfuse = 0.41.1-3
           Available: python2-llfuse-1.0-1.el7.x86_64 (epel)
               python-llfuse = 1.0-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

#22 Updated by Nico César over 3 years ago

test df8d16f045c8ed9d5fa7de35c05ab4d4fe234853

I'm following the instructions and we use sudo a lot, expecting the user to be the one excecuting the instructions. but sometimes it back fires:

[root@centos-2gb-nyc2-01 ruby-2.1.7]# sudo gem install bundler
sudo: gem: command not found
[root@centos-2gb-nyc2-01 ruby-2.1.7]# gem install bundler
Fetching: bundler-1.12.5.gem (100%)

this is from install-sso.html under "Build and install Ruby" on CentOS 7.2

we should use sudo -i gem ... instead.

But this is ready to merge event with that issue in place

#23 Updated by Brett Smith over 3 years ago

Nico Cesar wrote:

this is from install-sso.html under "Build and install Ruby" on CentOS 7.2

we should use sudo -i gem ... instead.

This is genuinely surprising. If you're using option #2 to build and install Ruby--so you're building from source, and not using RVM--I would expect gem to be somewhere under $PATH after sudo make install. Where did it go?

#24 Updated by Brett Smith over 3 years ago

Brett Smith wrote:

This is genuinely surprising. If you're using option #2 to build and install Ruby--so you're building from source, and not using RVM--I would expect gem to be somewhere under $PATH after sudo make install. Where did it go?

I see: on Red Hat, the /usr/local paths are added to $PATH in /etc/profile. OK, added sudo -i to gem install lines in 2bf563e.

#25 Updated by Nico César over 3 years ago

test 61284297350777ec0e752f272c0bd6ffbde81af7 / 9309-centos-7-rails-scripts-wip

I notice that #!/bin/sh was used on common-test-rails-server-package.sh then later some bashisms like ${PACKAGE_NAME%-server} and ${PACKAGE_NAME%.sh} .. probably the safest is to explicitly use a /bin/bash she-bang. but otherwise I find the script to be ok. (probably I'm too picky with this, I had some problems in the past with weird setups )

Besides that, the re-organization looked good to me.

I haven't done a try all the combinatios of packages and distros for build/rails-package-scripts/postinst.sh which I assume we'll be iterating over time to see if this works everywhere in present and future configurations.

Ready to merge-

#26 Updated by Brett Smith over 3 years ago

Nico Cesar wrote:

I notice that #!/bin/sh was used on common-test-rails-server-package.sh then later some bashisms like ${PACKAGE_NAME%-server} and ${PACKAGE_NAME%.sh}

?? Those are not bashisms. man dash, search for "Parameter Expansion"

I haven't done a try all the combinatios of packages and distros for build/rails-package-scripts/postinst.sh which I assume we'll be iterating over time to see if this works everywhere in present and future configurations.

FWIW the package tests do go through the postinst, and I ran them for centos6, centos7, and debian8 without issues.

#27 Updated by Nico César over 3 years ago

review arvados-dev @ commit:101791be5e0f127a46971d2c2b72d45e4f9e4fef

this is Ready to merge.

I'll do the deploy in CI so

https://ci.curoverse.com/view/Arvados%20build%20pipeline/job/build-packages-python-ruby/

goes back to blue

#28 Updated by Brett Smith over 3 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF