Project

General

Profile

Hacking prerequisites » History » Revision 61

Revision 60 (Peter Amstutz, 07/26/2019 08:10 PM) → Revision 61/77 (Peter Amstutz, 07/26/2019 08:10 PM)

{{>toc}} 

 h1. Hacking prerequisites 

 The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied. 

 h2. Host options 

 h3. Starting on your workstation 

 If your workstation is a debian stretch system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies". 

 Other linux distributions should work too with some modifications, but it's probably easier to use a VM. 

 h3. Starting on a VM 

 Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian stretch on it. The instructions below assume you have just a few basic requirements: 
 * SSH server 
 * sudo (@apt-get install sudo@) 
 * A user account with sudo privileges 

 h3. Starting in a docker container 

 _[[Arvbox]] provides a preinstalled Docker-based dev environment.    The following instructions are for creating a dev environment inside Docker from scratch._ 

 This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container. 

 See http://docker.io for more about installing docker. On debian it looks something like this. 

 <pre> 
 sudo apt-get install docker-ce 
 sudo adduser $USER docker 
 # {log out & log back in} 
 groups 
 # {should include "docker"} 
 </pre> 

 Start up a new container with debian 9 (stretch), make a new user and log in as that user: 

 <pre> 
 docker run -it --privileged debian:9 bash 
 apt-get update 
 apt-get -y install sudo 
 adduser me 
 adduser me sudo 
 sudo -u me -i 
 </pre> 

 The &quot;--privileged&quot; is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work). 

 h2. Install dev environment 

 <pre> 
 # only on debian 9 (stretch), to permit ruby 2.3 compilation (see https://github.com/rbenv/ruby-build/wiki#openssl-usrincludeopensslasn1_mach102-error-error-this-file-is-obsolete-please-update-your-software): 
 sudo apt-get install --no-install-recommends libssl1.0-dev 

 # other systems: 
 sudo apt-get install --no-install-recommends libssl-dev 

 # all systems: 

 sudo apt-get install --no-install-recommends \ 
     bison build-essential cadaver fuse gettext git gitolite3 graphviz \ 
     iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl \ 
     libcrypt-ssleay-perl libcurl3 libcurl3-gnutls libcurl4-openssl-dev curl \ 
     libjson-perl libpcre3-dev libpq-dev libpython2.7-dev libreadline-dev \ 
     libxslt1.1 libwww-perl linkchecker lsof net-tools nginx perl-modules \ 
     postgresql postgres-contrib python python-epydoc pkg-config sudo virtualenv \ 
     wget xvfb zlib1g-dev libgnutls28-dev python3-dev \ 
     r-base r-cran-testthat libxml2-dev pandoc cython bsdmainutils 

 # ruby 2.5 
 ( 
  set -e 
  mkdir -p ~/src 
  cd ~/src 
  wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz 
  tar xzf ruby-2.5.5.tar.gz 
  cd ruby-2.5.5 
  ./configure --disable-install-doc 
  make 
  sudo make install 
  sudo gem install bundler 
 ) 

 # go >= 1.12 
 sudo apt-get install golang-1.12 || \ 
 ( 
  set -e 
  wget https://storage.googleapis.com/golang/go1.12.5.linux-amd64.tar.gz 
  sudo tar -C /usr/local -xzf go1.12.5.linux-amd64.tar.gz 
  cd /usr/local/bin 
  sudo ln -s ../go/bin/* . 
 ) 

 # phantomjs 1.9.8 
 ( 
  set -e 
  PJS=phantomjs-1.9.8-linux-x86_64 
  wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2 
  sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2 
  sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/ 
 ) 

 # geckodriver 
 ( 
  set -e 
  GD=v0.24.0 
  wget -P /tmp https://github.com/mozilla/geckodriver/releases/download/$GD/geckodriver-$GD-linux64.tar.gz 
  sudo tar -C /usr/local/bin -xzf /tmp/geckodriver-$GD-linux64.tar.gz geckodriver 
 ) 

 # npm 
 ( 
  set -e 
  wget -O- https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.xz | sudo tar -C /usr/local -xJf - 
  sudo ln -s ../node-v6.11.2-linux-x64/bin/{node,npm} /usr/local/bin/ 
 ) 
 </pre> 

 Note: For ubuntu, virtualenv is python-virtualenv 

 h2. Get the arvados source tree and test scripts 

 <pre> 
 cd 
 git clone https://github.com/curoverse/arvados.git 
 </pre> 

 ...or, if you're a committer with your public key on our git server: 

 <pre> 
 cd 
 git clone git@git.curoverse.com:arvados.git 
 </pre> 

 h2. Start Postgres 

 _If you're running in a docker container_ you'll need to start Postgres manually: 

 <pre> 
 sudo /etc/init.d/postgresql start 
 </pre> 

 (If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.) 

 h2. Ensure entropy 

 If you're running in a VM, you might run out of entropy, which will make some tests run very slowly. The easiest solution is to install haveged. 

 <pre> 
 sudo apt-get install haveged 
 </pre> 

 h2. Setup groups 

 Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them. 

 h2. Create a Postgres user 

 Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed. 

 <pre> 
 newpw=`tr -cd a-zA-Z </dev/urandom |head -c32` 
 sudo -u postgres psql -c "create user arvados with superuser encrypted password '$newpw'" 
 mkdir ~/arvados-test-config 
 cat > ~/arvados-test-config/config.yml <<EOF 
 Clusters: 
   zzzzz: 
     PostgreSQL: 
       Connection: 
         client_encoding: utf8 
         dbname: arvados_test 
         host: localhost 
         password: $newpw 
         user: arvados 
 EOF 
 </pre> 

 

 h2. Run tests 

 <pre> 
 time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados CONFIGSRC=~/arvados-test-config CONFIGSRC=~/arvados-test-config/ 
 </pre> 

 During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on. 

 <pre> 
 mkdir -p ~/.cache/arvados-build 
 time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build 
 </pre>