Support #5461: following manual install instructions on laptop - Arvados

Actions

Copy link

Support #5461

closed

following manual install instructions on laptop

Added by Albert Vilella about 9 years ago. Updated over 4 years ago.

Status:

Closed

Priority:

Normal

Assigned To:

Category:

Target version:

Due date:

Story points:

Description

Trying to install arvados locally on an Ubuntu laptop. Details below:

http://doc.arvados.org/install/install-manual-prerequisites.html (click Next bottom right)

sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql sudo wget zlib1g-dev

_{$ cd $HOME # (or wherever you want to install)}$ git clone https://github.com/curoverse/arvados.git

_{$ cd arvados/services/api}/arvados/services/api$ bundle install

My application.yml file looks something like this:

```
development: # Mandatory site secrets. See application.default.yml for more info.
secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
uuid_prefix: foobar
workbench_address: https://localhost:3031

production: # Mandatory site secrets. See application.default.yml for more info.
secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
uuid_prefix: barfoo
workbench_address: https://localhost:9899

workbench_address: https://workbench.foobar.arvadosapi.com
```

I created a keepstore.key file that is 128 characters long.

I followed the instructions here: http://doc.arvados.org/install/install-keepstore.html

Results:

```
[hpenvy15 ~/arvados/services/api] $ keepstore -permission-key-file="/home/avilella/arvados/services/api/keepstore.key"
2015/03/13 11:56:49 Keep started: pid 4460
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.ZeLkn1xGDR/keep
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.XJotjPWD5P/keep
2015/03/13 11:56:49 Error setting up arvados client Missing required environment variable ARVADOS_API_HOST
```

I went to https://console.developers.google.com and created a project and got the following (dummy values shown):

```
Project ID: smthg-standard-12345 Project Number: 123456789012
CLIENT SECRET 1234567890QWERTYadsfghxy
API KEY AIzaSyDCmGeVk5ZXdMbfTLNzOovnzsmSJ84RquE
```

I then continued with the database setup. This is what my database.yml file looks like:

```
#development:

adapter: postgresql
encoding: utf8
database: arvados_development
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost #
#test:
adapter: postgresql
encoding: utf8
database: arvados_test
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost

production:
adapter: postgresql
encoding: utf8
database: arvados_production
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost
```

I then tried to initialize the production database like this:

```
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
psql:/home/avilella/arvados/services/api/db/structure.sql:22: ERROR: must be owner of extension plpgsql
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Uuid not a valid Arvados uuid 'barfoo-tpzed-000000000000000'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:56:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `block in save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:208:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:41:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `block in create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `block in scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/scoping.rb:98:in `with_scope'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:126:in `first_or_create!'
/home/avilella/arvados/services/api/lib/current_api_client.rb:64:in `block in system_user'
/home/avilella/arvados/services/api/lib/current_api_client.rb:222:in `check_cache'
/home/avilella/arvados/services/api/lib/current_api_client.rb:57:in `system_user'
/home/avilella/arvados/services/api/app/models/database_seeds.rb:4:in `install'
/home/avilella/arvados/services/api/db/seeds.rb:5:in `<top (required)>'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `block in load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/railties-3.2.17/lib/rails/engine.rb:525:in `load_seed'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/railties/databases.rake:347:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)
```

Any ideas?

Actions

Copy link

Updated by Albert Vilella about 9 years ago

Same content as above but properly quoted with pre tag:

Trying to install arvados locally on an Ubuntu laptop. Details below:

http://doc.arvados.org/install/install-manual-prerequisites.html (click Next bottom right)

sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql sudo wget zlib1g-dev

_{$ cd $HOME # (or wherever you want to install)}$ git clone https://github.com/curoverse/arvados.git

_{$ cd arvados/services/api}/arvados/services/api$ bundle install

My application.yml file looks something like this:

development:
  # Mandatory site secrets. See application.default.yml for more info.
  secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  uuid_prefix: foobar
  workbench_address: https://localhost:3031

production:
  # Mandatory site secrets. See application.default.yml for more info.
  secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  uuid_prefix: barfoo
  workbench_address: https://localhost:9899
#  workbench_address: https://workbench.foobar.arvadosapi.com

I created a keepstore.key file that is 128 characters long.

I followed the instructions here: http://doc.arvados.org/install/install-keepstore.html

Results:

[hpenvy15 ~/arvados/services/api] $ keepstore -permission-key-file="/home/avilella/arvados/services/api/keepstore.key" 
2015/03/13 11:56:49 Keep started: pid 4460
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.ZeLkn1xGDR/keep
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.XJotjPWD5P/keep
2015/03/13 11:56:49 Error setting up arvados client Missing required environment variable ARVADOS_API_HOST

I went to https://console.developers.google.com and created a project and got the following (dummy values shown):

Project ID: smthg-standard-12345 Project Number: 123456789012
CLIENT SECRET    1234567890QWERTYadsfghxy
API KEY    AIzaSyDCmGeVk5ZXdMbfTLNzOovnzsmSJ84RquE

I then continued with the database setup. This is what my database.yml file looks like:

#development:                                                                                                                                                                                                        
#  adapter: postgresql
#  encoding: utf8
#  database: arvados_development
#  username: arvados
#  password: 12345678900qwertyasdfgxy
#  host: localhost
#
#test:
#  adapter: postgresql
#  encoding: utf8
#  database: arvados_test
#  username: arvados
#  password: 12345678900qwertyasdfgxy
#  host: localhost

production:
  adapter: postgresql
  encoding: utf8
  database: arvados_production
  username: arvados
  password: 12345678900qwertyasdfgxy
  host: localhost

I then tried to initialize the production database like this:

[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
psql:/home/avilella/arvados/services/api/db/structure.sql:22: ERROR:  must be owner of extension plpgsql
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Uuid not a valid Arvados uuid 'barfoo-tpzed-000000000000000'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:56:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `block in save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:208:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:41:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `block in create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `block in scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/scoping.rb:98:in `with_scope'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:126:in `first_or_create!'
/home/avilella/arvados/services/api/lib/current_api_client.rb:64:in `block in system_user'
/home/avilella/arvados/services/api/lib/current_api_client.rb:222:in `check_cache'
/home/avilella/arvados/services/api/lib/current_api_client.rb:57:in `system_user'
/home/avilella/arvados/services/api/app/models/database_seeds.rb:4:in `install'
/home/avilella/arvados/services/api/db/seeds.rb:5:in `<top (required)>'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `block in load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/railties-3.2.17/lib/rails/engine.rb:525:in `load_seed'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/railties/databases.rake:347:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)

Actions

Copy link

Updated by Peter Amstutz about 9 years ago

Hi avilella, your site prefix is too long. It needs to be 5 characters and "barfoo" is 6.

Actions

Copy link

Updated by Albert Vilella about 9 years ago

See below script as it is now

#/bin/bash

Install dependencies
sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
1. 1

Install Ruby and Bundler
pushd $HOME
mkdir -p src
pushd src
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
tar xzf ruby-2.1.5.tar.gz
cd ruby-2.1.5
./configure
make
sudo make install
sudo gem install bundler
popd
rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
1. 2

Install service API
pushd arvados/services/api
bundle install
cp i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
sudo mkdir -p /var/lib/arvados/git
sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
sed -i "s/bogus/gamma/g" config/application.yml
sed -n -l 1 's/$^.*blob_signing_key: $$.*$/\2/p' config/application.yml | uniq > keepstore.key
keepstore --permission-key-file=keepstore.key &
prefix=`arv --format=uuid user current | cut -d -f1`
"Site prefix is '$prefix'"
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk"
}
EOF

3
echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
sudo /usr/bin/apt-get update
sudo /usr/bin/apt-get install keepstore

ruby -e 'puts rand(2**128).to_s(36)'

TODO: hard coded here: d47b6bmmpmm5hceqwsda9at5x
sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
cp -i config/database.yml.sample config/database.yml
sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
RAILS_ENV=production bundle exec rake db:setup
sudo su postgres
createdb arvados_production -E UTF8 -O arvados
createdb arvados_development -E UTF8 -O arvados
exit
RAILS_ENV=production bundle exec rake db:structure:load
RAILS_ENV=production bundle exec rake db:seed
RAILS_ENV=development bundle exec rake db:structure:load
RAILS_ENV=development bundle exec rake db:seed

omniauth
cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
sed -i "s/^$APP_SECRET = $$.*$$/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
bundle exec rails server --port=3030
popd

Install Phusion wheezy
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
sudo apt-get install apt-transport-https ca-certificates
echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
sudo apt-get update
sudo apt-get install nginx-extras passenger

Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
sudo service nginx restart
echo "server {" >> /etc/nginx/sites-available/foo
echo " listen 80;" >> /etc/nginx/sites-available/foo
echo " root `pwd`/public;" >> /etc/nginx/sites-available/foo
echo " index index.html index.htm;" >> /etc/nginx/sites-available/foo
echo "}" >> /etc/nginx/sites-available/foo

ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/
popd

Install workbench
pushd arvados/apps/workbench
bundle install
sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz

apt-get install libsqlite3-dev
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
pushd ~/arvados/services/api
sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
popd
arvados_login_base: https://prefix_uuid.your.domain/login
arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1
popd

Install SSO
pushd $HOME
git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
pushd sso-devise-omniauth-provider
bundle install
RAILS_ENV=production bundle exec rake db:create
RAILS_ENV=production bundle exec rake db:migrate
cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
sed -i "s/^$.*secret_token = $.*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb

review http://doc.arvados.org/install/install-sso.html
https://console.developers.google.com/project/foo-arvados/apiui/credential#
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/environments/production.rb config/environments/development.rb
openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer
RAILS_ENV=production bundle exec rails console
popd
popd

keep
unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut d -f1` && echo "Site prefix is '$prefix'"
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk"
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"workbench.gamma.foo.com",
"service_port":443,
"service_ssl_flag":true,
"service_type":"proxy"
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" {"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
"script_version":"master",
"script":"hash",
"repository":"arvados"}
EOF

Install Perl SDK
sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

Install Python SDK
sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

TODO: extras at bottom of http://doc.arvados.org/install/install-manual-prerequisites.html

Actions

Copy link

Updated by Albert Vilella about 9 years ago

Sorry, again now with pre tags:


#/bin/bash
# Install dependencies
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
### 1

## Install Ruby and Bundler
pushd $HOME
mkdir -p src
pushd src
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
tar xzf ruby-2.1.5.tar.gz
cd ruby-2.1.5
./configure
make
sudo make install
sudo gem install bundler
popd
rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
### 2

# Install service API
pushd arvados/services/api
bundle install
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
sudo mkdir -p /var/lib/arvados/git
sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
sed -i "s/bogus/gamma/g" config/application.yml
sed -n -l 1 's/\(^.*blob_signing_key: \)\(.*\)/\2/p' config/application.yml | uniq > keepstore.key
keepstore --permission-key-file=keepstore.key &
prefix=`arv --format=uuid user current | cut -d- -f1`
"Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk" 
}
EOF

### 3
echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
  sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
  sudo /usr/bin/apt-get update
  sudo /usr/bin/apt-get install keepstore

ruby -e 'puts rand(2**128).to_s(36)'
# TODO: hard coded here: d47b6bmmpmm5hceqwsda9at5x
sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
cp -i config/database.yml.sample config/database.yml
sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
RAILS_ENV=production bundle exec rake db:setup
sudo su postgres
createdb arvados_production -E UTF8 -O arvados
createdb arvados_development -E UTF8 -O arvados
exit
RAILS_ENV=production bundle exec rake db:structure:load 
RAILS_ENV=production bundle exec rake db:seed
RAILS_ENV=development bundle exec rake db:structure:load 
RAILS_ENV=development bundle exec rake db:seed

### omniauth
cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
sed -i "s/^\(APP_SECRET = \)\(.*$\)/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
bundle exec rails server --port=3030
popd

## Install Phusion wheezy
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
sudo apt-get install apt-transport-https ca-certificates
echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
sudo apt-get update
sudo apt-get install nginx-extras passenger
# Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
sudo service nginx restart
echo "server {" >> /etc/nginx/sites-available/foo
echo "        listen   80;" >> /etc/nginx/sites-available/foo
echo "        root `pwd`/public;" >> /etc/nginx/sites-available/foo
echo "        index index.html index.htm;" >> /etc/nginx/sites-available/foo
echo "}" >> /etc/nginx/sites-available/foo

ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/
popd

## Install workbench
pushd arvados/apps/workbench
bundle install
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz

apt-get install libsqlite3-dev
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
pushd ~/arvados/services/api
sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
popd
arvados_login_base: https://prefix_uuid.your.domain/login
arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1
popd

## Install SSO
pushd $HOME
git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
pushd sso-devise-omniauth-provider
bundle install
RAILS_ENV=production bundle exec rake db:create
RAILS_ENV=production bundle exec rake db:migrate
cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
sed -i "s/^\(.*secret_token = \).*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb
# review http://doc.arvados.org/install/install-sso.html
# https://console.developers.google.com/project/foo-arvados/apiui/credential#
cp -i config/environments/production.rb.example config/environments/production.rb
# cp -i config/environments/production.rb config/environments/development.rb
openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer
RAILS_ENV=production bundle exec rails console
popd
popd

### keep
unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut -d- -f1` && echo "Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"keep0.gamma.foo.com",
 "service_port":25107,
 "service_ssl_flag":false,
 "service_type":"disk" 
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"workbench.gamma.foo.com",
 "service_port":443,
 "service_ssl_flag":true,
 "service_type":"proxy" 
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" 
{"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
 "script_version":"master",
 "script":"hash",
 "repository":"arvados"}
EOF

# Install Perl SDK
sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

# Install Python SDK
sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

# TODO: extras at bottom of http://doc.arvados.org/install/install-manual-prerequisites.html

Actions

Copy link

Updated by Davide Fiorentino lo Regio about 9 years ago

#/bin/bash
# Install dependencies
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
### 1

## Install Ruby and Bundler
pushd $HOME
    mkdir -p src
    pushd src
        wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
        tar xzf ruby-2.1.5.tar.gz
        cd ruby-2.1.5
        ./configure
        make
        sudo make install
        sudo gem install bundler
    popd
    rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
### 2

# Install service API
pushd arvados/services/api
    bundle install
    cp -i config/environments/production.rb.example config/environments/production.rb
    cp -i config/application.yml.example config/application.yml
    sudo mkdir -p /var/lib/arvados/git
    sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
    sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
    sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
    sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
    sed -i "s/bogus/gamma/g" config/application.yml
    sed -n -l 1 's/\(^.*blob_signing_key: \)\(.*\)/\2/p' config/application.yml | uniq > keepstore.key

    echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
     sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
     sudo /usr/bin/apt-get update
     sudo /usr/bin/apt-get install keepstore
    keepstore --permission-key-file=keepstore.key &

    prefix=`arv --format=uuid user current | cut -d- -f1`
    "Site prefix is '$prefix'" 
    read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
        {
         "service_host":"keep0.gamma.foo.com",
         "service_port":25107,
         "service_ssl_flag":false,
         "service_type":"disk" 
        }
    EOF

### 3
    ruby -e 'puts rand(2**128).to_s(36)'
    # d47b6bmmpmm5hceqwsda9at5x
    sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
    cp -i config/database.yml.sample config/database.yml
    sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
    RAILS_ENV=production bundle exec rake db:setup
    sudo su postgres
    createdb arvados_production -E UTF8 -O arvados
    createdb arvados_development -E UTF8 -O arvados
    exit
    RAILS_ENV=production bundle exec rake db:structure:load    
    RAILS_ENV=production bundle exec rake db:seed
    RAILS_ENV=development bundle exec rake db:structure:load    
    RAILS_ENV=development bundle exec rake db:seed

    cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
    sed -i "s/^\(APP_SECRET = \)\(.*$\)/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
    bundle exec rails server --port=3030
popd

## Install Phusion
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
    sudo apt-get install apt-transport-https ca-certificates
    echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
    sudo apt-get update
    sudo apt-get install nginx-extras passenger
    # Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
popd

## Install workbench
pushd arvados/apps/workbench
    bundle install
    sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz \
    libsqlite3-dev

    cp -i config/environments/production.rb.example config/environments/production.rb
    cp -i config/application.yml.example config/application.yml
    pushd ~/arvados/services/api
    sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
    popd

    # set these
    arvados_login_base: https://prefix_uuid.your.domain/login
    arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1

popd

## Install SSO
pushd $HOME
    git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
    pushd sso-devise-omniauth-provider
        bundle install
        RAILS_ENV=production bundle exec rake db:create
        RAILS_ENV=production bundle exec rake db:migrate
        RAILS_ENV=development bundle exec rake db:create
        RAILS_ENV=development bundle exec rake db:migrate
        cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
        sed -i "s/^\(.*secret_token = \).*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb
# review http://doc.arvados.org/install/install-sso.html
# https://console.developers.google.com/project/foo-arvados/apiui/credential#
        cp -i config/environments/production.rb.example config/environments/production.rb
        cp -i config/environments/production.rb config/environments/development.rb
        openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer

        # RAILS_ENV=production bundle exec rails console
        RAILS_ENV=production bundle exec rails console
    popd
popd

unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut -d- -f1` && echo "Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"keep0.gamma.foo.com",
 "service_port":25107,
 "service_ssl_flag":false,
 "service_type":"disk" 
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"workbench.gamma.foo.com",
 "service_port":443,
 "service_ssl_flag":true,
 "service_type":"proxy" 
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" 
{"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
 "script_version":"master",
 "script":"hash",
 "repository":"arvados"}
EOF

sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Status changed from New to Closed

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Arvados

Custom queries

Support #5461

following manual install instructions on laptop

Updated by Albert Vilella about 9 years ago

Updated by Peter Amstutz about 9 years ago

Updated by Albert Vilella about 9 years ago

Updated by Albert Vilella about 9 years ago

Updated by Davide Fiorentino lo Regio about 9 years ago

Updated by Peter Amstutz over 4 years ago