Project

General

Profile

Actions

Task #15042

closed

Story #13996: [API] Migrate RailsAPI to new cluster config file

API server packaging includes its own copy of config.defaults.yml

Added by Peter Amstutz over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
04/04/2019
Due date:
% Done:

0%

Estimated time:

Description

13996-new-api-config

There's a symlink:

arvados/services/api/config/config.defaults.yml -> ../../../lib/config/config.defaults.yml

This gets included in the debian package as a symlink, where it is no longer point to a valid target. The deb/rpm packaging should make a copy of config.defaults.yml as a regular file.

Actions #1

Updated by Peter Amstutz over 3 years ago

  • Tracker changed from Bug to Task
  • Parent task set to #13996
Actions #2

Updated by Peter Amstutz over 3 years ago

  • Description updated (diff)
Actions #3

Updated by Fernando Monserrat over 3 years ago

@peter I'm not entirely sure about what needs to be done here. currently we have config.defaults.yml -> ../../../lib/config/config.defaults.yml it should be removed, right? and you want the file config.defaults.yml copied by the deb/rpm packaging exactly where?

Actions #4

Updated by Peter Amstutz over 3 years ago

Fernando Monserrat wrote:

@peter I'm not entirely sure about what needs to be done here. currently we have config.defaults.yml -> ../../../lib/config/config.defaults.yml it should be removed, right? and you want the file config.defaults.yml copied by the deb/rpm packaging exactly where?

I want the packaging to take the contents of ../../../lib/config/config.defaults.yml and package a regular file at config.defaults.yml, because if the packaging includes a symlink, it won't be valid when the package is installed.

Actions #5

Updated by Nico César over 3 years ago

Let's see if I understand...

we need config.defaults.yml to be a regular file in the package (without marking it as a config file so it gets overwritten)

This is what I see in the dpkg:

⌁ [nico:/tmp] $ wget http://apt.arvados.org/pool/xenial-dev/main/a/arvados-api-server/arvados-api-server_1.3.1.20190314194218-8_amd64.deb
⌁ [nico:/tmp] $ dpkg -c arvados-api-server_1.3.1.20190314194218-8_amd64.deb | grep config|grep yml
-rw-r--r-- 0/0           23739 2019-03-14 16:47 ./var/www/arvados-api/current/config/application.default.yml
-rw-r--r-- 0/0             807 2019-03-14 16:47 ./var/www/arvados-api/current/config/database.yml.example
-rw-r--r-- 0/0             311 2019-03-14 16:47 ./var/www/arvados-api/current/config/locales/en.yml
-rw-r--r-- 0/0            1588 2019-03-14 16:47 ./var/www/arvados-api/current/config/application.yml.example
-rw-r--r-- 0/0             213 2019-03-14 17:11 ./var/www/arvados-api/current/vendor/cache/themes_for_rails-61154877047d/test/dummy_app/config/locales/en.yml
-rw-r--r-- 0/0             469 2019-03-14 17:11 ./var/www/arvados-api/current/vendor/cache/themes_for_rails-61154877047d/test/dummy_app/config/database.yml

should this be /etc/arvados/config.defaults.yml? or somewhere else? (/var/www/arvados-api/current/config/config.defaults.yml for example)

Peter, FPM is not as easy to make this kind of changes, so having explicitly what you want to see in the deb (or rpm) package will help us to trace back what do we have to do here.

Actions #6

Updated by Peter Amstutz over 3 years ago

Nico César wrote:

Let's see if I understand...

we need config.defaults.yml to be a regular file in the package (without marking it as a config file so it gets overwritten)

This is what I see in the dpkg:

[...]

should this be /etc/arvados/config.defaults.yml? or somewhere else? (/var/www/arvados-api/current/config/config.defaults.yml for example)

/var/www/arvados-api/current/config/config.defaults.yml

  1. We want a unified defaults file that every component uses
  2. For development, we want to be able to run the API server in-place, so it symlinks to the common defaults file within the source tree, and it is just generally easier if it can look for it relative to the rails root
  3. We could also have a separate package containing just the defaults file, but tom's thought is that would complicate upgrades because every package would have to depend on the defaults file package
  4. We can search other places, if there's a way with fpm to specify "copy file X to location Y" and ignore symlinks

Peter, FPM is not as easy to make this kind of changes, so having explicitly what you want to see in the deb (or rpm) package will help us to trace back what do we have to do here.

The goal is to get the file in the source tree at arvados/lib/config/config.defaults.yml to appear inside /var/www/arvados-api/current/config/ in the packaging (it doesn't even have to be called config.defaults.yml, so long as the name is consistent.)

Actions #7

Updated by Peter Amstutz over 3 years ago

  • Project changed from 40 to Arvados
  • Status changed from New to In Progress
  • Start date set to 04/04/2019
Actions #8

Updated by Fernando Monserrat over 3 years ago

Plan is to remove the link and then copy the file from arvados/lib/config/config.defaults.yml to /var/www/arvados-api/current/config/ when it deploys the API package and this config file will be used by sso also .
I've been looking at the scripts and it will not be as easy as adding a parameter on fpm, so far looking at the scripts i think this is achieved here https://github.com/curoverse/arvados/blob/master/build/rails-package-scripts/postinst.sh#L112-L125 , but i need to be sure yet about this .

Actions #9

Updated by Peter Amstutz over 3 years ago

Fernando Monserrat wrote:

Plan is to remove the link and then copy the file from arvados/lib/config/config.defaults.yml to /var/www/arvados-api/current/config/ when it deploys the API package and this config file will be used by sso also .
I've been looking at the scripts and it will not be as easy as adding a parameter on fpm, so far looking at the scripts i think this is achieved here https://github.com/curoverse/arvados/blob/master/build/rails-package-scripts/postinst.sh#L112-L125 , but i need to be sure yet about this .

https://fpm.readthedocs.io/en/latest/source/dir.html

Does rails packaging use the "dir" package type? Can we do something like this?

fpm -s dir ... $WORKSPACE/lib/config/config.defaults.yml=/var/www/arvados-api/current/config/config.defaults.yml
Actions #10

Updated by Fernando Monserrat over 3 years ago

@Peter that is the easy way to do it using fpm and I wish we could use this but it is not the case for our scripts stack that build our packages. or maybe i dont find an easy way to do it
https://github.com/curoverse/arvados/blob/master/build/run-library.sh#L336

Actions #11

Updated by Ward Vandewege over 3 years ago

Actions #12

Updated by Ward Vandewege over 3 years ago

  • Status changed from In Progress to Feedback
Actions #13

Updated by Peter Amstutz over 3 years ago

  • Assigned To changed from Fernando Monserrat to Ward Vandewege

Ward Vandewege wrote:

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Awesome, thank you!

One thought, if we want to avoid munging the working directory, couldn't we (a) exclude the symlink (b) tell fpm to copy the file to a different name in the package (c) have arvados_config.rb check for both files?

Actions #14

Updated by Ward Vandewege over 3 years ago

Peter Amstutz wrote:

Ward Vandewege wrote:

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Awesome, thank you!

One thought, if we want to avoid munging the working directory, couldn't we (a) exclude the symlink (b) tell fpm to copy the file to a different name in the package (c) have arvados_config.rb check for both files?

b) + c) could work, yes. a) doesn't work (that was the first thing I tried)

Actions #15

Updated by Peter Amstutz over 3 years ago

Ward Vandewege wrote:

Peter Amstutz wrote:

Ward Vandewege wrote:

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Awesome, thank you!

One thought, if we want to avoid munging the working directory, couldn't we (a) exclude the symlink (b) tell fpm to copy the file to a different name in the package (c) have arvados_config.rb check for both files?

b) + c) could work, yes. a) doesn't work (that was the first thing I tried)

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

Actions #16

Updated by Ward Vandewege over 3 years ago

Peter Amstutz wrote:

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

fpm doesn't let you a) exclude the existing file/symlink and b) add a file with the same name to the package.

But, you were suggesting to do a + b + c from note #13, I see what you mean now. That would work. I didn't want to add yet another lookup path... seems more messy.

Actions #17

Updated by Peter Amstutz over 3 years ago

Ward Vandewege wrote:

Peter Amstutz wrote:

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

fpm doesn't let you a) exclude the existing file/symlink and b) add a file with the same name to the package.

Right but you could add the file with a different name, and the arvados_config.rb code would know to look for the file under two different possible names.

Actions #18

Updated by Peter Amstutz over 3 years ago

  • Status changed from Feedback to Resolved
  • Remaining (hours) set to 0.0
Actions

Also available in: Atom PDF