Arvados

@peter I'm not entirely sure about what needs to be done here. currently we have config.defaults.yml -> ../../../lib/config/config.defaults.yml it should be removed, right? and you want the file config.defaults.yml copied by the deb/rpm packaging exactly where?

Fernando Monserrat wrote:

@peter I'm not entirely sure about what needs to be done here. currently we have config.defaults.yml -> ../../../lib/config/config.defaults.yml it should be removed, right? and you want the file config.defaults.yml copied by the deb/rpm packaging exactly where?

I want the packaging to take the contents of ../../../lib/config/config.defaults.yml and package a regular file at config.defaults.yml, because if the packaging includes a symlink, it won't be valid when the package is installed.

Let's see if I understand...

we need config.defaults.yml to be a regular file in the package (without marking it as a config file so it gets overwritten)

This is what I see in the dpkg:

⌁ [nico:/tmp] $ wget http://apt.arvados.org/pool/xenial-dev/main/a/arvados-api-server/arvados-api-server_1.3.1.20190314194218-8_amd64.deb
⌁ [nico:/tmp] $ dpkg -c arvados-api-server_1.3.1.20190314194218-8_amd64.deb | grep config|grep yml
-rw-r--r-- 0/0           23739 2019-03-14 16:47 ./var/www/arvados-api/current/config/application.default.yml
-rw-r--r-- 0/0             807 2019-03-14 16:47 ./var/www/arvados-api/current/config/database.yml.example
-rw-r--r-- 0/0             311 2019-03-14 16:47 ./var/www/arvados-api/current/config/locales/en.yml
-rw-r--r-- 0/0            1588 2019-03-14 16:47 ./var/www/arvados-api/current/config/application.yml.example
-rw-r--r-- 0/0             213 2019-03-14 17:11 ./var/www/arvados-api/current/vendor/cache/themes_for_rails-61154877047d/test/dummy_app/config/locales/en.yml
-rw-r--r-- 0/0             469 2019-03-14 17:11 ./var/www/arvados-api/current/vendor/cache/themes_for_rails-61154877047d/test/dummy_app/config/database.yml

should this be /etc/arvados/config.defaults.yml? or somewhere else? (/var/www/arvados-api/current/config/config.defaults.yml for example)

Peter, FPM is not as easy to make this kind of changes, so having explicitly what you want to see in the deb (or rpm) package will help us to trace back what do we have to do here.

Nico César wrote:

Let's see if I understand...

we need config.defaults.yml to be a regular file in the package (without marking it as a config file so it gets overwritten)

This is what I see in the dpkg:

[...]

should this be /etc/arvados/config.defaults.yml? or somewhere else? (/var/www/arvados-api/current/config/config.defaults.yml for example)

/var/www/arvados-api/current/config/config.defaults.yml

1. We want a unified defaults file that every component uses
2. For development, we want to be able to run the API server in-place, so it symlinks to the common defaults file within the source tree, and it is just generally easier if it can look for it relative to the rails root
3. We could also have a separate package containing just the defaults file, but tom's thought is that would complicate upgrades because every package would have to depend on the defaults file package
4. We can search other places, if there's a way with fpm to specify "copy file X to location Y" and ignore symlinks

Peter, FPM is not as easy to make this kind of changes, so having explicitly what you want to see in the deb (or rpm) package will help us to trace back what do we have to do here.

The goal is to get the file in the source tree at arvados/lib/config/config.defaults.yml to appear inside /var/www/arvados-api/current/config/ in the packaging (it doesn't even have to be called config.defaults.yml, so long as the name is consistent.)

Plan is to remove the link and then copy the file from arvados/lib/config/config.defaults.yml to /var/www/arvados-api/current/config/ when it deploys the API package and this config file will be used by sso also .
I've been looking at the scripts and it will not be as easy as adding a parameter on fpm, so far looking at the scripts i think this is achieved here https://github.com/curoverse/arvados/blob/master/build/rails-package-scripts/postinst.sh#L112-L125 , but i need to be sure yet about this .

Fernando Monserrat wrote:

Plan is to remove the link and then copy the file from arvados/lib/config/config.defaults.yml to /var/www/arvados-api/current/config/ when it deploys the API package and this config file will be used by sso also .
I've been looking at the scripts and it will not be as easy as adding a parameter on fpm, so far looking at the scripts i think this is achieved here https://github.com/curoverse/arvados/blob/master/build/rails-package-scripts/postinst.sh#L112-L125 , but i need to be sure yet about this .

https://fpm.readthedocs.io/en/latest/source/dir.html

Does rails packaging use the "dir" package type? Can we do something like this?

fpm -s dir ... $WORKSPACE/lib/config/config.defaults.yml=/var/www/arvados-api/current/config/config.defaults.yml

@Peter that is the easy way to do it using fpm and I wish we could use this but it is not the case for our scripts stack that build our packages. or maybe i dont find an easy way to do it
https://github.com/curoverse/arvados/blob/master/build/run-library.sh#L336

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Peter Amstutz wrote:

Ward Vandewege wrote:

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Awesome, thank you!

One thought, if we want to avoid munging the working directory, couldn't we (a) exclude the symlink (b) tell fpm to copy the file to a different name in the package (c) have arvados_config.rb check for both files?

b) + c) could work, yes. a) doesn't work (that was the first thing I tried)

Ward Vandewege wrote:

Peter Amstutz wrote:

Ward Vandewege wrote:

I've pushed b076649ae4540da0d83c8cbce633a58f5802ae37 to fix this.

Awesome, thank you!

One thought, if we want to avoid munging the working directory, couldn't we (a) exclude the symlink (b) tell fpm to copy the file to a different name in the package (c) have arvados_config.rb check for both files?

b) + c) could work, yes. a) doesn't work (that was the first thing I tried)

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

Peter Amstutz wrote:

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

fpm doesn't let you a) exclude the existing file/symlink and b) add a file with the same name to the package.

But, you were suggesting to do a + b + c from note #13, I see what you mean now. That would work. I didn't want to add yet another lookup path... seems more messy.

Ward Vandewege wrote:

Peter Amstutz wrote:

I'm confused, doesn't your fix work by renaming the symlink then adding it to the exclude list?

fpm doesn't let you a) exclude the existing file/symlink and b) add a file with the same name to the package.

Right but you could add the file with a different name, and the arvados_config.rb code would know to look for the file under two different possible names.