Story #16264

Handle R SDK dependencies better

Added by Peter Amstutz 4 months ago. Updated 6 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Our build was broken for a while because Rcpp (a transitive dependency of a package we depend on) had a new release that is broken on R 3.3.

We were running our tests on Debian 9, which ships with R 3.3.

A fix was committed to upstream Rcpp, so the problem resolved itself, but we should have a more stable strategy for R dependencies.

Recompiling R dependencies every time is also very expensive.

Options

Use Debian packages

When we did the R SDK originally (about two years ago) most of the packages we needed were not in Debian. Since then, it appears they have been added to Debian 10, and are also available in a Debian 9 backport:

https://packages.debian.org/search?keywords=r-cran-devtools&searchon=names&suite=all&section=all

Of course this requires targeting Debian for the test builds, but we do that already. To get the maximum benefit it also requires that all the R packages we need are in Debian.

Pinned install

R doesn't have anything like a Gemfile.lock or "pip freeze". However someone has created a tool that computes an install plan so each package is installed at exactly each version we want.

https://www.ccdatalab.org/blog/2019/8/9/pinning-transitive-r-dependencies-for-fun-and-reproducible-builds

This works on any R install. The drawback is that we still end up compiling the whole world every time.


Related issues

Related to Arvados - Bug #16267: arvbox build uses arvados-server installIn Progress

Related to Arvados - Story #16053: [boot] subcommand to install/update dev and runtime dependenciesResolved03/31/2020

Related to Arvados - Bug #16586: ArvadosR: Dependency package(s) 'XML' not available.Resolved

History

#1 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 4 months ago

  • Related to Bug #16267: arvbox build uses arvados-server install added

#3 Updated by Peter Amstutz 4 months ago

Currently I think the solution is for #16053 to include installing the appropriate R packages from Debian 10.

#4 Updated by Peter Amstutz 4 months ago

  • Related to Story #16053: [boot] subcommand to install/update dev and runtime dependencies added

#5 Updated by Peter Amstutz 4 months ago

  • Category set to SDKs

#6 Updated by Peter Amstutz 4 months ago

  • Target version changed from 2020-04-08 Sprint to 2020-04-22

#7 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-04-22 to 2020-05-06 Sprint

#8 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-05-06 Sprint to 2020-05-20 Sprint

#9 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2020-05-20 Sprint to 2020-06-03 Sprint

#10 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint

#11 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint

#12 Updated by Peter Amstutz 26 days ago

  • Target version changed from 2020-07-01 Sprint to 2020-07-15

#13 Updated by Peter Amstutz 12 days ago

  • Target version changed from 2020-07-15 to 2020-07-29 Sprint

#14 Updated by Peter Amstutz 6 days ago

  • Related to Bug #16586: ArvadosR: Dependency package(s) 'XML' not available. added

#15 Updated by Peter Amstutz 6 days ago

  • Description updated (diff)

Also available in: Atom PDF