Bug #21321
closed
R SDK install is flaky - stringi doesn't download completely
Description
It looks like one of our dependencies, stringi, is unusually large and doesn't always successfully download. I'm not sure what, if anything, we can do about that.
======= install sdk/R Loading required namespace: devtools Loading required namespace: roxygen2 Loading required namespace: markdown Loading required namespace: XML rlang (0.4.10 -> 1.1.2 ) [CRAN] cli (2.3.0 -> 3.6.2 ) [CRAN] vctrs (0.3.6 -> 0.6.5 ) [CRAN] stringi (1.5.3 -> 1.8.3 ) [CRAN] magrittr (2.0.1 -> 2.0.3 ) [CRAN] lifecycle (0.2.0 -> 1.0.4 ) [CRAN] glue (1.4.2 -> 1.6.2 ) [CRAN] ps (1.5.0 -> 1.7.5 ) [CRAN] processx (3.4.5 -> 3.8.3 ) [CRAN] callr (3.5.1 -> 3.7.3 ) [CRAN] R6 (2.5.0 -> 2.5.1 ) [CRAN] withr (2.4.1 -> 2.5.2 ) [CRAN] rprojroot (2.0.2 -> 2.0.4 ) [CRAN] pkgbuild (1.2.0 -> 1.4.3 ) [CRAN] fs (1.5.0 -> 1.6.3 ) [CRAN] desc (1.2.0 -> 1.4.3 ) [CRAN] crayon (1.4.0 -> 1.5.2 ) [CRAN] yaml (2.2.1 -> 2.3.8 ) [CRAN] xfun (0.20 -> 0.41 ) [CRAN] highr (0.8 -> 0.10 ) [CRAN] evaluate (0.14 -> 0.23 ) [CRAN] cpp11 (NA -> 0.4.7 ) [CRAN] xml2 (1.3.2 -> 1.3.6 ) [CRAN] stringr (1.4.0 -> 1.5.1 ) [CRAN] purrr (0.3.4 -> 1.0.2 ) [CRAN] pkgload (1.1.0 -> 1.3.3 ) [CRAN] knitr (1.31 -> 1.45 ) [CRAN] commonmark (1.7 -> 1.9.0 ) [CRAN] brew (1.0-6 -> 1.0-10) [CRAN] roxygen2 (7.1.1 -> 7.2.3 ) [CRAN] Installing 30 packages: rlang, cli, vctrs, stringi, magrittr, lifecycle, glue, ps, processx, callr, R6, withr, rprojroot, pkgbuild, fs, desc, crayon, yaml, xfun, highr, evaluate, cpp11, xml2, stringr, purrr, pkgload, knitr, commonmark, brew, roxygen2 Installing packages into ‘/home/jenkins/tmp/R_LIBS’ (as ‘lib’ is unspecified) trying URL 'http://cran.wustl.edu/src/contrib/rlang_1.1.2.tar.gz' Content type 'application/x-gzip' length 763521 bytes (745 KB) ================================================== downloaded 745 KB trying URL 'http://cran.wustl.edu/src/contrib/cli_3.6.2.tar.gz' Content type 'application/x-gzip' length 569771 bytes (556 KB) ================================================== downloaded 556 KB trying URL 'http://cran.wustl.edu/src/contrib/vctrs_0.6.5.tar.gz' Content type 'application/x-gzip' length 969066 bytes (946 KB) ================================================== downloaded 946 KB trying URL 'http://cran.wustl.edu/src/contrib/stringi_1.8.3.tar.gz' Content type 'application/x-gzip' length 11917752 bytes (11.4 MB) ======================================== downloaded 9.2 MB Error in download.file(url, destfile, method, mode = "wb", ...) : (converted from warning) downloaded length 9617408 != reported length 11917752 Error in download.packages(pkgs, destdir = tmpd, available = available, : (converted from warning) download of package ‘stringi’ failed Calls: <Anonymous> ... with_envvar -> force -> force -> i.p -> download.packages Execution halted ======= sdk/R install -- FAILED
Files
Updated by Brett Smith 11 months ago
- Has duplicate Bug #21498: CRAN package download failures causing test failures added
Updated by Brett Smith 11 months ago
- Related to Feature #21494: Get Java and R SDKs out of the critical path of main branch builds added
Updated by Brett Smith 11 months ago
This has been happening more frequently lately, it's getting aggravating.
Updated by Peter Amstutz 11 months ago
Thoughts:
- Are there mirrors we can use?
- Can we use a HTTP proxy to cache packages?
- Are some of these packages available in Debian and can be pre-installed on the worker?
Updated by Brett Smith 11 months ago
Peter Amstutz wrote in #note-4:
- Are there mirrors we can use?
https://cran.r-project.org/mirrors.html - Using https://cloud.r-project.org/ would probably be a good start. Basically all the mirrors are academic, which is kind of heartwarming but also, um, makes this problem make more sense. If I had to pick a single one I would expect to be most reliable, probably OSUOSL.
Updated by Peter Amstutz 11 months ago
- Target version changed from Future to Development 2024-02-28 sprint
Updated by Brett Smith 11 months ago
- Status changed from New to In Progress
21321-cran-mirror @ 30a07d66f784564de0817dd5fae7a82f5a3928b1 - developer-run-tests: #4044
This does every part of the ticket that can be done in the repo. Anything else would be an ops task.
The branch checklist: literally all it does is change one URL string, no change in scale, no compatibility considerations, there is no R style guide, there is no UI, I am trying to abbreviate this and I have already typed more than appears in the diff.
Updated by Brett Smith 11 months ago
Peter Amstutz wrote in #note-4:
- Are some of these packages available in Debian and can be pre-installed on the worker?
sdk/R/install_deps.R is a little script that checks if it can load each dependency, and installs it if not. Our dependencies are packaged:
$ for pkg in devtools roxygen2 knitr markdown xml; do apt search "r-cran-$pkg"; done r-cran-devtools/stable,now 2.4.5-1 all [installed] Tools to Make Developing R Packages Easier r-cran-roxygen2/stable,now 7.2.3-1 amd64 [installed] in-line documentation for GNU R r-cran-knitr/stable,now 1.42+dfsg-1 all [installed] GNU R package for dynamic report generation using Literate Programming r-cran-markdown/stable,now 1.5+dfsg-1 all [installed] GNU R package providing R bindings to the Sundown Markdown rendering library r-cran-xml/stable,now 3.99-0.13-1 amd64 [installed] GNU R package for XML parsing and generation
So yes, this is an option, at least on bookworm. However, if we do this, it makes our test strategy for the R SDK different from every other SDK, where we resolve and install dependencies as part of the test. It would make the tests a little less valuable. I would want to at least run for a little while with the cloud mirror before we go that route.
Updated by Tom Clegg 11 months ago
Brett Smith wrote in #note-8:
21321-cran-mirror @ 30a07d66f784564de0817dd5fae7a82f5a3928b1 - developer-run-tests: #4044
This LGTM, thanks.
Updated by Brett Smith 10 months ago
- Status changed from In Progress to Resolved
With the work done between this and #21494, I think we can declare victory on this: CRAN failures are no longer a regular hurdle for branch review and merging. We can revisit some of the other ideas in this ticket later if desired but I don't think there's any immediate need for them.