Arvados

https://cran.r-project.org/web/packages/googleAuthR/README.html

https://github.com/fmuhic/ArvadosSDK

Here's an example curl line to get collection contents via WebDAV:

curl -v -X PROPFIND -H "Authorization: OAuth2 4invqy35tf70t7hmvdc83ges8ug9cklhgqq1l8gj2cjn18teuq" https://collections.4xphq.arvadosapi.com/c=4xphq-4zz18-9d5b0qm4fgijeyi/_/

Hi Fuad:

I've created a branch for the R sdk work. The code should go in arvados/sdk/R

Instructions:

git clone git@git.curoverse.com:arvados.git
git checkout --track -b 11876-r-sdk origin/11876-r-sdk

Please use "git push" to synchronize your changes daily.

Your commit messages will need a Developer Certificate of Origin (DCO), that means the commit message needs to contain this line:

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Fuad:

Please put the "DCO" line after the main commit message, othewise it clutters up the commit log.

Fuad:

In order to find the keep-web (WebDAV) server for a cluster, you will need to look it up in the discovery document.

This is located on the API server at /discovery/v1/apis/arvados/v1/rest

This is a JSON document which stores configuration parameters for the cluster, which the SDK should fetch during initialization. Specifically the key keepWebServiceUrl is the base URL for the WebDAV service, for example "https://downloads.4xphq.arvadosapi.com/".

However, I just today noticed that this configuration value is missing. I have a branch to fix it, tracked in issue #12752. It will be merged soon but will probably require a few days to get deployed.

Hi Fuad,

Can you add some examples / test cases for doing various operations with the R SDK, and update the README with instructions for running them?

In addition to collections, we also need support for querying projects.

A project is simply a "group" record with the field "group_class" = "project"

http://doc.arvados.org/api/methods/groups.html

The REST API is otherwise the same as collections (same get/put/post/delete semantics and filters), so depending on how you implemented it, the quickest way to implement this may be a cut and paste of the collections code with the endpoint changed.

Code review comments: 11876-r-sdk @ 99bec637f6d4384a8d6f3c2cb27eb32d13c14f21

• Needs unit testing, and integration into run-test.sh. The run-tests.sh framework will provide a test server for R SDK tests to communicate with. Recommended test framework for R seems to be "testthat"

https://github.com/r-lib/testthat

• This should be callable without any filters:
  

  arv$listCollections()
  Error in names(filters) <- c("collection") : 
  

• I'm getting an error creating a collection:
  

  collection <- Collection$new(arv, "c97qk-4zz18-klkpkv1ign5kcdu")
  Error in curl::curl_fetch_memory(url, h) : <url> malformed
  

  
  On further research, it looks like the upstream cluster has "keepWebServiceUrl" misconfigured, so this isn't really due to a bug in your code (although the error reporting should be better.)

• I think it is later, so you can remove this comment?
  

          #Todo(Fudo): Hardcoded credentials to WebDAV server. Remove them later
  

• You should add something similar to the "listAll" function in the Python SDK. This uses the "offset" query parameter to get all items when the number of items is > maximum API limit.
  https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/sdk/python/arvados/util.py#L375

• Instead of having two different modes for Collection$add, I suggest renaming the first one to "create" and having it return an ArvadosFile:
  

  collectionFile <- collection$create("main.cpp", "cpp/src/")
  

  
  Then the "add" method only does one thing.

• Instead of "collection$getFileContent()" I suggest calling it "getFileListing()"

It looks like the R concept of a generalized IO object is called a "connection":

https://www.rdocumentation.org/packages/base/versions/3.4.3/topics/connections

https://stackoverflow.com/questions/30445875/what-exactly-is-a-connection-in-r

Figure out how to expose ArvadosFile as a "connection" so that it can be used directly for loading and saving data.

On packaging: http://r-pkgs.had.co.nz/

note to self

.libPaths( "/var/lib/arvados/Rstuff")
devtools::install_dev_deps()

I pushed a commit to 11876-r-sdk which adds support for running the R SDK unit tests as part of the overall Arvados test suite. I also noticed several tests are failing, can you confirm?

                   ********** Running sdk/R tests **********

> results <- devtools::test()
Loading ArvadosR
Loading required package: testthat
Testing ArvadosR
Arvados API: .........................
ArvadosFile: ........................
Collection: .........................
CollectionTree: ...............
Http Parser: .....
Http Request: ...
REST service: ........1....2.......3.....4.....5.....................................
Subcollection: .................................
Utility function: ..........

Failed -------------------------------------------------------------------------
1. Error: getResource raises exception if response contains errors field (@test-RESTService.R#76) 
is.character(regexp) is not TRUE
1: expect_that(REST$getResource("collections", resourceUUID), throws_error(404)) at /usr/src/arvados/sdk/R/tests/testthat/test-RESTService.R:76
2: condition(object)
3: expect_error(x, regexp, ...)
4: expect_match(error$message, regexp, ..., info = info)
5: stopifnot(is.character(regexp), length(regexp) == 1)
6: stop(sprintf(ngettext(length(r), "%s is not TRUE", "%s are not all TRUE"), ch), call. = FALSE, 
       domain = NA)

2. Error: listResources raises exception if response contains errors field (@test-RESTService.R#117) 
is.character(regexp) is not TRUE
1: expect_that(REST$listResources("collections"), throws_error(404)) at /usr/src/arvados/sdk/R/tests/testthat/test-RESTService.R:117
2: condition(object)
3: expect_error(x, regexp, ...)
4: expect_match(error$message, regexp, ..., info = info)
5: stopifnot(is.character(regexp), length(regexp) == 1)
6: stop(sprintf(ngettext(length(r), "%s is not TRUE", "%s are not all TRUE"), ch), call. = FALSE, 
       domain = NA)

3. Error: deleteCollection raises exception if response contains errors field (@test-RESTService.R#190) 
is.character(regexp) is not TRUE
1: expect_that(REST$deleteResource("collections", resourceUUID), throws_error(404)) at /usr/src/arvados/sdk/R/tests/testthat/test-RESTService.R:190
2: condition(object)
3: expect_error(x, regexp, ...)
4: expect_match(error$message, regexp, ..., info = info)
5: stopifnot(is.character(regexp), length(regexp) == 1)
6: stop(sprintf(ngettext(length(r), "%s is not TRUE", "%s are not all TRUE"), ch), call. = FALSE, 
       domain = NA)

4. Error: updateResource raises exception if response contains errors field (@test-RESTService.R#238) 
is.character(regexp) is not TRUE
1: expect_that(REST$updateResource("collections", resourceUUID, newResourceContent), 
       throws_error(404)) at /usr/src/arvados/sdk/R/tests/testthat/test-RESTService.R:238
2: condition(object)
3: expect_error(x, regexp, ...)
4: expect_match(error$message, regexp, ..., info = info)
5: stopifnot(is.character(regexp), length(regexp) == 1)
6: stop(sprintf(ngettext(length(r), "%s is not TRUE", "%s are not all TRUE"), ch), call. = FALSE, 
       domain = NA)

5. Error: createResource raises exception if response contains errors field (@test-RESTService.R#288) 
is.character(regexp) is not TRUE
1: expect_that(REST$createResource("collections", resourceContent), throws_error(404)) at /usr/src/arvados/sdk/R/tests/testthat/test-RESTService.R:288
2: condition(object)
3: expect_error(x, regexp, ...)
4: expect_match(error$message, regexp, ..., info = info)
5: stopifnot(is.character(regexp), length(regexp) == 1)
6: stop(sprintf(ngettext(length(r), "%s is not TRUE", "%s are not all TRUE"), ch), call. = FALSE, 
       domain = NA)

DONE ===========================================================================
> any_error <- any(as.data.frame(results)$error)
> if (any_error) {
+   q("no", 1)
+ } else {
+   q("no", 0)
+ }

             ********** !!!!!! sdk/R tests FAILED !!!!!! **********

Tests failing on CI: https://ci.curoverse.com/job/developer-run-tests-remainder/581/consoleFull

for the record, if dependencies are there this will work:

> install.packages('http://r.arvados.org/ArvadosR_0.0.3.tar.gz', repos = NULL, type="source")

this is a small step towards getting it into Bioconductor Package system