Story #13076

R SDK autogenerate API to access all Arvados object types

Added by Peter Amstutz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
03/22/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Release relationship:
Auto

Subtasks

Task #13088: Review 13076-r-autogen-apiResolvedPeter Amstutz

Associated revisions

Revision f98e0188
Added by Fuad Muhic over 1 year ago

Merge branch '13076-r-autogen-api'

refs #13076

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <>

Revision 5af6a8e3 (diff)
Added by Peter Amstutz over 1 year ago

Fix R SDK online doc generation refs #13076

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz over 1 year ago

  • Target version changed from Arvados Future Sprints to 2018-02-28 Sprint

#3 Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Fuad Muhic

#4 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-02-28 Sprint to 2018-03-14 Sprint

#5 Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2018-03-14 Sprint to 2018-03-28 Sprint

#6 Updated by Peter Amstutz over 1 year ago

Building R documentation isn't working:

2018-03-23_17:29:17.62809 > pkgdown::build_site()
2018-03-23_17:29:17.62811 Error in switch(fun, `<-` = paste0(fun_name(expr[[2]]), "<-"), S3method = ,  : 
2018-03-23_17:29:17.62811   EXPR must be a length 1 vector
2018-03-23_17:29:17.62811 Calls: <Anonymous> ... <Anonymous> -> .Call -> .f -> <Anonymous> -> .Call -> .f
2018-03-23_17:29:17.62811 Execution halted

Unfortunately the error is pretty cryptic.

#7 Updated by Peter Amstutz over 1 year ago

Could you document how to run autoGenAPI.R ?

#8 Updated by Peter Amstutz over 1 year ago

It looks like getProject(), listProjects() and so forth are obsoleted by the autogenerated API (that's good). However, it looks like instead of turning them into stubs that call groups.get() and groups.list(), they are gone entirely, but still documented in README.Rmd.

Also, README.Rmd is lacking any documentation on using the new autogenerated API.

#9 Updated by Peter Amstutz over 1 year ago

  • The methods "index", "show" and "destroy" are aliases for the preferred names "list", "get" and "delete". We should probably fix the discovery document so it doesn't publish redundant methods, but in the mean time I suggest filtering out "index", "show" and "destroy".
  • For development and testing, we have an environment variable ARVADOS_API_HOST_INSECURE which specifies that the HTTP client should not try to validate the server's TLS certificate (this allows us to use self-signed certificates generated on the fly by the test suite). Please see if you can support this flag in the R SDK, it is probably a configuration option to httr or curl.

#10 Updated by Peter Amstutz over 1 year ago

I haven't tried every function, but spot-checking various autogenerated list(), get(), delete(), create() and update() everything works well so far!

#11 Updated by Peter Amstutz over 1 year ago

Fuad: 13076-r-autogen-api back to you with above comments.

#12 Updated by Fuad Muhic over 1 year ago

Methods projects.get, projects.create, projects.delete, projects.update and projects.list are now defined on Arvados class (they wrap groups.list ... by adding "group_class" attribute where necessary).

Methods that end on "index", "show" and "destroy" are removed from auto-generated Arvados class (reduced file by more than 2000 lines).

When environment variable ARVADOS_API_HOST_INSECURE is set to TRUE all http request will contain ssl_verifypeer options field set to FALSE.

README.Rmd is updated to reflect changes made in Arvados class.

autoGenAPI.R file contains one public function called generateAPI which doesn't receive arguments. This function,
when called, will generate Arvados.R file which contains main Arvados class. My idea was to generate Arvados.R file by calling generateAPI and put that file into ./R folder of project. Then we need to call document function (defined inside devtools package) to generate documentation. After that we can build package (without autoGenAPI.R file). generateAPI can be modified to take discovery document and/or file path, where to put Arvados.R file, as arguments.

Currently generateAPI is exported as public function (since I don't know what exactly to do with it and I need to use it) and can be called directly.

Most of the documentation that correspond to method arguments is missing from discovery document. So executing, for example, ?jobs.list will not give a very clear picture what method arguments represent. Also documentation for methods is preformatted. For example, documentation for collections.list looks like this:

List Collections.

The list method returns a
&lt;a href="/api/resources.html"&gt;resource list&lt;/a&gt; of
matching Collections. For example:
                   {
                    "kind":"arvados#collectionList",
                    "etag":"",
                    "self_link":"",
                    "next_page_token":"",
                    "next_link":"",
                    "items":[
                       ...
                    ],
                    "items_available":745,
                    "_profile":{
                     "request_time":0.157236317
                    }
                    

Not sure If it's possible to can put this in R documentation as description.

Most of the class descriptions (example discoverDocument.schemas.Collection.properties) contain duplicated uuid field.

#13 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-03-28 Sprint to 2018-04-11 Sprint

#14 Updated by Tom Morris over 1 year ago

  • Target version changed from 2018-04-11 Sprint to 2018-04-25 Sprint

#15 Updated by Peter Amstutz over 1 year ago

Currently the main blocker for merge is the online HTML documentation generation. pkgdown is failing as noted in https://dev.arvados.org/issues/13076#note-6 so either we need to determine why pkgdown is failing (and fix it) or stop using pkgdown and do something else. I would be fine with only processing the contents of README.Rmd and not worrying about method-level documentation. I think processing Rmd files requires using "knitR"

#16 Updated by Peter Amstutz over 1 year ago

By the way, the current documentation generated is implemented in arvados/doc/Rakefile.

#17 Updated by Fuad Muhic over 1 year ago

Peter Amstutz wrote:

By the way, the current documentation generated is implemented in arvados/doc/Rakefile.

I added createDoc.R script which can be run by $Rscript createDoc.R README.Rmd README.html.
I requires knitr and markdown packages. Could you update Rakefile?

#18 Updated by Peter Amstutz over 1 year ago

Alright, I spent some time with it and now it autogenerates mostly reasonable online documentation including both the README and function/method documentation. I pushed to the 13076-r-autogen-api branch.

Please go ahead and merge to master.

#19 Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to Resolved

#20 Updated by Tom Morris about 1 year ago

  • Release set to 13

Also available in: Atom PDF