Arvados

h1. Hacking Ruby SDK

{{toc}}

h2. Overview

|@lib/arvados.rb@          |Google API client.  Built from the discovery document.|

|@lib/arvados/keep.rb@     |Keep client functions.  Parse locators and manifests.|

h2. Running tests

Run @bundle exec rake test@.

Note: We have never (more or less) tested the Google API client.  The test infrastructure currently has no way to test against a development API server, like other SDKs do.  If you want to add this, be aware that you could create a knotty dependency loop: the API server needs the Ruby SDK to run, and the Ruby SDK needs to run the API server for tests.  If you can, it would be better to test the API client with mock responses.  See @test_api.py@ in the Python SDK for an illustration of this approach.

h2. Building and testing gems

As of this writing, the API server and Workbench use the Ruby SDK.  When you modify the SDK, you should build and install the Gem, and then test our servers that use it, to make sure you haven't accidentally introduced a bug.  To do that:

<pre>

sdk/ruby$ # Commit your changes to the Gem code first, then:

sdk/ruby$ gem build arvados.gemspec

sdk/ruby$ gem install arvados-0.1.YYYYMMDDHHMMSS.gem

# Then, for each server you're testing:

server$ bundle update arvados

server$ bundle exec rake test

</pre>

h2. Arranging a build

When new code is pushed to git, Jenkins builds the latest SDK Gem and runs all the tests with it.  However, it will crash if a server's Gemfile refers to a nonexistent Gem version.  To ensure a successful test, you must make sure that Gemfiles in your push refers to Gem versions that have been published.  There are two main ways to do this.

h3. Separate development approach

This approach requires you to think about Arvados components as separate pieces of software, but it's easier to wrap your head around.  Change the SDK separately from other components: make your changes, test them, push them for review, and merge them to master.  After Jenkins builds them, you can put the new version number in Gemfiles for server components that need it, in a later branch or commit.

h3. One-shot approach

If you're in a rush or it just doesn't make sense to have two separate branches reviewed, you can make a single push to master that builds new Gems and refers to them in Gemfiles.  You do this by making a commit with a known timestamp, effectively controlling the Gem version Jenkins builds so you can refer to it in that same commit.  The process is:

<pre>

$ gem_ts=$(date '+%Y-%m-%dT%H:%M:%S')

$ echo 0.1.$(echo "$gem_ts" | tr -dc 0-9)

# That prints your Gem version.  For each server that needs it:

    server$ # Edit the server's Gemfile.

    server$ bundle update arvados

    server$ git add Gemfile Gemfile.lock

# After you've updated all necessary servers:

$ GIT_COMMITTER_DATE=$gem_ts git commit

</pre>

In order for this to work, your new commit must be the most recent commit when it's pushed to the git server.  If new commits are pushed while you're working on this, discard the commit you made and start over.

h2. Use notes

When calling client.execute() on the Google API client, be careful to use @:body_object@ (which serializes the the provided hash to JSON) instead of @:body@ (which is the raw text.)

The @each_*@ methods in Keep::Manifest yield results as they go.  This is important for performance.  Many clients only look at the first few files.  Yielding results as they're available makes it possible for those clients to get information without parsing an entire manifest, which could be very large.