Hacking API server¶

• Table of contents
• Hacking API server
  • Source tree layout
  • Unlike a typical Rails project...
  • Running in development mode
  • Headaches to avoid
  • Features
    • Authentication
    • Permissions
    • Error handling
    • Routing
    • Tests
    • Discovery document
  • Development patterns
    • Add a model
    • Add an attribute to a model
    • Add a controller
    • Add a controller action
    • Add a configuration parameter
    • Add a test fixture

Source tree layout¶

Everything is in /services/api.

Key pieces to know about before going much further:

/	Usual Rails project layout
/app/controllers/application_controller.rb	Controller superclass with most of the generic API features like CRUD, authentication
/app/controllers/arvados/v1/	API methods other than generic CRUD (users#current, jobs#queue, ...)
/app/models/arvados_model.rb	Default Arvados model behavior: permissions, etag, uuid

Unlike a typical Rails project...¶

• Most responses are JSON. Very few HTML views. We don't normally talk to browsers, except during authentication.
• We assign UUID strings (see lib/assign_uuid.rb and app/models/arvados_model.rb)
• The Links table emulates a graph database a la RDF. Much of the interesting information in Arvados is recorded as a Link between two other entities.
• For the most part, relations among objects are not expressed with the usual ActiveRelation features like belongs_to and has_many.
• Permissions: see below.

Running in development mode¶

SDKs really want your server to offer SSL. One way is to generate a self-signed certificate.

openssl req -new -x509 -nodes -out ~/self-signed.pem -keyout ~/self-signed.key -days 3650 -subj '/CN=arvados.example.com'

Save something like this at ~/bin/apiserver, make it executable, make sure ~/bin is in your path:

#!/bin/sh
set -e
cd ~/arvados/services/api
export RAILS_ENV=development
rvm-exec 2.0.0 bundle install
exec rvm-exec 2.0.0 bundle exec passenger start --ssl --ssl-certificate ~/self-signed.pem --ssl-certificate-key ~/self-signed.key

Headaches to avoid¶

If you make a change that affects the discovery document, you need to clear a few caches before your client will see the change.

• Restart API server or: touch tmp/restart.txt
• Clear API server disk cache: rake tmp:cache:clear
• Clear SDK discovery doc cache on client side: rm -r ~/.cache/arvados/

Features¶

Authentication¶

Involves

• UserSessionsController (in app/controllers/, not .../arvados/v1): this is an exceptional case where we actually talk to a browser.

Permissions¶

Object-level permissions, aka ownership and sharing

• Models have their own idea of create/update permissions. Controllers don't worry about this.
• ArvadosModel updates/enforces modified_by_* and owner_uuid
• Lookups are not (yet) permission-restricted in the default scope, though. Controllers need to use Model.readable_by(user).
• ApplicationController uses an around_filter that verifies the supplied api_token and makes current_user available everywhere. If you need to override create/update permissions, use act_as_system_user do ... end.
• Unusual cases: KeepDisks and Collections can be looked up by inactive users (otherwise they wouldn't be able to read & clickthrough user agreements).

Controller-level permissions

• ApplicationController#require_auth_scope_all checks token scopes: currently, unless otherwise specified by a subclass controller, nothing is allowed unless scopes includes "all".
• ApplicationController has an admin_required filter available (not used by default)

Error handling¶

• "Look up object by uuid, and send 404 if not found" is enabled by default, except for index/create actions.

Routing¶

• API routes are in the :arvados → :v1 namespace.
• Routes like /jobs/queue have to come before resources :jobs (otherwise /jobs/queue will match jobs#get(id=queue) first). (Better, we should rearrange these to use resources :jobs do ... like in Workbench.)
• We use the standard Rails routes like /jobs/:id but then we move params[:id] to params[:uuid] in our before_filters.

Tests¶

• Run tests with rvm-exec 2.0.0 bundle exec rake test RAILS_ENV=test
• Functional tests need to authenticate themselves with authorize_with :active (where :active refers to an ApiClientAuthorization fixture)
• Big deficit of tests, especially unit tests. This is a bug! It doesn't mean we don't want to test things.

Discovery document¶

• Mostly, but not yet completely, generated by introspection (descendants of ArvadosModel are inspected at run time). But some controllers/actions are skipped, and some actions are renamed (e.g., Rails calls it "show" but everyone else calls it "get").
• Handled by Arvados::V1::SchemaController#index (used to be in #discovery_document before #1750). See config/routes.rb
• Must be available to anonymous clients.
• Has no tests! We test it by trying all of our SDKs against it.

Development patterns¶

Add a model¶

In shell:

• rails g model FizzBuzz

In app/models/fizzbuzz.rb:

• Change base class from ActiveRecord::Base to ArvadosModel.
• Add some more standard behavior.

include AssignUuid
include KindAndEtag
include CommonApiTemplate

In db/migrate/{timestamp}_create_fizzbuzzes.rb:

• Add the generic attribute columns.
• Run t.timestamps and add (at least!) a :uuid index.

class CreateFizzBuzz < ActiveRecord::Migration
  def change
    create_table :fizzbuzzes do |t|
      t.string :uuid, :null => false
      t.string :owner_uuid, :null => false
      t.string :modified_by_client_uuid
      t.string :modified_by_user_uuid
      t.datetime :modified_at
      t.text :properties

      t.timestamps
    end
    add_index :humans, :uuid, :unique => true
  end
end

Apply the migration:

• rake db:migrate
• RAILS_ENV=test rake db:migrate (to migrate your test database too)
• Inspect the resulting db/schema.rb and include it in your commit.
• Don't forget to git add the new migration and model files.

Add an attribute to a model¶

• Generate migration as usual
  

  rails g migration AddBazQuxToFooBar baz_qux:column_type_goes_here
  

• Consider adding null constraints and a default value to the add_column statement in the migration in db/migrate/timestamp_add_baz_qux_to_foo_bar.rb:
  

  , null: false, default: false

• Consider adding an index
• You probably want to add it to the API response template so clients can see it: app/models/model_name.rb → api_accessible :user ...
• Sometimes it's only visible to privileged users; see ping_secret in app/models/keep_disk.rb
• If it's a serialized attribute, add serialize :the_attribute_name, Hash to the model. Always specify Hash or Array!
• Run rake db:migrate and inspect your db/schema.rb and include the new schema.rb in the same commit as your db/migrate/*.rb migration script.
• Run rake tmp:cache:clear and touch tmp/restart.txt in your dev apiserver, to force it to generate a new REST discovery document.

Add a controller¶

• rails g controller Arvados::V1::FizzBuzzesController
• Avoid adding top-level controllers like app/controllers/fizz_buzzes_controller.rb.
• Avoid adding top-level routes. Everything should be in namespace :arvados → namespace :v1 except oddballs like login/logout actions.

Add a controller action¶

Add a route in config/routes.rb.

• Choose an appropriate HTTP method: GET has no side effects. POST creates something. PUT replaces/updates something.
• Use the block form:
  

  resources :fizz_buzzes do
    # If the action operates on an object, i.e., a uuid is required,
    # this generates a route /arvados/v1/fizz_buzzes/{uuid}/blurfl
    post 'blurfl', on: :member
    # If not, this generates a route /arvados/v1/fizz_buzzes/flurbl
    get 'flurbl', on: :collection
  end
  

In app/controllers/arvados/v1/fizz_buzzes_controller.rb:

• Add a method to the controller class.
• Skip the "find_object" before_filters if it's a collection action.
• Specify required/optional parameters using a class method _action_requires_parameters.
  

  skip_before_filter :find_object_by_uuid, only: [:flurbl]
  skip_before_filter :render_404_if_no_object, only: [:flurbl]
  
  def blurfl
    @object.do_whatever_blurfl_does!
    show
  end
  
  def self._flurbl_requires_parameters
    {
      qux: { type: 'integer', required: true, description: 'First flurbl qux must match this qux.' }
    }
  end
  def flurbl
    @object = model_class.where('qux = ?', params[:qux]).first
    show
  end
  

Add a configuration parameter¶

• Add it to config/application.default.yml with a sensible default value.
• If there is no sensible default value, like secret_token: specify ~ (i.e., nil) in application.default.yml and put a default value in the test section of config/application.yml.example that will make tests pass.
• If there is a sensible default value for development/test but not for production, like return address for notification email messages, specify the test/dev default in the common section application.default.yml but specify ~ (nil) in the production section. This prevents someone from installing or updating a production server with defaults that don't make sense in production!
• Use Rails.configuration.config_setting_name to retrieve the configured value. There is no need to check whether it is nil or missing: in those cases, "rake config:check" would have failed and the application would have refused to start.

Add a test fixture¶

Generate last part of uuid from command line:

ruby -e 'puts rand(2**512).to_s(36)[0..14]'
j0wqrlny07k1u12

Generate uuid from rails console:

Group.generate_uuid
=> "xyzzy-j7d0g-8nw4r6gnnkixw1i"