Project

General

Profile

Hacking API server » History » Version 7

Tom Clegg, 05/09/2014 04:57 PM

1 1 Tom Clegg
h1. Hacking API server
2
3
{{toc}}
4
5
h2. Source tree layout
6
7
Everything is in @/services/api@.
8
9
Key pieces to know about before going much further:
10
11
|/|Usual Rails project layout|
12
|/app/controllers/application_controller.rb|Controller superclass with most of the generic API features like CRUD, authentication|
13
|/app/controllers/arvados/v1/|API methods other than generic CRUD (users#current, jobs#queue, ...)|
14
|/app/models/arvados_model.rb|Default Arvados model behavior: permissions, etag, uuid|
15
16
h2. Unlike a typical Rails project...
17
18
* Most responses are JSON. Very few HTML views. We don't normally talk to browsers, except during authentication.
19
* We assign UUID strings (see lib/assign_uuid.rb and app/models/arvados_model.rb)
20
* The @Links@ table emulates a graph database a la "RDF":http://www.rdfabout.com/quickintro.xpd. Much of the interesting information in Arvados is recorded as a Link between two other entities.
21
* For the most part, relations among objects are not expressed with the usual ActiveRelation features like belongs_to and has_many.
22
* Permissions: see below.
23
24
h2. Running in development mode
25
26
Save something like this at @~/bin/apiserver@, make it executable, make sure ~/bin is in your path:
27
28 7 Tom Clegg
<pre>
29
#!/bin/sh
30 1 Tom Clegg
set -e
31
cd ~/arvados/services/api
32 7 Tom Clegg
if ! [ -e self-signed.key ]
33
then
34
  openssl req -new -x509 -nodes -out ./self-signed.pem -keyout ./self-signed.pem -days 3650 -subj /CN=localhost
35
fi
36
if [ -e /usr/local/rvm/bin/rvm ]
37
then
38
  rvmexec="rvm-exec 2.1.1"
39
else
40
  rvmexec=""
41
fi
42
export ARVADOS_WEBSOCKETS=true
43 1 Tom Clegg
export RAILS_ENV=development
44 7 Tom Clegg
$rvmexec bundle install
45
exec $rvmexec bundle exec passenger start -d -p3030 --ssl --ssl-certificate self-signed.pem --ssl-certificate-key self-signed.key
46
</pre>
47
48 1 Tom Clegg
49
h2. Headaches to avoid
50
51
If you make a change that affects the discovery document, you need to clear a few caches before your client will see the change.
52
* Restart API server or: @touch tmp/restart.txt@
53
* Clear API server disk cache: @rake tmp:cache:clear@
54
* Clear SDK discovery doc cache on client side: @rm -r ~/.cache/arvados/@
55 4 Tom Clegg
56
Do not store symbol keys (or values) in serialized attributes.
57
* Rails supplies @params@ as a HashWithIndifferentAccess so @params['foo']@ and @params[:foo]@ are equivalent. This is usually convenient. However, here we often copy arrays and hashes from @params@ to the database, and from there to API responses. JSON does not have HashWithIndifferentAccess (or symbols) and we want these serialized attributes to behave predictably everywhere.
58
* API server's policy is that serialized attributes (like @properties@ on a link) always have strings instead of symbols: these attributes look the same in the database, in the API server Rails application, in the JSON response sent to clients, and in the JSON objects received from clients.
59
* There is no validation (yet!) to check for this.
60
61 1 Tom Clegg
62
h2. Features
63
64
h3. Authentication
65
66
Involves
67
* UserSessionsController (in app/controllers/, not .../arvados/v1): this is an exceptional case where we actually talk to a browser.
68
69
h3. Permissions
70
71
Object-level permissions, aka ownership and sharing
72
* Models have their own idea of create/update permissions. Controllers don't worry about this.
73
* ArvadosModel updates/enforces modified_by_* and owner_uuid
74
* Lookups are not (yet) permission-restricted in the default scope, though. Controllers need to use Model.readable_by(user).
75
* ApplicationController uses an around_filter that verifies the supplied api_token and makes current_user available everywhere. If you need to override create/update permissions, use @act_as_system_user do ... end@.
76
* Unusual cases: KeepDisks and Collections can be looked up by inactive users (otherwise they wouldn't be able to read & clickthrough user agreements).
77
78
Controller-level permissions
79
* ApplicationController#require_auth_scope_all checks token scopes: currently, unless otherwise specified by a subclass controller, nothing is allowed unless scopes includes "all".
80
* ApplicationController has an admin_required filter available (not used by default)
81
82
h3. Error handling
83
84
* "Look up object by uuid, and send 404 if not found" is enabled by default, except for index/create actions.
85
86
h3. Routing
87
88
* API routes are in the @:arvados@ &rarr; @:v1@ namespace.
89
* Routes like @/jobs/queue@ have to come before @resources :jobs@ (otherwise @/jobs/queue@ will match @jobs#get(id=queue)@ first). (Better, we should rearrange these to use @resources :jobs do ...@ like in Workbench.)
90
* We use the standard Rails routes like @/jobs/:id@ but then we move params[:id] to params[:uuid] in our before_filters.
91
92
h3. Tests
93
94
* Run tests with @rvm-exec 2.0.0 bundle exec rake test RAILS_ENV=test@
95
* Functional tests need to authenticate themselves with @authorize_with :active@ (where @:active@ refers to an ApiClientAuthorization fixture)
96
* Big deficit of tests, especially unit tests. This is a bug! It doesn't mean we don't want to test things.
97
98
h3. Discovery document
99
100
* Mostly, but not yet completely, generated by introspection (descendants of ArvadosModel are inspected at run time). But some controllers/actions are skipped, and some actions are renamed (e.g., Rails calls it "show" but everyone else calls it "get").
101
* Handled by Arvados::V1::SchemaController#index (used to be in #discovery_document before #1750). See @config/routes.rb@
102
* Must be available to anonymous clients.
103
* Has no tests! We test it by trying all of our SDKs against it.
104
105
h2. Development patterns
106
107
h3. Add a model
108
109
In shell:
110
* @rails g model FizzBuzz@
111
112
In @app/models/fizzbuzz.rb@:
113
* Change base class from @ActiveRecord::Base@ to @ArvadosModel@.
114
* Add some more standard behavior.
115
116
<pre><code class="ruby">
117
include AssignUuid
118
include KindAndEtag
119
include CommonApiTemplate
120
</code></pre>
121
122
In @db/migrate/{timestamp}_create_fizzbuzzes.rb@:
123
* Add the generic attribute columns.
124
* Run @t.timestamps@ and add (at least!) a @:uuid@ index.
125
126
<pre><code class="ruby">
127
class CreateFizzBuzz < ActiveRecord::Migration
128
  def change
129
    create_table :fizzbuzzes do |t|
130
      t.string :uuid, :null => false
131
      t.string :owner_uuid, :null => false
132
      t.string :modified_by_client_uuid
133
      t.string :modified_by_user_uuid
134
      t.datetime :modified_at
135
      t.text :properties
136
137
      t.timestamps
138
    end
139
    add_index :humans, :uuid, :unique => true
140
  end
141
end
142
</code></pre>
143
144
Apply the migration:
145
* @rake db:migrate@
146
* @RAILS_ENV=test rake db:migrate@ (to migrate your test database too)
147
* Inspect the resulting @db/schema.rb@ and include it in your commit.
148
* Don't forget to @git add@ the new migration and model files.
149
150
h3. Add an attribute to a model
151
152
* Generate migration as usual
153
<pre>
154
rails g migration AddBazQuxToFooBar baz_qux:column_type_goes_here
155
</pre>
156
* Consider adding null constraints and a default value to the @add_column@ statement in the migration in @db/migrate/timestamp_add_baz_qux_to_foo_bar.rb@:
157
<pre><code class="ruby">, null: false, default: false</code></pre>
158
* Consider adding an index
159
* You probably want to add it to the API response template so clients can see it: @app/models/model_name.rb@ &rarr; @api_accessible :user ...@
160
* Sometimes it's only visible to privileged users; see @ping_secret@ in @app/models/keep_disk.rb@
161
* If it's a serialized attribute, add @serialize :the_attribute_name, Hash@ to the model. Always specify Hash or Array!
162 3 Tom Clegg
* Run @rake db:migrate@ and inspect your @db/schema.rb@ and include the new @schema.rb@ in the *same commit* as your @db/migrate/*.rb@ migration script.
163 1 Tom Clegg
* Run @rake tmp:cache:clear@ and @touch tmp/restart.txt@ in your dev apiserver, to force it to generate a new REST discovery document.
164
165
166
h3. Add a controller
167
168
* @rails g controller Arvados::V1::FizzBuzzesController@
169
* Avoid adding top-level controllers like @app/controllers/fizz_buzzes_controller.rb@.
170
* Avoid adding top-level routes. Everything should be in @namespace :arvados@ &rarr; @namespace :v1@ except oddballs like login/logout actions.
171
172
h3. Add a controller action
173
174
Add a route in @config/routes.rb@.
175
* Choose an appropriate HTTP method: GET has no side effects. POST creates something. PUT replaces/updates something.
176
* Use the block form:
177
<pre><code class="ruby">
178
resources :fizz_buzzes do
179
  # If the action operates on an object, i.e., a uuid is required,
180
  # this generates a route /arvados/v1/fizz_buzzes/{uuid}/blurfl
181
  post 'blurfl', on: :member
182
  # If not, this generates a route /arvados/v1/fizz_buzzes/flurbl
183
  get 'flurbl', on: :collection
184
end
185
</code></pre>
186
187
In @app/controllers/arvados/v1/fizz_buzzes_controller.rb@:
188
189
* Add a method to the controller class.
190
* Skip the "find_object" before_filters if it's a collection action.
191
* Specify required/optional parameters using a class method @_action_requires_parameters@.
192
<pre><code class="ruby">
193
skip_before_filter :find_object_by_uuid, only: [:flurbl]
194
skip_before_filter :render_404_if_no_object, only: [:flurbl]
195
196
def blurfl
197
  @object.do_whatever_blurfl_does!
198
  show
199
end
200
201
def self._flurbl_requires_parameters
202
  {
203
    qux: { type: 'integer', required: true, description: 'First flurbl qux must match this qux.' }
204
  }
205
end
206
def flurbl
207
  @object = model_class.where('qux = ?', params[:qux]).first
208
  show
209
end
210
</code></pre>
211
212
h3. Add a configuration parameter
213 6 Tom Clegg
214 1 Tom Clegg
* Add it to @config/application.default.yml@ with a sensible default value. (Don't fall back to default values at time of use, or define defaults in other places!)
215
* If there is no sensible default value, like @secret_token@: specify @~@ (i.e., nil) in @application.default.yml@ *and* put a default value in the @test@ section of @config/application.yml.example@ that will make tests pass.
216
* If there is a sensible default value for development/test but not for production, like return address for notification email messages, specify the test/dev default in the @common@ section @application.default.yml@ but specify @~@ (nil) in the @production@ section. This prevents someone from installing or *updating a production server* with defaults that don't make sense in production!
217 6 Tom Clegg
* Use @Rails.configuration.config_setting_name@ to retrieve the configured value. There is no need to check whether it is nil or missing: in those cases, "rake config:check" would have failed and the application would have refused to start.
218 2 Tom Clegg
219
220
h3. Add a test fixture
221
222
Generate last part of uuid from command line:
223
<pre><code class="ruby">ruby -e 'puts rand(2**512).to_s(36)[0..14]'
224
j0wqrlny07k1u12</code></pre>
225
Generate uuid from @rails console@:
226
<pre><code class="ruby">Group.generate_uuid
227
=> "xyzzy-j7d0g-8nw4r6gnnkixw1i"
228 1 Tom Clegg
</code></pre>