Project

General

Profile

Hacking API server » History » Version 14

Radhika Chippada, 05/26/2014 11:22 AM
Added command to kill api server

1 1 Tom Clegg
h1. Hacking API server
2
3
{{toc}}
4
5
h2. Source tree layout
6
7
Everything is in @/services/api@.
8
9
Key pieces to know about before going much further:
10
11
|/|Usual Rails project layout|
12
|/app/controllers/application_controller.rb|Controller superclass with most of the generic API features like CRUD, authentication|
13
|/app/controllers/arvados/v1/|API methods other than generic CRUD (users#current, jobs#queue, ...)|
14
|/app/models/arvados_model.rb|Default Arvados model behavior: permissions, etag, uuid|
15
16
h2. Unlike a typical Rails project...
17
18
* Most responses are JSON. Very few HTML views. We don't normally talk to browsers, except during authentication.
19
* We assign UUID strings (see lib/assign_uuid.rb and app/models/arvados_model.rb)
20
* The @Links@ table emulates a graph database a la "RDF":http://www.rdfabout.com/quickintro.xpd. Much of the interesting information in Arvados is recorded as a Link between two other entities.
21
* For the most part, relations among objects are not expressed with the usual ActiveRelation features like belongs_to and has_many.
22
* Permissions: see below.
23
24
h2. Running in development mode
25
26 12 Tom Clegg
First, take care of the dependencies documented at http://doc.arvados.org/install/install-api-server.html.
27
28 1 Tom Clegg
Save something like this at @~/bin/apiserver@, make it executable, make sure ~/bin is in your path:
29
30 7 Tom Clegg
<pre>
31
#!/bin/sh
32 1 Tom Clegg
set -e
33
cd ~/arvados/services/api
34 7 Tom Clegg
if ! [ -e self-signed.key ]
35
then
36 8 Tom Clegg
  # Generate a self-signed SSL key
37
  openssl req -new -x509 -nodes -out ./self-signed.pem -keyout ./self-signed.key -days 3650 -subj /CN=localhost
38 7 Tom Clegg
fi
39
if [ -e /usr/local/rvm/bin/rvm ]
40
then
41
  rvmexec="rvm-exec 2.1.1"
42
else
43
  rvmexec=""
44
fi
45
export ARVADOS_WEBSOCKETS=true
46 1 Tom Clegg
export RAILS_ENV=development
47 7 Tom Clegg
$rvmexec bundle install
48 11 Tom Clegg
exec $rvmexec bundle exec passenger start -p3030 --ssl --ssl-certificate self-signed.pem --ssl-certificate-key self-signed.key
49 7 Tom Clegg
</pre>
50 1 Tom Clegg
51 9 Tom Clegg
Notes:
52
* Here we use passenger instead of webrick (which is what we'd get with "@rails server@") in order to serve websockets from the same process as https. (You also have the option of serving https and wss from separate processes -- see @services/api/config/application.default.yml@ -- but the simplest way is to run both in the same process by setting @ARVADOS_WEBSOCKETS=true@.)
53
* Webrick can make its own self-signed SSL certificate, but passenger expects you to provide a certificate & key yourself. The above script generates a key/certificate pair the first time it runs, and leaves it in @services/api/self-signed.*@ to reuse next time.
54
* @bundle install@ ensures your installed gems satisfy the requirements in Gemfile and (if you have changed it locally) update Gemfile.lock appropriately. This should do the right thing after you change Gemfile yourself, or change Gemfile/Gemfile.lock by updating/merging master.
55
* If you're relying on rvm to provide a suitable version of Ruby, "@rvm-exec@" should do the right thing here. You can change the @2.1.1@ argument to the version you want to use (hopefully >= 2.1.1). If your system Ruby is >= 2.1.1, rvm is unnecessary.
56 14 Radhika Chippada
* You can kill the server by running
57
58
<pre><code class="ruby">
59
passenger stop --pid-file ~/arvados/services/api/tmp/pids/passenger.3030.pid
60
</code></pre>
61 9 Tom Clegg
62 1 Tom Clegg
h2. Headaches to avoid
63
64
If you make a change that affects the discovery document, you need to clear a few caches before your client will see the change.
65
* Restart API server or: @touch tmp/restart.txt@
66
* Clear API server disk cache: @rake tmp:cache:clear@
67
* Clear SDK discovery doc cache on client side: @rm -r ~/.cache/arvados/@
68 4 Tom Clegg
69
Do not store symbol keys (or values) in serialized attributes.
70
* Rails supplies @params@ as a HashWithIndifferentAccess so @params['foo']@ and @params[:foo]@ are equivalent. This is usually convenient. However, here we often copy arrays and hashes from @params@ to the database, and from there to API responses. JSON does not have HashWithIndifferentAccess (or symbols) and we want these serialized attributes to behave predictably everywhere.
71
* API server's policy is that serialized attributes (like @properties@ on a link) always have strings instead of symbols: these attributes look the same in the database, in the API server Rails application, in the JSON response sent to clients, and in the JSON objects received from clients.
72
* There is no validation (yet!) to check for this.
73
74 1 Tom Clegg
75
h2. Features
76
77
h3. Authentication
78
79
Involves
80
* UserSessionsController (in app/controllers/, not .../arvados/v1): this is an exceptional case where we actually talk to a browser.
81
82
h3. Permissions
83
84
Object-level permissions, aka ownership and sharing
85
* Models have their own idea of create/update permissions. Controllers don't worry about this.
86
* ArvadosModel updates/enforces modified_by_* and owner_uuid
87
* Lookups are not (yet) permission-restricted in the default scope, though. Controllers need to use Model.readable_by(user).
88
* ApplicationController uses an around_filter that verifies the supplied api_token and makes current_user available everywhere. If you need to override create/update permissions, use @act_as_system_user do ... end@.
89
* Unusual cases: KeepDisks and Collections can be looked up by inactive users (otherwise they wouldn't be able to read & clickthrough user agreements).
90
91
Controller-level permissions
92
* ApplicationController#require_auth_scope_all checks token scopes: currently, unless otherwise specified by a subclass controller, nothing is allowed unless scopes includes "all".
93
* ApplicationController has an admin_required filter available (not used by default)
94
95
h3. Error handling
96
97
* "Look up object by uuid, and send 404 if not found" is enabled by default, except for index/create actions.
98
99
h3. Routing
100
101
* API routes are in the @:arvados@ &rarr; @:v1@ namespace.
102
* Routes like @/jobs/queue@ have to come before @resources :jobs@ (otherwise @/jobs/queue@ will match @jobs#get(id=queue)@ first). (Better, we should rearrange these to use @resources :jobs do ...@ like in Workbench.)
103
* We use the standard Rails routes like @/jobs/:id@ but then we move params[:id] to params[:uuid] in our before_filters.
104
105
h3. Tests
106
107 10 Tom Clegg
* Run tests with @rvm-exec 2.1.1 bundle exec rake test@
108
* If prompted, migrate your test database by running @RAILS_ENV=test rvm-exec 2.1.1 bundle exec rake db:migrate@
109
* As above, you can leave out @rvm-exec 2.1.1@ if your system Ruby version is suitable. But don't leave out @bundle exec@.
110
* Run just the unit tests with @[...] rake test:units@ (or @test:functionals@ or @test:integration@).
111
* Run just a single test class (file) by specifying the file, like @[...] rake TEST=test/unit/owner_test.rb@ (save time in your "did that fix the failing test?" phase!)
112 1 Tom Clegg
* Functional tests need to authenticate themselves with @authorize_with :active@ (where @:active@ refers to an ApiClientAuthorization fixture)
113 10 Tom Clegg
* There is a deficit of tests, especially unit tests. This is a bug! It doesn't mean we don't want to test things.
114
115 1 Tom Clegg
116
h3. Discovery document
117
118
* Mostly, but not yet completely, generated by introspection (descendants of ArvadosModel are inspected at run time). But some controllers/actions are skipped, and some actions are renamed (e.g., Rails calls it "show" but everyone else calls it "get").
119
* Handled by Arvados::V1::SchemaController#index (used to be in #discovery_document before #1750). See @config/routes.rb@
120
* Must be available to anonymous clients.
121
* Has no tests! We test it by trying all of our SDKs against it.
122
123
h2. Development patterns
124
125
h3. Add a model
126
127
In shell:
128
* @rails g model FizzBuzz@
129
130
In @app/models/fizzbuzz.rb@:
131
* Change base class from @ActiveRecord::Base@ to @ArvadosModel@.
132
* Add some more standard behavior.
133
134
<pre><code class="ruby">
135 13 Peter Amstutz
include HasUuid
136 1 Tom Clegg
include KindAndEtag
137
include CommonApiTemplate
138
</code></pre>
139
140
In @db/migrate/{timestamp}_create_fizzbuzzes.rb@:
141
* Add the generic attribute columns.
142
* Run @t.timestamps@ and add (at least!) a @:uuid@ index.
143
144
<pre><code class="ruby">
145
class CreateFizzBuzz < ActiveRecord::Migration
146
  def change
147
    create_table :fizzbuzzes do |t|
148
      t.string :uuid, :null => false
149
      t.string :owner_uuid, :null => false
150
      t.string :modified_by_client_uuid
151
      t.string :modified_by_user_uuid
152
      t.datetime :modified_at
153
      t.text :properties
154
155
      t.timestamps
156
    end
157
    add_index :humans, :uuid, :unique => true
158
  end
159
end
160
</code></pre>
161
162
Apply the migration:
163
* @rake db:migrate@
164
* @RAILS_ENV=test rake db:migrate@ (to migrate your test database too)
165
* Inspect the resulting @db/schema.rb@ and include it in your commit.
166
* Don't forget to @git add@ the new migration and model files.
167
168
h3. Add an attribute to a model
169
170
* Generate migration as usual
171
<pre>
172
rails g migration AddBazQuxToFooBar baz_qux:column_type_goes_here
173
</pre>
174
* Consider adding null constraints and a default value to the @add_column@ statement in the migration in @db/migrate/timestamp_add_baz_qux_to_foo_bar.rb@:
175
<pre><code class="ruby">, null: false, default: false</code></pre>
176
* Consider adding an index
177
* You probably want to add it to the API response template so clients can see it: @app/models/model_name.rb@ &rarr; @api_accessible :user ...@
178
* Sometimes it's only visible to privileged users; see @ping_secret@ in @app/models/keep_disk.rb@
179
* If it's a serialized attribute, add @serialize :the_attribute_name, Hash@ to the model. Always specify Hash or Array!
180 3 Tom Clegg
* Run @rake db:migrate@ and inspect your @db/schema.rb@ and include the new @schema.rb@ in the *same commit* as your @db/migrate/*.rb@ migration script.
181 1 Tom Clegg
* Run @rake tmp:cache:clear@ and @touch tmp/restart.txt@ in your dev apiserver, to force it to generate a new REST discovery document.
182
183
184
h3. Add a controller
185
186
* @rails g controller Arvados::V1::FizzBuzzesController@
187
* Avoid adding top-level controllers like @app/controllers/fizz_buzzes_controller.rb@.
188
* Avoid adding top-level routes. Everything should be in @namespace :arvados@ &rarr; @namespace :v1@ except oddballs like login/logout actions.
189
190
h3. Add a controller action
191
192
Add a route in @config/routes.rb@.
193
* Choose an appropriate HTTP method: GET has no side effects. POST creates something. PUT replaces/updates something.
194
* Use the block form:
195
<pre><code class="ruby">
196
resources :fizz_buzzes do
197
  # If the action operates on an object, i.e., a uuid is required,
198
  # this generates a route /arvados/v1/fizz_buzzes/{uuid}/blurfl
199
  post 'blurfl', on: :member
200
  # If not, this generates a route /arvados/v1/fizz_buzzes/flurbl
201
  get 'flurbl', on: :collection
202
end
203
</code></pre>
204
205
In @app/controllers/arvados/v1/fizz_buzzes_controller.rb@:
206
207
* Add a method to the controller class.
208
* Skip the "find_object" before_filters if it's a collection action.
209
* Specify required/optional parameters using a class method @_action_requires_parameters@.
210
<pre><code class="ruby">
211
skip_before_filter :find_object_by_uuid, only: [:flurbl]
212
skip_before_filter :render_404_if_no_object, only: [:flurbl]
213
214
def blurfl
215
  @object.do_whatever_blurfl_does!
216
  show
217
end
218
219
def self._flurbl_requires_parameters
220
  {
221
    qux: { type: 'integer', required: true, description: 'First flurbl qux must match this qux.' }
222
  }
223
end
224
def flurbl
225
  @object = model_class.where('qux = ?', params[:qux]).first
226
  show
227
end
228
</code></pre>
229
230
h3. Add a configuration parameter
231 6 Tom Clegg
232 1 Tom Clegg
* Add it to @config/application.default.yml@ with a sensible default value. (Don't fall back to default values at time of use, or define defaults in other places!)
233
* If there is no sensible default value, like @secret_token@: specify @~@ (i.e., nil) in @application.default.yml@ *and* put a default value in the @test@ section of @config/application.yml.example@ that will make tests pass.
234
* If there is a sensible default value for development/test but not for production, like return address for notification email messages, specify the test/dev default in the @common@ section @application.default.yml@ but specify @~@ (nil) in the @production@ section. This prevents someone from installing or *updating a production server* with defaults that don't make sense in production!
235 6 Tom Clegg
* Use @Rails.configuration.config_setting_name@ to retrieve the configured value. There is no need to check whether it is nil or missing: in those cases, "rake config:check" would have failed and the application would have refused to start.
236 2 Tom Clegg
237
238
h3. Add a test fixture
239
240
Generate last part of uuid from command line:
241
<pre><code class="ruby">ruby -e 'puts rand(2**512).to_s(36)[0..14]'
242
j0wqrlny07k1u12</code></pre>
243
Generate uuid from @rails console@:
244
<pre><code class="ruby">Group.generate_uuid
245
=> "xyzzy-j7d0g-8nw4r6gnnkixw1i"
246 1 Tom Clegg
</code></pre>