Project

General

Profile

Hacking Workbench » History » Version 22

Tom Clegg, 12/10/2014 03:23 AM

1 1 Tom Clegg
h1. Hacking Workbench
2
3
{{toc}}
4
5
h2. Source tree layout
6
7
Everything is in @/apps/workbench@.
8
9
Key pieces to know about before going much further:
10
11
|/|Usual Rails project layout|
12
|/app/controllers/application_controller.rb|Controller superclass with authentication setup, error handling, and generic CRUD actions|
13
|/app/controllers/*.rb|Actions other than generic CRUD (users#activity, jobs#generate_provenance, ...)|
14
|/app/models/arvados_base.rb|Default Arvados model behavior and ActiveRecord-like accessors and introspection features|
15
|/app/models/arvados_resource_list.rb|ActiveRelation-like class (what you get from Model.where() etc.)|
16
17 12 Tom Clegg
h2. Background resources
18
19
Workbench is a Rails 4 application.
20
* "Getting started with Rails":http://guides.rubyonrails.org/getting_started.html at rubyonrails.org
21 1 Tom Clegg
* "AJAX in Rails 3.1":http://blog.madebydna.com/all/code/2011/12/05/ajax-in-rails-3.html blog post (still relevant in Rails 4)
22 20 Tom Clegg
23
Javascript readings
24
* http://javascript.crockford.com/code.html
25
* http://javascript.crockford.com/private.html
26
27
Angular readings
28
* http://www.ng-newsletter.com/posts/beginner2expert-how_to_start.html
29
* https://docs.angularjs.org/guide/introduction
30
* https://github.com/johnpapa/angularjs-styleguide
31 12 Tom Clegg
32 1 Tom Clegg
h2. Unlike a typical Rails project...
33
34 7 Peter Amstutz
* ActiveRecord in Workbench doesn't talk to the database directly, but instead queries the Arvados API as REST client.
35
* The Arvados query API is somewhat limited and doesn't accept SQL statements, so Workbench has to work harder to get what it needs.
36 1 Tom Clegg
* Workbench itself only has the privileges of the Workbench user: when making Arvados API calls, it uses the API token provided by the user.
37
38
h2. Unlike what you might expect...
39
40
* Workbench doesn't use the Ruby SDK. It uses a sort of baked-in Rails SDK.
41
** TODO: move it out of Workbench into a gem.
42
** TODO: use the Ruby SDK under the hood.
43
44
h2. Running in development mode
45
46 2 Misha Zatsman
h3. SSL certificates
47
48 4 Tom Clegg
You can get started quickly with SSL by generating a self-signed certificate:
49 1 Tom Clegg
50
 openssl req -new -x509 -nodes -out ~/self-signed.pem -keyout ~/self-signed.key -days 3650 -subj '/CN=arvados.example.com'
51
52
Alternatively, download a set from the bottom of the [[API server]] page.
53 2 Misha Zatsman
54
h3. Download and configure
55 1 Tom Clegg
56 2 Misha Zatsman
Follow "these instructions":http://doc.arvados.org/install/install-workbench-app.html to download the source and configure your workbench instance.
57 3 Misha Zatsman
58 4 Tom Clegg
h3. Start the server
59 1 Tom Clegg
60 4 Tom Clegg
Save something like the following at @~/bin/workbench@, make it executable[1], make sure @~/bin@ is in your path[2]:
61 1 Tom Clegg
62
 #!/bin/sh
63
set -e
64
cd ~/arvados/apps/workbench
65
export RAILS_ENV=development
66 5 Tom Clegg
bundle install --path=vendor/bundle
67 4 Tom Clegg
exec bundle exec passenger start -p 3031 --ssl --ssl-certificate ~/self-signed.pem --ssl-certificate-key ~/self-signed.key
68 1 Tom Clegg
69
The first time you run the above it will take a while to install all the ruby gems. In particular @Installing nokogiri@ takes a while
70
71
Once you see:
72
73
 =============== Phusion Passenger Standalone web server started ===============
74
75
You can visit your server at:
76
77 4 Tom Clegg
 @https://{ip-or-host}:3031/@
78
79 6 Misha Zatsman
You can kill your server with @ctrl-C@ but if you get disconnected from the terminal, it will continue running. You can kill it by running
80
81
 @ps x |grep nginx |grep master@
82
83
And then
84
85
 @kill ####@
86
87
Replacing #### with the number in the left column returned by ps
88
89 4 Tom Clegg
fn1. @chmod +x ~/bin/workbench@
90
91
fn2. In Debian systems, the default .profile adds ~/bin to your path, but only if it exists when you log in. If you just created ~/bin, doing @exec bash -login@ or @source .profile@ should make ~/bin appear in your path.
92
93
h2. Running tests
94
95
The test suite brings up an API server in test mode, and runs browser tests with Firefox.
96
97 17 Tom Clegg
Make sure API server has its dependencies in place and its database schema up-to-date.
98 4 Tom Clegg
99 1 Tom Clegg
<pre>
100 17 Tom Clegg
(
101
 set -e
102
 cd ../../services/api
103
 RAILS_ENV=test bundle install --path=vendor/bundle
104
 RAILS_ENV=test bundle exec rake db:migrate
105
)
106 4 Tom Clegg
</pre>
107
108
Install headless testing tools.
109
110
<pre>
111
sudo apt-get install xvfb iceweasel
112
</pre>
113
114
(Install firefox instead of iceweasel if you're not using Debian.)
115
116 10 Tom Clegg
Install phantomjs. (See http://phantomjs.org/download.html for latest version.)
117
118
<pre>
119 16 Tom Clegg
wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2
120
sudo tar -C /usr/local -xjf /tmp/phantomjs-1.9.8-linux-x86_64.tar.bz2
121
sudo ln -s ../phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/local/bin/
122 10 Tom Clegg
</pre>
123
124 4 Tom Clegg
Run the test suite.
125
126
<pre>
127
RAILS_ENV=test bundle exec rake test
128 1 Tom Clegg
</pre>
129 9 Tom Clegg
130 13 Tom Clegg
h3. When tests fail...
131
132
When an integration test fails (or skips) a screenshot is automatically saved in @arvados/apps/workbench/tmp/workbench-fail-1.png@, etc.
133
134 16 Tom Clegg
By default, @rake test@ just shows F when a test fails (and E when a test crashes) and doesn't tell you which tests had problems until the entire test suite is done. During development it makes more sense to use @TESTOPTS=-v@. This reports after each test the test class and name, outcome, and elapsed time:
135 13 Tom Clegg
* <pre>
136 1 Tom Clegg
$ RAILS_ENV=test bundle exec rake test TESTOPTS=-v
137 13 Tom Clegg
[...]
138
ApplicationControllerTest#test_links_for_object = 0.10 s = .
139
[...]
140
Saved ./tmp/workbench-fail-2.png
141 16 Tom Clegg
CollectionsTest#test_combine_selected_collection_files_into_new_collection = 10.89 s = F
142 13 Tom Clegg
[...]
143
</pre>
144 9 Tom Clegg
145 19 Tom Clegg
h3. Iterating on a single test
146
147
Sometimes you want to poke at the code and re-run a single test to confirm that you made it pass. You don't want to reboot everything just to make Minitest notice that you edited your test.
148
149 22 Tom Clegg
Since #3781 there is a @singletest@ function for this:
150 19 Tom Clegg
151
<pre>
152
arvados/apps/workbench$ RAILS_ENV=test bundle exec irb -Ilib:test
153 22 Tom Clegg
>>> load 'test/test_helper.rb'
154
>>> singletest 'integration/pipeline_instances_test.rb', 'Create and run a pipeline'
155 19 Tom Clegg
...
156 22 Tom Clegg
PipelineInstancesTest#test_Create_and_run_a_pipeline = 38.54 s = .
157 19 Tom Clegg
...
158
159 22 Tom Clegg
>>> singletest 'integration/pipeline_instances_test.rb', 'Create and run a pipeline'
160 19 Tom Clegg
...
161 22 Tom Clegg
PipelineInstancesTest#test_Create_and_run_a_pipeline = 29.58 s = .
162 19 Tom Clegg
...
163
</pre>
164
165 1 Tom Clegg
h2. Loading state from API into models
166
167
If your model makes an API call that returns the new state of an object, load the new attributes into the local model with @private_reload@:
168
169
<pre><code class="ruby">
170
  api_response = $arvados_api_client.api(...)
171
  private_reload api_response
172
</code></pre>
173
174
h2. Features
175
176
h3. Authentication
177
178
ApplicationController uses an around_filter to make sure the user is logged in, redirect to Arvados to complete the login procedure if not, and store the user's API token in Thread.current[:arvados_api_token] if so.
179
180
The @current_user@ helper returns User.current if the user is logged in, otherwise nil. (Generally, only special pages like "welcome" and "error" get displayed to users who aren't logged in.)
181
182
h3. Default filter behavior
183
184
@before_filter :find_object_by_uuid@
185
186
* This is enabled by default, @except :index, :create@.
187
* It renames the @:id@ param to @:uuid@. (The Rails default routing rules use @:id@ to accept params in path components, but @params[:uuid]@ makes more sense everywhere else in our code.)
188
* If you define a collection method (where there's no point looking up an object with the :id supplied in the request), skip this.
189
190
<pre><code class="ruby">
191
  skip_before_filter :find_object_by_uuid, only: [:action_that_takes_no_uuid_param]
192
</code></pre>
193
194
h3. Error handling
195
196
ApplicationController has a render_error method that shows a standard error page. (It's not very good, but it's better than a default Rails stack trace.)
197
198
In a controller you get there like this
199
200
<pre><code class="ruby">
201
  @errors = ['I could not achieve what you wanted.']
202
  render_error status: 500
203
</code></pre>
204
205
You can also do this, anywhere
206
207
<pre><code class="ruby">
208
  raise 'My spoon is too big.'
209
</code></pre>
210
211
The @render_error@ method sends JSON or HTML to the client according to the Accept header in the request (it sends JSON if JavaScript was requested), so reasonable things happen whether or not the request is AJAX.
212
213
h2. Development patterns
214
215
h3. Add a model
216
217
Currently, when the API provides a new model, we need to generate a corresponding model in Workbench: it's not smart enough to pick up the list of models from the API server's discovery document.
218
219
_(Need to fill in details here)_
220
# @rails generate model ....@
221
# Delete migration
222 8 Peter Amstutz
# Change base class to ArvadosBase
223
# @rails generate controller ...@ 
224 1 Tom Clegg
225
Model _attributes_, on the other hand, are populated automatically.
226
227
h3. Add a configuration knob
228
229
Same situation as API server. See [[Hacking API Server]].
230
231
h3. Add an API method
232
233
Workbench is not yet smart enough to look in the discovery document for supported API methods. You need to add a method to the appropriate model class before you can use it in the Workbench app.
234
235
h3. Writing tests
236
237 18 Tom Clegg
In integration tests, this makes your tests flaky because the result depends on whether the page has finished loading:
238
* <pre><code class="ruby">
239
assert page.has_selector?('a', text: 'foo')  # Danger!
240
</code></pre>
241
* Instead, do this:
242
* <pre><code class="ruby">
243
assert_selector('a', text: 'foo')
244
</code></pre>
245
* This lets Capybara wait for the selector to appear.
246
247 1 Tom Clegg
248
h3. AJAX using Rails UJS (remote:true with JavaScript response)
249
250
This pattern is the best way to make a button/link that invokes an asynchronous action on the Workbench server side, i.e., before/without navigating away from the current page.
251
252
# Add <code class="ruby">remote: true</code> to a link or button. This makes Rails put a <code class="html">data-remote="true"</code> attribute in the HTML element. Say, in @app/views/fizz_buzzes/index.html.erb@:
253
<pre><code class="ruby">
254
<%= link_to "Blurfl", blurfl_fizz_buzz_url(id: @object.uuid), {class: 'btn btn-primary', remote: true} %>
255
</code></pre>
256
# Ensure the targeted action responds appropriately to both "js" and "html" requests. At minimum:
257
<pre><code class="ruby">
258
class FizzBuzzesController
259
  #...
260
  def blurfl
261
    @howmany = 1
262
    #...
263
    respond_to do |format|
264
      format.js
265
      format.html
266
    end
267
  end
268
end
269
</code></pre>
270
# The @html@ view is used if this is a normal page load (presumably this means the client has turned off JS).
271
#* @app/views/fizz_buzz/blurfl.html.erb@
272
<pre><code>
273
<p>I am <%= @howmany %></p>
274
</code></pre>
275
# The @js@ view is used if this is an AJAX request. It renders as JavaScript code which will be executed in the browser. Say, in @app/views/fizz_buzz/blurfl.js.erb@:
276
<pre><code class="javascript">
277
window.alert('I am <%= @howmany %>');
278
</code></pre>
279
# The browser opens an alert box:
280
<pre>
281
I am 1
282
</pre>
283
# A common task is to render a partial and use it to update part of the page. Say the partial is in @app/views/fizz_buzz/_latest_news.html.erb@:
284
<pre><code class="javascript">
285
var new_content = "<%= escape_javascript(render partial: 'latest_news') %>";
286
if ($('div#latest-news').html() != new_content)
287
   $('div#latest-news').html(new_content);
288
</code></pre>
289
290
*TODO: error handling*
291
292
h3. AJAX invoked from custom JavaScript (JSON response)
293
294
(and error handling)
295
296
h3. Add JavaScript triggers and fancy behavior
297
298
Some guidelines for implementing stuff nicely in JavaScript:
299
* Don't rely on the DOM being loaded before your script is loaded.
300
** If you need to inspect/alter the DOM as soon as it's loaded, make a setup function that fires on "document ready" and "ajax:complete".
301
** jQuery's delegated event pattern can help keep your code clean. See http://api.jquery.com/on/
302
<pre><code class="javascript">
303
// worse:
304
$('table.fizzbuzzer tr').
305
    on('mouseover', function(e, xhr) {
306
        console.log("This only works if the table exists when this setup script is executed.");
307
    });
308
// better:
309
$(document).
310
    on('mouseover', 'table.fizzbuzzer tr', function(e, xhr) {
311
        console.log("This works even if the table appears (or has the fizzbuzzer class added) later.");
312
    });
313
</code></pre>
314
315
* If your code really only makes sense for a particular view, rather than embedding @<script>@ tags in the middle of the page,
316
** use this:
317
<pre><code class="ruby">
318
<% content_for :js do %>
319
console.log("hurray, this goes in HEAD");
320
<% end %>
321
</code></pre>
322
** or, if your code should run after [most of] the DOM is loaded:
323
<pre><code class="ruby">
324
<% content_for :footer_js do %>
325
console.log("hurray, this runs at the bottom of the BODY element in the default layout.");
326
<% end %>
327
</code></pre>
328
329
* Don't just write JavaScript on the @fizz_buzzes/blurfl@ page and rely on the fact that the only @table@ element on the page is the one you want to attach your special behavior to. Instead, add a class to the table, and use a jQuery selector to attach special behavior to it.
330
** In @app/views/fizz_buzzes/blurfl.html.erb@
331
<pre>
332
<table class="fizzbuzzer">
333
 <tr>
334
  <td>fizz</td><td>buzz</td>
335
 </tr>
336
</table>
337
</pre>
338
** In @app/assets/javascripts/fizz_buzzes.js@
339
<pre><code class="javascript">
340
<% content_for :js do %>
341
$(document).on('mouseover', 'table.fizzbuzzer tr', function() {
342
    console.log('buzz');
343
});
344
<% end %>
345
</code></pre>
346
** Advantage: You can reuse the special behavior in other tables/pages/classes
347
** Advantage: The JavaScript can get compiled, minified, cached in the browser, etc., instead of being rendered with every page view
348
** Advantage: The JavaScript code is available regardless of how the content got into the DOM (regular page view, partial update with AJAX)
349
350 14 Phil Hodgson
* If the result of clicking on some link invokes Javascript that will ultimately change the content of the current page using @window.location.href=@ then it is advisable to add to the link the @force-cache-reload@ CSS class. By doing so, when a user uses the browser-back button to return to the original page, it will be forced to reload itself from the server, thereby reflecting the updated content. (Ref: https://arvados.org/issues/3634)
351
352 11 Tom Clegg
h3. Invoking chooser
353
354
Example from @app/views/projects/_show_contents.html.erb@:
355
356
<pre>
357
    <%= link_to(
358
          choose_collections_path(
359
            title: 'Add data to project:',
360
            multiple: true,
361
            action_name: 'Add',
362
            action_href: actions_path(id: @object.uuid),
363
            action_method: 'post',
364
            action_data: {selection_param: 'selection[]', copy_selections_into_project: @object.uuid, success: 'page-refresh'}.to_json),
365
          { class: "btn btn-primary btn-sm", remote: true, method: 'get', data: {'event-after-select' => 'page-refresh'} }) do %>
366
      <i class="fa fa-fw fa-plus"></i> Add data...
367
    <% end %>
368
</pre>
369
370
Tour:
371 1 Tom Clegg
372
(TODO)
373
374 15 Tom Clegg
h3. Infinite scroll
375
376
When showing a list that might be too long to render up front in its entirety, use the infinite-scroll feature.
377
378
Links/buttons that flip to page 1, 2, 3, etc. (e.g., <code class="ruby">render partial: "paging"</code>) are deprecated.
379
380
The comments that should be at the top of source:apps/workbench/app/assets/javascripts/infinite_scroll.js, when we write them, will tell you how to do it.
381
382
383
h3. Filtering lists
384
385
When a list is displayed, and the user might want to filter them by selecting a category or typing a search string, use @class="filterable"@. It's easy!
386
387
The comments at the top of source:apps/workbench/app/assets/javascripts/filterable.js tell you how to do it.
388
389 1 Tom Clegg
h3. Tabs/panes on index & show pages
390
391
(TODO)
392
393
h3. User notifications
394
395
(TODO)
396
397
h3. Customizing breadcrumbs
398
399
(TODO)
400
401
h3. Making a page accessible before login
402
403
(TODO)
404
405
h3. Making a page accessible to non-active users
406
407
(TODO)
408 21 Phil Hodgson
409
h3. Developing and Testing the Job Log
410
411
To assist with developing and testing the live job log that updates itself via websockets, there is a rake task that will "replay" a log from a file as if it had been generated by a real job. _Note that this is done within the API Server context_, so you must first switch the current directory appropriately (@cd services/api@). The task takes up to three arguments:
412
413
* log path and filename
414
** The relative path to the log file you want to "replay".
415
* time multipler (optional)
416
** The speed factor at which this log replay should be simulated. The default is 1.0, or normal speed. Higher numbers will proportionately increase the speed of the simulation. For example "4" will make it so that log entries that normally would have appeared over the course of four minutes will appear over the course of one minute. Numbers between 0 and 1 will slow down the simulation.
417
* simulated job uuid (optional)
418
** By providing a job UUID to simulate, the rake task will replace the job UUID in the log file with this job UUID. This means that you can be observing the effects on the Log tab of a particular job but use the log file output from another job.
419
420
Note that as with all rake tasks, if there are confusing characters in the list of arguments, including spaces separating the arguments, you will need to enclose the rake argument in quotation marks.
421
422
Example:
423
424
<pre>
425
  rake "replay_job_log[path/to/your.log, 2.0, qr1hi-8i9sb-nf3qk0xzwwz3lre]"'
426
</pre>
427
428
A typical testing iteration using this task would work as follows:
429
430
# Delete the entries from the LOGS table for the Job UUID you will be observing.
431
# Refresh the browser page showing the Job's Log to clear the graph and graph contents.
432
# Run the rake task
433
# Enjoy and/or write beautiful code improvements
434
# Repeat