Arvados

h1. Hacking Workbench

{{toc}}

h2. Source tree layout

Everything is in @/apps/workbench@.

Key pieces to know about before going much further:

|/|Usual Rails project layout|

|/app/controllers/application_controller.rb|Controller superclass with authentication setup, error handling, and generic CRUD actions|

|/app/controllers/*.rb|Actions other than generic CRUD (users#activity, jobs#generate_provenance, ...)|

|/app/models/arvados_base.rb|Default Arvados model behavior and ActiveRecord-like accessors and introspection features|

|/app/models/arvados_resource_list.rb|ActiveRelation-like class (what you get from Model.where() etc.)|

h2. Background resources

Workbench is a Rails 4 application.

* "Getting started with Rails":http://guides.rubyonrails.org/getting_started.html at rubyonrails.org

* "AJAX in Rails 3.1":http://blog.madebydna.com/all/code/2011/12/05/ajax-in-rails-3.html blog post (still relevant in Rails 4)

Javascript readings

* http://javascript.crockford.com/code.html

* http://javascript.crockford.com/private.html

Angular readings

* http://www.ng-newsletter.com/posts/beginner2expert-how_to_start.html

* https://docs.angularjs.org/guide/introduction

* https://github.com/johnpapa/angularjs-styleguide

h2. Unlike a typical Rails project...

* ActiveRecord in Workbench doesn't talk to the database directly, but instead queries the Arvados API as REST client.

* The Arvados query API is somewhat limited and doesn't accept SQL statements, so Workbench has to work harder to get what it needs.

* Workbench itself only has the privileges of the Workbench user: when making Arvados API calls, it uses the API token provided by the user.

h2. Unlike what you might expect...

* Workbench doesn't use the Ruby SDK. It uses a sort of baked-in Rails SDK.

** TODO: move it out of Workbench into a gem.

** TODO: use the Ruby SDK under the hood.

h2. Running in development mode

h3. SSL certificates

You can get started quickly with SSL by generating a self-signed certificate:

 openssl req -new -x509 -nodes -out ~/self-signed.pem -keyout ~/self-signed.key -days 3650 -subj '/CN=arvados.example.com'

Alternatively, download a set from the bottom of the [[API server]] page.

h3. Download and configure

Follow "these instructions":http://doc.arvados.org/install/install-workbench-app.html to download the source and configure your workbench instance.

h3. Start the server

Save something like the following at @~/bin/workbench@, make it executable[1], make sure @~/bin@ is in your path[2]:

 #!/bin/sh

set -e

cd ~/arvados/apps/workbench

export RAILS_ENV=development

bundle install --path=vendor/bundle

exec bundle exec passenger start -p 3031 --ssl --ssl-certificate ~/self-signed.pem --ssl-certificate-key ~/self-signed.key

The first time you run the above it will take a while to install all the ruby gems. In particular @Installing nokogiri@ takes a while

Once you see:

 =============== Phusion Passenger Standalone web server started ===============

You can visit your server at:

 @https://{ip-or-host}:3031/@

You can kill your server with @ctrl-C@ but if you get disconnected from the terminal, it will continue running. You can kill it by running

 @ps x |grep nginx |grep master@

And then

 @kill ####@

Replacing #### with the number in the left column returned by ps

fn1. @chmod +x ~/bin/workbench@

fn2. In Debian systems, the default .profile adds ~/bin to your path, but only if it exists when you log in. If you just created ~/bin, doing @exec bash -login@ or @source .profile@ should make ~/bin appear in your path.

h2. Running tests

The test suite brings up an API server in test mode, and runs browser tests with Firefox.

Make sure API server has its dependencies in place and its database schema up-to-date.

<pre>

(

 set -e

 cd ../../services/api

 RAILS_ENV=test bundle install --path=vendor/bundle

 RAILS_ENV=test bundle exec rake db:migrate

)

</pre>

Install headless testing tools.

<pre>

sudo apt-get install xvfb iceweasel

</pre>

(Install firefox instead of iceweasel if you're not using Debian.)

Install phantomjs. (See http://phantomjs.org/download.html for latest version.)

<pre>

wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2

sudo tar -C /usr/local -xjf /tmp/phantomjs-1.9.8-linux-x86_64.tar.bz2

sudo ln -s ../phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/local/bin/

</pre>

Run the test suite.

<pre>

RAILS_ENV=test bundle exec rake test

</pre>

h3. When tests fail...

When an integration test fails (or skips) a screenshot is automatically saved in @arvados/apps/workbench/tmp/workbench-fail-1.png@, etc.

By default, @rake test@ just shows F when a test fails (and E when a test crashes) and doesn't tell you which tests had problems until the entire test suite is done. During development it makes more sense to use @TESTOPTS=-v@. This reports after each test the test class and name, outcome, and elapsed time:

* <pre>

$ RAILS_ENV=test bundle exec rake test TESTOPTS=-v

[...]

ApplicationControllerTest#test_links_for_object = 0.10 s = .

[...]

Saved ./tmp/workbench-fail-2.png

CollectionsTest#test_combine_selected_collection_files_into_new_collection = 10.89 s = F

[...]

</pre>

h3. Iterating on a single test

Sometimes you want to poke at the code and re-run a single test to confirm that you made it pass. You don't want to reboot everything just to make Minitest notice that you edited your test.

Here's how:

<pre>

arvados/apps/workbench$ RAILS_ENV=test bundle exec irb -Ilib:test

load 'test/integration/pipeline_instances_test.rb'

>>> Minitest.run(%w(-v -n test_Create_and_run_a_pipeline))

...

PipelineInstancesTest#test_Create_and_run_a_pipeline = 14.72 s = E

...

(Edit test/integration/pipeline_instances_test.rb here)

>>> begin

      Object.send :remove_const, :PipelineInstancesTest

      ::Minitest::Runnable.runnables.reject! { true }

      load 'test/integration/pipeline_instances_test.rb'

      Minitest.run(%w(-v -n test_Create_and_run_a_pipeline))

    end

...

PipelineInstancesTest#test_Create_and_run_a_pipeline = 38.54 s = .

...

</pre>

h2. Loading state from API into models

If your model makes an API call that returns the new state of an object, load the new attributes into the local model with @private_reload@:

<pre><code class="ruby">

  api_response = $arvados_api_client.api(...)

  private_reload api_response

</code></pre>

h2. Features

h3. Authentication

ApplicationController uses an around_filter to make sure the user is logged in, redirect to Arvados to complete the login procedure if not, and store the user's API token in Thread.current[:arvados_api_token] if so.

The @current_user@ helper returns User.current if the user is logged in, otherwise nil. (Generally, only special pages like "welcome" and "error" get displayed to users who aren't logged in.)

h3. Default filter behavior

@before_filter :find_object_by_uuid@

* This is enabled by default, @except :index, :create@.

* It renames the @:id@ param to @:uuid@. (The Rails default routing rules use @:id@ to accept params in path components, but @params[:uuid]@ makes more sense everywhere else in our code.)

* If you define a collection method (where there's no point looking up an object with the :id supplied in the request), skip this.

<pre><code class="ruby">

  skip_before_filter :find_object_by_uuid, only: [:action_that_takes_no_uuid_param]

</code></pre>

h3. Error handling

ApplicationController has a render_error method that shows a standard error page. (It's not very good, but it's better than a default Rails stack trace.)

In a controller you get there like this

<pre><code class="ruby">

  @errors = ['I could not achieve what you wanted.']

  render_error status: 500

</code></pre>

You can also do this, anywhere

<pre><code class="ruby">

  raise 'My spoon is too big.'

</code></pre>

The @render_error@ method sends JSON or HTML to the client according to the Accept header in the request (it sends JSON if JavaScript was requested), so reasonable things happen whether or not the request is AJAX.

h2. Development patterns

h3. Add a model

Currently, when the API provides a new model, we need to generate a corresponding model in Workbench: it's not smart enough to pick up the list of models from the API server's discovery document.

_(Need to fill in details here)_

# @rails generate model ....@

# Delete migration

# Change base class to ArvadosBase

# @rails generate controller ...@ 

Model _attributes_, on the other hand, are populated automatically.

h3. Add a configuration knob

Same situation as API server. See [[Hacking API Server]].

h3. Add an API method

Workbench is not yet smart enough to look in the discovery document for supported API methods. You need to add a method to the appropriate model class before you can use it in the Workbench app.

h3. Writing tests

In integration tests, this makes your tests flaky because the result depends on whether the page has finished loading:

* <pre><code class="ruby">

assert page.has_selector?('a', text: 'foo')  # Danger!

</code></pre>

* Instead, do this:

* <pre><code class="ruby">

assert_selector('a', text: 'foo')

</code></pre>

* This lets Capybara wait for the selector to appear.

h3. AJAX using Rails UJS (remote:true with JavaScript response)

This pattern is the best way to make a button/link that invokes an asynchronous action on the Workbench server side, i.e., before/without navigating away from the current page.

# Add <code class="ruby">remote: true</code> to a link or button. This makes Rails put a <code class="html">data-remote="true"</code> attribute in the HTML element. Say, in @app/views/fizz_buzzes/index.html.erb@:

<pre><code class="ruby">

<%= link_to "Blurfl", blurfl_fizz_buzz_url(id: @object.uuid), {class: 'btn btn-primary', remote: true} %>

</code></pre>

# Ensure the targeted action responds appropriately to both "js" and "html" requests. At minimum:

<pre><code class="ruby">

class FizzBuzzesController

  #...

  def blurfl

    @howmany = 1

    #...

    respond_to do |format|

      format.js

      format.html

    end

  end

end

</code></pre>

# The @html@ view is used if this is a normal page load (presumably this means the client has turned off JS).

#* @app/views/fizz_buzz/blurfl.html.erb@

<pre><code>

<p>I am <%= @howmany %></p>

</code></pre>

# The @js@ view is used if this is an AJAX request. It renders as JavaScript code which will be executed in the browser. Say, in @app/views/fizz_buzz/blurfl.js.erb@:

<pre><code class="javascript">

window.alert('I am <%= @howmany %>');

</code></pre>

# The browser opens an alert box:

<pre>

I am 1

</pre>

# A common task is to render a partial and use it to update part of the page. Say the partial is in @app/views/fizz_buzz/_latest_news.html.erb@:

<pre><code class="javascript">

var new_content = "<%= escape_javascript(render partial: 'latest_news') %>";

if ($('div#latest-news').html() != new_content)

   $('div#latest-news').html(new_content);

</code></pre>

*TODO: error handling*

h3. AJAX invoked from custom JavaScript (JSON response)

(and error handling)

h3. Add JavaScript triggers and fancy behavior

Some guidelines for implementing stuff nicely in JavaScript:

* Don't rely on the DOM being loaded before your script is loaded.

** If you need to inspect/alter the DOM as soon as it's loaded, make a setup function that fires on "document ready" and "ajax:complete".

** jQuery's delegated event pattern can help keep your code clean. See http://api.jquery.com/on/

<pre><code class="javascript">

// worse:

$('table.fizzbuzzer tr').

    on('mouseover', function(e, xhr) {

        console.log("This only works if the table exists when this setup script is executed.");

    });

// better:

$(document).

    on('mouseover', 'table.fizzbuzzer tr', function(e, xhr) {

        console.log("This works even if the table appears (or has the fizzbuzzer class added) later.");

    });

</code></pre>

* If your code really only makes sense for a particular view, rather than embedding @<script>@ tags in the middle of the page,

** use this:

<pre><code class="ruby">

<% content_for :js do %>

console.log("hurray, this goes in HEAD");

<% end %>

</code></pre>

** or, if your code should run after [most of] the DOM is loaded:

<pre><code class="ruby">

<% content_for :footer_js do %>

console.log("hurray, this runs at the bottom of the BODY element in the default layout.");

<% end %>

</code></pre>

* Don't just write JavaScript on the @fizz_buzzes/blurfl@ page and rely on the fact that the only @table@ element on the page is the one you want to attach your special behavior to. Instead, add a class to the table, and use a jQuery selector to attach special behavior to it.

** In @app/views/fizz_buzzes/blurfl.html.erb@

<pre>

<table class="fizzbuzzer">

 <tr>

  <td>fizz</td><td>buzz</td>

 </tr>

</table>

</pre>

** In @app/assets/javascripts/fizz_buzzes.js@

<pre><code class="javascript">

<% content_for :js do %>

$(document).on('mouseover', 'table.fizzbuzzer tr', function() {

    console.log('buzz');

});

<% end %>

</code></pre>

** Advantage: You can reuse the special behavior in other tables/pages/classes

** Advantage: The JavaScript can get compiled, minified, cached in the browser, etc., instead of being rendered with every page view

** Advantage: The JavaScript code is available regardless of how the content got into the DOM (regular page view, partial update with AJAX)

* If the result of clicking on some link invokes Javascript that will ultimately change the content of the current page using @window.location.href=@ then it is advisable to add to the link the @force-cache-reload@ CSS class. By doing so, when a user uses the browser-back button to return to the original page, it will be forced to reload itself from the server, thereby reflecting the updated content. (Ref: https://arvados.org/issues/3634)

h3. Invoking chooser

Example from @app/views/projects/_show_contents.html.erb@:

<pre>

    <%= link_to(

          choose_collections_path(

            title: 'Add data to project:',

            multiple: true,

            action_name: 'Add',

            action_href: actions_path(id: @object.uuid),

            action_method: 'post',

            action_data: {selection_param: 'selection[]', copy_selections_into_project: @object.uuid, success: 'page-refresh'}.to_json),

          { class: "btn btn-primary btn-sm", remote: true, method: 'get', data: {'event-after-select' => 'page-refresh'} }) do %>

      <i class="fa fa-fw fa-plus"></i> Add data...

    <% end %>

</pre>

Tour:

(TODO)

h3. Infinite scroll

When showing a list that might be too long to render up front in its entirety, use the infinite-scroll feature.

Links/buttons that flip to page 1, 2, 3, etc. (e.g., <code class="ruby">render partial: "paging"</code>) are deprecated.

The comments that should be at the top of source:apps/workbench/app/assets/javascripts/infinite_scroll.js, when we write them, will tell you how to do it.

h3. Filtering lists

When a list is displayed, and the user might want to filter them by selecting a category or typing a search string, use @class="filterable"@. It's easy!

The comments at the top of source:apps/workbench/app/assets/javascripts/filterable.js tell you how to do it.

h3. Tabs/panes on index & show pages

(TODO)

h3. User notifications

(TODO)

h3. Customizing breadcrumbs

(TODO)

h3. Making a page accessible before login

(TODO)

h3. Making a page accessible to non-active users

(TODO)

h3. Developing and Testing the Job Log

To assist with developing and testing the live job log that updates itself via websockets, there is a rake task that will "replay" a log from a file as if it had been generated by a real job. _Note that this is done within the API Server context_, so you must first switch the current directory appropriately (@cd services/api@). The task takes up to three arguments:

* log path and filename

** The relative path to the log file you want to "replay".

* time multipler (optional)

** The speed factor at which this log replay should be simulated. The default is 1.0, or normal speed. Higher numbers will proportionately increase the speed of the simulation. For example "4" will make it so that log entries that normally would have appeared over the course of four minutes will appear over the course of one minute. Numbers between 0 and 1 will slow down the simulation.

* simulated job uuid (optional)

** By providing a job UUID to simulate, the rake task will replace the job UUID in the log file with this job UUID. This means that you can be observing the effects on the Log tab of a particular job but use the log file output from another job.

Note that as with all rake tasks, if there are confusing characters in the list of arguments, including spaces separating the arguments, you will need to enclose the rake argument in quotation marks.

Example:

<pre>

  rake "replay_job_log[path/to/your.log, 2.0, qr1hi-8i9sb-nf3qk0xzwwz3lre]"'

</pre>

A typical testing iteration using this task would work as follows:

# Delete the entries from the LOGS table for the Job UUID you will be observing.

# Refresh the browser page showing the Job's Log to clear the graph and graph contents.

# Run the rake task

# Enjoy and/or write beautiful code improvements

# Repeat