Arvados

https://github.com/DatabaseCleaner/database_cleaner

4533-remote-reset @ 8bb8524 75634ad

• Updating bundle (524fba5) makes test suites run in random order by default, which (over time) reveals order dependencies between suites.

Now at bc20697 with one more test (missed a git add somewhere!), master merged, tests passing.

• This branch includes modifications to the Workbench test suites to make use of the new API. (It turns out Workbench test classes are order-dependent otherwise.)
• This branch does not include changes to the other test suites (Python, FUSE, et al.) because they don't seem to need them. As long as that holds true, it'll be easier to invoke this API from run-tests.sh in #4534 instead.

Reviewing bc20697

Code¶

• I think it would be better to read and set Rails.env rather than the process environment directly.
• Tests with their own "put things back the way they were" code (e.g., "reset fails when not in test mode") should do so with begin/ensure, in case the tested code crashes.
• remote_reset_test.rb seems to be testing a lot more than the database reset. Is it necessary to do GETs on the old and new specimens before a reset? Can't we trust the assert_response :success after the respective delete and create operations?

Philosophy¶

• I'm not sure I understand your earlier comment about the Python and FUSE tests not needing this. The reason they don't "need" it is because they restart the API server between test suites. Refactoring to use this API would provide a noticeable performance improvement and improve test isolation.
• I'm not sure I see the point in the configuration knob. As far as I can tell, its only purpose is to let you break the tests by setting it false in the testing environment. In production, it doesn't give me any additional security, because I'm never running in the test environment. It seems to me that we should use Rails.env alone to decide whether or not to advertise the route, deny permission, etc.
• Is there a reason not to reset the database in ActiveSupport::TestCase#teardown, other than performance? Rigging things up so that there's a line you must add to all of your test classes to do the right thing seems error-prone to me. We now have a much nicer tool to provide better test isolation; why not make full use of it?

Thanks.

Brett Smith wrote:

• I think it would be better to read and set Rails.env rather than the process environment directly.

Agreed, fixed.

• Tests with their own "put things back the way they were" code (e.g., "reset fails when not in test mode") should do so with begin/ensure, in case the tested code crashes.

Fixed.

• remote_reset_test.rb seems to be testing a lot more than the database reset. Is it necessary to do GETs on the old and new specimens before a reset? Can't we trust the assert_response :success after the respective delete and create operations?

I guess we can, but I thought it wouldn't hurt to check. I added a "perhaps redundant" comment there to make it more clear that it's just part of confirming setup, and that the real feature test happens down below.

• I'm not sure I understand your earlier comment about the Python and FUSE tests not needing this. The reason they don't "need" it is because they restart the API server between test suites. Refactoring to use this API would provide a noticeable performance improvement and improve test isolation.

Ah, good point. They don't need this in that they currently work (as opposed to Workbench, which fails tests without it if you upgrade Minitest), but they do need it in that they're currently getting by (slowly) with an inferior workaround. So we should say run-tests.sh and Python/fuse tests will be using this.

• I'm not sure I see the point in the configuration knob. As far as I can tell, its only purpose is to let you break the tests by setting it false in the testing environment. In production, it doesn't give me any additional security, because I'm never running in the test environment. It seems to me that we should use Rails.env alone to decide whether or not to advertise the route, deny permission, etc.

• it seems OK to have multiple safety belts.
• somehow it seems more readable to always using config to decide which way to behave, but in this special case add the env==test check at the last minute as a sanity check.
• the config knob mentions the existence of the feature in a place where people will normally look when discovering the difference between test/dev/prod environments.
• if someone wants to use env=test for something where it useful to have this turned off, we might as well give them an easy way.
• IIRC Ward asked for something like this.

• Is there a reason not to reset the database in ActiveSupport::TestCase#teardown, other than performance? Rigging things up so that there's a line you must add to all of your test classes to do the right thing seems error-prone to me. We now have a much nicer tool to provide better test isolation; why not make full use of it?

Performance is the only reason. I think it takes ~1 second. Perhaps defaulting to once per test, and allowing a test class to override that and get once-per-class or none at all, would be better?

(It should be easy to do this by default after each test, or class -- as long as we have a way for the diagnostics test classes to skip it.)

Now at e65d698

Tom Clegg wrote:

Brett Smith wrote:

• remote_reset_test.rb seems to be testing a lot more than the database reset. Is it necessary to do GETs on the old and new specimens before a reset? Can't we trust the assert_response :success after the respective delete and create operations?

I guess we can, but I thought it wouldn't hurt to check.

I guess this was a philosophical comment too. :) Checking more than necessary hurts the diagnostic value of a test: if you introduce a bug, more tests will fail, and you'll have more work to do to find the common thread between them to identify the bug. They also make the tests slower. Stuff like this is part of the reason why our builds are taking 45 minutes now.

I realize that there are plenty of worse offenders across our test suite, and that this branch is overall going to improve our test runtime noticeably. But it seems like the best way to improve test discipline is to practice at every opportunity.

• I'm not sure I understand your earlier comment about the Python and FUSE tests not needing this. The reason they don't "need" it is because they restart the API server between test suites. Refactoring to use this API would provide a noticeable performance improvement and improve test isolation.

Ah, good point. They don't need this in that they currently work (as opposed to Workbench, which fails tests without it if you upgrade Minitest), but they do need it in that they're currently getting by (slowly) with an inferior workaround. So we should say run-tests.sh and Python/fuse tests will be using this.

Yeah, that sounds good to me.

• I'm not sure I see the point in the configuration knob.

• it seems OK to have multiple safety belts.

Doing this correctly requires a lot of development discipline. How often have a project's developers confused the relationship between two security limits over time? It's easy to forget the exact semantics of each knob, forget how they relate, forget which components are responsible for enforcing each, etc. You have a very clear model in your head right now that says "Always check the configuration, except just prior to running the method," but will that always remain clear to future readers?

I submit that one belt would actually be more secure: there's much less risk that someone will make a mistake writing if Rails.env == "test" than they will correctly choosing whether to check the configuration or the environment at a given point in the code. If you still want to go ahead with this, I'd at least suggest writing one method that checks all the booleans, and then call it from everywhere else (route setup, exception at the top of the controller method).

• IIRC Ward asked for something like this.

I remember this, but the way Rails layers its configuration settings makes the end result come out looking funny from a deployment perspective. Assume I'm an admin with a normal production setup. If I want to make my setup "more secure" (i.e., make sure the feature is always disabled), I have to add an entire new test section to my application.yml—and when would that really be relevant, since I don't have a test database or other infrastructure set up? It would be nicer if there were a "super-common" section that overrode all the environment-specific ones, but there's not.

• Is there a reason not to reset the database in ActiveSupport::TestCase#teardown, other than performance? Rigging things up so that there's a line you must add to all of your test classes to do the right thing seems error-prone to me. We now have a much nicer tool to provide better test isolation; why not make full use of it?

Performance is the only reason. I think it takes ~1 second. Perhaps defaulting to once per test, and allowing a test class to override that and get once-per-class or none at all, would be better?

I like that idea, yeah. Thanks.

Discussed this some in the office with Ward. We came to the conclusion that what would be helpful is additional bounds checking in the controller method itself to verify that the instance really is, and always has been, a test instance. For example, system_user.uuid == system_user_uuid. That would help prevent the reset button from being used on an instance where the environment has accidentally been switched to test, or something like that. A little creativity could be applied to the exact test(s) added. But with that addition, Ward's sold on removing the configuration knob.

Now resetting after each test case unless otherwise specified.

• 300 runs
• 76eaaac → Pass: workbench tests (1053s) (one reset per class)
• a69a1cb → Pass: workbench tests (1261s) (one reset per test)
• f3460b2 → Pass: workbench tests (1217s) (one reset per test, with a couple of classes set to one-per-class)

(Along the way I noticed we had achieved a state where different FoosControllerTest classes were given in functional/foos_* and controllers/foos_*. I consolidated everything into controllers/, except one test which actually seemed better off in helpers/.)

[...] you'll have more work to do to find the common thread between them to identify the bug. [...]

Indeed. Based on these comments and IRC discussion, I've split "do basics still work after reset?" into its own test in e3914c4.

Stuff like this is part of the reason why our builds are taking 45 minutes now.

I do take your point that efficiency of test suites is valuable, but I wouldn't quite go that far. My impression is that our build/test slowness has a lot more to do with proliferation of expensive operations like "shutdown rails server; import fixtures; startup rails server" and using selenium to test things that should be [made to be] testable by smaller hammers. If it were just a problem of too many controller invocations and single-row pg queries, wouldn't we be grumbling about 45 seconds instead of 45 minutes?

On that note, I wonder what our rule should be about when to use "one api-reset per class" vs. "one api-reset per test", given that the obvious/safe option costs 3 minutes of database resetting per Workbench test suite?

Brett Smith wrote:

Discussed this some in the office with Ward. We came to the conclusion that what would be helpful is additional bounds checking in the controller method itself to verify that the instance really is, and always has been, a test instance. For example, system_user.uuid == system_user_uuid. That would help prevent the reset button from being used on an instance where the environment has accidentally been switched to test, or something like that. A little creativity could be applied to the exact test(s) added. But with that addition, Ward's sold on removing the configuration knob.

1. Rails.env == 'test', and
2. Every user record in the database has either
   1. uuid equal to a user uuid in test/fixtures/users.yml, or
   2. email ending with @example.com (this lets Workbench tests create test users without making the database unresettable).

Tom Clegg wrote:

Stuff like this is part of the reason why our builds are taking 45 minutes now.

I do take your point that efficiency of test suites is valuable, but I wouldn't quite go that far. My impression is that our build/test slowness has a lot more to do with proliferation of expensive operations like "shutdown rails server; import fixtures; startup rails server" and using selenium to test things that should be [made to be] testable by smaller hammers.

I think we're in pretty broad agreement here. You're right that general test overhead is a bigger performance problem. And I view some of those Selenium tests you're talking about as the bigger version of this problem: some of them follow a pattern of checking lots of semi-related things. I don't want to overstate my case; I'm just saying, it all adds up.

On that note, I wonder what our rule should be about when to use "one api-reset per class" vs. "one api-reset per test", given that the obvious/safe option costs 3 minutes of database resetting per Workbench test suite?

I'm still in favor of resetting after every test. To me, improved test isolation seems worth the cost. It's very disruptive to us when state between tests breaks the build and requires us to diagnose and fix it. If it saves us even one incident of that, I think it's paid off. And again, I expect overall test time to go down when the Python SDK/FUSE tests are refactored to use this reset. (By contrast, asserting functionality that isn't being directly tested doesn't seem worth it to me, because we pay more to make the tests less focused.)

9d28157 removes the config knob and adds a database-content sanity check.

The knob is still listed and documented in the common section of application.default.yml.

Other comments on 9d28157:

• In the database controller tests, I think the test would be best written this way:

   test "route not found when not in test mode" do
      authorize_with :admin
      env_was = Rails.env
      begin
        Rails.env = 'production'
        Rails.application.reload_routes!
        assert_raises ActionController::RoutingError do
          post :reset
        end
      ensure
        Rails.env = env_was
        Rails.application.reload_routes!
      end
    end
  

  This removes a route reload before the begin that seems to be a noop, and tightens the assertion to make sure the exception comes from the place we expect.

• To match, does the integration test "reset fails when Rails.env != 'test'" need to reload routes in its ensure block?
• Why was the system_user fixture added? I thought this was covered by DatabaseSeeds.
• Is it common Ruby style to put require statements right before the feature is used? DatabaseController does this with active_record/fixtures, although it's not the only place in our code. If it's expected, it's fine; it just throws me off because I think every other language I've worked with encourages you to move those to the top of the file.
• Why are database_reset_test.rb and remote_reset_test.rb separate?
• I believe the rationale for moving Workbench tests around is that tests/functional was the Rails 3 layout, and tests/controllers is the Rails 4 layout, so everything is moved to the latter. Is that right?

Thanks.

Brett Smith wrote:

I'm still in favor of resetting after every test. To me, improved test isolation seems worth the cost. It's very disruptive to us when state between tests breaks the build and requires us to diagnose and fix it. If it saves us even one incident of that, I think it's paid off.

Perhaps a better way to optimize this would be for the API server to remember whether anything has actually changed since the last reset? (Not going to do that now.) Meanwhile, do you think I should remove the few reset_api_fixtures :after_each_test, false I put in?

The knob is still listed and documented in the common section of application.default.yml.

Oops. Fixed.

Other comments on 9d28157:

• In the database controller tests, I think the test would be best written this way:

[...]

This removes a route reload before the begin that seems to be a noop, and tightens the assertion to make sure the exception comes from the place we expect.

Fixed.

• To match, does the integration test "reset fails when Rails.env != 'test'" need to reload routes in its ensure block?

It did it in teardown, after restore_configuration -- but since it's only needed by that one test, I moved it to an ensure block.

• Why was the system_user fixture added? I thought this was covered by DatabaseSeeds.

Its email is not @example.com so it looked like real data, and our other seeds are also present as fixtures so I figured becoming consistent would be a reasonable way to fix it. Alternatively, we could whitelist the two seed users, and remove the seed users/groups from fixtures. (If any tests refer to the seeds by fixture name rather than CurrentApiClient, or something like that, it probably wouldn't be hard to fix.)

• Is it common Ruby style to put require statements right before the feature is used? DatabaseController does this with active_record/fixtures, although it's not the only place in our code. If it's expected, it's fine; it just throws me off because I think every other language I've worked with encourages you to move those to the top of the file.

In this case I thought (but haven't actually verified) this prevents activesupport/fixtures from becoming a runtime dependency in non-test environments. Perhaps it's already a runtime dependency anyway (e.g., inextricable from other bits of activesupport), but I didn't think it was necessary to assume that.

• Why are database_reset_test.rb and remote_reset_test.rb separate?

Hm, I think just because I fragmented my development across two machines and missed a "git add" somewhere. I've consolidated them into database_reset_test.rb.

• I believe the rationale for moving Workbench tests around is that tests/functional was the Rails 3 layout, and tests/controllers is the Rails 4 layout, so everything is moved to the latter. Is that right?

Yes. The impetus for touching anything is that we had stuff spread across both, sometimes using the same class names, which is really bad for dev sanity (which teardown(s)??). The rationale for choosing tests/controllers instead of tests/functional as the winner is that Rails 4 says so.

Now at af3d57d

Tom Clegg wrote:

Brett Smith wrote:

I'm still in favor of resetting after every test. To me, improved test isolation seems worth the cost. It's very disruptive to us when state between tests breaks the build and requires us to diagnose and fix it. If it saves us even one incident of that, I think it's paid off.

Perhaps a better way to optimize this would be for the API server to remember whether anything has actually changed since the last reset? (Not going to do that now.) Meanwhile, do you think I should remove the few reset_api_fixtures :after_each_test, false I put in?

I don't feel strongly about that. I thought it was important to have the default be "reset between each test," but I'm fine with low-impact test suites opting out of that as an optimization. af3d57d looks good to merge to me, thanks.

Now this happens sometimes:

   (0.7ms)  ALTER TABLE "api_client_authorizations" ENABLE TRIGGER ALL;ALTER TABLE "api_clients" [...]
#<ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 28078 waits for AccessExclusiveLock on relation 922935 of database 922763; blocked by process 24889.
Process 24889 waits for RowExclusiveLock on relation 922764 of database 922763; blocked by process 28078.
HINT:  See server log for query details.

Hard to know, but might be fixed by 34d7bf5 on 4533-lock-tables-first

4533-lock-tables-first looks okay to me at 34d7bf56. Would it make sense to use something like http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html for locking the records? It would be nice if there's a Rails locking convention that we could apply here rather than baking Postgres lock requests into the code.

Tim Pierce wrote:

4533-lock-tables-first looks okay to me at 34d7bf56. Would it make sense to use something like http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html for locking the records? It would be nice if there's a Rails locking convention that we could apply here rather than baking Postgres lock requests into the code.

We're deleting all rows, so row-locking would probably get a bit inefficient and wouldn't cover insertions. (Is there a table lock API in Rails?) AFAICT this whole action breaks so many of Rails' expectations about what happens in a controller action (or while the application is running!) that Rails' APIs can be expected to let us down. I'm definitely OK with leaving this as a "Postgres-only" action at least long enough to find out whether it even solves Jenkins' deadlock problem...

Saw deadlock error on workbench tests at around 2PM (have latest from master). API server log for that error:

(0.7ms)  SELECT COUNT(DISTINCT "authorized_keys"."id") FROM "authorized_keys" WHERE (authorized_keys.uuid in ('zzzzz-tpzed-xurymjxw79nv3jz') OR authorized_keys.owne
r_uuid in ('zzzzz-tpzed-xurymjxw79nv3jz','zzzzz-j7d0g-rew6elm53kancon','zzzzz-j7d0g-22xp1wpjul508rk','zzzzz-j7d0g-v955i6s2oi1cbso','zzzzz-j7d0g-axqo7eu9pwvna1x','zzzzz-j7d0g-fut
rprojviewgrp','zzzzz-j7d0g-fffffffffffffff','zzzzz-j7d0g-ptt1ou6a9lxrv07','zzzzz-j7d0g-cx2al9cqkmsf1hs','zzzzz-j7d0g-0077nzts8c178lw','zzzzz-j7d0g-swqu6hmi4pa7bk7','zzzzz-j7d0g-
anonymouspublic') OR authorized_keys.uuid IN (SELECT head_uuid FROM links WHERE link_class='permission' AND tail_uuid IN ('zzzzz-tpzed-xurymjxw79nv3jz', 'zzzzz-j7d0g-rew6elm53ka
ncon', 'zzzzz-j7d0g-22xp1wpjul508rk', 'zzzzz-j7d0g-v955i6s2oi1cbso', 'zzzzz-j7d0g-axqo7eu9pwvna1x', 'zzzzz-j7d0g-futrprojviewgrp', 'zzzzz-j7d0g-fffffffffffffff', 'zzzzz-j7d0g-pt
t1ou6a9lxrv07', 'zzzzz-j7d0g-cx2al9cqkmsf1hs', 'zzzzz-j7d0g-0077nzts8c178lw', 'zzzzz-j7d0g-swqu6hmi4pa7bk7', 'zzzzz-j7d0g-anonymouspublic'))) AND (authorized_keys.authorized_use
r_uuid='zzzzz-tpzed-xurymjxw79nv3jz')
Completed 200 OK in 10.0ms (Views: 0.5ms | ActiveRecord: 2.6ms)
   (0.6ms)  UPDATE "api_client_authorizations" SET "last_used_at" = '2014-12-04 18:48:09.099640', "last_used_by_ip_address" = '127.0.0.1', "updated_at" = '2014-12-0
4 18:48:09.105870', "scopes" = '---
- all
' WHERE "api_client_authorizations"."id" = 230334899
   (2.2ms)  COMMIT
Started POST "/database/reset" for 127.0.0.1 at 2014-12-04 13:48:09 0500
Processing by DatabaseController#reset as /*
  Parameters: {"api_token"=>"1a9ffdcga2o7cw8q12dndskomgs1ygli3ns9k2o9hgzgmktc78", "reader_tokens"=>"[]"}
WARNING: Can't verify CSRF token authenticity
  ApiClientAuthorization Load (1.3ms)  SELECT "api_client_authorizations". FROM "api_client_authorizations" WHERE (api_token='3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2fod
9zru30k1jqdmi' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) LIMIT 1
  User Load (0.8ms)  SELECT "users".* FROM "users" WHERE (email is null or email not like '%@example.com')
  ApiClient Load (0.8ms)  SELECT "api_clients".* FROM "api_clients" WHERE "api_clients"."id" IN (243589807)
  User Load (0.7ms)  SELECT "users".* FROM "users" WHERE "users"."id" IN (186579971)
   (0.3ms)  BEGIN
  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE "users"."uuid" = 'zzzzz-tpzed-xurymjxw79nv3jz' LIMIT 1
   (0.4ms)  BEGIN
   (999.9ms)  UPDATE "api_client_authorizations" SET "last_used_at" = '2014-12-04 18:48:09.125006', "updated_at" = '2014-12-04 18:48:09.135593', "scopes" = '--
- all
' WHERE "api_client_authorizations"."id" = 186579971
  ApiClientAuthorization Load (993.9ms)  SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2f
od9zru30k1jqdmi' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) LIMIT 1
   (1006.7ms)  LOCK TABLE "api_client_authorizations","api_clients","authorized_keys","collections","groups","jobs","keep_disks","keep_services","links","logs","nod
es","pipeline_instances","pipeline_templates","repositories","specimens","traits","users","virtual_machines" IN ACCESS EXCLUSIVE MODE
PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 28249 waits for AccessExclusiveLock on relation 3602829 of database 3602657; blocked by process 16323.
Process 16323 waits for RowExclusiveLock on relation 3602658 of database 3602657; blocked by process 28249.
HINT:  See server log for query details.
: LOCK TABLE "api_client_authorizations","api_clients","authorized_keys","collections","groups","jobs","keep_disks","keep_services","links","logs","nodes","pipeline_instances"," 
pipeline_templates","repositories","specimens","traits","users","virtual_machines" IN ACCESS EXCLUSIVE MODE
   (1.0ms)  ROLLBACK
#<ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 28249 waits for AccessExclusiveLock on relation 3602829 of database 3602657; blocked by process 16323.
Process 16323 waits for RowExclusiveLock on relation 3602658 of database 3602657; blocked by process 28249.
HINT:  See server log for query details.
: LOCK TABLE "api_client_authorizations","api_clients","authorized_keys","collections","groups","jobs","keep_disks","keep_services","links","logs","nodes","pipeline_instances"," 
pipeline_templates","repositories","specimens","traits","users","virtual_machines" IN ACCESS EXCLUSIVE MODE>
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/postgresql_adapter.rb:650:in `async_exec'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/postgresql_adapter.rb:650:in `block in execute'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute'
/home/radhika/arvados/services/api/app/controllers/database_controller.rb:36:in `block in reset'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/test_after_commit-0.2.3/lib/test_after_commit.rb:12:in `block in transaction_with_transactional_fixtures'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/test_after_commit-0.2.3/lib/test_after_commit.rb:9:in `transaction_with_transactional_fixtures'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/transactions.rb:208:in `transaction'
/home/radhika/arvados/services/api/app/controllers/database_controller.rb:32:in `reset'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/abstract_controller/base.rb:167:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/rendering.rb:10:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:546:in `_run__2967369565115406249__process_action__3863524393400387349__callbacks'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `__run_callback'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/abstract_controller/callbacks.rb:17:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/rescue.rb:29:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/notifications.rb:123:in `block in instrument'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activesupport-3.2.17/lib/active_support/notifications.rb:123:in `instrument'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/activerecord-3.2.17/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/abstract_controller/base.rb:121:in `process'
/home/radhika/.rvm/gems/ruby-2.1.1/gems/actionpack-3.2.17/lib/abstract_controller/rendering.rb:45:in `process'