Bug #11917

Error fetching collection (HttpError 422) during call to arvados.util.listdir_recursive

Added by Joshua Randall 3 months ago. Updated 2 months ago.

Status:FeedbackStart date:06/29/2017
Priority:NormalDue date:
Assignee:Tom Clegg% Done:

100%

Category:API
Target version:Arvados Future Sprints
Story pointsSRemaining (hours)0.00 hour
Velocity based estimate-

Description

2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr 2017-06-27 16:17:34 arvados.arvados_fuse[45815] ERROR: Error fetching collection 'f23f6b30aa7d658b565af6d4e2
e5368a+9409': Failed to retrieve collection 'f23f6b30aa7d658b565af6d4e2e5368a+9409' from either API server (<HttpError 422 when requesting https://api.arvados.sanger.ac.uk/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409?alt=json returned "#<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>">) or Keep (f23f6b30aa7d658b565af6d4e2e5368a+9409 not found:  http://humgen-01-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-01-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-01-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ).
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr Traceback (most recent call last):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 99, in <module>
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     main()
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 54, in main
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     cram_file = gatk_helper.mount_gatk_cram_input(input_param="input")
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/hgi_arvados/gatk_helper.py", line 81, in mount_gatk_cram_input
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     for f in arvados.util.listdir_recursive(input_dir):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/usr/lib/python2.7/dist-packages/arvados/util.py", line 332, in listdir_recursive
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     for ent in sorted(os.listdir(dirname)):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr OSError: [Errno 2] No such file or directory: '/keep/f23f6b30aa7d658b565af6d4e2e5368a+9409'
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 child 29639 on humgen-02-01.16 exit 1 success=
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 failure (#1, permanent) after 35 seconds


Subtasks

Task #11940: Review 11917-dont-clear-cacheResolvedRadhika Chippada


Related issues

Related to Arvados - Bug #11470: [API] container_requests#update alternately responds 422 ... New 04/12/2017

Associated revisions

Revision d084ca24
Added by Tom Clegg 3 months ago

Merge branch '11917-dont-clear-cache'

refs #11917

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Joshua Randall 3 months ago

From production.log:

[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] /usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `initialize'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `open'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `rescue in block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1162:in `block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `each'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `touch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:58:in `probe_stat_in'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:32:in `atomic_write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/file_store.rb:83:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache.rb:115:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:391:in `block in write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:166:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:389:in `write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:588:in `save_block_result_to_cache'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:299:in `fetch'
/var/www/arvados-api/current/lib/sweep_trashed_collections.rb:21:in `sweep_if_stale'
/var/www/arvados-api/current/app/models/collection.rb:441:in `where'
/var/www/arvados-api/current/app/controllers/arvados/v1/collections_controller.rb:28:in `find_object_by_uuid'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:432:in `block in make_lambda'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `block in halting_and_conditional'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/builder.rb:63:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/etag.rb:24:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.4.1/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] Error 1498576650+aad9240b: 422
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] {"method":"GET","path":"/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409","format":"json","control
ler":"Arvados::V1::CollectionsController","action":"show","status":422,"duration":4.1,"view":0.61,"db":0.0,"params":{"alt":"json"},"@timestamp":"2017-06-27T15:17:30.34520886
4Z","@version":"1","message":"[422] GET /arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409 (Arvados::V1::CollectionsController#show)"}

#2 Updated by Tom Clegg 3 months ago

This code looks like a supported use of Rails cache:

    Rails.cache.fetch('SweepTrashedCollections', expires_in: exp) do
      need = true
    end

#3 Updated by Tom Clegg 3 months ago

In activesupport-4.2.5.2/lib/active_support/cache/file_store.rb

        def write_entry(key, entry, options)
          file_name = key_file_path(key)
          return false if options[:unless_exist] && File.exist?(file_name)
          ensure_cache_path(File.dirname(file_name))
          File.atomic_write(file_name, cache_path) {|f| Marshal.dump(entry, f)}
          true
        end

Perhaps empty cache directories are being deleted between "ensure_cache_path" and "File.atomic_write"?

#4 Updated by Tom Clegg 3 months ago

We do this in source:services/api/config/environment.rb:

# Initialize the rails application
Server::Application.initialize!
begin
  Rails.cache.clear
rescue Errno::ENOENT => e
  # Cache directory does not exist? Then cache is clear, proceed.
  Rails.logger.warn "In Rails.cache.clear, ignoring #{e.inspect}" 
end

I'm guessing this is what's clearing the cache directories. It's possible this is running every time Passenger brings up a new worker. It's very likely it is also running every time a rake task is invoked by cron. We should fix this so it only runs once at server startup, and never on periodic rake tasks... or perhaps so it doesn't run at all.

#5 Updated by Tom Clegg 3 months ago

#6 Updated by Tom Clegg 3 months ago

  • Category set to API
  • Status changed from New to In Progress
  • Assignee set to Tom Clegg

#7 Updated by Tom Morris 3 months ago

  • Target version set to 2017-07-19 sprint

#8 Updated by Radhika Chippada 3 months ago

Is the cache LRU? http://guides.rubyonrails.org/caching_with_rails.html says: "As the cache will grow until the disk is full, it is recommended to periodically clear out old entries." But I couldn't tell how / if this is being done in our server.

#9 Updated by Tom Clegg 3 months ago

We use expiry times and a finite number of keys, so the cache shouldn't grow indefinitely.

We should add a cache-clearing thing that does "delete all entries matching *" instead of the non-race-safe "clear" method. And choose a better moment to run it, so it doesn't clear every time someone runs a rails-based maintenance cron job...

#10 Updated by Tom Morris 3 months ago

"expiry times" == "clear out old entries"

Perhaps we could put a rake tmp:cache:clear in the API server boot script if we're really concerned about stale cache space from e.g. a server crash, but otherwise it sounds like the combination of limited keys and expiry times should do the trick.

#11 Updated by Tom Clegg 2 months ago

  • Status changed from In Progress to Feedback

#12 Updated by Tom Clegg 2 months ago

  • Target version changed from 2017-07-19 sprint to Arvados Future Sprints
  • Story points set to 0.0

Also available in: Atom PDF