Arvados

According to the log message (and c97qk workbench dashboard) the queue is empty, in which case wishlist=0 is correct. Was there a job queued that has since been cancelled?

Tom Clegg wrote:

According to the log message (and c97qk workbench dashboard) the queue is empty, in which case wishlist=0 is correct. Was there a job queued that has since been cancelled?

There is a job queued. Also listed on the workbench dashboard. How come you didn't see it?

https://workbench.c97qk.arvadosapi.com/pipeline_instances/c97qk-d1hrv-stxx1wltwlzssp5

Looks like this is yet another instance of "you can't run two job dispatchers at the same time":

/etc/service/crunch-dispatch-jobs-1/log/main

@400000005657da2437923c04.s:2015-11-25_16:06:38.86565 dispatch: job c97qk-8i9sb-3k4o5augc494rm0 end
@400000005657da2437923c04.s:2015-11-25_16:25:43.88271 dispatch: waiting for nodes for c97qk-8i9sb-3k4o5augc494rm0
@400000005657da2437923c04.s:2015-11-25_17:18:32.54096 dispatch: c97qk-8i9sb-3k4o5augc494rm0: crunch-dispatch was stopped during job's tempfail retry loop
@400000005657da2437923c04.s:2015-11-25_17:18:32.92287 dispatch: tried to mark job c97qk-8i9sb-3k4o5augc494rm0 as failed but it was already locked by someone else

vs

/etc/service/crunch-dispatch-jobs-0/log/main

2015-11-25_16:03:17.20886 dispatch: ["git", "--git-dir=/var/lib/arvados/internal.git", "tag", "c97qk-8i9sb-3k4o5augc494rm0", "4b84269f160c4d64cb46b2d3bc2a404a962e793f"]
2015-11-25_16:03:17.21574 
2015-11-25_16:03:17.22272 dispatch: sudo -E -u crunch LD_LIBRARY_PATH= PATH=/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/bin:/usr/local/rvm/gems/ruby-2.1.5/bin:/usr/local/rvm/gems/ruby-2.1.5@global/bin:/usr/local/rvm/rubies/ruby-2.1.5/bin:/usr/local/rvm/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/arvados/src/services/crunch PERLLIB=/usr/local/arvados/src/sdk/perl/lib PYTHONPATH= RUBYLIB=/usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib GEM_PATH= salloc --chdir=/ --immediate --exclusive --no-kill --job-name=c97qk-8i9sb-3k4o5augc494rm0 --nodelist=compute2 /usr/local/arvados/src/services/crunch/crunch-job --job-api-token 5oqv77bzqieoondrt63uyjkwvve9ffgnedi2gdev6rl2ny1f --job c97qk-8i9sb-3k4o5augc494rm0 --git-dir /var/lib/arvados/internal.git
2015-11-25_16:03:17.22952 dispatch: job c97qk-8i9sb-3k4o5augc494rm0
2015-11-25_16:03:17.22956 dispatch: child 20641 start 2015-11-25_16:03:17
2015-11-25_16:03:17.84845 dispatch: update compute2 state to {:state=>"alloc", :job=>"c97qk-8i9sb-3k4o5augc494rm0"}
2015-11-25_16:03:18.14644 c97qk-8i9sb-3k4o5augc494rm0 ! salloc: error: Unable to allocate resources: Requested nodes are busy
2015-11-25_16:03:18.17740 dispatch: child 20641 exit 75
2015-11-25_16:03:18.17747 dispatch: job c97qk-8i9sb-3k4o5augc494rm0 end
2015-11-25_16:06:34.85513 dispatch: update compute2 state to {:state=>"alloc", :job=>nil}
2015-11-25_16:06:36.77112 dispatch: update compute2 state to {:state=>"down", :job=>nil}
2015-11-25_17:18:31.61005 Received TERM signal
2015-11-25_17:18:33.00783 Stopping crunch-dispatch-jobs-0
2015-11-25_17:18:34.15911 /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/spec_set.rb:92:in `block in materialize': Could not find curb-0.8.8 in any of the sources (Bundler::GemNotFound)
2015-11-25_17:18:34.15922       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/spec_set.rb:85:in `map!'
2015-11-25_17:18:34.15924       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/spec_set.rb:85:in `materialize'
2015-11-25_17:18:34.15926       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/definition.rb:140:in `specs'
2015-11-25_17:18:34.15928       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/definition.rb:185:in `specs_for'
2015-11-25_17:18:34.15930       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/definition.rb:174:in `requested_specs'
2015-11-25_17:18:34.15932       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/environment.rb:18:in `requested_specs'
2015-11-25_17:18:34.15934       from /usr/local/rvm/gems/ruby-2.1.5/gems/bundler-1.10.6/lib/bundler/runtime.rb:13:in `setup'
...

Ward Vandewege wrote:

There is a job queued. Also listed on the workbench dashboard. How come you didn't see it?

https://workbench.c97qk.arvadosapi.com/pipeline_instances/c97qk-d1hrv-stxx1wltwlzssp5

The job you're referring to (c97qk-8i9sb-3k4o5augc494rm0) was in the Running state. That's not the same thing as Queued.

I can't really figure out a good way to test this (I tried, I can at least verify there are no syntax errors that prevent execution), but I understand and agree with the logic, so LGTM.