Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-06-05T15:06:00ZArvados
Redmine Arvados - Feature #15317 (Resolved): [Keep] expose backend latency stats in the Prometheus metricshttps://dev.arvados.org/issues/153172019-06-05T15:06:00ZWard Vandewegeward@curii.com
<a name="keepstore"></a>
<h2 >keepstore<a href="#keepstore" class="wiki-anchor">¶</a></h2>
<ul>
<li>bucketed by block size (1 MiB < 4 MiB < 8 MiB < 16 MiB < 32 MiB < 64 MiB)</li>
<li>time to first byte from backend</li>
<li>transfer speed from backend (start of request to last byte)</li>
<li>transfer speed from backend (first byte to last byte)</li>
<li>time between first byte from backend and first byte sent to client</li>
</ul>
<a name="keep-web"></a>
<h2 >keep-web<a href="#keep-web" class="wiki-anchor">¶</a></h2>
<ul>
<li>bucketed by block size (1 MiB < 4 MiB < 8 MiB < 16 MiB < 32 MiB < 64 MiB)
<ul>
<li>time to first byte of a block from keepstore</li>
<li>transfer speed of a block from keepstore (start of request to last byte)</li>
<li>transfer speed of a block from keepstore (first byte to last byte)</li>
<li>transfer speed of a block (from disk cache)</li>
</ul>
</li>
<li>bucketed by file size (1 MiB < 10 MiB < 100 MiB < 1000 MiB)
<ul>
<li>time from start of request to first byte on file (cached & uncached manifest)</li>
<li>transfer speed of a whole file (start of request to last byte)</li>
<li>transfer speed of a whole file (first byte to last byte)</li>
<li>time between first byte from keepstore and first byte sent to client</li>
</ul></li>
</ul> Arvados - Bug #7385 (Closed): [Workbench] Pagination sets limit on last page, causing unintended ...https://dev.arvados.org/issues/73852015-09-24T18:56:37ZWard Vandewegeward@curii.com
<p>When viewing the last page on a paginated resource (for instance, collections) in workbench, a limit=remainder is introduced in the url. That limit is then applied to generate all the links to the pages at the top and bottom of the list.</p>
<p>The effect is that when there is a list of 102 items, and the default pagination is 100, you see a page 1 and 2 link on the first page.</p>
<p>If you click on the '2' for page 2, you see the 2 last items in the list, but you now also see a list of 51 pages and each of those pages has just 2 items.</p> Arvados - Bug #7312 (New): [API] CSS+images missing on login_failure.htmlhttps://dev.arvados.org/issues/73122015-09-11T15:22:06ZWard Vandewegeward@curii.com
<p>Try for example <a class="external" href="https://qr1hi.arvadosapi.com/static/login_failure.html">https://qr1hi.arvadosapi.com/static/login_failure.html</a>.</p> Arvados - Bug #7166 (New): [API] specifying an invalid filter should result in an errorhttps://dev.arvados.org/issues/71662015-08-28T20:21:04ZWard Vandewegeward@curii.com
<p>This invalid filter</p>
<pre>api_client.collections().list(filters=[['portable_data_hash', 'somepdhoranother+size']], select=['portable_data_hash', 'uuid'], limit=100000).execute()</pre>
<p>is simply ignored; the query above returns all collections. Our SDK should probably throw an exception - or at least print a warning when an invalid filter is ignored.</p>
<pre>api_client.collections().list(filters=[['portable_data_hash', '=', 'somepdhoranother+size']], select=['portable_data_hash', 'uuid'], limit=100000).execute()</pre>
<p>the above filter is correct and works as expected.</p> Arvados - Feature #7127 (New): [API] Job log cleaner should not clean logs when the job's log is ...https://dev.arvados.org/issues/71272015-08-25T18:56:30ZWard Vandewegeward@curii.comArvados - Bug #6635 (New): [Documentation] document installation of node managerhttps://dev.arvados.org/issues/66352015-07-15T17:27:00ZWard Vandewegeward@curii.com
<p>For example, it requires a system token created with</p>
<pre><code>arv api_client_authorization create_system_auth \<br /> --scopes "[\"GET /arvados/v1/jobs/queue\",<br /> \"GET /arvados/v1/nodes\",<br /> \"PUT /arvados/v1/nodes\",<br /> \"PUT /arvados/v1/nodes/\",<br /> \"POST /arvados/v1/nodes\",<br /> \"POST /arvados/v1/nodes/\"]"</code></pre> Arvados - Bug #6076 (New): [API] walk api server installations and ensure modified_at for collect...https://dev.arvados.org/issues/60762015-05-19T17:09:17ZWard Vandewegeward@curii.com
<p>Until this is done, datamanager needs to ask for groups of collections with overlap. That is not the most efficient, but it is also potentially problematic in light of <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [API] Be careful about memory usage when responding to collections#index queries (Resolved)" href="https://dev.arvados.org/issues/5834">#5834</a> which dynamically adjust the result set based on result size (and only guarantees a minimum of 1 result).</p> Arvados - Bug #6020 (Closed): [Workbench] The collection information box values for content addre...https://dev.arvados.org/issues/60202015-05-14T03:05:33ZWard Vandewegeward@curii.comArvados - Task #5651 (Closed): Fix reported scaling factor for failed jobs (see notes)https://dev.arvados.org/issues/56512015-04-03T14:19:25ZWard Vandewegeward@curii.com
<p>For example, <a href="https://arvadosapi.com/su92l-d1hrv-wo4tw2o83t7o0h8">su92l-d1hrv-wo4tw2o83t7o0h8</a>, where the scaling factor (and compute time used by the failed job) keeps increasing.</p> Arvados - Task #5649 (Closed): [Workbench] We need better job graphs, less spaghettihttps://dev.arvados.org/issues/56492015-04-03T02:02:36ZWard Vandewegeward@curii.com
<p>Our job graphs are not useful. Here's an example for a job with just 8 tasks:</p>
<p><img src="https://dev.arvados.org/attachments/download/519/20150402-job-graphs.png" alt="" /></p>
<p>This is just meaningless spaghetti.</p>
<p>We need</p>
<p>a) a legend<br />b) Y axis with units<br />c) fewer things on the graph / possibly more graphs</p>
<p>For what it's worth, the GCE developer console splits out Disk / CPU / Memory into 3 different graphs.</p> Arvados - Bug #4957 (New): [SDKs] arv keep put blows up when trying to put a danging symlinkhttps://dev.arvados.org/issues/49572015-01-10T19:49:14ZWard Vandewegeward@curii.com
<p>For example:</p>
<pre>
$ arv keep put mirobot_updater
Traceback (most recent call last):
File "/usr/local/bin/arv-put", line 4, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 411, in main
bytes_expected = expected_bytes_for(args.paths)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 333, in expected_bytes_for
bytesum += os.path.getsize(os.path.join(path, filename))
File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: 'mirobot_updater/resources/app/node_modules/.bin/serialportlist'
</pre>
<p>and</p>
<pre>
$ ls mirobot_updater/resources/app/node_modules/.bin/serialportlist -laF
lrwxrwxrwx 1 wardtest wardtest 35 Jan 10 19:41 mirobot_updater/resources/app/node_modules/.bin/serialportlist -> ../serialport/bin/serialportList.js
</pre>
<p>arv keep put should just skip things it can not save, and ideally list at the end of the run which file(s) it skipped.</p> Arvados - Bug #4767 (New): [API] Discovery doc should fix its incorrect "list" method description...https://dev.arvados.org/issues/47672014-12-10T03:30:30ZWard Vandewegeward@curii.com
<p>Currently, "list" is advertised with a hard-coded and outdated list of parameters ("order" is advertised as a string parameter). "index" is advertised correctly using introspection.</p>
<p>(We might also be able to remove some of the other hard-coded method descriptions too, and let the "Check for Rails routes" code generate them instead.)</p> Arvados - Bug #4452 (New): [Workbench] Report a Problem" menu item does not like being opened in ...https://dev.arvados.org/issues/44522014-11-06T19:39:19ZWard Vandewegeward@curii.com
<p>Oh... fiddlesticks.</p>
<p>Sorry, I had some trouble handling your request.</p>
<pre><code>Missing template actions/report_issue_popup, application/report_issue_popup with {:locale=>[:en], :formats=>[:html], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby, :coffee]}. Searched in: * "/var/www/workbench.qr1hi.arvadosapi.com/releases/20141105135933/themes/curoverse/views" * "/var/www/workbench.qr1hi.arvadosapi.com/releases/20141105135933/app/views" * "/var/www/workbench.qr1hi.arvadosapi.com/shared/vendor_bundle/ruby/2.1.0/gems/wiselinks-1.2.1/app/views"</code></pre> Arvados - Bug #4427 (Closed): [Workbench] Each active job/pipeline on the dashboard should show h...https://dev.arvados.org/issues/44272014-11-05T14:02:43ZWard Vandewegeward@curii.com
<p>"Local jobs" will be easy to spot: they will have no worker nodes are allocated.</p>
<p>This answers the question "are these jobs actually consuming compute node resources?"</p>
<p>Distinguishing local jobs from batch jobs later, after they have completed, is a separate story.</p> Arvados - Feature #4193 (New): [SDK] Make arv keep docker smarter about storing docker layers, th...https://dev.arvados.org/issues/41932014-10-14T19:34:19ZWard Vandewegeward@curii.com