Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-01-25T16:44:16ZArvados
Redmine Arvados - Bug #21417 (Resolved): Stop trying to read image timestamp from docker metadata in arv-...https://dev.arvados.org/issues/214172024-01-25T16:44:16ZTom Cleggtom@curii.com
<p>This part of <a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/sdk/python/arvados/commands/keepdocker.py">source:sdk/python/arvados/commands/keepdocker.py</a> should go away so it doesn't crash on new image tarball formats:</p>
<pre>
json_file = image_tar.extractfile(image_tar.getmember(json_filename))
image_metadata = json.loads(json_file.read().decode('utf-8'))
json_file.close()
image_tar.close()
link_base = {'head_uuid': coll_uuid, 'properties': {}}
if 'created' in image_metadata:
link_base['properties']['image_timestamp'] = image_metadata['created']
</pre>
<p>See <a class="issue tracker-6 status-3 priority-4 priority-default closed" title="Idea: test-provision-debian11 fails loading workflow Docker image (Resolved)" href="https://dev.arvados.org/issues/21408">#21408</a> for example.</p>
<p>(Tom & Peter discussed offline, came to the conclusion that saving the image timestamp is not important enough to justify maintaining the code.)</p> Arvados - Bug #20754 (Resolved): docker-cleaner possibly brokenhttps://dev.arvados.org/issues/207542023-07-18T17:50:24ZTom Cleggtom@curii.com
<p>observed in systemd journal on a pirca compute instance while troubleshooting something else:</p>
<pre>
Stopped Arvados Docker Image Cleaner.
Started Arvados Docker Image Cleaner.
Traceback (most recent call last):
File "/usr/bin/arvados-docker-cleaner", line 8, in <module>
sys.exit(main())
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 365, in main
run(config, docker.Client(version='1.14'))
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 345, in run
images = DockerImages.from_daemon(config['Quota'], docker_client)
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/arvados_docker/cleaner.py", line 68, in from_daemon
for image in docker_client.images():
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/api/image.py", line 39, in images
res = self._result(self._get(self._url("/images/json"), params=params),
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/utils/decorators.py", line 47, in inner
return f(self, *args, **kwargs)
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/docker/client.py", line 112, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/share/python3/dist/arvados-docker-cleaner/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'
arvados-docker-cleaner.service: Main process exited, code=exited, status=1/FAILURE
arvados-docker-cleaner.service: Failed with result 'exit-code'.
</pre>
<p>It seems to be in a crash-and-restart loop.</p> Arvados - Bug #19967 (Resolved): crunch-run periodically updates container costhttps://dev.arvados.org/issues/199672023-01-20T16:25:15ZTom Cleggtom@curii.com
<p>Currently the container cost is recorded as 0 in this case.</p>
<p>This should ensure the cost of a cancel-retry-cancel-retry-cancel-fail cycle is reflected properly.</p>
<p>Have crunch-run periodically update the price while the container is running, perhaps each time it gets a SIGUSR2 after loadPrices (i.e., on each dispatcher probe).</p> Arvados - Bug #8904 (Resolved): Make test suite work on debian 7 (wheezy) by using pip<8 for python3https://dev.arvados.org/issues/89042016-04-07T02:14:57ZTom Cleggtom@curii.com
<p>debian wheezy has python 3.2, but pip 8 no longer supports python 3.2.</p>
<pre>
File "/etmp/2arvados-build/VENV3DIR/lib/python3.2/site-packages/pip/_vendor/progress/bar.py", line 48
empty_fill = u'∙'
^
SyntaxError: invalid syntax
</pre> Tapestry - Feature #8701 (Closed): Arvados job/pipeline generates ClinVar HTML report from VCFhttps://dev.arvados.org/issues/87012016-03-15T17:59:55ZTom Cleggtom@curii.comTapestry - Feature #8700 (Closed): Arvados job/pipeline generates GET-Evidence HTML report from VCFhttps://dev.arvados.org/issues/87002016-03-15T17:59:43ZTom Cleggtom@curii.comArvados - Task #6116 (Closed): Review 5883-oauth2-update at get-evidence.githttps://dev.arvados.org/issues/61162015-05-20T20:12:49ZTom Cleggtom@curii.comArvados - Bug #5074 (Resolved): [SDKs] CLI client and Ruby SDK should recover from (and avoid cre...https://dev.arvados.org/issues/50742015-01-23T20:51:23ZTom Cleggtom@curii.com
<p>The current code creates a race condition while writing a file. It should create a temp file, write it, flush/close it, and rename it into place. If anything fails, it should delete the temp file.</p>
<p>Additionally, if a cache file exists (and is new enough, etc.) but is unreadable or unparseable, the SDK crashes. Instead, it should print a warning, then retrieve the discovery document from the server and continue.</p>
<p>See <a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/sdk/cli/bin/arv#L55">source:sdk/cli/bin/arv#L55</a> and <a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/sdk/ruby/lib/arvados.rb#L108">source:sdk/ruby/lib/arvados.rb#L108</a></p>
<p>Bonus points: de-duplicate this copy-and-pasted code!</p> Arvados - Bug #4836 (Resolved): [Workbench] Ensure first tab content loads when shown, even if lo...https://dev.arvados.org/issues/48362014-12-17T14:48:04ZTom Cleggtom@curii.com
<p>These two links show me one collection in the Data collections tab.</p>
<ul>
<li><a class="external" href="https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62">https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62</a></li>
<li><a class="external" href="https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62#Data_collections">https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62#Data_collections</a></li>
</ul>
<p>However, if I load this page...</p>
<ul>
<li><a class="external" href="https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62#Jobs_and_pipelines">https://workbench.4xphq.arvadosapi.com/projects/4xphq-j7d0g-o65lwls3n5duo62#Jobs_and_pipelines</a></li>
</ul>
<p>...the tab heading says "Data collections (1)" but when I click it, I get a table with name and description headings but no collections.</p>
<p>It looks like the "load content for collections table" trigger fires during initial page load (and only if that tab is active), and should also (instead?) fire when that tab/table is first exposed.</p> Arvados - Bug #4562 (Resolved): [Documentation] Wiki page: explain appropriate use cases for arv-...https://dev.arvados.org/issues/45622014-11-17T21:20:04ZTom Cleggtom@curii.com
<p>These general approaches need to be introduced and explained between <a class="external" href="http://doc.arvados.org/user/tutorials/intro-crunch.html">http://doc.arvados.org/user/tutorials/intro-crunch.html</a> and <a class="external" href="http://doc.arvados.org/user/tutorials/running-external-program.html">http://doc.arvados.org/user/tutorials/running-external-program.html</a>.</p>
Executive summary
<ul>
<li>arv-run makes sense for simple fan-out commands</li>
<li>run-command makes sense when you already have a command line tool installed in a docker image, and you just want to invoke it as part of a compute workflow/pipeline</li>
<li>writing your own crunch script makes sense if you want your code to be in revision control, you want more control of concurrency patterns, you need better performance, or you'd just rather write everything in python.</li>
<li>third option (use run-command to wrap something that lives in your own git tree) isn't very well supported but as a workaround you could copy the run-command stuff into your own git tree.</li>
</ul>
<p>This page should clearly explain the limitations of arv-run and run-command so that users know when to switch between the approaches to run things.</p> Arvados - Bug #4501 (Resolved): [FUSE] arv-mount --by-id should expose the /by_id directory as we...https://dev.arvados.org/issues/45012014-11-12T20:30:02ZTom Cleggtom@curii.comArvados - Bug #4408 (Resolved): [Workbench] Anonymous collection download links should not redire...https://dev.arvados.org/issues/44082014-11-04T14:08:49ZTom Cleggtom@curii.com
To reproduce:
<ol>
<li>Visit collection page.</li>
<li>Click "Create sharing link".</li>
<li>Open the resulting link in a new/incognito browser session. (The file listing page works.)</li>
<li>Click a link. You get a Google login page.
<ul>
<li>(Also, unsurprisingly, pasting the supplied "wget" command line into a shell prompt causes wget to follow redirect links to a Google login page.)</li>
</ul></li>
</ol>
TODO:
<ul>
<li>Augment the "can download an entire collection" integration test so it downloads a <em>file</em>, not just the list of files, as mentioned in the TODO comment. Make sure the test reproduces this bug (fails). (It is possible the bug is in the "create sharing link" process, not the download handler.)</li>
<li>Fix the bug to make the test pass.</li>
</ul> Arvados - Bug #4388 (Resolved): [SDKs] Manifest#each_line should not crash when encountering a bl...https://dev.arvados.org/issues/43882014-11-01T18:32:07ZTom Cleggtom@curii.com
<p>While displaying the collections tab on "show Home project":</p>
<pre>
Started GET "/projects/qr1hi-tpzed-tpj2ff66551eyym?partial=contents_rows&filters=%5B%5B%22uuid%22%2C%22is_a%22%2C%22arvados%23collec
tion%22%5D%5D" for 192.168.200.73 at 2014-11-01 14:29:42 -0400
Processing by ProjectsController#show as JSON
Parameters: {"partial"=>"contents_rows", "filters"=>"[[\"uuid\",\"is_a\",\"arvados#collection\"]]", "id"=>"qr1hi-tpzed-tpj2ff66551
eyym"}
...
#<NoMethodError: undefined method `gsub' for nil:NilClass>
/usr/local/lib/ruby/gems/2.1.0/gems/arvados-0.1.20140917180103/lib/arvados/keep.rb:116:in `unescape'
/usr/local/lib/ruby/gems/2.1.0/gems/arvados-0.1.20140917180103/lib/arvados/keep.rb:104:in `block in each_line'
/usr/local/lib/ruby/gems/2.1.0/gems/arvados-0.1.20140917180103/lib/arvados/keep.rb:102:in `each_line'
/usr/local/lib/ruby/gems/2.1.0/gems/arvados-0.1.20140917180103/lib/arvados/keep.rb:102:in `each_line'
/usr/local/lib/ruby/gems/2.1.0/gems/arvados-0.1.20140917180103/lib/arvados/keep.rb:137:in `files'
/home/tom/src/arvados/apps/workbench/app/models/collection.rb:42:in `total_bytes'
/home/tom/src/arvados/apps/workbench/app/models/collection.rb:38:in `content_summary'
/home/tom/src/arvados/apps/workbench/app/views/application/_show_object_description_cell.html.erb:1:in `_app_views_application__show_object_description_cell_html_erb___1631925922960951774_69892807124400'
</pre> Arvados - Bug #3720 (Resolved): [API] Fetching collection record is very slow when manifest is no...https://dev.arvados.org/issues/37202014-08-27T14:59:46ZTom Cleggtom@curii.com
<p>This makes nginx give up on Workbench and say "502 Bad Gateway".</p> Arvados - Task #3374 (Resolved): Test patched Python packagehttps://dev.arvados.org/issues/33742014-07-26T15:27:51ZTom Cleggtom@curii.com