Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422024-03-20T20:22:22ZArvados
Redmine Arvados - Bug #21612 (New): a-c-r with --debug can try to log entire input/output objects, which ...https://dev.arvados.org/issues/216122024-03-20T20:22:22ZBrett Smithbrett.smith@curii.com
<p>User got this error while running aws-s3-bulk-download.cwl with >6K input URLs, using <code>a-c-r --submit --debug</code>.</p>
<p>I don't think it actually interfered with the workflow's run at all, but it clogs the logs and looks scary.</p>
<p>IMO a-c-r (along with the rest of our code) should not try to log data that can be arbitrarily large.</p>
<p>Three instances where this came up:</p>
<pre>
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.7/logging/__init__.py", line 1037, in emit
stream.write(msg + self.terminator)
BlockingIOError: [Errno 11] write could not complete without blocking
Call stack:
File "/usr/bin/arvados-cwl-runner", line 8, in <module>
sys.exit(main())
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 440, in main
input_required=not workflow_op)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 1302, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 874, in arv_executor
self.start_run(runnable, runtimeContext)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 248, in start_run
self.workflow_eval_lock, self.stop_polling)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/task_queue.py", line 85, in add
task()
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 202, in run
self.output_callback(cast(Optional[CWLObjectType], ev), "success")
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 321, in wrapped_callback
cb(obj, st)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 429, in receive_output
output_callback(output, processStatus)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 564, in receive_output
_logger.debug("[%s] produced output %s", step.name, json_dumps(jobout, indent=4))
</pre>
<pre>--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.7/logging/__init__.py", line 1037, in emit
stream.write(msg + self.terminator)
BlockingIOError: [Errno 11] write could not complete without blocking
Call stack:
File "/usr/bin/arvados-cwl-runner", line 8, in <module>
sys.exit(main())
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 440, in main
input_required=not workflow_op)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 1302, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 863, in arv_executor
for runnable in jobiter:
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 175, in job
yield from job.job(builder.job, output_callbacks, runtimeContext)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 821, in job
for newjob in step.iterable:
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 751, in try_make_job
yield from jobs
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 77, in job
yield from self.step.job(joborder, output_callback, runtimeContext)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 462, in job
runtimeContext,
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 175, in job
yield from job.job(builder.job, output_callbacks, runtimeContext)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 821, in job
for newjob in step.iterable:
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 735, in try_make_job
json_dumps(inputobj, indent=4),
</pre>
<pre>--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.7/logging/__init__.py", line 1037, in emit
stream.write(msg + self.terminator)
BlockingIOError: [Errno 11] write could not complete without blocking
Call stack:
File "/usr/bin/arvados-cwl-runner", line 8, in <module>
sys.exit(main())
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/__init__.py", line 440, in main
input_required=not workflow_op)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/main.py", line 1302, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 874, in arv_executor
self.start_run(runnable, runtimeContext)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 248, in start_run
self.workflow_eval_lock, self.stop_polling)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/task_queue.py", line 85, in add
task()
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 202, in run
self.output_callback(cast(Optional[CWLObjectType], ev), "success")
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/arvados_cwl/executor.py", line 321, in wrapped_callback
cb(obj, st)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow.py", line 429, in receive_output
output_callback(output, processStatus)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 582, in receive_output
self.do_output_callback(final_output_callback)
File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 541, in do_output_callback
_logger.debug("[%s] outputs %s", self.name, json_dumps(wo, indent=4))
</pre> Arvados - Bug #21260 (New): abort processing on timeout and/or client hangup in Rails APIhttps://dev.arvados.org/issues/212602023-12-04T19:41:43ZPeter Amstutzpeter.amstutz@curii.com
<p>Relevant notes on #21160</p>
<p><a class="external" href="https://dev.arvados.org/issues/21160#note-8">https://dev.arvados.org/issues/21160#note-8</a></p>
<p><a class="external" href="https://dev.arvados.org/issues/21160#note-10">https://dev.arvados.org/issues/21160#note-10</a></p>
<p>Summary: controller enforces request timeout using a context (supposed to be API.RequestTimeout that defaults to 5 minutes but I am seeing the controller context expire after 1 minute -- might also be a bug?)</p>
<p>However, Rails / Postgres don't get any signal to stop processing. As a result the request continues processing (despite being cut loose by controller).</p>
<p>When controller cancels the session, the client gets 500 Internal Server Error. This is treated as a retryable response.</p>
<p>As a result, the client retries the expensive request <em>which is still running</em>, and the retry takes up a second request handler slot.</p>
<p>This can cascade with the retry timing out, blocked by the first request (if there are locks involved) resulting in another retry which ties up a third request handler slot, and so on.</p>
<p>To make the system more stable, we should have a mechanism that terminates long-running requests in Rails when they exceed a certain runtime and/or the client hangs up.</p>
<p>We might want to use this:</p>
<p><a class="external" href="https://github.com/ankane/slowpoke">https://github.com/ankane/slowpoke</a></p>
<p>This specifically supports passenger and tells passenger to abandon the Ruby process on timeout (which is fine, because we use passenger in forked multiprocess mode since threaded mode is "enterprise only").</p> Arvados - Bug #21187 (New): a-c-r should detect and warn when arv:IntermediateOutput outputTTL is...https://dev.arvados.org/issues/211872023-11-09T19:31:33ZTom Cleggtom@curii.com
<p>Currently, if outputTTL is set too low and a workflow tries to use intermediate data after it has already been trashed, a-c-r may read a intermediate collection manifest successfully (before trash time) but then fail to save it later (after trash time) in a combined collection. In that case the user ends up getting a python stack trace ending in a 403 error (invalid blob signature).</p>
<p>a-c-r should warn the user when the duration the current workflow has been running exceeds outputTTL (this is probably a good indicator the user should increase outputTTL even if it hasn't actually broken anything yet)</p>
<p>a-c-r should also report a more helpful error message when it fails to create a collection due to expired blob signatures. This could be done by checking for a 403 error from the create call and/or checking the expiry times (given as hexadecimal unix times) on the blob signatures in the manifest text.</p> Arvados - Bug #20977 (New): a-c-r crashes with "Secret store only accepts strings" if you try to ...https://dev.arvados.org/issues/209772023-09-19T20:47:15ZBrett Smithbrett.smith@curii.com
<p>e.g., if you run <code>arvados-cwl-runner --create-workflow aws-s3-bulk-cp.cwl</code> from our <code>aws-cli-cwl</code> repository:</p>
<pre>INFO bin/arvados-cwl-runner 2.6.3, arvados-python-client 2.6.3, cwltool 3.1.20230601100705
INFO Resolved 'aws-s3-bulk-cp.cwl' to 'file:///aws-cli-cwl/aws-s3-bulk-cp.cwl'
ERROR Unhandled error:
Secret store only accepts strings
Traceback (most recent call last):
File "lib/python3.9/site-packages/cwltool/main.py", line 1264, in main
initialized_job_order_object = init_job_order(
File "lib/python3.9/site-packages/cwltool/main.py", line 456, in init_job_order
secret_store.store(
File "lib/python3.9/site-packages/cwltool/secrets.py", line 34, in store
job[j] = self.add(job[j])
File "lib/python3.9/site-packages/cwltool/secrets.py", line 22, in add
raise Exception("Secret store only accepts strings")
Exception: Secret store only accepts strings
</pre> Arvados - Bug #18334 (New): Accept release info changes in docker recipeshttps://dev.arvados.org/issues/183342021-11-04T15:02:18ZTom Cleggtom@curii.com
<p>In some circumstances, "apt-get update" stops working due to existence of a future debian version.</p>
<p>This can break cmd/arvados-package tests.</p>
<pre>
$ docker run --rm -it arvados-package-deps-debian:10 bash
root@7d1560822db7:/# apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Reading package lists... Done
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.11'
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
</pre>
<p>Proposed fix: "apt-get --allow-releaseinfo-change update" in scripts.</p> Arvados - Bug #17434 (New): project not shown in hierarchy on the left of wb2https://dev.arvados.org/issues/174342021-03-01T21:07:39ZDaniel Kutyła
<p>Project that is being shared is not visible in the left tree menu</p> Arvados - Bug #17396 (New): Favorites copy dialog further issueshttps://dev.arvados.org/issues/173962021-02-16T22:31:18ZDaniel Kutyła
<p><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: Favorites in copy dialog is different to favorite list (Resolved)" href="https://dev.arvados.org/issues/17306">#17306</a> solved main issues reported by our users but there are some stuff that came out while fixing those:</p>
<p>- We can not copy element which is located in a subdir without copying other elements as parent folder will not be selected and thus it will be removed <br />- For a large collections files copy operation might be very slow as we remove non selected elements in a post copy operation</p> Arvados - Bug #17300 (New): array of array of File causes 'NoneType' object has no attribute 'res...https://dev.arvados.org/issues/173002021-01-29T14:19:14ZStanislaw Adaszewski
<p>When running an ExpressionTool which has an output of the following type:</p>
<pre>
outputs:
subject_images:
type:
type: array # subjects
items:
type: array # timepoints
items: File
</pre>
<p>I receive the following error:</p>
<pre>
ERROR Unhandled error:
'NoneType' object has no attribute 'resolved'
Traceback (most recent call last):
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/main.py", line 833, in main
logger=_logger)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 767, in arv_executor
self.final_output, self.final_output_collection = self.make_output_collection(self.output_name, storage_classes, self.output_tags, self.final_output)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 467, in make_output_collection
adjustFileObjs(outputObj, rewrite)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/pathmapper.py", line 48, in adjustFileObjs
visit_class(rec, ("File",), op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 221, in visit_class
visit_class(rec[d], cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 224, in visit_class
visit_class(d, cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 224, in visit_class
visit_class(d, cls, op)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/utils.py", line 219, in visit_class
op(rec)
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 461, in rewrite
fileobj["location"] = generatemapper.mapper(fileobj["location"]).target
File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/pathmapper.py", line 332, in mapper
return MapperEnt(v.resolved, v.target + src[i:], v.type, v.staged)
AttributeError: 'NoneType' object has no attribute 'resolved'
</pre>
<p>If I replace type: File with type: string and store locations instead of File objects the request finishes successfully and returns the expected list of locations.</p> Arvados - Bug #16228 (New): NullPointerException while uploading file using Java SDKhttps://dev.arvados.org/issues/162282020-03-10T07:28:41ZKrzysztof Majewski
<p>When trying to upload a files using Java SDK I get the following error:</p>
<pre>
2020-03-10 08:07:23 ERROR - Cannot upload file to Keep server.
org.arvados.client.exception.ArvadosApiException: java.net.ConnectException: Failed to connect to keep1.ardev.roche.com/10.115.164.103:25107
at org.arvados.client.api.client.BaseApiClient.getResponseBody(BaseApiClient.java:76)
at org.arvados.client.api.client.BaseApiClient.newCall(BaseApiClient.java:49)
at org.arvados.client.api.client.KeepServerApiClient.upload(KeepServerApiClient.java:42)
at org.arvados.client.logic.keep.FileTransferHandler.put(FileTransferHandler.java:35)
at org.arvados.client.logic.keep.KeepClient.lambda$put$1(KeepClient.java:110)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.net.ConnectException: Failed to connect to keep1.ardev.roche.com/10.115.164.103:25107
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:270)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:176)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:236)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:109)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:77)
at okhttp3.internal.connection.Transmitter.newExchange$okhttp(Transmitter.kt:162)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:35)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:82)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:84)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:71)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:112)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:87)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:184)
at okhttp3.RealCall.execute(RealCall.kt:66)
at org.arvados.client.api.client.BaseApiClient.getResponseBody(BaseApiClient.java:63)
... 10 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:126)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:268)
... 29 more
</pre>
<p>The keep1 server is taken from: <a class="external" href="https://api.ardev.roche.com/arvados/v1/keep_services/accessible">https://api.ardev.roche.com/arvados/v1/keep_services/accessible</a>. If I manipulate the response and remove it, leaving just keep2 everything works fine.</p>
<p>This is my code used for this example:</p>
<pre>
String serverUrl = "ardev.roche.com";
ExternalConfigProvider config = ExternalConfigProvider.builder()
.apiToken("api_token")
.apiHost(format("api.%s", serverUrl))
.apiPort(443)
.keepWebHost(format("collections.%s", serverUrl))
.keepWebPort(443)
.apiProtocol("https")
.apiHostInsecure(true)
.fileSplitSize(64)
.fileSplitDirectory(new File("C:\\file-split"))
.numberOfCopies(2)
.numberOfRetries(0)
.build();
List<File> files = asList(new File("C:\\samples\\sample.fcs"));
Collection collection = new CollectionsApiClient(config).get("collection_uuid");
new ArvadosFacade(config).uploadToExistingCollection(files, collection.getUuid());
</pre> Arvados - Bug #15183 (New): [Workbench2] No errors reported when file is missing a blockhttps://dev.arvados.org/issues/151832019-05-01T21:39:26ZTom Morristfmorris@veritasgenetics.com
<p>When looking at collection <a href="https://arvadosapi.com/su92l-4zz18-44qy35pg0lxshkf">su92l-4zz18-44qy35pg0lxshkf</a>/cwl.input.yml the file is cwl.output.json is listed as containing 361 bytes, but when downloaded from Workbench, it produces a zero byte file, but without giving any error or warning.</p>
<pre>
$ md5sum keep/by_id/su92l-4zz18-44qy35pg0lxshkf/*
a890894f5a748d2a93440fd046c73069 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.input.yml
md5sum: keep/by_id/su92l-4zz18-44qy35pg0lxshkf/cwl.output.json: Input/output error
d41d8cd98f00b204e9800998ecf8427e keep/by_id/su92l-4zz18-44qy35pg0lxshkf/hu27FD1F_var-GS000034212-ASM.bed
b02868b9ac1264bd1f04b61d04518f57 keep/by_id/su92l-4zz18-44qy35pg0lxshkf/workflow.cwl
</pre> Arvados - Bug #11581 (New): a-c-r should log the command used to run the pipelinehttps://dev.arvados.org/issues/115812017-04-27T15:20:55ZBryan Coscabcosca@curii.com
<p>for example, somewhere in the logs would be something like</p>
<p>arvados-cwl-runner --submit --project-uuid="" --name="" ...</p>
<p>It would help with training other users.</p> Arvados - Bug #10408 (New): 4xphq keep balance api request causes oom on api serverhttps://dev.arvados.org/issues/104082016-10-31T17:37:05ZWard Vandewegeward@curii.com
<p>On 4xphq, with the newly installed keep-balance (cf. #9966), I'm seeing:</p>
<p>keep-balance logs:</p>
<pre>
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 timer went off
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 starting next run
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 Run: start
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 skipping <a href="https://arvadosapi.com/4xphq-bi6l4-rbtrws2jxul6i4t">4xphq-bi6l4-rbtrws2jxul6i4t</a> with service type "proxy"
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 GetCurrentState: start
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 <a href="https://arvadosapi.com/4xphq-bi6l4-8009ayh02ibjfpx">4xphq-bi6l4-8009ayh02ibjfpx</a> (keep1.4xphq.arvadosapi.com:25107, s3): retrieve index
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 <a href="https://arvadosapi.com/4xphq-bi6l4-dk9mjspdg2v8mhq">4xphq-bi6l4-dk9mjspdg2v8mhq</a> (keep0.4xphq.arvadosapi.com:25107, s3): retrieve index
Oct 31 17:24:37 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:37 collections: 0/25457
Oct 31 17:24:38 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:38 collections: 1000/25457
Oct 31 17:24:39 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:39 collections: 2000/25457
Oct 31 17:24:51 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:51 <a href="https://arvadosapi.com/4xphq-bi6l4-8009ayh02ibjfpx">4xphq-bi6l4-8009ayh02ibjfpx</a> (keep1.4xphq.arvadosapi.com:25107, s3): add 37685 replicas to map
Oct 31 17:24:51 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:51 <a href="https://arvadosapi.com/4xphq-bi6l4-8009ayh02ibjfpx">4xphq-bi6l4-8009ayh02ibjfpx</a> (keep1.4xphq.arvadosapi.com:25107, s3): done
Oct 31 17:24:52 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:52 <a href="https://arvadosapi.com/4xphq-bi6l4-dk9mjspdg2v8mhq">4xphq-bi6l4-dk9mjspdg2v8mhq</a> (keep0.4xphq.arvadosapi.com:25107, s3): add 37685 replicas to map
Oct 31 17:24:52 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:24:52 <a href="https://arvadosapi.com/4xphq-bi6l4-dk9mjspdg2v8mhq">4xphq-bi6l4-dk9mjspdg2v8mhq</a> (keep0.4xphq.arvadosapi.com:25107, s3): done
Oct 31 17:25:13 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:25:13 GetCurrentState: took 35.815118053s
Oct 31 17:25:13 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:25:13 Run: took 35.846179352s
Oct 31 17:25:13 4xphq.arvadosapi.com keep-balance[8022]: 2016/10/31 17:25:13 run failed: request failed: {https <nil> 4xphq.arvadosapi.com:443 /arvados/v1/collections %!s(bool=false) filters=%5B%5B%22modified_at%22%2C%22%5Cu003e%3D%22%2C%222014-11-07T15%3A42%3A32.50418Z%22%5D%2C%5B%22uuid%22%2C%22%21%3D%22%2C%22<a href="https://arvadosapi.com/4xphq-4zz18-ttsnywvm921kkw8">4xphq-4zz18-ttsnywvm921kkw8</a>%22%5D%5D&limit=100000&order=modified_at%2C+uuid&select=%5B%22uuid%22%2C%22manifest_text%22%2C%22modified_at%22%2C%22portable_data_hash%22%2C%22replication_desired%22%5D }: 422 Unprocessable Entity
</pre>
<p>api server logs:</p>
<pre>
#<NoMemoryError: failed to allocate memory>
/var/www/arvados-api/current/app/controllers/application_controller.rb:188:in `dump'
/var/www/arvados-api/current/app/controllers/application_controller.rb:188:in `send_json'
/var/www/arvados-api/current/app/controllers/application_controller.rb:494:in `render_list'
/var/www/arvados-api/current/app/controllers/application_controller.rb:75:in `index'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/abstract_controller/base.rb:167:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:590:in `_run__36149969870514075__process_action__2372069873549583241__callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/abstract_controller/callbacks.rb:17:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/notifications.rb:123:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/notifications.rb:123:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.17/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/abstract_controller/base.rb:121:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/abstract_controller/rendering.rb:45:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal.rb:203:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_controller/metal.rb:246:in `block in action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/routing/route_set.rb:73:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/routing/route_set.rb:36:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/routing/route_set.rb:608:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.1.1/lib/omniauth/strategy.rb:177:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.1.1/lib/omniauth/strategy.rb:157:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.1.1/lib/omniauth/builder.rb:48:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/head.rb:14:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/flash.rb:242:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.17/lib/active_record/query_cache.rb:64:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `_run__2636124107984033352__call__75134247275516573__callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.3.6/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.17/lib/rails/rack/logger.rb:16:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:22:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.17/lib/rails/rack/logger.rb:16:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.17/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.17/lib/rails/engine.rb:484:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.17/lib/rails/application.rb:231:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
Error 1477935314+0c4830bc: 422
</pre>
<p>Once <a class="issue tracker-6 status-3 priority-4 priority-default closed" title="Idea: [API] [keep-balance] Option to return unsigned manifests from collections#index (Resolved)" href="https://dev.arvados.org/issues/6830">#6830</a> is done, revisit keep-balance on 4xphq to see if that fixed the problem.</p> GET-Evidence - Bug #5872 (In Progress): 23andMe report pipeline is brokenhttps://dev.arvados.org/issues/58722015-04-30T22:52:53ZAbram Connellyabram.connelly@gmail.com
<p>When participants upload their 23andMe data, this kicks off a pipeline run through Arvados (now) through the same code path that the dataset submission goes through from Tapestry. The pipeline is failing to correctly generate reports for 23andMe data.</p>
<p>Porting the pipeline over to Arvados, the format for the input was changed and that's why the pipelines are failing on 23andMe data. I think the best thing to do going forward is to be more liberal about what input collection is specified so that 23andMe reports along with older reports can be run (or re-run) without having to worry about updating to the new format.</p>
<p>The old format just specifies a portable data hash. The new format expects <PDH>/<filename>.</p> GET-Evidence - Bug #835 (New): Error when saving edits or submittinghttps://dev.arvados.org/issues/8352012-02-06T16:44:19ZMadeleine Ballmpball@gmail.com
<p>When saving or submitting we're seeing this error message now: "Save failed -- received bogus response from server."</p>
<p>Seems to date to changes made in commit <a class="changeset" title="if session is expired when "save draft" is clicked, pop-up a login window instead of falsely proc..." href="https://dev.arvados.org/projects/get-evidence/repository/8/revisions/5dc9adee7bd5cde0d1c9b16652cf0b5e8d6229d3">5dc9adee</a> ?</p>
<p>Sasha pointed out that the changes still get saved, but it's still a serious bug -- especially since we just sent the paper out for review and were planning to encourage people to edit...</p> GET-Evidence - Bug #468 (New): latest-flat incorrectly displays non single-base substitution vari...https://dev.arvados.org/issues/4682011-02-15T11:15:42ZMadeleine Ballmpball@gmail.com
<p>I haven't checked, but this problem likely extends to deletion & other new length changing or multiple-aa variants.</p>
<p>This bug blocks creating a flatfile version of gff_getevidence_map.py</p>
<p>For example, FIG4-K278Shift is showing up like this (2nd and 3rd columns are incorrect):<br />FIG4 278 278 pathogenic Moderate clinical importance, Uncertain pathogenic recessive 0 0 0 0 0 3 Y -- - Y 0 Y 3 - 2 - Y - - - - Y 4 Y 2N 1 0 This variant is predicted to cause a frameshift and may cause Charcot-Marie-Tooth Disease Type 4J in an autosomal recessive manner. Other variants in this gene which cause frameshift and premature termination have been implicated in causing this disease when compound heterozygous with another FIG4 variant.</p>
<p>While MYL2-A13T shows up like this:<br />MYL2 Ala13Thr A13T pathogenic Low clinical importance, Uncertain pathogenic dominant 1 1 1 6 6 3 Y 4Y 0 Y ! Y 4 Y 3 Y Y Y - - Y Y 1 - Familial Hypertrophic Cardiomyopathy 4596 1 455 3.054 4 N 0 0 This rare variant is implicated in causing late-onset familial hypertrophic cardiomyopathy. The variant has been found in five affected Caucasian individuals (in four families), but affected non-carriers and unaffected carriers have also been observed. No statistically significant enrichment of this variant in cases vs. controls has been shown.</p>